Evaluation of Cross-Survey Research Methods for the Estimation of Low-Incidence Populations

Abstract

This study evaluates the accuracy, precision, and stability of three different methods of cross-survey analysis in order to determine their suitability for estimating the proportions of low-incidence populations. Population parameters of size and demographic distribution are necessary for planning and policy development. The estimation of these parameters for low-incidence populations poses a number of methodological challenges. Cross-survey analysis methodologies offer an alternative to generate useful, low-incidence population estimates not readily available in today's census without conducting targeted, costly surveys to estimate group size directly. The cross-survey methods evaluated in the study are meta-analysis of complex surveys (MACS), pooled design-based cross-survey (PDCS), and Bayesian multilevel regression with post-stratification (BMRP). The accuracy and precision of these methods were assessed by comparing the estimates of the proportion of the adult Jewish population in Canada generated by each method with benchmark estimates. The stability of the estimates, in turn, was determined by cross-validating estimates obtained with data from two random stratified subsamples drawn from a large pool of US surveys. The findings of the study indicate that, under the right conditions, cross-survey methods have the potential to produce very accurate and precise estimates of low-incidence populations. The study did find that the level of accuracy and precision of these estimates varied depending on the cross-survey method used and on the conditions under which the estimates were produced. The estimates obtained with PDCS and BMRP methodologies were more accurate than the ones generated by the MACS approach. The BMRP approach generated the most accurate estimates. The pooled design-based cross-survey method generated relatively accurate estimates across all the scenarios included in the study. The precision of the estimates was found to be related to the number of surveys considered in the analyses. Overall, the findings clearly show that cross-survey analysis methods provide a useful alternative for estimation of low-incidence populations. More research is needed to fully understand the factors that affect the accuracy and precision of estimates generated by these cross-survey methods.