Investigating Adaptive Nonresponse Follow-up Strategies for Small Businesses through Embedded Experiments (Plus) Katherine Jenny Thompson Stephen Kaputa Economic Statistical Methods Division Any views expressed are those of the authors and not necessarily those of the U.S. Census Bureau.
Acknowledgments • Laura Bechtel • Daniel Whitehead • Alfred “Dave” Tuttle • Jennifer Beck • Michael Padgett • Eddie Salyers • Robert Struble
Timeline for Our Research 2014 (and 2016) Nonrespondent Subsampling Methods
Timeline for Our Research 2015 2014 (and 2016) Contact Strategy for Nonrespondent Nonresponse Subsampling Follow-up Methods (NRFU)
Timeline for Our Research 2016 2015 Adaptive Collection 2014 (and 2016) Contact Design Strategy for Nonrespondent Nonresponse Subsampling Follow-up Methods (NRFU)
Timeline for Our Research 2016 2015 Adaptive Collection 2014 (and 2016) Contact Design Strategy for Nonrespondent Nonresponse Subsampling Follow-up Methods (NRFU)
Context • Research on survey design and data collection features for 2017 Economic Census – Survey design – simulation – Data collection – field tests in annual business surveys • Adaptive collection design protocols considered for NRFU – Nonrespondent subsampling – Targeted allocation
Timeline for Our Research 2016 2015 Adaptive Collection Design Field Test 2014 (and 2016) Contact Strategy for (Embedded Nonresponse Experiment) Follow-up (NRFU) Nonrespondent Field Test Subsampling (Embedded Methods Experiment) Simulation Studies
Economic Census • Conducted every five years • Covers eighteen non-farm sectors • Surveys over 4 million establishments • Produces industry and geographic estimates (benchmark measures of the economy) • Provides data for sampling frames
Business Organization Structures (Simplified) Single Unit (SU) Single-Unit Company (SU) Establishment • Operate in one primary industry • 1 Economic Census questionnaire
Business Organization Structures (Simplified) Single Unit (SU) Multi Unit (MU) Single-Unit Multi-Unit Company Company (SU) Establishment Establishment Establishment Establishment 1 2 n • • Can operate in more than one industry Operate in one primary industry • More than 1 Economic Census • 1 Economic Census questionnaire questionnaire
Economic Programs • Collection Procedures and NRFU – Designed to obtain respondent data from the largest cases – Small units rarely receive personal contact • Multi-unit establishments (Economic Census) – Company and establishment level NRFU strategies – Post-data collection completeness and coverage procedures • Mandated lower bound on the unit response rate
Consequences • Limited research on data collection strategies for smaller businesses • Distinct possibility (prior to our research) that realized set of respondents were not a random subsample – Potential for nonresponse bias in survey totals
Annual Survey of Manufactures (ASM) • Alternative to Economic Census in off- census years (manufacturing sector) – Similar electronic questionnaire (same items) – Similar editing/imputation procedures • Different sampling design – Census: Stratified SRS-WOR – ASM: Stratified PPS-WOR
Timeline 2016 2015 2014 (and 2016) Nonrespondent Subsampling Methods Simulation Studies
Motivation • Find allocation method for nonrespondent subsampling of Single-unit businesses • Maintain mandated unit response rates – Targeted industry-specific r esponse rates • Reduce – Cost of NRFU – Nonresponse bias
Nonresponse Subsampling • Stratified systematic sample design – Sorted by measure of size • Allocation – Equal probability (1-in-K) sampling – Optimal Allocation that… • minimizes deviation between industry unit response rates (Min-URR) • minimizes deviation between industry sampling intervals (Min-K)
Optimal Allocation • Formulated as quadratic programs with linear constraints on overall and industry level response rates • Subsampling only applied in designated subdomains (“ Noncertainty Single- units”)
Relative Bias of the Estimate Combined ratio estimator with an estimated conversion rate 1-in-2 Nonrespondent Subsample
Relative Bias of the Estimate Combined ratio estimator with an estimated conversion rate 1-in-2 Nonrespondent subsample • Systematic subsample with 1-in-K allocation not effective
Relative Bias of the Estimate Subsampled Subdomains Combined ratio estimator with an estimated conversion rate 1-in-2 Nonrespondent subsample • Slight increase in bias • Additional contacts (rounds) not helpful
Mean Squared Error Combined ratio estimator with an estimated conversion rate 1-in-2 Nonrespondent subsample • Large increase in variance • Additional contacts not helpful
Conclusions • Developed clever allocation strategies for NRFU with potential to decrease nonresponse bias – Quality cost – increased variance – $ cost – little savings • Could offset variance gains with increased response from subsampled cases • Needed new contact strategy for these “hard to reach” establishments
Timeline for Our Research 2016 2015 Adaptive Collection Design Field Test 2014 (and 2016) Contact Strategy for (Embedded Nonresponse Experiment) Follow-up (NRFU) Nonrespondent Field Test Subsampling (Embedded Methods Experiment) Simulation Studies
Motivation • Find effective NRFU contact strategy – Small businesses only – Improve response rates for chronic nonresponders • Reduce nonresponse bias improve estimates
Embedded Experiment 2014 Annual Survey of Manufactures (ASM) Single-unit establishments Certainty (larger) Noncertainty (smaller) – Domain of interest (target) Split panel design
Contact Strategies by Treatment Panel 1 st NRFU 2 nd NRFU 3 Rd NRFU 4 th NRFU Panel Initial Mail Control Letter Letter Form Letter Letter (C) Treatment 1 Letter Letter Certified Form Letter (T1) Letter Treatment 2 Letter Letter Letter/Flyer Form Letter (T2) Same Treatment for all panels
Contact Strategies by Treatment Panel 1 st NRFU 2 nd NRFU 3 Rd NRFU 4 th NRFU Panel Initial Mail Control Letter Letter Form Letter Letter (C) Treatment 1 Letter Letter Certified Form Letter (T1) Letter Treatment 2 Letter Letter Letter/Flyer Form Letter (T2) New treatment in each panel
Treatment 2: Letter/Flyer
Analyses Conducted Statistic/Indicator Examines effects on Proxy Unit Response Rate Unit Response Cox Proportional Hazards Regression Model Length of Time to Respond Parameters Hazard Ratio Probability of Responding Balance Indicator Representativeness of Distance Indicator Respondent Sample Fraction of Missing Information (FMI) Nonresponse bias for specific outcome variables
Proxy Unit Response Rate Certainty Cases T1: Certified Letter at NRFU 2 T2: Letter/Flyer at NRFU 2 C: Control
Proxy Unit Response Rate Certainty Cases T1: Certified Letter at NRFU 2 Same T2: Letter/Flyer at NRFU 2 C: Control
Proxy Unit Response Rate Certainty Cases T1: Certified Letter at NRFU 2 T2: Letter/Flyer at NRFU 2 C: Control Different New NRFU treatment introduced
Proxy Unit Response Rate Certainty Cases • Letter/flyer slightly more effective than current procedure • Certified letter most effective treatment
Proxy Unit Response Rate Certainty Cases Noncertainty Cases
Proxy Unit Response Rate Certainty Cases Noncertainty Cases • Letter/flyer consistently less effective than current procedure • Certified letter most effective treatment
Survival Analysis: Cox Proportional Hazards Model Length of time to respond Certainty Units: Reduced for T2 (Letter/Flyer) Noncertainty Units: Reduced for T1 (Cert Letter) Probability of (eventually) responding Certainty units: Increases for T2 (Letter/Flyer) Noncertainty Units: Increases for T1 (Cert Letter)
Fraction of Missing Information (FMI) • Bounded between (0,1) • Multiple Imputation • Proxy Pattern-Mixture (PPM) Model • Gamma PPM Model (Andridge and Thompson 2015) • Predict outcome variable from frame measure of size (Proxy) • Obtained different models for respondents and nonrespondents (Pattern-Mixture model) • Measured sensitivity by range of response mechanisms • MAR:missing at random – “Best Case” • not missing at random – “Worst Case” NMAR:
FMI Results – Noncertainty Units Strength of Proxy
FMI Results – Noncertainty Units Unit Nonresponse Rate (UNR) FMI < UNR
FMI Results – Noncertainty Units FMI lowest in T1 FMI highest for T2
FMI Results – Noncertainty Units Difference (spread) indicative of Sensitivity to response mechanism Strength of proxy
2014 Experiment Conclusion Certified letter effective Increases response Reduced NR bias Letter/flyer current procedure No effects on nonresponse No (discernable) effects on NR bias
Recommend
More recommend
