Nonresponse Bias J. Michael Brick, Westat Roger Tourangeau, Westat - - PowerPoint PPT Presentation

Responsive Designs to Reduce Nonresponse Bias

• J. Michael Brick, Westat

Roger Tourangeau, Westat

Adaptive Survey Design Workshop March 14, 2018

Premise

• Increasing nonresponse rates lead to increased

chance of nonresponse bias in estimates and increased data collection costs

• Responsive/adaptive design is a tool to help

conduct surveys efficiently in this environment while lowering nonresponse bias

2

Premise

• Increasing nonresponse rates lead to increased

chance of nonresponse bias in estimates and increased data collection costs

• Responsive/adaptive design is a tool to help

conduct surveys efficiently in this environment while lowering nonresponse bias

• Requires–understanding relationship between

nonresponse rates and bias AND of field

• perations available to reduce nonresponse bias

3

Outline

• Re-examine relationship between nonresponse

rate and nonresponse bias

• Discuss why the nonresponse rate to bias

relationship is not stronger

• Implications for Responsive Design
• Conclusions

4

Nonresponse Rates and Bias

• The theory
• Both the mean and standard deviation of the propensities

(f) are important.

• E.g., the cv(f) =0.5 when f=.80 and cv(f) = 2.0 (4 times

larger) when f=.20….huge increase in f needed to keep bias the same

5 , ,

( , ) ( ) ( )

y y i i r y y

Cov y NR Bias y cv

f f f

   f f f f     

Empirical Studies

• Contrary to expectations, widely cited studies show

little or no relationship

—Curtin, Presser, and Singer (2000) —Groves (2006) —Groves and Peytcheva (2008) —Keeter, Miller, Kohut, Groves, and Presser (2000) —Keeter, Kennedy, Dimock, Best, and Craighill (2006) —Merkle and Edelman (2002)

6

Groves and Petcheva (2008) Meta-Analysis

• Most comprehensive and influential study
• Two conclusions

—Little or no relationship between bias and rate —Tremendous within-study variability in bias

• Second conclusion suggests no study-level

indicator is informative about nonresponse bias

• G&P provided their data set for our re-analysis: 959

relbias estimates from 59 studies

7

Groves and Peytcheva (2008)

• Looked at 59 studies with bias estimates (959 estimates)

8

Reanalysis— G&P Data, by Sample Size

9

• 5

5 10 15 20 25 30 35 40 15 25 35 45 55 65 75 85 95

Mean Absolute Relbias Study Response Rate

10

Correlation between Response Rates and RelBias Measures at the Estimate and Study Level Correlation Unweighted Estimate-Level Correlations Response rate and absolute relbias

• .191 (n=953)

Unweighted Study-Level Correlations Response rate and mean absolute relbias Study-Level Correlations—Weighted by Number of Estimates Response rate and mean absolute relbias Study-Level Correlations—Weighted by Study Sample Size Response rate and mean absolute relbias

• .255 (n=57)
• .402 (n=57)
• .413 (n=57)

Reanalysis Conclusions

• There is a relationship between nonresponse rate

and bias at the study-level

• Some additional study-level characteristics beside

nonresponse rate are important (e.g., method of estimating bias)

• Big differences in bias by study; study accounts for

much of the variance, about a quarter

11

Four Models for Relationship (1)

• Random propensities: Propensities essentially

random, product of many transient characteristics

• f respondents

— “… We’re sort of lucky. The mechanisms that produce the decision to participate or not participate in a survey are myriad; …the covariance between the decision to participate and what we’re measuring tends to be small..” (Groves, 2017)

• Design-driven propensities: Response propensities

largely determined by study-level design features that are largely unrelated to characteristics of the sample members

12

Four Models for Relationship (2)

• Demographic-driven propensities: Propensities

determined by respondent characteristics unrelated to survey variables or corrected by weighting

—Low bias

• Correlated propensities: Response propensities

determined by design features and characteristics

• f the sample members; some groups (high

education, voters, civically engaged, altruistic) consistently respond at high rates

—High bias

13

Large Biases

• A set of variables related to a sense of civic obligation and volunteering

are highly related both to survey participation and these variables.

• Survey estimates involving these variables such as reports about voting

are at substantial risk for large biases (Tourangeau, Groves, & Redline, 2010)

14

Subgroup Entire Sample (Frame Data) Respondents (Frame Data) Bias

Nonresponse

Overall 43.7 (2689) 57.0 (904) 13.3 Telephone Mail 43.2 (1020) 43.9 (1669) 57.4 (350) 56.7 (554) 14.2 12.8

Theory and Empirical Results

• Substantial biases generally rare, although rel-biases may

be large

• Large biases often appear for a specific set of variables

that are correlated with unit response (e.g., volunteering)

• Weighting using correlates like education help reduce bias

somewhat

• For most estimates, random propensities and design-driven

propensities seem reasonable models

15

Responsive/Adaptive Design

• Responsive designs: Designs with multiple phases, with

aim of reducing bias by getting more representative set of respondents (Groves and Heeringa 2006)

• Adaptive designs: Designs tailored from the outset (Luiten

and Schouten, 2013) or adapted continuously throughout the field period (Peytchev, Riley, Rosen, Murphy, and Lindblad 2010) to achieve more balanced set of respondents

16

Implications

17

• Average study-level nonresponse bias can be reduced by

lowering overall nonresponse rate

• Large decrease in nonresponse rate needed to achieve

reductions in bias

• Targeting data collection efforts to reduce variation in

response propensities may be more effective strategy

• Since large nonresponse bias generally associated with

specific variables, any efforts to target those cases my be most promising strategy

Data Collection Tools

18

• Avoid designs that give equal effort to all cases
• Adaptive/responsive designs employ unequal efforts

—Change/vary modes —Change/vary incentive levels —Vary level of effort for different cases or subgroups —Two-phase sampling and focus effort

• Auxiliary data related to propensities (e.g.,

volunteering) are extremely important but often unavailable

Conclusions

• There is a relationship between nonresponse rates and

bias, but it is not strong

• Models suggest that many estimates are likely to have

small biases if data collection efforts are reasonable

• Targeting efforts to reduce variation in propensities is

worthwhile if possible

• Variables with worst biases may be hard to affect

because auxiliaries unavailable in most cases

• Weighting helps too

19

THANK YOU!!!! mikebrick@westat.com

20