Structure of Responsive and Adaptive Design (R)evolutions In Memory - - PowerPoint PPT Presentation

Structure of Responsive and Adaptive Design (R)evolutions

In Memory of Professor Stephen E. Fienberg, 1942-2016 Responsive and Adaptive Design Workshop March 14, 2018 Asaph Young Chun U.S. Census Bureau

Guest Editor, Journal of Official Statistics Special Issue on Responsive and Adaptive Survey Design

What is RAD?

RAD = Wonderful, extraordinary!

(Youth slang)

The JOS Special Issue in Memory of Professor Stephen Fienberg (1942-2016)

• University Professor of Statistics and Social

Science at Carnegie Mellon University

• Transformative contributor to JOS for decades

(e.g. Fienberg 1994; Fienberg and Makov 1998)

• Guest Editor of JOS Special Issue on Disclosure

Limitation Methods (Fienberg and Willenborg 1998)

• Plenary Speaker in JOS 30th Anniversary

Conference (Fienberg 2015)

Outline

• Introduction
• Reflections on RAD
• Overview of the Special Issue
• Comments Tailored to Two Papers
• Challenges Remaining for RAD
• Conclusions

Introduction

• A rapidly changing survey environment

requires a nimble, flexible design

• Birth of responsive and adaptive survey

design (Groves and Heeringa 2006; Wagner 2008)

• RAD is being evolved

Triple Phenomena to Watch

• Computerization of survey data collection

enables real-time analysis of paradata (Couper, 1998)

• Methods from fields as diverse as machine

learning, operations research, and Bayesian statistics are useful (Early, Mankoff and Fienberg, 2017)

• Evidence-driven policy makers as well as survey

researchers have renewed their attention to administrative records (Chun 2009; Chun, Larsen, Reiter, and Durrant, forthcoming 2018)

Reflections on RAD

• Birth of RAD is a natural reaction to the basic

rationale of survey design that addresses response and measurement errors in population subgroups

• Systematic approach to adaptive design evolved

(Schouten et al. 2013)

• Evolution of RAD is due to:

– increasing pressure on response rates, – use of paradata, – IT-driven data collection methods

Responsive vs. Adaptive

• Responsive survey design originates from

settings with less auxiliary data, long data collection periods and detailed quality-cost constraints

• Adaptive survey design comes from

settings with richer auxiliary data, short data collection periods and structural variation

What Drives RAD?

• Use of Auxiliary data
• Design features/interventions
• Explicit quality and cost metrics
• Quality-cost optimization

What Stems Growth of Responsive & Adaptive Design?

• Literature tends to be produced by survey

statisticians and not by survey managers.

• Survey designs demand for more complex

monitoring and case management systems, as well as explicit C-Q control

• Number of success stories is limited

Overview of JOS Special Issue

• Provide formalized rules for adaptation.
• Examine the impact of responsive and

adaptive designs on the quality of estimates

• Consider adaptive design tailored to panel

surveys

• Examine responsive and adaptive designs

for establishment surveys

Rules for adaptive design

• Paiva, T., Reiter, J.

Stop or Continue Data Collection: A Nonignorable Missing Data Approach for Continuous Variables

• Lewis, T.

Univariate Tests for Phase Capacity: Tools for Identifying When to Modify a Survey’s Data Collection Protocol

• Early, K., Mankoff, J., Fienberg, S.

Dynamic question ordering in online surveys.

• Vandenplas, C., Loosveldt, G., Beullens, K.

Fieldwork Monitoring for the European Social Survey: an illustration with Belgium and the Czech Republic in Round 7

• Burger, J., Perryck, K., Schouten, B.

Robustness of adaptive survey designs to inaccuracy of design parameters

Impact of responsive and adaptive designs on the quality of estimates

• Lundquist, P., Särndal, C.

Inconsistent regression and nonresponse bias: Exploring their relationship as a function of response imbalance

• Brick, M., Tourangeau, R.

Responsive survey designs for reducing nonresponse bias

Comments on Brick, Tourangeau

• Reanalysis of Groves and Peytcheva (2008)
• Response Propensity Model Typology is

viable

• Responsive designs are useful for reducing

NRB by employing unequal efforts (e.g, modes, incentives, LOE for subgroups)

• (Suggestion) Conduct the next round of meta

analysis with NRB studies conducted since 2008, including studies employing AR (Chun, 2009; Chun and Scheuren, 2011)

Groves and Cooper (1998)

• “The decision to participate may be

affected by characteristics of the survey design and topic as well as by characteristics of the population sampled…We shouldn’t expect there to be a simple relationship between nonresponse error and nonresponse rate.” (p 319)

Questions to Brick, Tourangeau

• How would you operationialize costs and

monitor cost-quality tradeoffs in justifying responsive designs that reduce NRB?

• Under what conditions would you make

more efforts to reduce the imbalance of the sample under collection or leverage postsurvey adjustment, such as weighting?

Adaptive design tailored to panel surveys

• Shlomo, N., Plewis, I.

Using Response Propensity Models to Improve the Quality of Response Data in Longitudinal Studies

• Lynn, P., Kaminska, O.

The implications of alternative allocation criteria in adaptive design for panel surveys

• Durrant, G., Maslovskaya, O., Smith, P.

Using prior wave information and paradata: Can they help to predict response outcomes and call sequence length in a longitudinal study?

Comments on Durrant, Maslovskaya, Smith

• Prior wave information and paradata are

useful in predictive analytics.

• Call outcomes of the most recent calls are

the most significant in predicting response

• utcomes in future wave (recency effect).
• (Suggestion) Contact/Participation propensity

models need to reflect the separate processes in their model specification (Groves and Cooper, 1998; Chun, 2009)

Questions to Durrant, Maslovskaya, Smith

• How would you dissect and integrate

contact propensity and response propensity to make your models more rigorous and predictive of outcomes of interest in light of a nonresponse lifecycle?

• What auxiliary data, including

administrative records, would you consider identifying and adding to your models to improve predictive power?

Responsive and adaptive design for establishment surveys

• Thompson, K.J., Kaputa, S.

Investigating Adaptive Nonresponse Follow-up Strategies for Small Businesses through Embedded Experiments

• McCarthy, J., Wagner, J., Sanders, H.

The Impact of Targeted Data Collection on Nonresponse Bias in an Establishment Survey: A Simulation Study of Adaptive Survey Design

RAD Prospects

• Build the toolkit of evidence-based designs
• Learn more about deploying features & cost

allocation differentially across pop subgroups

• Have survey managers work more closely with

statisticians, survey methodologists, cost experts

• Develop rules of phase switching and of

stopping data collection (e.g. AD in clinical trials)

• Design and test cost-quality tradeoff models

(Groves, 1989)

Conclusions

• RAD is evolving today
• Further innovation and cross-fertilization is

required

• Hope the JOS articles be a catalyst of

further innovation in RAD

Thank you for listening!

In Memory of Professor Stephen E. Fienberg 1942-2016

Asaph Young Chun US CENSUS BUREAU Asaph.young.chun@census.gov