Thoughts on origins, developments and future directions . Journal of - - PowerPoint PPT Presentation

Responsive Design after 15 Years: Thoughts on origins, developments and future directions.

Journal of Official Statistics (JOS) and DC-AAPOR Workshop on Responsive and Adaptive Survey Design Washington, DC March 14, 2018 Steven G. Heeringa, University of Michigan sheering@umich.edu

1

Why are these guys smiling?

2

Answer

It was 1981

3

Friendly Perspectives

• Groves, R,M. (2011). “Three Eras of Survey

Research”, Public Opinion Quarterly, Vol 75, No. 5,pp 861-871.

– 1930-1960: The Era of Invention – 1960-1990: The Era of Expansion – 1990-Present: ? – “Survey research is not dying, it is changing.”

Why’s of Responsive Design (ca 2000)

• One-off surveys are mounted with large

uncertainties (e.g., eligibility of frame elements, effort required to contact sample units, cooperation rate, new survey measures, length of interview).

• Most survey budgets are relatively fixed at start of

project.

• Some survey errors are functions of effort during
• production. Hence, total quality is out of control of

researcher unless designs are permitted to change based on production experience.

5

Antecedents (prior to 2000)

• Replication in sample release
• Experiments embedded in surveys
• Matrix sampling, randomized measurement
• Double sampling, two-phase sampling
• Subsampling for nonresponse

– Hansen and Hurwitz (1946), many others

• Sequential analysis, adaptive trials

– Wald (1947)

• TSE, Survey Quality, TQM

– Morganstein and Marker (1997)

6

2004

“We make no pretense that our concept of responsive design represents a theoretical breakthrough. Nor do we wish to claim that we have substantially refined methods for sample selection, questionnaire design or survey data collection procedures.”

“The theme of this paper is that the entire survey process, from design through data collection should be responsive to both anticipated uncertainties that exist before data collection begins and to real time information obtained throughout the survey data collection.”

7

Essential Survey Conditions that Supported the Responsive Design Concept

• Complex scientific survey designs that included substantial uncertainty in

design parameters and operational features.

• Cost structures that were highly dependent on decisions being made in

the field or data collection centers, often without “evidence base”

• Resistance to survey participation

– Declines in overall levels of response – Heterogeneity of response mechanisms (Groves and Couper, 1998).

• Advent of “Real time” systems for sample management, data acquisition,

paradata capture in distributed interviewing settings.

• Increasing availability of augmented sample frames, administrative data,

GIS, and other survey-related technologies.

8

RSD Met the Real World: Essential survey conditions continued to change

– Continued escalation of costs – Continued decline in survey participation – Mixed modes (often sequential) introduced to adapt to technology, cost factors – Heterogeneity of survey designs and conditions: RSD features that worked better in large scale, protracted in-person surveys (e.g. National Survey

• f Family Growth) were less successful when applied

under different modes, time constraints.

9

The Real World Responds: Response to changing conditions and new opportunities

– Initially, a rush to using RSD/ASD to tackle nonresponse, reduce end-game costs. – Adaptive survey design for repeated data collections to exploit historical paradata to address time constraints – Predictive modeling of individual sample element propensities, outcomes, paradata promise ? – Increasing use of metrics, analytic approaches to gauging RASD total survey error.

10

Major advances in RASD theory and methods of the past decade reflected in this workshop.

• Responsive survey design has moved on from simply

“plugging the dike” to empirical and theoretical investigations focused on optimizing the cost and error trade-off.

• Adaptive survey design—borrowing strength from previous

study replications.

• RASD- Moving to individualized tailoring in real time.
• Deepening understanding of the mechanisms that produce

nonresponse bias, the heterogeneity of bias, the role of responsive balancing in respondent samples, stopping rules.

11

RASD in Survey-assisted Applications

12

New (Primary) Data Needs Ancillary Data (Big or Small) RASD Data Collection Optimal Estimation And Inference

Design, Measurement, Inference RASD Adaptation to Available Data Resources

Traditional Survey Design, Measurement, Inference Model-Assisted Survey Design, Inference

Survey-Assisted Modeling Modeling, Data Mining, “Analytics”

Information Content of Available Data

High Low

13

“Survey-Assisted Modeling”

• Survey “assist” provides:

– Model “training”: Timely estimates of model parameters relating outcomes of interest to the covariate information in the big data systems – Model refinement: More complete information on multivariate associations, mediating and moderating effects and chronological/spatial variation in the big data models (e.g. Army STARRS Suicide Model). – “Compensation” for noncoverage, non-observation or missing data in the existing data inputs to the model. – “Understanding of error structure” in large scale data systems as it relates to the modeling objective.

14

Future Directions for RASD Development

• RASD methodology will increasingly be called upon to support

systems of model-based estimation, inference and prediction.

• Optimal, cost efficient designs for such applications will need

to developed in the context of the statistical information that is present in existing sources of data and the specific aims of a combined analysis based on multiple data sources.

• Sampling statisticians and survey methodologists will need to

transfer their general knowledge of RASD and total survey error to problems of survey-assisted modeling.

15

Thank you!

16