Lessons Learned from Implementing Response Propensity Models in - - PowerPoint PPT Presentation

Lessons Learned from Implementing Response Propensity Models in the 2013 Census Test

Gina Walejko, PhD U.S. Census Bureau, Center for Survey Measurement

The views expressed on statistical, methodological, technical, or

• perational issues are those of the author and not necessarily those of

the U.S. Census Bureau.

Outline

• 2013 Census Test
• Day 0 Model
• Case Prioritization

Model

• Systems infrastructure
• Takeaways
• Goals of test
• Case control & tracking
• Model
• Test execution

2

2013 Census Test

• Operational study of Census NRFU procedures
• In field Nov. - Dec. 2013
• 2,077 cases in Philadelphia
• Used existing Census Bureau survey infrastructure
• One goal included implementing automated

propensity model to assign “high priority” cases to CAPI interviewers

• Score propensity to complete on next attempt
• Assign highest seven scores to interviewers daily

3

2013 Census Test

• Operational study of NRFU procedures
• In field Nov. - Dec. 2013
• 2,077 cases in Philadelphia
• Used existing Census Bureau infrastructure
• One goal included implementing automated

propensity model to assign “high priority” cases to CAPI interviewers

• Score propensity to complete on next attempt
• Assign highest seven scores to each interviewer

4

CAPI interviewers assigned instructions based on daily model output.

Day 0 Model

• Day 0 Model
• Used for cases with no contact attempt data
• Used 2010 decennial information to predict

household’s likelihood of response on the first personal visit

• Housing unit status variables, refusal indicators,

respondent information, and household characteristics

• Ran model once on all cases

5

Case Prioritization Model

𝑚𝑝𝑕 𝑞𝑗𝑑 1 − 𝑞𝑗𝑑 = +𝛾1𝑦𝑗𝑑1 + ⋯ + 𝛾𝑙𝑦𝑗𝑑𝑙

• 𝑦𝑗𝑑1, … , 𝑦𝑗𝑑𝑙 are the covariates on the next

contact attempt, c, for the ith case associated with that contact.

• 𝛾1 + ⋯ + 𝛾𝑙 are regression parameter

estimated from all prior contact attempts.

6

How?

• Load frame file
• Collect paradata
• Know workload
• Execute program
• Data setup
• Model
• Business rules
• Generate instructions
• Put instructions on server
• Transmit instructions to interviewers

7

Daily

Available in same place

Lessons Learned

8

• 1. Goals of Model-Based Intervention
• What is the purpose of intervention?
• e.g. reduce cost, reduce NR bias, increase RR
• What are the anomalies of your data

collection?

• E.g. proxy interviews on occupied housing units

(decennial Census)

• E.g. vacant housing units (decennial Census, ACS)

9

• 2. Case Control & Tracking

Considerations

• If working with multiple modes, must identify cases in each.
• Mode switches (e.g. CATI to CAPI)
• Late returns (i.e. in CAPI but completed via self-response)
• Decide how to handle reassignments.
• Two copies of one case (e.g. one on interviewer laptop and one

awaiting supervisor review)

• May be reassigned after instructions already delivered to interviewer
• For data analyses:
• If you cannot guarantee interviewer will receive instructions, attached

what interviewer saw to contact attempt data.

• Save daily propensities.
• For monitoring purposes:
• Consider what data needs to be stored for reports

10

• 3. Model Considerations
• If modeling contacts, decide how to manage

certain attempts.

• Non-sample unit members
• Telephone contact attempts
• Decide how to handle certain case dispositions.
• Vacants and cases that are not housing units
• Cases closed by supervisor’s action
• Late returns completed via another mode of contact
• No contacts (i.e. case “untouched” by interviewer)

11

• 4. Execution
• Obtain interviewer

compliance.

• Design of case

management systems (two approaches)

• Train (untrain?),

incentivize, supervise, monitor.

• Account for

non-compliance in CAPI simulations.

12

Percent Days with Compliant Transmissions Attempted All High Priority Cases 45.22 Did Not Attempt All High Priority Cases but Did Not Attempt Other Cases 6.96 Did Not Attempt All High Priority Cases and Attempted Other Cases 47.83

Questions? gina.k.walejko@census.gov Thanks to:

Tamara Adams, Karen Bagwell, Stephanie Coffey, Jaya Damineni, Chandra Erdman, Susanne Johnson, Ganesan Kakkan, Scott Konicki, Peter Miller, Shadana Myers, and many others!

13

Back-up Slides

14

15

Case Prioritization Model (con’t)

16

Type of input Description Frame data Case’s initial propensity to respond predicted by Day 0 Model Study treatment Whether or not the sample unit is in multi-unit structure Paradata Mode of each contact attempt (telephone or in-person) Total number of contact attempts already made on sample unit If contact made with household member during current or any previous contact attempts If potential respondent expressed reluctance during current or previous contact attempts If contact performed during “peak” hours, weekend or after 6:00 p.m. on weekday

• Admin. records

If more than one person in the housing unit If all sample unit members are less than 30 years-old If all sample unit members are 70+ years-old If there are children under 5 years-old in the house

Systems Testing

• Use an initial test to generate data to build program.

Conduct subsequent test that includes instructions generated by program.

• Examine diverse set of scenarios.
• Interview: e.g. multiple refusals, partial complete, case put on

hold, case reassigned, case recycled from different mode

• Interviewer: e.g. didn’t work first day of field period, got let go

before finishing all cases, didn’t follow expected procedures

• Systems: e.g. instructions did not generate, data transmissions

did not work

• Test exact scenarios.
• Be prepared to adjust scenarios if results aren’t what were

expected.

17

Day 0 Model Details

• Three main-effects stepwise models run on 2010 NRFU cases in

the Philadelphia MSA to determine variables significant in predicting likelihood of response at the first, second, and third personal visit

• Some manual examination and variable changes made to

increase model parsimony

• Due to the high predictive value of the main-effect models, 2-

way interactions excluded

• Data split into two panels, and parameters calculated on one

panel and scored on the other panel to test model

• Determined that first personal visit model most appropriate

because we wanted to predict likelihood of a completed response at the first contact.

• Predictive value was determined using concordance, how often

the model correctly predicted that a response occurred within the 1st, 2nd, or 3rd visit.

18