Measurement Error in American Community Survey Paradata and 2014 - - PowerPoint PPT Presentation

Measurement Error in American Community Survey Paradata and 2014 Community Survey Paradata and 2014 Redesign of the Contact History Instrument Instrument

M tt Vi il Matt Virgile Center for Survey Measurement U.S. Census Bureau DC AAPOR Summer Conference Preview/Review 2015 A t 3rd 2015 August 3rd, 2015

Disclaimer: This talk is to inform interested parties of research and to encourage discussion. The views expressed on statistical, methodological, technical, or operational issues are those of the author and not necessarily those of the

1

stat st ca , et odo og ca , tec ca , o ope at o a ssues a e t ose o t e aut o a d ot ecessa y t ose o t e U.S. Census Bureau.

CHI Background g

• The Contact History Instrument (CHI) is a data collection instrument

used since 2004 by Census interviewers to collect data on Computer- y p Assisted Personal Interview (CAPI) contact attempts in demographic surveys Launched automatically when interviewers exit the questionnaire for

• Launched automatically when interviewers exit the questionnaire for

a given survey, but may also be launched manually

• Call record data
• Case ID
• Interviewer code
• Date and time of contact attempt

Date and time of contact attempt

• Contact mode
• Contact strategies by interviewer

Fi l t

• Final outcome
• Interviewer-respondent interactions
• Respondent and nonrespondent reluctance/concerns about interview

2

Timeline: CHI Launch Dates by Survey

Launch Date Survey January 2004 National Health Interview Survey (NHIS) April 2005 Consumer Expenditure Quarterly Survey (CEQ) March 2006 Consumer Expenditure Diary Survey (CED) July 2009 Current Population Survey (CPS) October 2010 National Crime Victimization Survey (NCVS) May 2011 Survey of Income and Program Participation (SIPP) May 2011 National Survey of Fishing Hunting and Wildlife-Associated Recreation (FHWAR) May 2011 National Survey of Fishing, Hunting and Wildlife-Associated Recreation (FHWAR) June 2011 American Housing Survey (AHS) August 2011 American Community Survey (ACS) January 2012 National Ambulatory Medical Care Survey (NAMCS) January 2012 National Hospital Ambulatory Medical Care Survey (NHAMCS)

Source: Nelson, Coombs, & Miller (2014)

3

Paradata Quality Issues y

• High frequency of “ghost records” in 2012 ACS (26%) with

empty fields pertaining to the contact attempt (Nelson, Coombs, & Miller 2014)

• Most due to interviewers selecting “just looking” option to exit the CHI

without entering info without entering info

• Unclear whether interviewers bypassed CHI after making a contact

attempt or whether they performed other actions that the CHI did not capture

• Possibility of response order effect on select screens (Bates &

Piani 2005) Piani 2005)

• Respondent Concerns and Contact Strategies screens each contain 23

response options

• Records may contain contact attempt data when no contact

attempt was made

• Interviewers may have been unclear what comprises an attempt
• Interviewers may have been unclear what comprises an attempt

4

2014 CHI Redesign C edes g

• In January 2014, the Census Bureau launched a redesign of

th CHI ith b t ti l h t f th ti the CHI with substantial changes to many of the questions (nCHI), including:

• Removal of “just looking” option exiting the CHI

Removal of just looking option exiting the CHI

• replaced with new “not attempting contact” option and followup

screens on what action(s) interviewer was performing other than a t t tt t contact attempt

• New options to differentiate between incoming and outgoing

telephone contact attempts

• implemented at interviewers’ request
• Changes in Respondent Concern categories
• d

d f 23 t 17 ti

• reduced from 23 to 17 options
• Changes in Contact Strategy categories
• reduced from 23 to 14 options

p

5

CHI vs nCHI Screens (1)

6

CHI vs nCHI Screens (2)

7

CHI vs nCHI Screens (3)

8

CHI vs nCHI Screens (4)

9

CHI vs nCHI Screens (5)

10

CHI vs nCHI Screens (6)

11

Research Question & Methodology

• How did the 2014 CHI redesign impact paradata

quality?

• Data analysis of CHI records from ACS
• Compare distributions for select variables between Jan-June

2013 (old CHI) and Jan-June 2014 (nCHI)

• Largest survey with over 1.3 million CHI records from over

350 000 h h ld i th i d 350,000 households per six-month period

• CAPI cases only – those with an unmailable address, or that

have not yet completed a mail internet or telephone survey have not yet completed a mail, internet, or telephone survey

12

1.6 100%

ACS CHI Mode Distribution ACS CHI Mode Distribution

21.7 23.5 80% 90% 17.9 19.3 70% 80% Ghost Record 50% 60% Ghost Record Not Attempting Contact Telephone (incoming) Telephone (outgoing) 58.6 57.4 30% 40% Telephone (outgoing) Telephone Personal Visit 10% 20% 0% Jan-June 2013 Jan-June 2014

13

Source: ACS Contact History Instrument, January 2013 - June 2013 and January 2014 - June 2014

30%

ACS nCHI: "Not Attempting Contact" Distribution ACS nCHI: "Not Attempting Contact" Distribution

25% 20% 15% 10% 5% 0% Reviewing/changing case info Readying case for transmission Opened case/CHI by mistake Locating activities ACS geocoding Other - specify Verification of Type B/C

14

Source: ACS Contact History Instrument, January 2014 - June 2014

15

Source: ACS Contact History Instrument, January 2013 - June 2013 and January 2014 - June 2014

16

Source: ACS Contact History Instrument, January 2013 - June 2013 and January 2014 - June 2014

4

ACS CHI Mode Distribution ACS CHI Mode Distribution (Average No. Records per HU) (Average No. Records per HU)

0 82 0 88 0.06 3.5 4 0 67 0.73 0.82 0.88 2 5 3 0.67 0.73 2 2.5 Ghost Record Not Attempting Contact Telephone 2.19 2.17 1 1.5 Telephone Personal Visit 0.5 1 Jan-June 2013 Jan-June 2014

17

Source: ACS Contact History Instrument, January 2013 - June 2013 and January 2014 - June 2014

Conclusions Conclusions

• The 2014 CHI redesign reduced measurement error

g and improved paradata quality with a 93% reduction in ghost records!

• “Not Attempting Contact” option provided new info
• n interviewer level of effort on activities in addition

to contact attempts

• Changes to Respondent Concerns and Contact

g p Strategies screens had minimal impact

• The 2014 CHI redesign did not reduce the number of

g “true” contact attempts per HU

18

Questions/Comments?

Matthew.Virgile@census.gov

19