Measurement Error in American Community Survey Paradata and 2014 - PowerPoint PPT Presentation

Measurement Error in American Community Survey Paradata and 2014 Community Survey Paradata and 2014 Redesign of the Contact History Instrument Instrument M tt Vi Matt Virgile il Center for Survey Measurement U.S. Census Bureau DC AAPOR Summer Conference Preview/Review 2015 t 3 rd 2015 A August 3 rd , 2015 Disclaimer: This talk is to inform interested parties of research and to encourage discussion. The views expressed on statistical, methodological, technical, or operational issues are those of the author and not necessarily those of the stat st ca , et odo og ca , tec ca , o ope at o a ssues a e t ose o t e aut o a d ot ecessa y t ose o t e U.S. Census Bureau. 1

CHI Background g The Contact History Instrument (CHI) is a data collection instrument  used since 2004 by Census interviewers to collect data on Computer- y p Assisted Personal Interview (CAPI) contact attempts in demographic surveys Launched automatically when interviewers exit the questionnaire for Launched automatically when interviewers exit the questionnaire for  a given survey, but may also be launched manually Call record data   Case ID  Interviewer code  Date and time of contact attempt Date and time of contact attempt  Contact mode  Contact strategies by interviewer  Final outcome Fi l t Interviewer-respondent interactions   Respondent and nonrespondent reluctance/concerns about interview 2

Timeline: CHI Launch Dates by Survey Launch Date Survey January 2004 National Health Interview Survey (NHIS) April 2005 Consumer Expenditure Quarterly Survey (CEQ) March 2006 Consumer Expenditure Diary Survey (CED) July 2009 Current Population Survey (CPS) October 2010 National Crime Victimization Survey (NCVS) May 2011 Survey of Income and Program Participation (SIPP) May 2011 May 2011 National Survey of Fishing Hunting and Wildlife-Associated Recreation (FHWAR) National Survey of Fishing, Hunting and Wildlife-Associated Recreation (FHWAR) June 2011 American Housing Survey (AHS) August 2011 American Community Survey (ACS) January 2012 National Ambulatory Medical Care Survey (NAMCS) January 2012 National Hospital Ambulatory Medical Care Survey (NHAMCS) Source: Nelson, Coombs, & Miller (2014) 3

Paradata Quality Issues y High frequency of “ghost records” in 2012 ACS (26%) with  empty fields pertaining to the contact attempt (Nelson, Coombs, & Miller 2014)  Most due to interviewers selecting “just looking” option to exit the CHI without entering info without entering info  Unclear whether interviewers bypassed CHI after making a contact attempt or whether they performed other actions that the CHI did not capture Possibility of response order effect on select screens (Bates &  Piani 2005) Piani 2005)  Respondent Concerns and Contact Strategies screens each contain 23 response options Records may contain contact attempt data when no contact  attempt was made   Interviewers may have been unclear what comprises an attempt Interviewers may have been unclear what comprises an attempt 4

2014 CHI Redesign 0 C edes g  In January 2014, the Census Bureau launched a redesign of the CHI with substantial changes to many of the questions th CHI ith b t ti l h t f th ti (nCHI), including:  Removal of “just looking” option exiting the CHI Removal of just looking option exiting the CHI  replaced with new “not attempting contact” option and followup screens on what action(s) interviewer was performing other than a contact attempt t t tt t  New options to differentiate between incoming and outgoing telephone contact attempts  implemented at interviewers’ request  Changes in Respondent Concern categories   reduced from 23 to 17 options d d f 23 t 17 ti  Changes in Contact Strategy categories  reduced from 23 to 14 options p 5

CHI vs nCHI Screens (1) 6

CHI vs nCHI Screens (2) 7

CHI vs nCHI Screens (3) 8

CHI vs nCHI Screens (4) 9

CHI vs nCHI Screens (5) 10

CHI vs nCHI Screens (6) 11

Research Question & Methodology How did the 2014 CHI redesign impact paradata  quality? Data analysis of CHI records from ACS   Compare distributions for select variables between Jan-June 2013 (old CHI) and Jan-June 2014 (nCHI)  Largest survey with over 1.3 million CHI records from over 350,000 households per six-month period 350 000 h h ld i th i d  CAPI cases only – those with an unmailable address, or that have not yet completed a mail internet or telephone survey have not yet completed a mail, internet, or telephone survey 12

ACS CHI Mode Distribution ACS CHI Mode Distribution 1.6 100% 90% 21.7 23.5 80% 80% 70% 19.3 17.9 Ghost Record Ghost Record 60% Not Attempting Contact Telephone (incoming) 50% Telephone (outgoing) Telephone (outgoing) 40% Telephone Personal Visit 58.6 57.4 30% 20% 10% 0% Jan-June 2013 Jan-June 2014 Source: ACS Contact History Instrument, January 2013 - June 2013 and January 2014 - June 2014 13

ACS nCHI: "Not Attempting Contact" Distribution ACS nCHI: "Not Attempting Contact" Distribution 30% 25% 20% 15% 10% 5% 0% Reviewing/changing Readying case for Opened case/CHI by Locating activities ACS geocoding Other - specify Verification of Type B/C case info transmission mistake Source: ACS Contact History Instrument, January 2014 - June 2014 14

Source: ACS Contact History Instrument, January 2013 - June 2013 and January 2014 - June 2014 15

Source: ACS Contact History Instrument, January 2013 - June 2013 and January 2014 - June 2014 16

ACS CHI Mode Distribution ACS CHI Mode Distribution (Average No. Records per HU) (Average No. Records per HU) 4 4 0.06 3.5 0 82 0.82 0.88 0 88 3 0 0.73 0.73 0.67 0 67 2 5 2.5 Ghost Record Not Attempting Contact 2 Telephone Telephone 1.5 Personal Visit 2.19 2.17 1 1 0.5 0 Jan-June 2013 Jan-June 2014 Source: ACS Contact History Instrument, January 2013 - June 2013 and January 2014 - June 2014 17

Conclusions Conclusions The 2014 CHI redesign reduced measurement error g  and improved paradata quality with a 93% reduction in ghost records! “Not Attempting Contact” option provided new info  on interviewer level of effort on activities in addition to contact attempts Changes to Respondent Concerns and Contact g p  Strategies screens had minimal impact The 2014 CHI redesign did not reduce the number of g  “true” contact attempts per HU 18

Questions/Comments? Matthew.Virgile@census.gov 19