Using process indicators to Using process indicators to assess and - PowerPoint PPT Presentation

1 Using process indicators to Using process indicators to assess and improve the quality of assess and improve the quality of LFS data collection procedures LFS data collection procedures Dag F. Gravem Statistics Norway 1

The background • For the 4th quarter of 2011, the new SIV case management system was introduced for the Norwegian LFS – Built on top of Blaise – Modifications made to Blaise • The old CAI system: mainly list-based CATI interviewing by locally based interviewers (CATI-L) – Electronic lists consisting of sampled families and their members • The SIV system: mainly database-based CATI interviewing by call centre interviewers (CATI-D) – Respondents distributed to ”one by one” – With list-based CATI follow-up by locally based interviewers 2

Implications of the SIV system • Respondents distributed to interviewers ”one by one” – Treated more as sampled individuals , and less as members of a sampled family • The relationship between respondent and interviewer becomes more impersonal – Respondents communicate with several different interviewers instead of one – Call centre interviewers communicate with many more respondents per session than locally based interviewers • Locally based interviewers receive follow-up cases only – This may influence their motivation • More process data is collected! 3

Process data collected for each contact attempt • Interviewer ID • Entry priority – Default – Appointment – Selected by interviewer • Date • Time • Result – Interview – No answer – Non-response • Phone number(s) called • Appointment data • Duration of interview: does not work • …and lots more (too much) 4

Some issues with the process data • Two different sources – Blaise CATI-D system: Automatically generated – CATI-L follow-up: Manually recorded by interviewers � The interviewer may not always follow procedures • Not completely integrated – Reflects the different routines and tasks of CATI-D and CATI-L interviewers • Need for more data – More detailed nonresponse information – Time used for each contact attempt • On the positive side: Process data is available in a file that also contains administrative data and questionnaire data! 5

The SIV system: Hopes and expectations regarding improved process quality • Improved data collection timeliness – At the CATI-D call centre, we may to a larger extent control which survey the interviewer is working on – Potentially leading to improved estimates • Reduced costs – Resources may be allocated more efficiently – Call centre interviewers are cheaper • Data collection more in line with responsive design – Easier to target underrepresented groups – Potentially increasing the representativity of the net sample • What do the process (and other) data tell us..? 6

Timeliness of the data collection No. of days after the end of the reference week that interviews are completed Q1 2011 Q2 2011 Q3 2011 Q4 2011 No. of interviews 19 702 19 334 19 026 18 619 Median value 3 4 4 4 Mean value 7.32 8.78 9.05 10.04 Std. deviation 9.79 10.95 11.82 12.31 7

Timeliness of the data collection • The timeliness of the data collection did not improve in the 4th quarter • It rather continued a negative timeliness trend throughout 2011, with – Fewer and fewer interviews – More and more time needed to complete interviews • Still, a weak point in the new system was detected: the transition from CATI-D to the CATI-L follow-up phase 8

Timeliness, 3rd and 4th quarter compared Number of interviews by day after the end of the reference week. Accumulated. 20000 18000 Interviews accumulated 16000 14000 12000 Q3 2011 10000 Q4 2011 8000 6000 4000 2000 0 1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97 Day after reference week 9

Timeliness, 3rd and 4th quarter compared • The 4th quarter data collection keeps pace with the 3rd quarter until the middle of the 2nd week of data collection. • The 4th quarter data collection then loses pace dramatically through the rest of the 2nd week of data collection • We never managed to catch up • This time coincides with the transferring of non-response from CATI-D to CATI-L – A lag was to be expected, but not this dramatic – Why weren’t we able to catch up? 10

Explanations • Routines were new to the CATI-L interviewers • CATI-L interviewers received cases in mid-week, and had less time to plan and start calling • CATI-L interviewers only received nonresponse follow-up, with cases that previously had been contacted by CATI-D interviewers. – This may have had a negative impact on motivation 11

Motivation: probability of getting an interview by contact attempt. LFS 4th quarter 2011 • CATI-L interviewers generally Contact Probability of get- received cases that had been attempt ting an interview tried, though some cases had not First 27 % been tried due to CATI-D Second 26 % undermanning • We now try to limit the number of Third 21 % contact attempts each case is Fourth 17 % exposed to during the CATI-D Fifth 14 % phase Sixth 13 % 12

Data collection timeliness: some improvement in the 1st quarter of 2012 No. of days after the end of the reference week that interviews are completed Q1 2011 Q2 2011 Q3 2011 Q4 2011 Q1 2012 No. of interviews 19 702 19 334 19 026 18 619 18 383 Median value 3 4 4 4 3 Mean value 7.32 8.78 9.05 10.04 6.95 Std. deviation 9.79 10.95 11.82 12.31 10.98 13

Representativity of the net sample • Response rates have been declining, what about representativity? • R-indicator: useful for measuring representativity over time for one survey – Use of different background variables – A value of 1 indicates perfect representation according to the variables used • LFS R-indicator variables – Gender – Age group (11 values) – Urbanity (39 values) – Level of education (3 values) • Household size ought to be included 14

Response rates and R-indicator compared R-indicator and response rates for the LFS data collection. 1 0,9 0,8 0,7 0,6 R-indcator 0,5 Response rate 0,4 0,3 0,2 0,1 0 Q4 2010 Q1 2011 Q2 2011 Q3 2011 Q4 2011 Q1 2012 15

But what about LFS specific representativity? • Traditionally, 1st wave respondents have been slightly underrepresented. – This was reinforced when SIV was introduced Quarterly response rates for the Norwegian LFS. 2009-2012. 90 85 80 75 Percent Wave 1 70 All 8 waves 65 60 55 50 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 2009 2009 2009 2009 2010 2010 2010 2010 2011 2011 2011 2011 2012 Quarter and year 16

LFS specific representativity, continued • With SIV, 1st wave respondents must be monitored extra closely – Recruiting to the panel is essential – In the 1st quarter of 2012, we started turning the ship… • We should also look at respondents’ reference weeks – The reference weeks towards the end of each quarter suffer from lower response rates • Should we make an LFS specific representativity indicator including these or other variables..? 17

The costs of the data collection Monthly interviewer wage payouts for LFS interviewing 2010-2012 500 Thousand 450 400 NOK 350 Payouts Poly. (Payouts) 300 250 200 0 1 2 0 0 0 1 1 1 2 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 2 y r y r y l y l y l i e i e i r r r r r r l l a p u a p u a p b b u A J u A J u A o o n n n t t a c a c a O O J J J Year and month 18

Man hours LFS Interviewer man hours. 2010-2012 8000 7000 6000 Man hours 5000 4000 Man hours worked 3000 2000 1000 0 0 0 0 0 1 1 1 1 2 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 2 2 2 2 2 2 2 2 2 1 2 3 4 1 2 3 4 1 Q Q Q Q Q Q Q Q Q Quarter and year 19

The costs of the data collection • Not adjusted for wage increases, holidays, interview length • Some of the increases may be attributed to training • The cost figure includes data for April 2012, which was not available at the time the paper was completed – A more linear increase in wage costs than suggested in the paper – Still too early to determine whether SIV in itself has had an effect • The number of man-hours seems to have increased more after SIV – Call centre interviewers have lower wages than locally based interviewers 20

Conclusions • Q: Did the SIV case management system lead to improved data collection quality? A: Inconclusive – The quality indicators presented in the paper have limitations – The new system creates and shapes data collection procedures, sometimes i unexpected ways – We work on re-shaping the system to fit the procedures we want, rather than having to adapt procedures to the system • Easier access to process data combined with administrative and survey data will enable us to do analyses of how the data collection influences data quality more directly – So far, we have had to focus on making systems and routines work – But how does e.g. the number of contact attempts influence bias, variance, mean squared error..? 21