ESSnet on Data Collection for Social Surveys Using Multi Modes (DCSS) Annemieke Luiten (CBS) and Karen Blanke (Destatis)
1. The ESSnet Project DCSS ‐ Initiated by Eurostat • Developing web data collection tools • Implementing and assessing the impact of mixed mode data ‐ Labor Force Survey (LFS) chosen as a concrete example ‐ Duration of the project: autumn 2012 – 2014 ‐ Consortium: • Partners (5): FI, NO, NL, UK, DE (coordinator) • Support-group-members (3): SE, DK, IT ESSnet DCSS - Lessons learnt
Design LFS1 LFS2+ EU-SILC ICT Census HBS Single mode Capi 10 2 11 4 2 4 Cati 4 6 1 8 Cawi 1 Papi 8 4 7 6 5 8 Papi + other 1 1 Pap 1 1 4 3 Registry 8 Capi + registry 2 1 Cati + registry 3 3 3 2 Cati - other + registry 1 Mixed mode interviewer - interviewer Capi - Cati 2 8 4 Cati- Capi - Papi 2 2 2 1 Cati - Papi 2 1 Capi - Papi 1 1 1 Cati - capi + registry 2 4 1 1 Mixed mode interviewer - noninterviewer Capi - pap 1 1 5 Cati - pap - other 1 Cawi - cati - pap 1 Cawi - capi - papi 1 Cawi - papi 2 Papi - pap 1 1 6 Cati - capi - papi - pap 1 Cati - papi - pap Capi - papi - pap 1 Cawi - papi - pap 1 1 1 Cawi - capi - cati - registry 1 Cati - capi - pap - registry Cawi - cati - registry 1 Papi - Pap - registry - other 1 1 Cawi - capi - registry 2 Cawi - Capi - Papi - registry 1 Cawi - Capi - Cati - pap - registry 1 Mixed mode noninterview - noninterview Cawi - pap 1 4 1 Cati - pap Papi - pap Cawi - pap - registry 1
Policy towards the implementation of web data collection for social statistics 6 Yes, within the next 5 years Yet, but NOT within the next 5 years 13 Not yet decided 3 No policy so far 1 More than half of NSIs have concrete CAWI plans (N=23) ESSnet DCSS - Lessons learn t
Conclusions Usability & concepts ‐ Implementation of web questionnaires is possible • It’s an additional mode with pos. & neg. impact • Respondents in standard employment: no problem ‐ Challenges • Loss of interviewer has impact on conveying concepts • Length & household approach demanding • Some LFS subgroups have problems • Coding occupation & economic sector ESSnet DCSS - Lessons learnt
Mode strategies (including web) 1. Sequential design; web first • Potential for substantial costs savings • However, risk of lower response rates • Challenging with fixed reference week 2. Concurrent design; respondent chooses Often used when other mode is paper • • Limits costs savings: respondents favour paper 3. Sequential design; web last • Expensive, • But offers potential to higher response rates ESSnet DCSS - Lessons learnt
Mode strategies (continued) 4. Other mode in wave 1, web in second (and later) waves • May suffer from high panel attrition (NL) • But could also work out fine (Canada) • Depends on mode combination 5. Adaptive Survey Design Not one design for all cases, but decisions based • on sample unit characteristics ESSnet DCSS - Lessons learnt
Web response - Web cannot (yet) be the only mode - response rates are low and biased. - Mixed mode designs on the other hand show response rates and representativeness that are similar to CAPI. - Web response rates are dependent on the design and a range of other influences - The same persons who respond in CATI / CAPI respond in web ESSnet DCSS - Lessons learnt
Mode effects – Measurement errors are an important source of differences between modes in some but not all surveys. – For the LFS mode effects can mostly be explained with common weighting variables – This is not always the case: ‐ Large mode effects in the Dutch Safety Monitor have led to a restriction in the modes to web and paper ‐ In the Finnish Consumer Sentiments Survey large mode differences led to the decision not to introduce web data collection ‐ Research on the British Opinions Survey showed that mode effects can be explained, but additional auxiliary variables are necessary ESSnet DCSS - Lessons learnt
Recommendations mode effects - Mode effects should be taken seriously, but not too seriously; they are one of many error sources. - We need to develop rules of thumb for choices in the survey design: - Is every sample unit subjected to the same modes, - Is it possible to adjust afterwards, - Can we stabilize findings. ESSnet DCSS - Lessons learnt
Adjustment - Methods for adjusting for measurement errors can be developed but - have limitations - rely on assumptions that are difficult to verify - Mixing modes may introduce instability due to variations in response mode composition. - Even though the measurement bias in the survey estimates will not be removed, applying adjustment methods is recommended to keep the measurement bias under control. ESSnet DCSS - Lessons learnt
Recommendations adjustment - Research has only just started, and further work needs to be done. - Try to prevent the necessity to adjust, by careful questionnaire design and pre-testing of questionnaires. - Auxiliary data need to be available and of good quality. These can be registry data or paradata that resemble registry information. - Time series will be compromised by the introduction of mixed mode designs ESSnet DCSS - Lessons learnt
The future – Many important challenges still remain – New challenges arise – Continuation as a Centre of Excellence? ‐ Guarantee the retaining and strengthening of the competence on modern data collection methods ‐ Coordination of the actions needed for the harmonisation of practices in the ESS. ‐ Support countries in developing efficient strategies ‐ Test other social surveys ‐ Develop general guidelines ESSnet DCSS - Lessons learnt
Recommend
More recommend
