Surveys Using Multi Modes (DCSS) Annemieke Luiten (CBS) and Karen - - PowerPoint PPT Presentation

Annemieke Luiten (CBS) and Karen Blanke (Destatis)

ESSnet on Data Collection for Social Surveys Using Multi Modes (DCSS)

• 1. The ESSnet Project DCSS

‐ Initiated by Eurostat

• Developing web data collection tools
• Implementing and assessing the impact of mixed

mode data ‐ Labor Force Survey (LFS) chosen as a concrete example ‐ Duration of the project: autumn 2012 – 2014 ‐ Consortium:

• Partners (5): FI, NO, NL, UK, DE (coordinator)
• Support-group-members (3): SE, DK, IT

ESSnet DCSS - Lessons learnt

Design LFS1 LFS2+ EU-SILC ICT Census HBS Single mode

Capi 10 2 11 4 2 4 Cati 4 6 1 8 Cawi 1 Papi 8 4 7 6 5 8 Papi + other 1 1 Pap 1 1 4 3 Registry 8 Capi + registry 2 1 Cati + registry 3 3 3 2 Cati - other + registry 1

Mixed mode interviewer - interviewer

Capi - Cati 2 8 4 Cati- Capi - Papi 2 2 2 1 Cati - Papi 2 1 Capi - Papi 1 1 1 Cati - capi + registry 2 4 1 1

Mixed mode interviewer - noninterviewer

Capi - pap 1 1 5 Cati - pap - other 1 Cawi - cati - pap 1 Cawi - capi - papi 1 Cawi - papi 2 Papi - pap 1 1 6 Cati - capi - papi - pap 1 Cati - papi - pap Capi - papi - pap 1 Cawi - papi - pap 1

Cawi - capi - cati - registry

1 1

Cati - capi - pap - registry

1 Cawi - cati - registry 1 Papi - Pap - registry - other 1 1 Cawi - capi - registry 2 Cawi - Capi - Papi - registry 1 Cawi - Capi - Cati - pap - registry 1

Mixed mode noninterview - noninterview

Cawi - pap 1 4 1 Cati - pap Papi - pap Cawi - pap - registry 1

Policy towards the implementation of web data collection for social statistics

 More than half of NSIs have concrete CAWI plans (N=23)

ESSnet DCSS - Lessons learnt

Yes, within the next 5 years Yet, but NOT within the next 5 years Not yet decided No policy so far

13 1 3 6

Conclusions Usability & concepts

‐ Implementation of web questionnaires is possible

• It’s an additional mode with pos. & neg. impact
• Respondents in standard employment: no problem

‐ Challenges

• Loss of interviewer has impact on conveying

concepts

• Length & household approach demanding
• Some LFS subgroups have problems
• Coding occupation & economic sector

ESSnet DCSS - Lessons learnt

Mode strategies (including web)

• 1. Sequential design; web first
• Potential for substantial costs savings
• However, risk of lower response rates
• Challenging with fixed reference week
• 2. Concurrent design; respondent chooses
• Often used when other mode is paper
• Limits costs savings: respondents favour paper
• 3. Sequential design; web last
• Expensive,
• But offers potential to higher response rates

ESSnet DCSS - Lessons learnt

Mode strategies (continued)

• 4. Other mode in wave 1, web in second (and later) waves
• May suffer from high panel attrition (NL)
• But could also work out fine (Canada)
• Depends on mode combination
• 5. Adaptive Survey Design
• Not one design for all cases, but decisions based
• n sample unit characteristics

ESSnet DCSS - Lessons learnt

Web response

• Web cannot (yet) be the only mode
• response rates are low and biased.
• Mixed mode designs on the other hand show response

rates and representativeness that are similar to CAPI.

• Web response rates are dependent on the design and a

range of other influences

• The same persons who respond in CATI / CAPI respond in

web

ESSnet DCSS - Lessons learnt

Mode effects

– Measurement errors are an important source of differences between modes in some but not all surveys. – For the LFS mode effects can mostly be explained with common weighting variables – This is not always the case:

‐ Large mode effects in the Dutch Safety Monitor have led to a restriction in the modes to web and paper ‐ In the Finnish Consumer Sentiments Survey large mode differences led to the decision not to introduce web data collection ‐ Research on the British Opinions Survey showed that mode effects can be explained, but additional auxiliary variables are necessary

ESSnet DCSS - Lessons learnt

Recommendations mode effects

ESSnet DCSS - Lessons learnt

• Mode effects should be taken seriously, but not too seriously;

they are one of many error sources.

• We need to develop rules of thumb for choices in the survey

design:

• Is every sample unit subjected to the same modes,
• Is it possible to adjust afterwards,
• Can we stabilize findings.

Adjustment

ESSnet DCSS - Lessons learnt

• Methods for adjusting for measurement errors can be

developed but

• have limitations
• rely on assumptions that are difficult to verify
• Mixing modes may introduce instability due to variations

in response mode composition.

• Even though the measurement bias in the survey

estimates will not be removed, applying adjustment methods is recommended to keep the measurement bias under control.

Recommendations adjustment

ESSnet DCSS - Lessons learnt

• Research has only just started, and further work needs to be

done.

• Try to prevent the necessity to adjust, by careful

questionnaire design and pre-testing of questionnaires.

• Auxiliary data need to be available and of good quality.

These can be registry data or paradata that resemble registry information.

• Time series will be compromised by the introduction of

mixed mode designs

The future

– Many important challenges still remain – New challenges arise – Continuation as a Centre of Excellence? ‐ Guarantee the retaining and strengthening of the competence on modern data collection methods ‐ Coordination of the actions needed for the harmonisation

• f practices in the ESS.

‐ Support countries in developing efficient strategies ‐ Test other social surveys ‐ Develop general guidelines

ESSnet DCSS - Lessons learnt