Integrating the Web Mode in Marc Plate Romana Riegler the Austrian - PowerPoint PPT Presentation

Integrating the Web Mode in Marc Plate Romana Riegler the Austrian Household Budget Survey 2014/15 Statistics Austria Directorate Social Statistics First Experiences with a New IT System for Data Brussels Collection in a Mixed Mode Design 12 Mar 2015 www.statistik.at We provide information

Introduction  „ EISNeu “ ( SurveyInfrastructureNew)  A new electronic survey infrastructure tool  Ongoing in-house development since 2013, integrates data collection and field work management of mixed mode surveys: CAWI, CAPI, CATI  Austrian Household Budget Survey 2014/15  Field work: Oct. 2014 – Nov. 2015  Probability sample of addresses from population register  Voluntary, 50 € voucher as incentive  Modes: CAWI (first time ever in Austrian HBS), PAPI/CAPI  All presented findings are preliminary and rely on a limited number of observations from the first months of field work www.statistik.at slide 2 | 12 March 2015

Participation procedure & different modes Notification letter CAWI self-registration, if not: automatic mode switch to CAPI/PAPI (visit by interviewer) 3 phases of the HBS: Questionnaires part I: household/personal ‚ 14 days of diary keeping (reporting period) ‚ Questionnaires part II: household/personal ‚  Concerns beforehand: Are households even willing to participate via CAWI (CAWI response  rate)? Will CAWI result in high break-off rates (no personal contact)?   Importance of the notification letter / process (ongoing experimentation) www.statistik.at slide 3 | 12 March 2015

Reporting periods 5-10 Gross sample=2853 households (~476 per reporting period) CAWI complete sampling losses, not quality neutral sampling losses, quality neutral 2 never started break-off 5,9% 91,3% 2,7% ? before new letter : 1,7 % before new letter: 1,1 % The households not registered for CAWI are switched to PAPI mode (= 100 %) Paper/Pencil (PAPI) complete sampling losses, not quality neutral sampling losses, not known quality neutral 2 yet other 1 refusal break-off 26,2% 39,2% 12,4% 0,9% 3,1% 18,1% TOTAL complete sampling losses, not quality neutral sampling losses, not known quality neutral 2 yet other 1 refusal break-off 29,9% 35,8% 11,4% 3,5% 2,9% 16,5% 1 E.g. language problems, no-one reached at housing unit, etc. 2 E.g. housing unit is vacant, not a private household, etc. www.statistik.at slide 4 | 12 March 2015

Differences in Sample Population The CAWI Sample, as compared to the PAPI Sample, is made up of: 30% MORE Employees 35% LESS Immigrants 65% LESS Retirees 60% MORE People under 45 years 25% MORE Households with Children 65% LESS People above 60 years 270% MORE People with academic degree 60% LESS People without qualification for university entrance Even when controlled for age, education still makes much difference www.statistik.at slide 5 | 12 March 2015

Differences in Key-Statistics by mode CAWI PAPI CAWI PAPI Expenditures for … Above Above No Expend. No Expend. Median Median Private Health Insurance 51% 49% 50% 58% Private Life Insurance 58% 42% 56% 68% Dental Health Care 45% 52% 67% 81% Electricity 52% 48% 47% 52% www.statistik.at slide 6 | 12 March 2015

Selection Effect or Mode Effect? Private Life Insurance CAWI PAPI CAWI PAPI . Expenditures Above Above No expend. No expend. Median Median Highest Education Lower secondary education - 31% - 82% Higher secondary education, without University qualification 53% 49% 39% 61% Higher secondary education, 66% 50% 38% 53% with University qualification Tertiary degree 69% 63% 46% 50% www.statistik.at slide 7 | 12 March 2015

CAWI: Cost effectiveness and high data quality?  Does self-administration via CAWI lead to excessive use of „don‘t know / don‘t want to answer “ and thus a high rate of missing values? HBS contains complex (housing) and possibly sensitive (income) questions Missing values (“don’t know” / refusal) for… PAPI CAWI Total monthly housing cost (rented flat/house) 4% 5% Monthly rent (rented flat/house) 36% 30% Monthly utility costs (rented flat/house) 31% 26% Electricity cost 5% 9% Household income (either exact or in categories*) 3% 9% *Income given only as category (of all who gave an income answer) 27% 12% www.statistik.at slide 8 | 12 March 2015

CAWI: Cost effectiveness and high data quality? Does self-administration via CAWI promote „ sloppyness “ (i.e. speeding through questionnaire)? Answering time for Q. Total housing cost (seconds) Cumulative Percent of CAWI Respondents www.statistik.at slide 9 | 12 March 2015

Summary & Outlook  Start of 2nd Mode CAPI (instead of PAPI) by April is already highly anticipated and will provide further insights  For implementing CAWI as a new mode, surveys need even stronger to be methodologically accompanied: Further explore, quantify and find ways to correct for the „pure“  mode effects. Find useful interpretation of CAWI/CAPI/CATI paradata to create  indicators for data quality. Triangulation of methods (additional use of qualitative methods)  needed to fully understand and compensate measurement errors. www.statistik.at slide 10 | 12 March 2015

Please address queries to: Marc Plate Romana Riegler Contact information: Guglgasse 13, 1110 Vienna phone: +43 (1) 71128-7972 Marc.Plate@statistik.gv.at Romana.Riegler@statistik.gv.at www.statistik.at slide 11 | 12 March 2015