The hard-to-survey in EU-SILC Nadja Lamei Statistics Austria, - - PowerPoint PPT Presentation

www.statistik.at We provide information

The hard-to-survey in EU-SILC

Challenges and potential solutions for the Austrian case Summer school 'Reaching out to hard-to- survey groups among the poor' 30 May – 3 June, 2016

Nadja Lamei Statistics Austria, Directorate Social Statistics

www.statistik.at slide 2 | 3 June 2016

Outline

• _ EU-SILC: the project, its political context and some

important facts of fieldwork

• _ The „hard-to-survey“: who are they and what is so

difficult about them?

• _ Any solutions?: strategies for different groups

Folie 3 | 30.03.2016

EU-SILC

European Community Statistics on Income and Living Conditions http://ec.europa.eu/eurostat/web/income-and-living- conditions/overview

www.statistik.at slide 4 | 3 June 2016

Objectives

Comparable statistical information on income and living conditions Some typical questions at hand:

• How are household incomes composed and distributed? What about the

income situation of families, singel-parents households, older persons…?

• For which groups of the population are certain goods and services not

affordable, e.g. going to the doctor‘s, owning a car, having a holiday…?

• How often are children socially excluded because of their parents‘ economic

situation? How often are social risks passed on between the generations?

• How much are tenants burdened by their rents?
• How satisfied are persons with their life, what roll does the financial

situation play in this?

• How much risk of poverty and social exclusion is there really in a wealthy

country like Austria?

EU-SILC as data source for (social) politics in MS and the EU!

www.statistik.at slide 5 | 3 June 2016

• Regulation (EC) No 1177/2003 of the European Parliament and of the

Council of 16 June 2003 concerning Community statistics on income and living conditions (EU-SILC)

• plus implementing regulations:

Definitions Fieldwork and imputation procedures Sampling and tracing rules List of permanent variables Quality reports New material deprivation items from 2016 onwards Yearly Modules Ongoing revision of the European legal documents > Integrated European Social Statistics (from 2019?)

• National Regulation of the Federal Minsistry of Labour, Social Affairs and

Consumer Protection (ELStV, BGBL. 277/II/2010)

Legal Background

www.statistik.at slide 6 | 3 June 2016

1999 Treaty of Amsterdam: Social Politics on the EU‘s Agenda 2000 European Councils of Lisbon and Nice: Poverty must me reduced until 2010 2001 European Council of Laeken: Decision on common indicators in the field

• f social protection and social inclusion

2010 Europe 2020-Strategy for smart, sustainable and inclusive growth: Emphasis on social situation than just economic indicators Targets and indicators: Europe-2020-Targets Fighting poverty and social exclusion in the EU: At least 20 million fewer people in or at risk of poverty and social exclusion Translated into National Action Plans

Political Background

www.statistik.at slide 7 | 3 June 2016

Measuring Poverty in Austria and the EU

„Official“ reporting on poverty since 1990ies 1995: AT‘s EU-accession, European Community Household Panel (ECHP) 2003: new instrument EU-SILC, continous reporting 2004: start of the integrated (cross-sectional and longitudinal) rotational design, 18 participating countries. Now: 28 EU-MS + Norway, Island, Turkey, Switzerland, Macedonia, Serbia and Montenegro. Further development of concepts:

• OECD vs. EU-scale for equivalised income
• Risk of poverty > risk of poverty and social exclusion

Further development of data and methods:

• Survey and administrative data combined
• CAPI, CATI and in the future also CAWI
• Optimizing field work, sampling, weighting > better representativeness and

validity Ever faster, many stakeholders!

www.statistik.at slide 8 | 3 June 2016

Latest results

Short English content: http://www.statistik.at/web_en/statistics/PeopleSociety/social_statistics/poverty_and_s

• cial_inclusion/index.html

More detailed German Version: http://www.statistik.at/web_de/statistiken/menschen_und_gesellschaft/soziales/armut _und_soziale_eingliederung/index.html

www.statistik.at slide 9 | 3 June 2016

Latest results (2)

Risk of poverty Low work intensity Severely materially deprived Risk of poverty or social exclusion

www.statistik.at slide 10 | 3 June 2016

Latest results (3)

Q: STATISTIK AUSTRIA/EUROSTAT, EU-SILC 2008-2015.

Risk of poverty or social exclusion

www.statistik.at slide 11 | 3 June 2016

Fieldwork for EU-SILC in Austria – overview

2003 Cross sectional survey 2004 Start of integrated rotational design 2005 2006 2007 Beginning of in-house field work 2008 2009 2010 National regulation 2011 Register use (for pension variables only) 2012 Extensive register use 2013 2014 2015 2016 2017 New survey infrastructure 2018 Modular questionnaire design and CAWI 2019 New European Legal Act? Mixed Mode Design, field work 100% in-house national financing (before: 2/3 Eurostat)

Some numbers…

• Min. effective sample size:

4,500 (cross-sectional) 3,250 (longitudinal) HHs. Actual sample size reached: ~6,000 HHs net/ year 4 wave panel: ~1,200 HHs Voluntary participation, all persons aged 16+ are surveyed Proxy rate ~ 10% Since 2012: 85% of total sum of household income from registers Rotational design

www.statistik.at slide 12 | 3 June 2016

First wave

• Dwellings /households – see in further detail next slide
• Personal computer assisted interviewing (CAPI)
• Gross sample 2016: 3,528 adresses
• Expected response rate: 65%

www.statistik.at slide 13 | 3 June 2016

Type of sampling: one-stage stratified probability sample Sampling units: dwellings registered in the central residence register (ZMR) Stratification criteria:

_Interviewer units (geographical units below NUTS2 level)

_Since 2016: also information on Household income from registers (first quartile or above) _Disproportional allocation per NUTS2 level according to expected response rates (based on average response of two preceding year)

Sampling: Selection of first wave sample

www.statistik.at slide 14 | 3 June 2016

Follow up waves (2-4)

• Persons sample

_ Tracing of sample persons (movers, splits) _ All households with at least one sample person take part in the survey (i.e. sample persons and “co-residents” are interviewed) _ Non eligible if moved to institutional household or outside Austria

• Computer assisted telephone interviewing (CATI) except for:

_ Households with no valid telephone number _ Households explicitly asking for a CAPI (f2f)-interview _ Method changes are possible throughout the fieldwork period

• Gross sample 2016: 4,787 households, 2-3% split off-households per year
• therof CAPI: 965
• therof CATI: 3,822 (80%)
• Expected response rate: 85%

www.statistik.at slide 15 | 3 June 2016

• Fieldwork period: February – June/July
• Information for households

_ Seprate letters for each wave also in Bosnian/Croatian/Serbian or Turkish _ Leaflet on EU-SILC _ First wave: Booklet „Austria in Figures“ _ Follow up waves: EU-SILC newsletter

• Website and information video: www.statistik.at/silcinfo
• Hotline and e-mail
• Information for local administrative units
• 15 Euro incentive (voucher) for each household upon successfull interview

Fieldwork

www.statistik.at slide 16 | 3 June 2016

Mode (only follow up waves)

planned assignment to mode mode changes response rate by mode changes response rate by planned mode actual assignment to mode response rate by actual mode 2% changes to CATI response rate: 62% n=4.699 (follow-up) n=3.941 (n=7.936 total cross section) (n=5.909 total cross section) 100% response rate: 86% response rate: 81% S: Statistics Austria, EU-SILC 2014. *Follow -up w aves excl. 8 households w hich w eren't processed (reported refusals betw een the w aves etc.). 60% CATI 40% CAPI 100% response rate: 83% response rate: 84% total response rate follow- up waves: 84% 18% changes to CAPI 82% remain CATI 98% remain CAPI 100% 100% response rate: 84% response rate: 86% response rate: 75% follow-up gross sample 100% 27% CAPI 73% CATI

Folie 18 | 30.03.2016

Who are those hard-to-survey?

… and what effect might they have on the statistic‘s outcome?

www.statistik.at slide 19 | 3 June 2016

Nonresponse and Total Survey Error

Groves et al. 2004:48

_ Item-Nonresponse: Missings in Variables Counter-measure -> Imputation

Item Non-Response 2014: 2% for Employment Income (from admin data > lack of identifier for linking but income receipt seems likely), <1% for Unemployment benefits (same reason as for

• empl. income),

10% for Self-employed income (surveyed)

_ Unit-Nonresponse: Missing Persons or complete Households Counter-measure -> Weighting

Is Nonresponse selective?

www.statistik.at slide 20 | 3 June 2016

Nonresponse Error and Bias

Two kinds of Nonresponse Error:

• Variance due to Nonresponse

Random deviation from one net sample compared to all potential net samples due to nonresponse (cf. Groves 2006)

• Nonresponse Bias
• Systematic deviation of the expected value of the estimate in the

net sample from the expected value in the gross sample due to

• nonresponse. (cf. Groves 2006, Eurostat 2009, Särndal & Lundström 2005)

Size of Nonresponse Error gets bigger with _ Lower response rates AND _ Bigger variance between respondents and nonrespondents

Important to know if nonresponse error is systematic, i.e. variable of interest is correlated with nonresponse

www.statistik.at slide 21 | 3 June 2016

Missingness at random?

MCAR MAR NMAR non-ignorable NR

www.statistik.at slide 22 | 3 June 2016

Who is surveyed – who not: Response rates

Rotational group

• 1. w ave
• 2. w ave
• 3. w ave
• 4. w ave

First w ave 2014 2013 2012 2011 Household non-response Total sample 3.229 1.887 1.473 1.347 7.936 Address not existent (DB120 = 23) 132 132 NRh - Household non-response rate in % 36,5 23,6 12,2 10,5 24,3 Rh - Household response rate in % 63,6 76,4 87,8 89,5 75,7 Individual non-response Eligible persons (RB245 = 1+2+3) 3.568 2.615 2.344 2.218 10.745 Personal interview s (RB250 = 11+12+13) 3.557 2.613 2.343 2.216 10.729 Rp - Complete personal interview s in % 99,7 99,9 100,0 99,9 99,9 Source: Statistics Austria, EU-SILC 2014 Total

www.statistik.at slide 23 | 3 June 2016

Resasons for drop out

Rotational group

• 1. w ave
• 2. w ave
• 3. w ave
• 4. w ave

First w ave 2014 2013 2012 2011 Household non-response Total sample 3.229 1.887 1.473 1.347 7.936 Address not existent (DB120 = 23) 132 132 NRh - Household non-response rate in % 36,5 23,6 12,2 10,5 24,3 Rh - Household response rate in % 63,6 76,4 87,8 89,5 75,7 Individual non-response Eligible persons (RB245 = 1+2+3) 3.568 2.615 2.344 2.218 10.745 Personal interview s (RB250 = 11+12+13) 3.557 2.613 2.343 2.216 10.729 Rp - Complete personal interview s in % 99,7 99,9 100,0 99,9 99,9 Source: Statistics Austria, EU-SILC 2014 Total

Rotational group

• 1. w ave
• 2. w ave
• 3. w ave
• 4. w ave

First w ave 2014 2013 2012 2011

• 2 Adress not used

41 25 13 10 89 11 Household sucessfully contacted 3.051 1.857 1.452 1.331 7.691 21 Adress cannot be found 5 5 8 6 24 23 Building does not exist 4 4 24 Not used for living purposes 19 19 25 Empty 75 75 26 No person w ith main residence 34 34 3.229 1.887 1.473 1.347 7.936

D002000 A 002000 Adr dres ess c cont

• ntac

act s stat atus us

Total Total

Rotational group

• 1. w ave
• 2. w ave
• 3. w ave
• 4. w ave

First w ave 2014 2013 2012 2011

• 2 non eligible adress

(D002000 <> 11) 178 30 21 16 245 11 sucessfull 1.968 1.442 1.293 1.206 5.909 21 noone at home 158 64 24 21 267 22 refusal 804 281 101 79 1.265 23 break-off during interview 12 18 3 33 24 language problems 18 2 20 25 no person at home qualified for an interview 2 2 26 entire household temporarily aw ay 15 8 6 7 36 27 household unable to respond (illness, disability…) 73 24 11 9 117 28 other reason for drop-out 1 18 14 9 42 3.229 1.887 1.473 1.347 7.936 Total

D003000 H 003000 Hous

• usehol

ehold c d cont

• ntac

act s stat atus us

Total

www.statistik.at slide 24 | 3 June 2016

Nonresponse-Analysis

Comprison of gross and net sample: (Rich) Sampling frame, Screenings, Interviewer debriefings, Follow up surveys for nonrespondents…) Estimate response rates for relevant groups Results from previous research shows (cf. Glaser/Kafka 2015):

First wave

S: Statistics Austria, EU-SILC 2010

60% 61% 58% 57% 61% 55% 62% 66% 67% 66% 61%

0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 1 2 3 4 5 decile 6 7 8 9 10 Total

Mean response rates by income decile

www.statistik.at slide 25 | 3 June 2016

Nonresponse-Analysis (2)

Results from previous research shows (cf. Glaser/Kafka 2015):

Panel

www.statistik.at slide 26 | 3 June 2016

Nonresponse-Analysis (3)

Results from previous research shows (cf. Glaser/Kafka 2015): Summary of results – Effects on panel participation Negative

• Being deprived
• Larger number of grown-ups in HH
• Main activity full time work or being

in education

• Household with foreigners
• Vienna
• Item-Nonresponse
• Proxyinterview

Positive

• Larger number of children in HH
• Partnership in the HH
• University degree
• Landline telephone
• No moving of the household
• Two-families or semi-detached

house

• Being contacted later during field

work No impact

• Equivalised HH income
• Tenure status
• Response burden

Folie 27 | 30.03.2016

Solutions for (better) treatment of the hard-to-survey?

What we already do and what could be done more

www.statistik.at slide 28 | 3 June 2016

Different measures at different times

During field work: _ close monitoring of field work and response rates _ measures for groups that are hard to reach/interview (several contact attempts, allowing for mode change, incentive, translations,…) After field work: _ weighting and calibration _ keeping-in-touch between waves

www.statistik.at slide 29 | 3 June 2016

Treatment of nonresponse bias - fieldwork

• High response means better net sample size
• BUT: Higher response can lead to higher bias if response rate

is very different for different (relevant) groups

• So effect of fieldwork measures (letter design, incentives,

number and mode of contact) has to be evaluated, on which groups does it have which effect?

• R-Indicator: “Indicator of Representativity”

“Definition (weak): A response subset is representative of a categorical variable X with H categories if the average response propensity over the categories is constant” (cf. Schouten et al., 2009)

www.statistik.at slide 30 | 3 June 2016

Weighting and calibration (1)

Sampledesign Nonresponse Adjustment Base weight Household X-weight Nonresponse Base weight Adjustment Household X-weight (t=1) (t>1) Nonresponse

…

Individual L-weight

panel attrition

www.statistik.at slide 31 | 3 June 2016

Weighting and calibration (2)

Marginal distributions used for calibration Household level _ Household size (1, 2, 3, 4+ HH members) _ Tenure Status _ NUTS2 Personal level: _ Age _ Sex _ No. of Persons with foreign citizenship (aged 16+) _ No. of Persons with receipt of unemployment benefits / employment inc. / pension inc. _ PLUS in LONGITUDINAL WEIGHTS: income below the median equivalized income, income below 60% of median equivalized income (individuals at-risk-of-poverty), Individuals belonging to the population not covered in the panel (migrants and newborns) household weight = design weight * non-response weight * adjustment weight

• design weight: inverse selection probability
• non-response weight: inverse estimated response probability
• adjustment weight: calibration to external sources

www.statistik.at slide 32 | 3 June 2016

What more?

Some ideas for further research: _ (Non)response and Mode effects _ Tailored field work strategies for different groups _ Nonresponse from wave to wave and its effect on poverty dynamics _ Better use of register data in Sampling and weighting (Optimal allocation, calibration, quantification of nonresponse bias and nonrespnse adjustment) _ Analyse nonresponse bias and measurement error together (Total Survey Error)

www.statistik.at slide 33 | 3 June 2016

Literature

• European Commission (2009): ESS Handbook for quality reports. Luxembourg: Office

for Official Publications of the European Communities. (Eurostat methodologies and working papers).

• Glaser, Thomas; Kafka, Elisabeth (2015): Analyse und Behebung von selektivem Bias –

EU-SILC in Österreich. In: Nonresponse Bias. Qualitätssicherung sozialwissenschaftlicher Umfragen. Schriftenreihe der ASI - Arbeitsgemeinschaft Sozialwissenschaftlicher Institute. 395-434.

• Groves, Robert M.; Fowler Jr., Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer,

Eleanor & Tourangeau, Roger (2004): Survey Methodology. Hoboken: Wiley. (Wiley series in survey methodology).

• Groves, Robert M. (2006): Nonresponse Rates and Nonresponse Bias in Household
• Surveys. Public Opinion Quarterly 70 (5, Special Issue), 646–675. DOI:

10.1093/poq/nfl033.

• Särndal, Carl-Erik; Lundström, Sixten (2005): Estimation in Surveys with
• Nonresponse. West-Sussex: Wiley. (Wiley series in survey methodology).
• Schouten, Barry; Cobben, Fannie; Bethlehem, Jelke (2009). Indicators for the

representativeness of survey response. Survey Methodology 35, 101-113.

www.statistik.at slide 34 | 3 June 2016 Please address queries to: Nadja Lamei Contact information: Guglgasse 13, 1110 Vienna phone: +43 (1) 71128-7336 nadja.lamei@statistik.gv.at

The hard-to-survey in EU-SILC

Challenges and potential solutions for the Austrian case

www.statistik.at slide 36 | 3 June 2016

Imputation of income components: Overview

calendar or imputation How long? number of months/times How much gross? How much net? net amount category? category imputation statistical imputation n.a. = no answer N/G = net/gross G/N = gross/net calendar etc. yes no n.a. n.a. n.a. n.a. n.a. N/G-conversion or G/N-conversion gross amount

Income component received?

www.statistik.at slide 37 | 3 June 2016

Imputation of income components: Methods

Imputation (only) for income variables Longitudinal and cross-sectional imputation Longitudinal: using the information of previous years:

• Little & Su (1989)
• Development of distribution of variables and income development of single case

Cross-sectional imputations

• Multiple linear regression models
• Median imputation
• Always: adding an artificial error term (reduction of variance)