Calculating the eligibility rate of sampling units with unknown - - PowerPoint PPT Presentation

Rita Lima Rita Ranaldi

lima@istat.it ranaldi@istat.it

Italian National Statistical Institute

Calculating the eligibility rate of sampling units with unknown eligibility

7th Workshop on Labour Force Survey Methodology

Madrid, 11 May 2012

7th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012

Classification of units according to eligibility

A core issue is the definition of “eligible unit” as it affects the calculation of response rate

7th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012

Definition of “eligible unit” in the IT LFS

A unit is eligible:

• if the name corresponds to the selected household and
• if it is a private household having usual residence in the

municipality The eligibility is unknown when it is not possible to collect sufficient information for a proper classification, e.g. no contact made during fieldwork period no one at home and no other information available (CAPI) unreachable due to wrong telephone number (CATI)

7th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012

7,5 15,9 9,3 1,5 2,4 1,9 5,4 6,8 5,9 0% 20% 40% 60% 80% 100% Total Wave 1 Wave 2-4 Total Wave 1 Wave 2-4 Total Wave 1 Wave 2-4

Eligible Not eligible Unknown eligibility

T O T A L C A P I C A T I

% of units with unknown eligibility

Units according to eligibility (shares)

IT LFS, by survey mode and wave - 1st quarter 2011

7th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012

Response rate

100 × ⋅ + + = UN EN EI EI RR α

where: EI Number of eligible interviews EN Number of eligible non-interviews UN Number of units with unknown eligibility α Estimated proportion of units of unknown eligibility that are actually eligible

• one of the most important quality indicators for the

social sample surveys

• used between surveys, years and countries to

compare survey quality

Estimating Eligibility Rates: a review

The Minimum and Maximum Allocation method

(MMA)

The Proportional Allocation method (CASRO) The American Association for Public Opinion

Research (AAPOR) approach

The Survival Analysis method (SAM)

7th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012

7th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012

The Minimum and Maximum Allocation method (MMA)

• Defines the lower and upper boundaries of the response

rate: all units with unknown status are actually eligible (upper) or non-eligible (lower)

• So it is possible to define a range of response rates by

setting:

=

upper

MMA

α

1 =

lower

MMA

α

7th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012

The Proportional Allocation method (CASRO)

• Proportion of eligible units amongst those whose eligibility

is unknown = Proportion of eligible units amongst the eligibility known sample units

• CASRO formula for α is:

NE EN EI EN EI

CASRO

+ + + = α

where: EI Number of eligible interviews EN Number of eligible non-interviews NE Number of not eligible units

7th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012

The American Association for Public Opinion Research (AAPOR) approach

• Estimation of the eligibility rate left to discretion of

researchers

− on the basis of the best available scientific information − the basis of the estimate must be explicitly stated and

explained

− if no information available, all units of unknown

eligibility should be considered as eligible

• AAPOR formula for α is:

1 =

AAPOR

α

7th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012

The Survival Analysis method (SAM)

• Estimates the eligibility rate by modeling the “time to

resolution (death)” of each sampling units using survival

• analysis. It uses the additional information on the number
• f attempts until resolution (eligible or ineligible).
• Being resolved as eligible or ineligible is comparable to

two different “causes of deaths”.

) ( ˆ ) ( ˆ ) ( ˆ ˆ

ineligible eligible eligible

S S S R + =

∞ unknown eligible tot SAM

n n n R − × =

∞

) ˆ ( α

where:

Ŝeligible(0)

survival function for resolving as eligible

Ŝineligible(0) survival function for resolving as

not eligible where:

ntot

total sample size

neligible

number of eligible units

nunknown number of units with unknown

eligibility

7th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012

An application from IT-LFS: main results /1

84,6 88,3 71,3 80,0 89,0 69,8 82,5 89,3 66,7

60,0 65,0 70,0 75,0 80,0 85,0 90,0 95,0 100,0 Total Wave 1 Wave 2.4 Total Wave 1 Wave 2.4 Total Wave 1 Wave 2.4

87,7 75,1

T O T A L C A P I C A T I

94,0 81,7 73,2 89,7 93,2 79,4 96,5

Ranges of the response rates according to MMA method in the IT LFS by survey mode and wave - 1st quarter 2011

7th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012

An application from IT-LFS: main results /2

1 wave

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 5 10 15 20 27 34 50 Number of call attempts Elegibility Rate waves 2-4

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 10 20 30 40 50 60 70 80 94 107 123 Number of call attempts Elegibility Rate

Eligibility rate for CATI mode of the IT-LFS according to SAM method by wave and number of call attempts – 1st quarter 2011

The main result is that, insisting endlessly to contact The main result is that, insisting endlessly to contact the units, at the end they are almost all eligible the units, at the end they are almost all eligible

7th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012

An application from IT-LFS: main results /3

Estimated eligibility rates and response rates according to different methods in the IT CATI LFS – 1st quarter 2011

There is no practical difference between the There is no practical difference between the different methods different methods

Eligibility rate (α) Response rate (RR) MMA

lower

MMA

upper

CASRO AAPOR SAM MMA

lower

MMA

upper

CASRO AAPOR SAM Total 1 0.9990 1 0.9990 84.558 93.210 84.566 84.558 84.566 Wave 1 1 0.9959 1 0.9967 66.728 79.439 66.771 66.728 66.763 Waves 2-4 1 0.9997 1 0.9996 89.313 96.545 89.315 89.313 89.316

7th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012

Conclusions /1

MMA method

advantage: advantage: comparability of the results drawback drawback:

: communication issues and the range of response rate

could be great

CASRO method

advantage: advantage: easy to apply and it does not inflate the response rate drawback: drawback: the assumption that the units with uncertain eligibility have the attributes as the units with known eligibility may be too strong

AAPOR approach

advantage: advantage: easy to apply drawback: drawback: it inflates the response rate and the assumption of considering all uncertain units as eligible may be strong

7th Workshop on Labour Force Survey Methodology – Madrid, 10-11 May 2012

Conclusions /2

SAM method

advantage: advantage: it estimates accurately the eligibility rate if we have large sample of units that are contacted more times drawback: drawback: its application to this problem is relatively new and it is a very complicated estimation method for the current practice

Under these conditions the CASRO method Under these conditions the CASRO method produces more similar results to the SAM method for produces more similar results to the SAM method for the IT LFS the IT LFS Moreover it seems the most appropriate one to Moreover it seems the most appropriate one to estimate the eligibility rate in case there is evidence estimate the eligibility rate in case there is evidence that that α α is less than 1 is less than 1

Thank you for your attention