During and after fieldwork: Fieldwork monitoring, quality control - - PowerPoint PPT Presentation

During and after fieldwork: Fieldwork monitoring, quality control and several ways to assess nonresponse bias

Joost Kappelhof

INGRID 2016

Overview

• Paradata & monitoring fieldwork
• Nonresponse and nonresponse bias
• Ways to assess nonresponse bias
• Several ways to deal with nonresponse
• Alternative quality indicators and nonresponse bias

INGRID 2016

Paradata

• Couper (1998) and can mean process data (e.g. call records, timers, etc.)

and/ or auxiliary information, such as sample frame data, interviewer

• bservations, etc.
• Various sources, such as sample frames, contact form s, timers, etc.
• Useful for:
• Find out if fieldwork has been carried out according to specifications
• Number of calls, Timing of calls
• Monitoring fieldwork
• When and how to intervene, interviewer performances, quality control
• Calculate response rates
• Detect fraud
• Nonresponse analysis, bias adjustment and future recommendations
• Number of calls, Timing of calls, Reasons for refusal, Compare early

and late contacts, Compare easy and difficult respondents, final disposition codes.

INGRID 2016

Example of how and what type of paradata is collected in the ESS

• Contact form s in the ESS
• Form to be completed, coded and keyed for every sample unit
• Different versions for address, household and individual sample

frames

• For address samples: household selection procedure (if 1+ )
• For address and household samples: person selection procedure (if

1+ ): Kish grid or first/ last birthday

• Detailed call records (to be coded and keyed)
• Lots of other information (characteristics dwelling, etc.)
• Different fieldwork procedures (phone calls allowed in Nordic

countries)

INGRID 2016

Contact form: for each sample unit

• interviewer number
• target person name and telephone number
• selection of households for address samples
• selection of persons for household samples
• Interviewer judgment on future cooperation (in case of

refusal)

• neighbourhood characteristic form
• type of dwelling
• physical state of building and neighbourhood
• presence of litter/ rubbish
• Vandalism/ graffiti in neighbourhood).

INGRID 2016

Call records

• for each call
• date, exact time, mode
• result (complete/ partial interview, different types of noncontact,

ineligible)

• for each contact
• Final outcome (interview, appointment, refusal (by target person or

someone else)

• different types of not being able to participate (language barrier)
• if ineligible
• why invalid address
• for each refusal
• reasons for refusal and judgments of possible cooperation at future calls
• sex and age of doorstep contact

INGRID 2016

Problems and challenges with the ESS CF and CF data

• Different privacy regulations
• Privacy: no reasons for nonresponse
• Privacy: no observational data
• Privacy: no information on nonrespondents at all
• Forms complicated
• Errors in some countries (final interview before refusal?)
• Codes were missing or unclear
• how do we know if someone has moved out of the country?
• Telephone calls do not fit (hard to keep or too many)
• Data sometimes too perfect
• Comparability of interviewer observations.
• Self-fulfilling prophecy with interviewer judgment future cooperation
• How far can we go?

INGRID 2016

Nonresponse

• Types of nonresponse
• Unit and item
• Types of missingness (Rubin, 1976)
• MCAR (Missing Completely At Random)
• E.g., The reason for the missing data occurs entirely at random. It is

independent of observable variables, such as gender, and the unobservable parameter of interest, such as income

• MAR (Missing At Random -given the observed data, the missingness

mechanism does not depend on the unobserved data)

• E.g., Men are less likely to answer a question about income, but the

missingness has nothing to do with income level after you adjust for gender

• MNAR (Missing Not At Random – nonignorable nonresponse)
• E.g., The reason an answer on income is missing is correlated in an

unknown way with the level of the income (and not on observed variables such as gender)

INGRID 2016

From nonresponse to assessing nonresponse bias

• (Unit) Nonresponse happens at the survey level
• Nonresponse bias is a characteristic of a variable,

not of a survey

• How do you know there is nonresponse bias?
• Can you measure nonresponse bias?
• Can you adjust for nonresponse bias?

INGRID 2016

Response models

Fixed Response Model

• The estimator:
• The expected value:
• The bias:
• The bias of the estimator is determined by:
• Average difference between respondents and nonrespondents.
• Relative size of the nonresponse stratum.

N R k k k R k 1

1 y a R Y n

=

=

∑

R R

E(y ) Y

=

NR R R R NR

N B(y ) Y Y (Y Y ) QK N

= − = − =

The Nonresponse Problem (Jelke Bethlehem, selection)

Response models

Random Response Model

• Estimator:
• Expected value:

where is the mean response probability.

• Bias of the estimator:

The bias is large if

• The target variable and response behaviour are correlated.
• There is much variation in response probabilities.
• The response rate is small.

=

ρ ≈ ≡ ρ

∑

N * k k r k 1

Y 1 E(y ) Y , N

ρ

Y Y R R

R S S B(y ) E(y ) Y

ρ ρ

= − = ρ

N R k k k R k 1

1 y a R Y n

=

=

∑

The Nonresponse Problem (Jelke Bethlehem, selection)

INGRID 2016

When to expect nonresponse bias?

• More men than women?
• More poor than rich?
• More employed than unemployed?
• Related to topic of survey
• Time use
• Employment
• Political interest
• Volunteer work
• Travel
• Interest in topic

INGRID 2016

5 approaches to assess nonresponse bias Groves, 2006

• 1. Compare response rates across subgroups
• 2. Comparisons to similar estimates from other sources
• 3. Look at variance within existing survey
• 4. Using rich sampling frame data or suppl. matched data
• 5. Contrasting alternative postsurvey adjustments for

nonresponse

INGRID 2016

• 1. Compare response rates across subgroups
• Presenting estimates of response rates on key subgroups of the

target population (e.g., age, race, gender, urbanicity subgroups). Generally, the researcher asserts that there is no evidence of “nonresponse bias” if the response rates are similar across subgroups.

• Easy, but can be misleading (not informative about possible

nonresponse biases in estimates based on other survey variables)

• Peytcheva, E., Groves, R.M. (2009). Using variation in response

rates of demographic subgroups as evidence of nonresponse bias in survey estimates. Journal of Official Statistics, 25, 193 – 201.

INGRID 2016

• 2. Comparisons to similar estimates from other sources
• For example, to compare the distributions of age, gender, and other

sociodemographic variables among respondents with those from gold standard (high-quality) survey or population statistics

• Pro:
• Estimates independent of the survey are compared
• Credibility of reference survey can give confidence about estimates
• f the survey in question
• Con:
• Key survey variables of the survey do not usually exist in the

external source

• Differences in measurement between survey and gold standard
• Coverage and nonresponse characteristics of the gold standard

survey are not completely known

INGRID 2016

• 3. Variance within existing survey
• For example, compare early and late respondents (differences in

contactability) or cooperative and reluctant respondent.

• Pro:
• Can be used in many different modes of data collection, with

diverse populations, on diverse topics.

• Con:
• Offers no direct information about the nonrespondents to the

survey (assumption of continuum of resistance)

• Requires process data and the construction of indicators on

contactability and reluctance can be tricky

INGRID 2016

How to measure contactability

INGRID 2016

Indicators of contactability

Number of calls Number of calls to first contact

INGRID 2016

Contactability

Call (nr, time, mode) First contact

INGRID 2016

Problems when measuring contactability

• Has the target respondent been contacted?
• Acknowledge timing and mode of calls
• What is optimised?
• Minimum number of contacts
• Costs of contacts
• Individual interviewer strategies
• Adapt calling pattern to at-home behaviour respondents

(from start or after first call)

• How are interviewers paid
• Rarely at home or away for longer period

INGRID 2016

How to measure reluctance

INGRID 2016

Indicators of reluctance

• Sampling frame
• Comparing reluctance across countries
• One contact
• Duration of recruitment
• Doorstep interaction comments Couper, 1997; Campanelli,

Sturgis and Purdon, 1997; Loosveldt and Storms, 2001

• Interviewer judgment (How difficult was it …

?)

• More contacts: Soft and hard refusals
• Number of contacts (1, 2+ )
• Reason for initial refusal (topic, survey, unspecific)
• Interviewer judgment: likelihood of future cooperation
• Extra incentives

INGRID 2016

Example of reasons for refusal: ESSR4 (Matsuo et al., 2010)

10 20 30 40 50 60 70 80 90 100

Poland Turkey Portugal France Ukraine Greece Czech republic Netherlands Bulgaria Norway Cyprus Belgium Latvia Estonia Russia Romania Spain Israel Ireland Hungary Switzerland Slovak Republic United Kingdom Finland Denmark Slovenia Croatia Sweden bad timing not interested subject unknown/ difficult waste time waste money privacy never do surveys co-operate too often no trust surveys bad experience dislike subject no approval not admit strangers

• ther

INGRID 2016

Problems when measuring contactability and reluctance in a comparative survey

• Different survey organizations and survey

traditions

• Different sampling frames
• Different contact forms
• Different legislations

INGRID 2016

• 4. Enrich sample with data from other sources
• E.g., sampling frame, interviewer observations, registers

(individual, regional),doorstep questionnaire, nr follow-up survey

• Pro:
• Identical measurements for both respondents and

nonrespondents are availabe and accurate estimates of nonresponse bias on those frame or external data variables can be constructed.

• Statistical relationships between those variables and survey

variables can be measured among respondents, to address in a partial way the likely nonresponse biases of survey variables.

• Con:
• The variables available are not all those of key interest;
• Possible missing values in other data source;
• Possible measurement error in the record data.

INGRID 2016

• 5. Contrasting alternative postsurvey adjustments for

nonresponse

• I.e., comparisons of unadjusted respondent-based estimates with

estimates utilizing some weighting class adjustments.

• Pro:
• A large set of alternative estimators measuring the same

population parameter can be compared based on different assumptions and in case of similar results between estimates, one can have more confidence in the size and direction of nonresponse

• bias. If they differ: caution.
• Con:
• Lack of a golden standard (what is correct if different adjustments

yield different estimates?)

INGRID 2016

Dealing with nonresponse: types of post survey adjustment

• Post-stratification
• Linear weighting
• Multiplicative weighting
• Calibration
• Multiple Imputation
• Etc.,..

INGRID 2016

Adjustment weighting (Jelke Bethlehem) 29

Post-stratification

Example Two auxiliary variables: Gender x Age. Weight of young female = 0.209 / 0.150 = 1.393. Now the weighted response is representative with respect to gender and age!

Response (n= 100) Population (N= 1000) Weights Male Female Male Female Male Female Young 23 15 Young 226 209 Young 0.983 1.393 Middle 16 17 Middle 152 144 Middle 0.950 0.847 Old 13 16 Old 133 136 Old 1.023 0.850

INGRID 2016

Considerations when applying post survey adjustment. An example from an evaluation of the post-stratification in the ESS

Positive

Uniform for all countries Adjust all nonresponse (refusal + noncontact) At least an idea of direction of nonresponse bias

Negative

Also sampling errors ViF Not effective when target vars do not correlate with PS vars Measurement errors of PS variables in ESS Different availability and quality of population info across countries Only minor differences in regression parameters found

INGRID 2016

‘Alternatives’ to weighting: responsive survey design and adaptive survey design

• attempt to adjust for nonresponse by design rather than post hoc

Characteristics of Responsive survey designs (Groves & Heeringa, 2006) :

• a long data collection with several instances for intervention
• refusal conversion is possible
• relatively little prior knowledge
• the focus is on learning during data collection and on both structural and

incidental deviations. Characteristics of Adaptive survey designs (Schouten et. al, 2013) :

• relatively short data collection periods
• limited intervention and limited possibility to convert refusers
• strong prior knowledge
• a focus on learning in between waves and on structural deviations only.

In fact, any design phase of responsive survey design, i.e. any period in between interventions, could be adaptive (Schouten & Schlomo, 2013).

INGRID 2016

How to go about it? (Schouten & Schlomo, 2015)

• 1. Choose proxy measures for survey quality, e.g. R-indicator and cv (= σ/ µ)
• f response propenisities, in case of a set of key variable(s);
• 2. Choose a set of candidate design features, e.g. modes or incentives;
• 3. Define cost constraints and other practical constraints;
• 4. Link available frame data, administrative data and paradata;
• 5. Form strata with the auxiliary variables for which design features can be

varied;

• 6. Estimate input parameters (e.g. contact and participation propensities,

costs);

• 7. Optimize the allocation of design features to the strata;
• 8. Conduct, monitor and analyse data collection;
• 9. In case of incidental deviation from anticipated quality or costs, return to

step 7;

• 10. In case of structural deviation from anticipated quality or costs, return to

step 6;

• 11. Adjust for nonresponse in the estimation.

INGRID 2016

Alternative quality indicators and nonresponse bias

• Examples of quality indicators are, (partial) R-indicator, maximal

absolute bias, CV of response propensity, FMI, balance indicators and proxy pattern-mixture analysis.

• They consider aspects such as the differences between

respondents and nonrespondents on characteristics that are

• bserved for the entire sample (i.e., paradata), and the

relationship between these fully observed covariates and the survey outcome of interest.

• Different assumptions (e.g. MAR or NMAR) and requirements (e.g.

sample frame data)

• If assumptions hold, nonresponse bias can be estimated

INGRID 2016

The Representativity or R-indicator

• What is it? ‘representativity’ indicator of a respondents sample,

but only with respect to the (complete frame) variables in the model.

• How does it work? It evaluates differences in the estimated

average response propensities (In essence one can view it as a measure that uses the variability between nonresponse adjustment weights).

• Allows for the estimation of maximal absolute standardized bias

(upper bound nonresponse bias for a hypothetical item), cv of response propensities and/ or (un/ conditional) partial R-indicators.

• For more detail (see : http: / / www.risq-project.eu/ )

INGRID 2016

Some references on alternative quality indicators

• Andridge, R. R. and R. J. Little. 2011. “Proxy Pattern-Mixture Analysis for Survey

Nonresponse.” Journal of Official Statistics 27: 153-180.

• Bethlehem, J., F

. Cobben, and B. Schouten. 2011. Handbook of Nonresponse in Household Surveys. Hoboken, New Jersey: John Wiley & Sons, Inc.

• Särndal, C.-E. 2011. “Three Factors to Signal Non Response Bias With Applications to

Categorical Auxiliary Variables.” International Statistical Review 79: 233-254. DOI: http: / / dx.doi.org/ 10.1111/ j.1751-5823.2011.00142.x

• Särndal, C.-E. and S. Lundström. 2010. “Design for Estimation: Identifying Auxiliary

Vectors to Reduce Nonresponse Bias.” Survey Methodology 36: 131-144.

• Schouten, B., J. Bethlehem, K. Beullens, Ø. Kleven, G. Loosveldt, A. Luiten, K. Rutar,
• N. Shlomo, and C. Skinner. 2012. “Evaluating, Comparing, Monitoring, and Improving

Representativity of Survey Response Through R-Indicators and Partial R-Indicators.” International Statistical Review 80: 382-399.

• Schouten, B., F

. Cobben, and J. Bethlehem. 2009. “Indicators of Representativeness

• f Survey Nonresponse.” Survey Methodology 35: 101-113.
• Wagner, J. 2010. “The Fraction of Missing Information as a Tool for Monitoring the

Quality of Survey Data.” Public Opinion Quarterly 74: 223-243. DOI: http: / / dx.doi.org/ 10.1093/ poq/ nfq007