Adaptive Design in an Establishment Survey: Adaptive Design in an - - PowerPoint PPT Presentation

Adaptive Design in an Establishment Survey: Adaptive Design in an Establishment Survey:

Strategic Targeting in the Agricultural Resource Management Survey (ARMS)

• Dr. Jaki McCarthy

T l Wil Tyler Wilson Research and Development Division, National Agriculture Statistics Service National Agriculture Statistics Service

ARMS

l b i l i l l

• Annual survey run by National Agricultural

Statistics Service (NASS) and Economic Research S i (ERS) f th U S D t t f Service (ERS) of the U. S. Department of Agriculture.

• Three stage survey:
• 1. First stage is a screening process
• 2. Second stage captures production expenses,

chemical use, and area-specific commodities

• 3. Third stage focuses on financial data like expenses

and income

• St

3 i th f f thi h

• Stage 3 is the focus of this research

ARMS Stage III g

S l f 30 0 000 f d h

• Sample of 30-40,000 farms and ranches
• Potentially sensitive topics like finance and

household characteristics

• Long survey, interviews can exceed an hour

g y,

• Relatively low history of response rates (50-60%)
• Mixed mode approach mail web with in-person
• Mixed mode approach mail, web, with in-person

interviews follow up

• Nonresponse propensity score to data to target
• Nonresponse propensity score to data to target

some operations

Using Trees to Predict Nonrespondents Nonrespondents

• Census Data imported and used as a proxy of ARMS III respondent

characteristics characteristics – Process repeated to generate alternative trees using each of 70 different variables as the starting point – Each tree generates different groups of nonrespondents Initial efforts targeted operations with >70% nonresponse in ANY – Initial efforts targeted operations with >70% nonresponse in ANY

• f the trees
• Too many records identified
• Some had low overall nonresponse

propensity

– Current efforts use average nonresponse across all trees Current efforts use average nonresponse across all trees – i.e. consistent nonrespondents

An Example Tree p

Initial Use of Models

• Models provide a “profile” of operations
• FO’s provided with least likely to respond

FO s provided with least likely to respond

• perations (>70%NR)

O’ di d fi ld ffi di d i

• FO’s directed to use field office directors, deputies,

and supervisory enumerators to recruit respondents Results: Results:

• Small sample sizes
• Slight rise in response rates, but unclear

A Different View?

Th I b bl O ti hi hl lik l t The Improbables – Operations highly unlikely to respond even when targeted

• 70% or higher nonresponse propensity scores
• 70% or higher nonresponse propensity scores

The Pursuadables – Operations who respond when p p targeted

• 50-69% nonresponse propensity scores

The Sure Things – Operations likely to respond whether targeted or not whether targeted or not

• 0-49% nonresponse propensity scores

Data Collection Driven by Nonresponse Propensity Scores Nonresponse Propensity Scores

• 1. Highly Unlikely to Respond and High Impact on Estimates
• 1. Highly Unlikely to Respond and High Impact on Estimates
• Small group assigned for recruitment to high level staff

(Regional Directors and State Statisticians) to large, hard to t ti get operations

• 2. Somewhat Unlikely to Respond
• Testing a drop off pick up (DOPU) distribution model that
• Testing a drop off, pick up (DOPU) distribution model that

has increased response rates in previous studies (Steel et al. 2001)

• First contact IN PERSON to solicit cooperation and drop off

questionnaire

• Inter ie er sched les SECOND IN PERSON appointment to
• Interviewer schedules SECOND IN PERSON appointment to

pick up completed questionnaire or conduct an interview

• 3. Likely Responders
• 3. Likely Responders
• Standard procedures

The Persuadables

• Test operations identified as unlikely, but not

impossible p

– 50-69% Nonresponse Propensity using classification trees (Earp and McCarthy 2010) classification trees (Earp and McCarthy 2010)

• Why investigate this subgroup?
• 1. With alternative collection strategies (incentives,
• 1. With alternative collection strategies (incentives,

face-to-face…) these operations could prove more likely responders more likely responders

Influencing the Persuadables

• Customized approach to the hand delivery

(drop off, pick up) survey distribution (Allred ( p , p p) y ( and Ross-Davis 2011)

• NASS’ hand delivery mode sought to…
• 1. Increase face to face interaction

2 Increase the use of token items and sponsorship

• 2. Increase the use of token items and sponsorship
• 3. Provide additional flexibility to respondent

h d f f l

• 4. Decrease the opportunities and ease of refusal

The Drop Off Approach p pp

Unique to this study…

• No pre mailing or phone call
• Must establish contact before
• Must establish contact before

‘dropping off’

• Required to leave the packet with

Required to leave the packet with respondent even if they refuse

• After 2 personal contact tries,

surveyor can default to standard procedures…

The Data Collection Bag

The bag helps facilitate the basic requirements of cooperation in a social exchange:

Reciprocation, consistency, social validation, authority, l k ( ld )

• Questionnaire

liking, scarcity (Groves, Cialdini, Couper 1992)

Questionnaire

• DOPU Cover Letter
• Brochure

Brochure

• Postcard
• Door Hangers

Door Hangers

• ERS Data Uses Fact Sheet
• NASDA Token Item

NASDA Token Item

• Privacy Envelope

Experimental Design p g

Targeted all operations with nonresponse propensity scores of 50 69% (n = 1552) scores of 50-69% (n = 1552) – Randomly split into treatment (n = 774) and l ( 778) control (n = 778) – Mandatory follow-up supplement sheet for treatment sample to verify how case was handled – Selected at a U.S. level, some regions had more , g than others

Data Collection Mode

Mode of Interview between Treatment and Control

83% 62% Face to Face 83% 4% 6% Telephone 24% 9% Mail 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Control Treatment

Were New Procedures Feasible in an Operational Environment an Operational Environment

Determinants of Feasibility

1. Training and outreach 2. Handling requirements (sample & materials) 3. Operational deadlines 4. Specific requests from respondents

All these determinants were tracked under a ‘catch All these determinants were tracked under a catch all’ question: Were alternative methods used?

85% Said No

Initial Treatment Sample D d Vi bl Deemed Viable N=658

Was Delivery Possible? y

Were surveyors able to establish contact to deliver y the data collection packet?

Contact Established in Treatment Sample

45% 40% 45% 50%

Contact Established in Treatment Sample

25% 30% 35% 12% 10% 15% 20%

57% (n 388) cases in our treatment sample were able to follow procedures and deliver

0% 5% First Attempt Second Attempt

57% (n=388) cases in our treatment sample were able to follow procedures and deliver the data collection packet during their two face to face attempts

Completion Rates p

Completion Rates

68% 79% 70% 80% 90% 68% 58% 50% 60% 70%

mpleted

41% 30% 40% 50%

Percentage Com

10% 20% 30% 0% 10% 1st Contact 2nd Contact No Contact Established Control

Treatment vs Control

Completion Rate p

Completion Percentage Treatment Control 60% 58%

No significant difference found between the drop off sample and the standard procedures sample H h bl t t bli h f t f However, when surveyors were able to establish face to face contact, cooperation rates were very high for this unlikely group

Cooperation Percentage when Contact Established Treatment 70%

Discussion

h d ff d f ll d d

• When drop off procedures were followed and

contact was achieved, response rates increased b 12% by 12%

• Was it operationally possible to make a second

in-person survey attempt?

• Less inaccessible and out of scope operations in

treatment…

– A possible indication of DOPU’s ability to decrease the ease of refusing the survey

• Mandatory this year

Conclusions

• When personally contacted using a drop off process,

p y g p p , 70% of operations previously identified as 50-69% unlikely to respond completed ARMS III using a drop y p p g p

• ff, pick up process
• Main Determinants:

Main Determinants:

– Face-to-face interaction – Incentives – Incentives – Fewer ‘easy’ ways to refuse Flexibility – Flexibility

• Need to follow up with Field Offices and surveyor to

see why they could not establish contact see why they could not establish contact

Further Research

M E – Measurement Error

• Do harder to get respondents provide less quality data?

H d d id l li d

• Harder to get respondents provide less quality data

(Dahlhamer 2012)

• Examination of raw edited and imputed rates between

Examination of raw, edited, and imputed rates between samples

– Response Bias p

• Are harder to get respondents different (with respect to

estimates) than easier to obtain respondents?

• Analysis of distributions

– Resources

• Is this an effective use of data collection dollars?

Acknowledgments and Citations

• Shiela Corley Head of Environment and Economic Survey Section NASS
• Shiela Corley, Head of Environment and Economic Survey Section, NASS
• Ryan Skipper, Team Leader Frames Maintenance Group, NASS
• Andrew Dau, Summary Estimation and Disclosure Methodology, NASS

l h k bl ff l

• Alex Minchenkov, Public Affairs Specialist, NASS

Allred, S. B., & Ross-Davis, A. (2011). The drop-off and pick-up method: An approach to reduce nonresponse bias in natural resource surveys. Small-Scale Forestry, 10(3), 305-318. D hlh J M Th I t ti f R P it d D t Q lit i th N ti l H lth I t i S Dahlhamer, J. M. The Intersection of Response Propensity and Data Quality in the National Health Interview Survey (NHIS). Earp, M., Mitchell, M., McCarthy, J., & Kreuter, F. (2014). Modeling Nonresponse in Establishment Surveys: Using an Ensemble Tree Model to Create Nonresponse Propensity Scores and Detect Potential Bias in an Agricultural Survey. Journal of Official Statistics, 30(4), 701-719. G Ci ldi i & C (1992) U d di h d i i i i i bli Groves, R. M., Cialdini, R. B., & Couper, M. P. (1992). Understanding the decision to participate in a survey. Public Opinion Quarterly, 56(4), 475-495. Steele, J., Bourke, L., Luloff, A. E., Liao, P. S., Theodori, G. L., & Krannich, R. S. (2001). The drop-off/pick-up method for household survey research. Community Development, 32(2), 238-250.