American Community Survey Jessica Holzberg Center for Behavioral - - PowerPoint PPT Presentation

Assessing the Burden of the American Community Survey

Jessica Holzberg Center for Behavioral Science Methods

DC-AAPOR Respondent Burden Workshop October 21, 2019

Disclaimer: This presentation is released to inform interested parties of research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau. CBDRB-FY19-CED001-B0018

Burden is “the product of an interaction between the nature

• f the task [objective burden]

and the way it is perceived by the respondent [subjective burden]” (Bradburn 1978, p. 36)

2

What might contribute to burden in a household survey?

3

Survey characteristics Respondent characteristics Respondent perceptions

Respondent burden

Adapted from Fricker (2016)

• Questions
• Length
• Response mode
• Contacts (#,

mode, message)

• Mandatory
• One-time or

longitudinal

• Demographics

(age, education, etc.)

• Attitudes
• Interest
• Motivation
• Household size

The American Community Survey (ACS)

• The ACS produces estimates on more than 35 topics:
• Demographic (age, sex, race, relationship to householder)
• Social (marital status, disability, school enrollment, veteran status)
• Economic (health insurance coverage, employment status,

income/earnings)

• Housing (tenure, year structure built, home heating fuel)
• The ACS takes an average of 40 minutes to complete
• Response to the survey is mandatory

4

The ACS (cont.)

• ACS samples approximately 3.5 million addresses annually (about

290,000 addresses per month)

• Response modes
• Self-response
• Internet
• Paper
• Interviewer-administered response (if no self-response received)
• CATI (until October 2017)
• CAPI

5

Present study

• Until recently, ACS research has focused on objective burden
• Goal: Develop questions to measure respondents’ subjective

perceptions of burden of the ACS

• Literature review
• Focus groups
• Cognitive interviews
• Questions may be field tested and implemented on the ACS as
• ptional, follow-on questions

6

Research questions

• 1. What is respondents’ overall level of perceived burden in being

contacted by and responding to the ACS?

• 2. Which features of the ACS contribute to or affect respondents’ level
• f perceived burden? How much does each of these survey features

contribute to respondents’ perceived burden?

• 3. Which respondent and household-level characteristics, if any,

contribute to variation in respondents’ perceived level of burden?

7

Focus groups (spring 2017)

• 10 focus groups with 46 ACS respondents held across 3 cities
• 5 self-response groups, 5 interviewer-administered response groups
• 3-8 respondents* per group
• Respondents recruited by telephone
• Recruitment targets focused on response mode; demographics collected

during focus groups

• Fairly even on sex, race (white v. nonwhite), education (less than

Bachelor’s degree v. Bachelor’s degree or higher), age (younger than

• v. older than 45)
• Most were non-Hispanic and lived in 1-2 person households

8

*Variation in size of the focus groups was by design as part of separate methodological research

Focus groups (cont.)

• Moderator’s guide

1. Open discussion of ACS experiences 2. 13 survey questions on the burden of the ACS, mostly adapted from 2013 Consumer Expenditure Survey

9

Takeaways from focus groups

• Respondents did not consider the ACS to be burdensome
• “Took a little time for me to do it, but I wouldn’t have exactly called it

burdensome.”

• Diversity in how respondent perceptions were formed
• “It only took about 10, 15 minutes. He asked me basic questions…”
• “It had a lot of questions.”

10

Takeaways from focus groups (cont.)

• Most respondents were thinking about features of the questionnaire

when considering the burden of the survey

• Length of the survey, question difficulty, question sensitivity
• Fewer comments on other aspects of survey, such as contacts
• “It was just the right amount for me because I kept forgetting.”
• Some evidence of potential differences in perceptions by respondent

characteristics

• “I think a lot depended [on] if you had 3 people in your household versus 7 in

your household.”

• “I just felt like it wasn’t my information to give, I guess.”

11

Questions for cognitive testing

• Overall burden of the ACS (two versions: “burdensome” v. “hassle”)
• Questionnaire
• Length of survey response
• Whether length of survey response met respondents’ expectations
• Difficulty answering ACS questions
• Contacts
• Number of contact attempts
• Appropriateness of contacts
• Level of effort put forth in responding to the ACS
• Perceptions of whether the ACS “invaded” their privacy

12

Cognitive interviews (fall 2017-summer 2018)

• 62 cognitive interviews across 2 rounds and 5 cities
• Prior ACS respondents and newly recruited respondents
• New respondents took ACS prior to cognitive interview
• Recruitment
• Prioritized diversity in demographics, household composition, response mode
• Prior ACS respondents recruited as in focus groups
• Traditional cognitive interview recruitment methods used for new

respondents (Craigslist, flyers, etc.)

13

Cognitive interviews (cont.)

• Demographics
• Fairly diverse on sex, race (white v. nonwhite), education (less than Bachelor’s

degree v. Bachelor’s degree or higher), age (younger than v. older than 45), household size, owning v. renting

• Most were non-Hispanic, income $50,000+, and lived only with related

household members

• Respondents answered burden question series and were debriefed on

their answers

• Question revisions made between rounds

14

Takeaways from cognitive interviews

• Similar to the focus groups
• Respondents did not consider the ACS to be burdensome
• “It didn’t bother me at all to do it.”
• No big differences in comprehension, response to two versions of the
• verall burden question
• “Hassle” more plain language but also perhaps too informal

15

Takeaways from cognitive interviews (cont.)

• Most respondents were thinking about the questionnaire when

considering the overall burden of the survey

• “It really did not take that much time to complete.”
• “The survey itself was pretty straightforward.”
• Some evidence of potential differences by response mode, household

composition and size

• “I don’t know if it’s unique to my situation or not but there were a lot of

questions I couldn’t answer…”

16

Selecting questions for further testing

• Questions generally well understood and able to be answered
• Asking questions about burden can increase burden of a survey
• Need to be selective, implement only a minimal number
• Priority given to questions that captured overall perceptions or

unique perceptions that other questions did not

17

Evolution of burden questions across cognitive interviews

Round 1 Round 2 Recommendations for field testing Overall burden of the ACS

• “Burdensome”
• “Hassle”

Overall burden of the ACS

• “Burdensome”
• “Hassle”

Overall burden of the ACS

• “Burdensome”
• “Hassle”

Questionnaire:

• Response length
• Perception of length
• Difficulty

Questionnaire:

• Response length
• Perception of length
• Difficulty

Questionnaire:

• Response length

Contacts

• Number of contact attempts
• Perception of contacts

Contacts

• Number of contact attempts
• Perception of contacts

Effort Effort Effort Whether ACS “invades” privacy Whether ACS “invades” privacy Whether ACS “invades” privacy

18

(The exact wording of retained questions was revised after round 1 and at the conclusion of cognitive testing)

Limitations

• Gap between ACS response and focus group/cognitive interview

participation

• Unlikely to have recruited the most burdened respondents

19

Conclusion

• Burden is a complex, multidimensional construct
• Generally, clear need for more research on measuring subjective

perceptions of survey burden

• Question wording: Difficult to come up with a broad term that captures

“burden”!

• Differences by survey and respondent characteristics
• Final report on ACS burden testing forthcoming

20

Official call coming soon: Journal of Official Statistics Special Issue on Respondent Burden

• 1. Quantitative research on the relationship between respondent

burden, response propensity, nonresponse bias, response rates, item nonresponse, and other data quality outcomes

• 2. Qualitative research on respondents’ subjective perceptions of

survey burden

• 3. Differences in respondent burden across different survey modes
• 4. The effects of objective versus subjective respondent burden and

the relationship between them

• 5. The effects of survey design on respondent burden
• 6. Techniques for reducing respondent burden

21

Jessica.Holzberg@census.gov

22