Health Interview Survey James Dahlhamer, Aaron Maitland, Ben - - PowerPoint PPT Presentation

National Center for Health Statistics

Does Reducing the Length Matter? An Assessment of Respondent Burden with the Redesigned U.S. National Health Interview Survey

James Dahlhamer, Aaron Maitland, Ben Zablotsky, Adena Galinsky National Center for Health Statistics, USA DC-AAPOR Workshop on Respondent Burden

This presentation does not represent official opinions or statistics of the National Center for Health Statistics or the Centers for Disease Control and Prevention

U.S. National Health Interview Survey (NHIS)

• Purpose: To monitor the health of the U.S. population through the collection

and analysis of data on a broad range of health topics

• Sample: Address-based, multi-stage, clustered national sample of housing

units from every state, to be representative of the civilian noninstitutionalized US population

• Mode: In-person interviews by U.S. Census Bureau interviewers, with follow-

up by telephone if needed

• Data collection: Continuous, with quarterly and annual data files
• Target sample size: Complete interviews for 27,000+ households
• Content redesign: January 2019

2019 NHIS Questionnaire Redesign

• Goals of the redesign

– Improve the relevancy of covered health topics, better meeting the needs

• f data users

– Focus on leading causes of morbidity and mortality, targets of health promotion initiatives, and risk and protective factors – Harmonize overlapping content with other federal heath surveys – Establish a long-term structure of ongoing and periodic topics – Reduce respondent burden

2019 NHIS Questionnaire Redesign: Reducing Burden

• Time-related burden

– Eliminated topics better collected by other federal surveys – Eliminated questions on rare conditions – Limited the length of sponsored content

• Cognitive and emotional burden

– Improved clarity of questions – Grouped similar content together to improve flow – Moved health questions to beginning, sensitive questions to end

• Design changes affecting both

– Household roster → Family interview → Sample child → Sample adult

Research Questions

• Per respondent assessments, did we reduce respondent burden with the

questionnaire redesign?

• Do respondent assessments of the ease or difficulty of answering and the

sensitivity of questions differ between the old and new questionnaires? What impacts do they have on respondent assessments of burden?

• To what extent did we reduce the length of an NHIS interview? What

impact does time spent in the interview have on respondent assessments

• f burden?

Burden Questions

• Main burden question:

– How burdensome was this survey to you? Would you say not at all burdensome, a little burdensome, moderately burdensome, very burdensome, or extremely burdensome?

• Ease or difficulty of answering:

– How easy or difficult was it for you to answer the questions in this survey? Would you say very easy, somewhat easy, somewhat difficult, or very difficult?

• Sensitivity of questions:

– How sensitive were the questions in this survey? Would you say not at all sensitive, a little sensitive, moderately sensitive, very sensitive, or extremely sensitive?

Data/Methods (1)

• Households randomly assigned to receive either the old or new questionnaire

in quarter 4 (October, November, December) of 2018

• Burden questions asked at the end of the sample adult or sample child

module in fully complete interviews

• Bivariate analyses of the following by questionnaire:

– Respondent assessment of burden – Respondent assessment of ease/difficulty of answering – Respondent assessment of the sensitivity of questions – Time spent in the interview

"How Burdensome Was This Survey to You?" by Questionnaire: NHIS, 2018, Quarter 4 (base weights)

53.1% 63.5% 29.6% 24.8% 12.1% 8.3% 3.1% 2.3% 2.0% 1.1%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Old Questionnaire New Questionnaire Not at all A little Moderately Very Extremely

Χ2 p-value < .01

68.9% 77.0% 25.4% 19.1% 5.2% 3.3% 0.5% 0.7%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Old Questionnaire New Questionnaire

"How Easy or Difficult Was it for You to Answer the Questions in This Survey?" by Questionnaire: NHIS, 2018, Quarter 4 (base weights)

Very easy Somewhat easy Somewhat difficult Very difficult

Χ2 p-value < .01

47.2% 50.5% 34.3% 32.2% 13.1% 12.4% 4.0% 3.8% 1.5% 1.2%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Old Questionnaire New Questionnaire

"How Sensitive Were the Questions in This Survey?“ by Questionnaire: NHIS, 2018, Quarter 4 (base weights)

Not at all A little Moderately Very Extremely

Χ2 p-value = .112

10.5% 26.8% 27.9% 40.8% 25.6% 20.3% 17.4% 7.4% 18.6% 4.7%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Old Questionnaire New Questionnaire

Time Sample Adult Spent in the Interview by Questionnaire: NHIS, 2018, Quarter 4 (base weights)

< 30 minutes 30 - < 45 minutes 45 - < 60 minutes 60 - < 75 minutes 75+ minutes

Χ2 p-value < .01

Data/Methods (2)

• Model respondent assessments of burden using logistic regression

– Dependent variable = moderately/very/extremely burdensome (high burden) vs. a little/not at all burdensome – Independent variable: questionnaire (new vs. old) – Covariates: perceptions of survey content, task difficulty, recruitment effort and respondent motivation, and sample adult and case characteristics

• Model was estimated separately for the old questionnaire (n=3,943) and new

questionnaire (n=3,940), and then applied to the full dataset (n=7,883) with an indicator for questionnaire

Covariates Included in Logistic Regressions of Respondent Burden (1)

• Perceptions of Survey Content

– Ease/difficulty of answering questions – Sensitivity of questions

• Measures of Task Difficulty

– Total minutes sample adult spent in interview

Covariates Included in Logistic Regressions of Respondent Burden (2)

• Measures of Recruitment Effort and Motivation

– Householder(s) expressed time constraints – Householder(s) made refusal-like statements – Householder(s) expressed privacy concerns – Proportion of sample adult responses don’t know or refused – Number of contact attempts – Case reassigned to different interviewer

Covariates Included in Logistic Regressions of Respondent Burden (3)

• Sample Adult and Case Characteristics

– Age – Sex – Education – Race/ethnicity and language of interview – Reported health status – Cognitive difficulty – Number of children – Mode of interview (FTF vs. phone)

Significant (p < .05) Adjusted Odds Ratios from Logistic Regressions of Respondent Burden: Questionnaire, Perception of Survey Task, and Task Difficulty Measures Combined Old New

New questionnaire (vs. old) 0.55

• Somewhat easy/somewhat difficult/very

difficult (vs. very easy) 2.42 2.51 2.28 Sensitivity of questions (ref = not at all) A little 1.56 1.78 1.33 Moderately/very/extremely 5.76 6.29 5.30 Total sample adult minutes in interview (ref = > 45 -- < 60) < 30 minutes > 30 -- < 45 minutes > 60 -- < 75 minutes > 75 minutes 1.31 1.34

Summary of Remaining Significant Effects from Logistic Regressions of Respondent Burden Combined Old New

Householder(s) expressed time constraints * * * Householder(s) made refusal-like statements * * Householder(s) expressed privacy concerns * * * Number of contact attempts * Sample adult item nonresponse * * * Mode of interview * * Male * Age * * * Education * Race/ethnicity and language * * Number of children * * *

Case reassigned to a different interviewer, cognitive difficulties, and reported health status were not associated with burden in any of the models.

Conclusions (1)

• Sample adults who received the new questionnaire reported lower levels of

burden compared with sample adults who received the old questionnaire

• Items in the new questionnaire appear to be easier to answer, but perceptions
• f sensitivity did not differ between the surveys
• Time was significantly reduced in the new instrument

– Time spent in the interview was associated with burden in the old, but NOT the new questionnaire

Conclusions (2)

• Respondent assessments of ease/difficulty of answering and sensitivity of

questions were the strongest predictors of burden – Design changes to reduce cognitive and emotional burden appear to be driving the reductions in burden observed with the new questionnaire

Thank you!

jdahlhamer@cdc.gov

Extra Slides

"How Burdensome Was This Survey to You?" by Questionnaire: NHIS, 2018, Quarter 4 (base weights)

53.1% 63.5% 29.6% 24.8% 12.1% 8.3% 3.1% 2.3% 2.0% 1.1%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Old Questionnaire New Questionnaire Not at all A little Moderately Very Extremely

17.2%* 11.7%*

*p < .05

66 90 88% 67%

0% 25% 50% 75% 100% 2001 2003 2005 2007 2009 2011 2013 2015 2017 30 60 90 120 150

Response Rate Minutes

NHIS Response Rate and Completed Interview Times

Mean Interview Time Response Rate

Length of Completed Interviews in Minutes: NHIS, 2018, Q4

Old Questionnaire New Questionnaire Mean 77.4 55.1 Median 70.4 48.9

0% 5% 10% 15% 20% 25% < 30 minutes 30 - < 45 minutes 45 - < 60 minutes 60 - < 75 minutes 75+ minutes

Percent of Sample Adults Reporting the Survey to Be Moderately, Very, or Extremely Burdensome by Time in the Interview and Questionnaire: NHIS, 2018, Quarter 4 (base weights)

Old Questionnaire New Questionnaire