Exploring the Relationship between Burden Factors and Survey - - PowerPoint PPT Presentation

▶
exploring the relationship between
SMART_READER_LITE
LIVE PREVIEW

Exploring the Relationship between Burden Factors and Survey - - PowerPoint PPT Presentation

Nov 04, 2023 453 likes •785 views

Exploring the Relationship between Burden Factors and Survey Response DC-AAPOR Workshop on Respondent Burden October 21, 2019 Morgan S. Earp Brandon Kopp John Dixon 1 U.S. B UREAU OF L ABOR S TATISTICS bls.gov Overview Respondent

slide-1
SLIDE 1

1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Exploring the Relationship between Burden Factors and Survey Response

DC-AAPOR Workshop on Respondent Burden October 21, 2019 Morgan S. Earp Brandon Kopp John Dixon

slide-2
SLIDE 2

2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Overview

 Respondent Burden  Current Population Survey

CPS Supplements

 Structural Equation Models  Conclusions  Limitations  Next Steps

slide-3
SLIDE 3

3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Respondent Burden

 In Bradburn’s (1978) model respondent burden includes five

dimensions:

length of the interview amount of effort required of the respondent amount of stress on the respondent survey topic saliency/importance frequency with which the respondent is being independently surveyed

slide-4
SLIDE 4

4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Respondent Burden

 In Bradburn’s (1978) model respondent burden includes four

dimensions:

length of the interview amount of effort required of the respondent amount of stress on the respondent survey topic saliency/importance frequency with which the respondent is being independently surveyed frequency with which the respondent is being independently surveyed X survey topic saliency/importance (interaction effect)

slide-5
SLIDE 5

5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Current Population Survey (CPS)

 The CPS is a monthly survey conducted by the US Census

Bureau and the Bureau of Labor Statistics

 The data are used to produce labor force status estimates  Several months out of the year, target populations of CPS

respondents are asked to complete additional questions referred to as topic specific supplemental surveys

There is concern that additional supplemental surveys may hurt response rates of both the CPS and other annual supplemental surveys

slide-6
SLIDE 6

6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CPS Supplement Population

 Several months out of the year, target populations of CPS respondents

are asked to complete additional questions referred to as topic specific supplemental surveys

CPS Sample

slide-7
SLIDE 7

7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CPS Supplement Population

 Several months out of the year, target populations of CPS respondents

are asked to complete additional questions referred to as topic specific supplemental surveys

CPS Sample CPS Respondents

slide-8
SLIDE 8

8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CPS Supplement Population

 Several months out of the year, target populations of CPS respondents

are asked to complete additional questions referred to as topic specific supplemental surveys

CPS Sample CPS Respondents

CPS Supplement Sample (Varies by supplement topic)

slide-9
SLIDE 9

9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CPS Supplements

 Several months out of the year, target populations of CPS respondents

are asked to complete additional questions referred to as topic specific supplemental surveys

CPS Sample CPS Respondents

CPS Supplement Respondents

CPS Supplement Sample (Varies by supplement topic)

slide-10
SLIDE 10

10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CPS Supplements

 Several months out of the year, target populations of CPS respondents

are asked to complete additional questions referred to as topic specific supplemental surveys

 The CPS topic specific supplemental surveys are used to collect

additional data on issues related to economic and social well being

 Between 2005 and 2015 the CPS had supplemental surveys on about 20

different topics; the frequency of these supplement surveys ranges from

  • nce a year to once every two years to a single administration
slide-11
SLIDE 11

11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

2016 CPS Supplements Example

 Displaced Workers & Job Tenure/Occupational Mobility (January) 91.7% RR | CPS Respondent

 Workers who lost a job in the last 3 years

 Fertility (June) 92.6% RR | CPS Respondent

 Number of children that women aged 15-50 have ever had

 Veterans (August) 87.4% RR | CPS Respondent

 Veterans of the United States

 School Enrollment (October) 92% RR | CPS Respondent

 Population 3 years old and older on school enrollment, junior or regular college attendance, and high school graduation

 Voting and Registration (November) 89.4% RR | CPS Respondent

 Demographic information on persons who did and did not register to vote

 Food Security (December) 80.1% RR | CPS Respondent

 Food expenditure, access to food, and food quality and safety

slide-12
SLIDE 12

14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Research Questions

 How are the different factors of respondent burden related to

CPS Supplement response rates?

 Does the impact of respondent burden factors vary across

different CPS Supplements?

 Does the number of previous supplements a

person/household was selected for affect their response to future supplements?

slide-13
SLIDE 13

15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Data

 We used characteristics reported by CPS respondents on the CPS as

proxy measures of burden.

 Only CPS respondents are eligible to be sampled for supplements

Therefore the analysis is limited to CPS respondents only

CPS Respondents

CPS Supplement Sample

CPS Supplement Respondents

slide-14
SLIDE 14

16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

 Effort  Stress / Sensitivity  Length

CPS Burden Proxy Items

Bradburn Factors CPS Items

 # of Adults  Self, Proxy, or Both  # of Adults x Self, Proxy, or Both  Refused HH Income  Refused Hispanic Ethnicity  Refused Marital Status  Refused Education  Interview Type  # of Months in Sample  # of Personal Contacts

slide-15
SLIDE 15

17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

 Effort  Stress / Sensitivity  Length

CPS Burden Proxy Items

Bradburn Factors CPS Items

 # of Adults  Self, Proxy, or Both  # of Adults x Self, Proxy, or Both  Refused HH Income  Refused Hispanic Ethnicity  Refused Marital Status  Refused Education  Interview Type  # of Months in Sample  # of Personal Contacts

Factor1 Factor2 Factor3 0.9980 0.0000

  • 0.0002

1.0000 0.0000

  • 0.0001

N/A N/A N/A 0.0000 0.9500 0.0068

  • 0.0000

0.9822

  • 0.0020
  • 0.0007

0.9708

  • 0.0019

0.0001 1.0000

  • 0.0001
  • 0.0059

0.0007 0.9459

  • 0.0001

0.0002 1.0000

  • 0.0000

0.0216

  • 0.8920
slide-16
SLIDE 16

Supplement Response Effort Sensitivity Length

# of Adults # of Personal Contacts Refused HH Income Refused Hispanic Ethnicity Refused Marital Status Refused Education Self, Proxy, or Both # of Adults X Proxy # of Months in Sample Interview Type

slide-17
SLIDE 17

19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CPS Supplements & Topic Saliency Indicators

 Veterans  School Enrollment  Voting and Registration  Food Security  Contingent Workers  Unbanked/Underbanked  Participation in the Arts  Volunteering & Civic Life  Computer & Internet Usage  Employment Status (UE/NILF, PT, or FT)  Disability Status  Home Ownership  Child Status  Education Level

CPS Supplements CPS Saliency Items

slide-18
SLIDE 18

Supplement Response Effort Sensitivity Length

# of Adults # of Personal Contacts Refused HH Income Refused Hispanic Ethnicity Refused Marital Status Refused Education Self, Proxy, or Both # of Adults X Proxy # of Months in Sample Interview Type Employment UE, PT, or FT Has Disability Owns Home Has Child Education Level # of Previous Supplements

slide-19
SLIDE 19

21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

SEM Models

 In order to examine these relationships, we used structural equation

modeling and we explored the invariance of item and factor loadings across the nine different CPS supplements

SEM can be thought of as a combination of factor analysis and multiple regression analysis

slide-20
SLIDE 20

Supplement Response Effort Sensitivity Length

# of Adults # of Personal Contacts Refused HH Income Refused Hispanic Ethnicity Refused Marital Status Refused Education Self, Proxy, or Both # of Adults X Proxy # of Months in Sample Interview Type

Factor Analysis

slide-21
SLIDE 21

Supplement Response Effort Sensitivity Length

# of Adults # of Personal Contacts Refused HH Income Refused Hispanic Ethnicity Refused Marital Status Refused Education Self, Proxy, or Both # of Adults X Proxy # of Months in Sample Interview Type Employment UE, PT, or FT Has Disability Owns Home Has Child Education Level # of Previous Supplements

Multiple Regression

slide-22
SLIDE 22

Supplement Response Effort Sensitivity Length

# of Adults # of Personal Contacts Refused HH Income Refused Hispanic Ethnicity Refused Marital Status Refused Education Self, Proxy, or Both # of Adults X Proxy # of Months in Sample Interview Type Employment UE, PT, or FT Has Disability Owns Home Has Child Education Level

Overall Impact of Burden on CPS Supplement Response Controlling for Number of Previous Supplements

  • 0.26
  • 0.69

0.13 0.15 0.21 0.15

  • 0.10
  • 0.03

# of Previous Supplements

  • 0.04
slide-23
SLIDE 23

Supplement Response Effort Sensitivity Length

Employment UE, PT, or FT Has Disability Owns Home Has Child Education Level

Overall Impact of Burden on CPS Supplement Response Controlling for Number of Previous Supplements

  • 0.26
  • 0.69

0.13 0.15 0.21 0.15

  • 0.10
  • 0.03

# of Previous Supplements

  • 0.04
slide-24
SLIDE 24
  • 0.26
  • 0.69

0.13 0.15 0.21 0.15

  • 0.1
  • 0.03
  • 0.04
  • 0.8
  • 0.7
  • 0.6
  • 0.5
  • 0.4
  • 0.3
  • 0.2
  • 0.1

0.1 0.2 0.3

Standardized Factor Loading ( r )

IMPACT OF BURDEN FACTORS, SALIENCY, & SURVEY SAMPLING FREQUENCY ON CPS SUPPLEMENT RESPONSE RATES

slide-25
SLIDE 25
  • 0.1
  • 0.02
  • 0.23

0.01 0.11

  • 0.29
  • 0.03
  • 0.37

0.06

  • 0.01

0.15 0.01

  • 0.12

0.28 0.05

  • 0.12

0.3

  • 0.5
  • 0.4
  • 0.3
  • 0.2
  • 0.1

0.1 0.2 0.3 0.4

Standardized Factor Loading ( r ) Veterans 2016 Veterans 2017 School Enrollment 2016 School Enrollment 2017 Food Security 2016 Food Security 2017

slide-26
SLIDE 26
  • 0.4
  • 0.3
  • 0.2
  • 0.1

0.1 0.2 0.3 0.4 0.5 0.6 0.7

Standardized Factor Loading ( r )

Veterans 2016 Veterans 2017 School Enrollment 2016 School Enrollment 2017 Food Security 2016 Food Security 2017

slide-27
SLIDE 27

33 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 33 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Conclusions

 The impact of factor loadings and saliency item loadings varied in

strength and direction across the supplements and across years for some supplements

We did see fairly consistent positive effects of education level, disability status, and child status

 Sensitivity had a consistent negative impact on response rates  The impact of previous supplements varied in direction and strength

slide-28
SLIDE 28

34 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 34 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Limitations

 Estimation with binary predictors with low frequency (CPS item refusal

flags) is difficult, so we were only able to utilize some of the refusal flags in our model.

 Burden is a complex phenomenon, other variables not available in this

study might be useful in describing it, specifically those in restricted microdata.

 There are a number of interaction effects that we were not able to test,

as well as potential nonlinear effects, issues with missing data, and issues with singularity.

slide-29
SLIDE 29

35 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 35 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Limitations Cont.

 The fit indices could have been better, but they were not bad

(MacCallum et al, 1996; Steiger, 2007)

H0: RMSEA = 0.065 H1: RMSEA = 0.072

slide-30
SLIDE 30

36 — U.S. BUREAU OF LABOR STATISTICS • bls.gov 36 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Next Steps

 Expand our dataset to include additional supplements over multiple years  Include panels that receive the March Supplement  Use the regression tree rpsm R Package to model CPS Supplement

response rates over time, to better assess interactions effects and handle issues with missing data, nonlinear relationships, and singularity

 Model month two through eight CPS response rates for month one CPS

respondents using similar proxy data

slide-31
SLIDE 31

Contact Information

37 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Morgan Earp earp.morgan@bls.gov Brandon Kopp kopp.Brandon@bls.gov John Dixon dixon.john@bls.gov