CHIS 2013 Sample Design and Survey Methodology TAC August 30, 2012 1 - - PowerPoint PPT Presentation

CHIS 2013 Sample Design and Survey Methodology TAC

August 30, 2012

1

Welcome

• Ninez Ponce, CHIS PI
• David Grant, CHIS Director
• A. A. Afifi, TAC Chair

2

Introductions

(In order of Adobe Connect roster)

1) Name 2) Where you work 3) Expertise

3

Meeting Goals

• Overview of CHIS methodology past and present
• Discuss methodological topics most relevant to

CHIS

• Sketch out design and experiment ideas as they

arise

• No burden to leave today with clear solutions to

specific challenges…work groups may follow

4

Meeting Themes

• CHIS Field Schedule
• Methodological experiments and implementing

methodological changes?

• Interview Length
• Shortening interview length; What role does

questionnaire length play in overall survey cost and data quality?

• Measurement
• Improve measurement maintaining data series?

5

Costs and Cost/Error Tradeoffs

• Adapt CHIS methodology to reduce overall

survey costs

• Proactive consideration (e.g., experiments and

tests that will reduce costs long‐term)

• Some cost‐savings incur up‐front costs
• Most cost decisions are cost/quality trade‐offs
• Further adaptive/responsive design approach to

data collection

6

The Interview

• Telephone survey on wide range of health issues

(see questionnaire content handouts)

• Only mode for all data collection
• Landline‐RDD, Cell‐RDD, Surname List
• Pre‐notification letters with $2 incentive when

address matched from records

• Westat interviewers conduct interviews from call

centers and home

7

Sample designed to represent…

• Residents of CA
• Stratification by CA counties and county groups
• In 2011, 44 strata representing 58 counties
• Most with targets of n=600 ea.
• Smallest counties (17) in 3 grouped strata
• Key ethnic and racial subgroups
• Initial proportionate allocation accounts for this
• Vietnamese and Korean oversample since 2001
• Am Indian/Alaskan Native in 2001 and “current”

8

Field Schedule

• Every two years since 2001
• In past, data collected w/in approx. 9‐month window
• 2011‐12 forward in quarterly “rolling” sample

starting in January of 2‐year cycle

• T4 currently in field, finishing end of Dec.

9

TCE Building Health Communities

• 14 geographically defined communities in CA
• Chosen by TCE as part of 10‐year health improvement

effort

• CHIS data collection is baseline and follow‐up

measurement (i.e., “panel in development”)

• CHIS 2009 oversampled numbers in these areas
• Expansion of sample to insure 400 completes per site
• Participants re‐contacted in 2012 and asked about

future participation (n100 per site)

• http://www.calendow.org/healthycommunities/
• http://www.healthpolicy.ucla.edu/healthprofiles09.aspx

10

Status of 2013 Planning

• SDSM TAC is one of 5 TACs
• Questionnaire being revised based on other TACs
• Room for experiments resulting from today
• Sample design
• Room for adjustments based on this meeting
• Increase/decrease in cell sample; alternative

sampling methods for hard‐to‐reach populations (e.g., RDS)

11

CHIS Performance Over Time

Royce Park

12

13

2001 2003 2005 2007 2009 Screener Interview

59.2 55.9 49.8 35.6 36.1

Extended Interview Household

‐ ‐ 59.3 59.4 54.7

Adult

63.7 59.9 54.0 52.8 49.0

Child

87.6 81.4 75.2 73.7 72.9

Adolescent

63.5 57.3 48.5 44.1 42.8

Overall Household

‐ ‐ 29.6 21.1 19.7

Adult

37.7 33.5 26.9 18.7 17.7

Child

33.0 27.3 25.2 16.8 15.7

Adolescent

23.9 19.2 14.2 10.2 7.9

CHIS Response Rates

* Landline/List Sample (excludes cell phone sample), Weighted, AAPOR RR4

CHIS Response Rates

Screener Interview Extended Household Extended Adult Extended Child Extended Teen Overal Household Overall Adult Overall Child Overall Teen 10 20 30 40 50 60 70 80 90

2001 2003 2005 2007 2009

Response Rate (%)

* Landline/List Sample (excludes cell phone sample), Weighted, AAPOR RR4

14

Screener & Extended Response Rates

49.8 35.6

Screener Interview Extended Household Extended Adult Extended Child Extended Teen 10 20 30 40 50 60 70 80 90

2001 2003 2005 2007 2009

Response Rate (%)

* Landline/List Sample (excludes cell phone sample), Weighted, AAPOR RR4

15

Overall Response Rates

Screener Interview Overal Household Overall Adult Overall Child Overall Teen 10 20 30 40 50 60 70 80 90

2001 2003 2005 2007 2009

Response Rate (%)

* Landline/List Sample (excludes cell phone sample), Weighted, AAPOR RR4

16

CHIS 2009 Screener Response Rate by Strata

17

25 30 35 40 45 50 San Francisco Santa Clara Solano San Mateo Los Angeles Ventura San Diego Riverside Contra Costa Lake El Dorado Fresno San Joaquin Sacramento Orange Nevada Alameda Imperial Santa Barbara Yolo San Bernardino Placer Kings Marin Sonoma Stanislaus Napa Mendocino Merced Madera Monterey Yuba Kern Santa Cruz Sutter San Benito Tulare Sierra Balance San Luis Obispo Shasta Butte Colusa, Glenn, Tehama Humboldt North Balance

Response Rate (%)

CA County

Overall State RR = 36.1

* Landline/List Sample (excludes cell phone sample), Weight, AAPOR RR4

CHIS 2009 Screener Response Rate by Strata

18

25 30 35 40 45 50 San Francisco Santa Clara Solano San Mateo Los Angeles Ventura San Diego Riverside Contra Costa Lake El Dorado Fresno San Joaquin Sacramento Orange Nevada Alameda Imperial Santa Barbara Yolo San Bernardino Placer Kings Marin Sonoma Stanislaus Napa Mendocino Merced Madera Monterey Yuba Kern Santa Cruz Sutter San Benito Tulare Sierra Balance San Luis Obispo Shasta Butte Colusa, Glenn, Tehama Humboldt North Balance

Response Rate (%)

CA County

* Landline/List Sample (excludes cell phone sample), Weight, AAPOR RR4

Overall State RR = 36.1

Smallest Counties

CHIS 2009 Screener Response Rate by Strata

19

25 30 35 40 45 50 San Francisco Santa Clara Solano San Mateo Los Angeles Ventura San Diego Riverside Contra Costa Lake El Dorado Fresno San Joaquin Sacramento Orange Nevada Alameda Imperial Santa Barbara Yolo San Bernardino Placer Kings Marin Sonoma Stanislaus Napa Mendocino Merced Madera Monterey Yuba Kern Santa Cruz Sutter San Benito Tulare Sierra Balance San Luis Obispo Shasta Butte Colusa, Glenn, Tehama Humboldt North Balance

Response Rate (%)

* Landline/List Sample (excludes cell phone sample), Weight, AAPOR RR4

Overall State RR = 36.1

Medium Counties

CA County

CHIS 2009 Screener Response Rate by Strata

20

25 30 35 40 45 50 San Francisco Santa Clara Solano San Mateo Los Angeles Ventura San Diego Riverside Contra Costa Lake El Dorado Fresno San Joaquin Sacramento Orange Nevada Alameda Imperial Santa Barbara Yolo San Bernardino Placer Kings Marin Sonoma Stanislaus Napa Mendocino Merced Madera Monterey Yuba Kern Santa Cruz Sutter San Benito Tulare Sierra Balance San Luis Obispo Shasta Butte Colusa, Glenn, Tehama Humboldt North Balance

Response Rate (%) Overall State RR = 36.1

* Landline/List Sample (excludes cell phone sample), Weight, AAPOR RR4

Largest Counties

CA County

Landline/List Cell phone Total 49.0 56.2 Sex Male 43.7 56.3 Female 53.8 56.1 Age 18 to 30 years 36.0 52.3 31 to 45 years 42.6 57.6 46 to 65 years 52.1 57.0 Over 65 years 60.4 64.3 Type of household With child and/or Adolescent 43.3 52.1 Without child or Adolescent 53.1 59.5 Number of adults in household 1 65.2 56.3 2 52.6 59.7 3 or more 39.5 51.3 Adult was screener respondent Yes 64.2 58.9 No 34.5 20.9

Adult Response Rate by Sample Adult Characteristics

How do we compare to other surveys

• RR’s difficult to compare due to disposition

definitions, RR calculations, and undocumented differences between surveys

• Comparable to CA BRFSS over the years
• 27.1% CA BRFSS RR v. 19.7
• BRFSS has single adult interview, CHIS considers

any one interview (Adult, Teen, Child) a “complete”

22

Address‐based Sampling Pilot

Matt Jans

23

ABS – Address‐based Sampling

• Pro: More complete coverage of HH population

than telephone frames

• Con: Unless survey questionnaire is mail,

requires switch to another mode for completion

• Challenge: How to obtain phone number, and

eventually completed phone surveys, from sampled addresses

24

ABS Pilot Design

• Implemented in 2 BHC sites
• Selected for lower survey saturation
• Spanish language prevalence
• “Short Questionnaire” (i.e., screener form)

mailed to all addresses

• Requests adult response
• A few basic health and demography questions and
• Data expected end‐of‐year

25

Test of Messaging and Outreach

• Random half in each site receive at 2nd and 3rd

mailing…

• TCE logo on the out‐going envelope
• Insert showing TCE/BHC support (1/3 page slip in

bright color or with color logo)

• Seeking support of Program Managers (PMs) in

the two communities

• Vouch for our survey with their members
• Distribute info via email list, etc.

26

Designed to Compare

Boyle Heights Compares to Usual CHIS Merced Compares to Merced BHC, 2010

‐Eligibility determined by usual CHIS rules and location within BHC area ‐Random half of 2nd and 3rd mailing receive additional TCE logo on outgoing envelope and TCE insert. ‐Yields 400 completed that fit usual CHIS protocol for comparison to CHIS data ‐Eligibility determined by BHC age and parenting status criteria, and location within BHC area ‐Random half of 2nd and 3rd mailing receive additional TCE logo on outgoing envelope and TCE insert. ‐Yields 400 completed that fit BHC eligibility criteria for comparison to BHC

• versample data

27

ABS – Comments and Questions

• Experiences moving respondents from mail to

another mode

• Dealing with DSF quality (e.g., non‐residential

addresses, drop‐points)

• Other ABS ideas

28

Meeting Second Half

TAC‐wide Discussion

29

Alternate Sampling Methods

• Geographic Stratification
• Other approaches than county stratification
• ABS
• Best practices
• Challenges moving Rs from mail to phone (or other

modes)

• Network sampling and related methods
• Synthetic estimates for costly areas

30

Multimode Data Collection

• We would expect higher response rates from
• ther modes. What novel combinations are cost‐

competitive?

• What techniques work best for long surveys via

the web? Mail? Combinations?

• T‐ACASI options: How long/how many

questions?

• Nonresponse follow‐up for key estimates by

mail?

31

Matrix Sampling

• Random subgroups of respondents receive

sections of the questionnaire; Missing data are imputed prior to release

• Models for best practices? Who has done this

with public release data successfully?

• How large/small can sampled question sets be?

How many units per question? How large should a “common question core” be, particularly as imputation core?

32

33

2001 2003 2005 2007 2009 2011‐2012*

Screener Interview 2.7 2.3 2.5 2.6 2.7 2.6 Extended Interview Adult 33.0 32.3 35.2 34.7 39.9 35.1 Child 14.5 14.0 15.0 17.3 15.9 15.0 Adolescent 20.1 21.5 19.6 19.8 18.0 22.9 Parental permission 2.0 2.2 2.3 2.4 2.7 3.0

* Preliminary

CHIS Questionnaire Duration (minutes)

CHIS Questionnaire Duration (minutes)

Screener Interview Extended Adult Extended Child Extended Adolescent Parental permission 0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0 45.0

2001 2003 2005 2007 2009 2011‐2012*

Minutes

* Preliminary

34

Parent and Teen Consent

• Focus group results
• Ask for teen participation more softly
• Allow parent to opt out of types/sets of questions
• Other reflections
• The order of selection and permission is important
• E.g., is “Your child has been selected. May we interview them”

different from “May we interview one of your children? Mary has been selected”?

• Don’t ask for teen names

35

Parent and Teen Consent

• Known methods for gaining parent consent?
• Novel incentives or incentive structures known to

work with youth?

• Raffle for iPad or similar? Smaller, cheaper pre‐paid

incentives?

• Multi‐phase incentives?
• E.g., parent given incentive for allowing teen (beyond
• riginal incentive); Teen given own incentive
• Message tailoring to avoid refusal at consent? To

gain cooperation with teen?

36

Adolescent Rates

75.9 58.5 83.2 74.6 63.5 42.8 23.9 7.9 84.5 75.1

10 20 30 40 50 60 70 80 90 2001 2003 2005 2007 2009

Rates (%)

Permission Rate** Completion Rate** (all interview types) Extended Adolescent RR* Overall Adolescent RR*

* Landline/List Sample (excludes cell phone sample), Weighted, AAPOR RR4 ** Unweighted rate

Adol Int Rate Given Parent Consent ** (all interview types)

37

Interview Optimization

• Optimizing work assigned as well as follow‐up

and call number/timing decisions

• Best call windows and time‐of day/week
• Behavior tailoring at first call and key points (e.g.,

consent)

• Do concepts of responsive/adaptive design apply

to phone surveys?

• Statistical Process Control models for phone call

management (in or out of survey research)

38

Prioritization of Projects

• What seems most tractable and interesting?
• Will follow‐up with quick survey to rate

39

In Closing

• Comments and summary available in a few

weeks

• Possibility of forming working groups
• May send poll for interest
• Welcome calls, comments, and collaborations
• mjans@ucla.edu, npark@ucla.edu, dgrant@ucla.edu,

nponce@ucla.edu

Thank You!

40