Matching an Internet g Panel Sample of Pregnant Women to a g - - PowerPoint PPT Presentation

Matching an Internet g Panel Sample of Pregnant Women to a g Probability Sample

Andrew Burkey Abt SRBI Carla L Black CDC* Charles DiSogra Abt SRBI John Sokolowski Abt SRBI Stacie M Greby CDC* Helen Ding CDC* K.P. Srinath Abt SRBI Sarah W Ball Abt Assoc. Sara MA Donahue Abt Assoc.

AAPOR – May 16, 2015 – Hollywood, FL DC-AAPOR – August 3 2015 – Washington DC

Immunization Services Division, National Center for Immunization and Respiratory Diseases, Centers for Disease Control and Prevention

DC-AAPOR August 3, 2015 Washington, DC

 Centers for Disease Control and Prevention (CDC) uses

Background Background

 Centers for Disease Control and Prevention (CDC) uses

influenza vaccination coverage data to:

M it th i t f i ti

• Monitor the impact of vaccination programs
• Identify groups in need of vaccination services

 Pregnant women (PW) are prioritized for influenza vaccination

• Pregnant women are at increased risk of influenza-

Pregnant women are at increased risk of influenza related severe illness and hospitalization

• Infants younger than 6 months are at high risk of severe
• Infants younger than 6 months are at high risk of severe

complications from the flu, but they are too young to be vaccinated.

Abt SRBI | pg 2

Surveying a rare group: pregnant women Surveying a rare group: pregnant women

 Probability-based survey could be costly and time-consuming  Probability-based survey could be costly and time-consuming

• Low prevalence of PW: <4% of all women

 Compromise: Use opt in Internet panels  Compromise: Use opt-in Internet panels

• Non-probability opt-in panel is fit for purpose
• Can access large numbers of PW with Black Hispanic over-samples

Can access large numbers of PW with Black, Hispanic over samples

• Large sample: approximately 2,000 completes per 2-week period
• Provide information quickly (all Web mode)

q y ( )

• Short 2-week data collection period in November and in April
• Are affordable for the task

 Survey Sampling Inc., opt-in Internet panel used for sample since

2010

Abt SRBI | pg 3

Limitations of non Limitations of non-

• probability sample

probability sample

 PW are all panel volunteers who use the Internet  Due to Internet use and multi-survey exposure PW in survey may  Due to Internet use and multi survey exposure, PW in survey may

be better informed than PW in general population

 Bias in estimate is unknown; poses threat to generalizability  Bias in estimate is unknown; poses threat to generalizability  Standard errors and confidence intervals cannot be used with non-

probability samples (AAPOR Task Force Report, June 2013)

Abt SRBI | pg 4

Study Objective Study Objective

Objective Objective Explore ways to mitigate the shortcomings of non-probability sample

• Assess the bias
• Assign a measure of precision to the estimate

Method M t h f S i 2013 b bilit PW ith Match cases from Spring 2013 non-probability PW survey with PW cases from a 2013 national probability survey Assumption Assumption

Purposely matched opt-in sample can resemble a probability sample

• Variance of estimate approximates that from a probability sample thus a

Variance of estimate approximates that from a probability sample thus a confidence interval may be derived

• Matched sample estimate may not be similar to the probability sample’s estimate

due to existence of inherent bias

Abt SRBI | pg 5

Panel bias is assumed Panel bias is assumed



Probability sample Matched panel sample

Abt SRBI | pg 6

The comparison probability sample The comparison probability sample

 2013 National Health Interview Survey (NHIS)

• In-person survey of population weighted to population totals
• Data collection throughout the year
• Data collection throughout the year
• Has some of the same descriptive variables as the Panel sample

 To best align with panel sample, NHIS sample limited to women:

• Asked questions about pregnancy

I i d F b J l 2013

• Interviewed February – July 2013
• Pregnant any time between August 2012 – March 2013
• Responded to questions about influenza vaccination status
• Responded to questions about influenza vaccination status

Abt SRBI | pg 7

Matching variables Matching variables

 Variables in both surveys and with differences in the distribution

between the panel and NHIS (age, race, education) don’t explain differences in vaccination coverage between the samples

 Variables used by other studies not available in both surveys,

liti l t ffili ti ki t t e.g., political party affiliation, smoking status



One difference between the panel and NHIS is Internet use

• Literature suggests that panel members are frequent internet

users.

• Frequent internet users have attitudinal and behavioral

differences compared to the general population

Abt SRBI | pg 8

Internet use: observed difference Internet use: observed difference

A diff b h l A difference between the two samples:

 Panel: All Internet users (online respondents – English only)  NHIS: Internet users and non-users (in-person, general population)

There may be other attitude, behavioral, socio-political differences

 Internet use can be defined for both samples

p

 Internet use was thought to be the most logical available measure to

define coverage differences between the two samples g p

 Opt-in panelists have been described as more active Internet users

Abt SRBI | pg 9

Flu Vaccination rate by Internet use Flu Vaccination rate by Internet use

Panel NHIS

Vaccination Rate for Panel and NHIS Samples (weighted*)

Panel NHIS Panel NHIS Total Panel Total NHIS

47 1% 34 9% V i ti R t

Panel NHIS Total Panel Total NHIS Non-daily Internet Daily Internet

47 1% 34 9% 31 0% 36 4% 47.1% 34.9%

• 28.8 - 41.0

Vaccination Rate 95% Conf. Interval 47.1% 34.9% 31.0% 36.4%

• 28.8 - 41.0

18.7- 43.4 29.4 - 43.4

* Both samples weighted to identical benchmarks

• The Panel sample is missing non-Internet users

The Panel sample is missing non Internet users

• Daily Internet has a higher observed vaccination rate as does the Panel
• Difference by Internet use suggests a reason why Panel rate is higher

Abt SRBI | pg 10

Difference by Internet use suggests a reason why Panel rate is higher

Matching strategy Matching strategy

Internet Use Propensity

 Leverage NHIS Internet use questions for matching  Develop a propensity score based on frequency of Internet usage

“Daily Internet” Use vs. “Less than daily/No Internet” Use y y

Abt SRBI | pg 11

Internet user propensity Internet user propensity

 Cases used in analysis: 2,035 panel cases + 394 NHIS cases  Panel cases: We assume 100% to be daily users (for our purpose)  Panel cases: We assume 100% to be daily users (for our purpose)  NHIS cases: 71% daily users, 29% less than daily/non-users  Combined data used in a logistic regression model predicting

likelihood of a “Less-than-Daily” user (i.e., propensity) y ( , p p y)

Abt SRBI | pg 12

Internet user propensity Internet user propensity

 An exploratory investigation pursued for the propensity model  Eleven variables evaluated for prediction of Less-than-Daily user  Stepwise analysis found six as significant for use (i.e., p<.05 level)

Effect DF p Chi-Square education level 3 96.36 <.0001 race/ethnicity 3 36.80 <.0001 p q type of phone used 3 20.88 <.001 age 5 16.93 <.001 home ownership 1 7.01 <.01

p >.05 = health insurance, marital status, region, income, employment status

child ≤ 4 yrs old in HH 1 4.46 <.05

Abt SRBI | pg 13

Vaccination and Internet propensity Vaccination and Internet propensity Vaccination and Internet propensity Vaccination and Internet propensity score score

Vaccination Rate and Quintiles of Internet Use Propensity

59% 60% 70%

Panel (47%) NHIS (35%)

n rate

52% 45% 42% 39% 50% 45% 0% 50% 60%

accinatio

36% 27% 30% % 30% 40%

Va

% 10% 20%

P l 2 035 NHIS 394 P t d d t h l b

Quintile

0% 1st High Use 2nd 3rd 4th 5th Low Use

Abt SRBI | pg 14

Panel = 2,035 cases; NHIS = 394 cases. Percents rounded to whole numbers

Matching rates Matching rates

 Matched cases from both samples on their propensity score value  82% of NHIS cases matched to 63% of Panel cases Source Matched Did Not Match Total Cases

C t P t C t P t C t Count Percent Count Percent Count

Panel 1,282 63% 753 37% 2,035 NHIS 325 82% 69 18% 394

Abt SRBI | pg 15

Ratio adjustment weights Ratio adjustment weights

P bl

 Problem

• Single NHIS case matched to multiple Panel cases
• Multiple NHIS cases matched to a single Panel case
• Multiple NHIS cases matched to a single Panel case
• Multiple NHIS cases matched to multiple Panel cases

 Solution  Solution

• A ratio adjustment for each of the matched Panel cases

( b f NHIS i t h) (number of NHIS cases in a match) (number of Panel cases in same match) Ratio Adj. Weight =

Example: When 2 NHIS cases match 5 Panel cases Ratio adjustment weight = 2/5 = 0.40

Abt SRBI | pg 16

Ratio adjustment weights Ratio adjustment weights

 Ratio adjust 1,282 matched Panel cases

47 (4%) had adj wgt =1 00 e g 1/1 2/2 3/3 etc 47 (4%) had adj. wgt. =1.00 e.g., 1/1, 2/2, 3/3, etc. 1,216 (95%) had adj. wgt. <1.00 e.g., 1/2, 1/3, 2/3, etc. 19 (<2%) had adj. wgt. >1.00 e.g., 2/1, 3/2, 4/3, etc.

 Sum of ratio adjusted weighted Panel cases = 325

j g

Abt SRBI | pg 17

Population weights for PW Population weights for PW

Base weights:

 Base weights:

• Panel = 1.00 x the ratio adjustment weight
• NHIS = NHIS sample adult file design weight

NHIS NHIS sample adult file design weight

(weight prior to NHIS post stratification adjustment)  Determined number of PW 18-44 years in the U.S. population by

y p p y age, race/ethnicity, and state (National Center for Health Statistics May 2013)

 Rake both samples to identical control totals for:  Rake both samples to identical control totals for:

• Age
• Race/Ethnicity

C i

• Census region

Abt SRBI | pg 18

Overview of method Overview of method

2 035 Panel cases 753 No Match Not used 2,035 Panel cases 1,282 Matched Ʃ ratio adjust=325 Population weighted 6.4 million PW

Compare

32 h d

Internet usage propensity score

394 NHIS cases 325 Matched Population weighted 6 4 million PW 394 NHIS cases 69 No Match 6.4 million PW

Abt SRBI | pg 19

Results Results

Source Influenza vaccination Standard 95% CI lower 95% CI upper Precision (% t ) Source vaccination rate (%) error lower bound upper bound (% pts) Panel 47.1

• Matched Panel

43 8 2 23 39 4 48 2 ±4 4 Matched Panel 43.8 2.23 39.4 48.2 ±4.4 NHIS 34.9 3.11 28.8 41.0 ±6.1 Estimated inherent bias = 8.9% Small overlap – but were tested to be different *

* Schenker, N. and Gentleman, J.F. “On Judging the Significance of Differences by Examining the Overlap

B t C fid I t l ” Th A i St ti ti i A t 2001 V l 55 N 3

Abt SRBI | pg 20

Between Confidence Intervals” The American Statistician, August 2001, Vol. 55, No. 3.

Examples: vaccination rates Examples: vaccination rates for two for two Examples: vaccination rates Examples: vaccination rates for two for two subgroups subgroups

NHIS Panel Matched Panel NHIS Panel Matched Panel

Race/Ethnicity Education

50% 60% 70%

NHIS Panel Matched Panel

50% 60% 70%

NHIS Panel Matched Panel

10% 20% 30% 40% 10% 20% 30% 40% 0% 10%

<HS HS HS+some col. BA or more

0% 10%

Hispanic White NH Black NH Other NH

Matching made estimates closer to NHIS estimates in most but not all (see <HS)

Abt SRBI | pg 21

PW matching procedure limitations PW matching procedure limitations

NHIS PW sample chosen as the available alternative reference for

 NHIS PW sample chosen as the available alternative reference for

comparison/validation

 NHIS PW sample is not a true comparable reference:  NHIS PW sample is not a true comparable reference:

• PW are a domain of a national, year-round sample of adults
• Design weights intended for a national cluster sample weighting scheme

 Variance estimations are not strictly based on NHIS sample design  Results hinge on Internet usage propensity  Propensity score matching restricted to available variables  All NHIS and Internet panel results are self report  All NHIS and Internet panel results are self-report  Possible mode effects between in-person NHIS and Internet panel

Abt SRBI | pg 22

PW matching procedure conclusions PW matching procedure conclusions

 Approximates a probability sample from a non-probability panel  Approximates a confidence interval around the panel estimate

(overall precision of ±4.4%)

 Provides non-arbitrary variance estimation for trend comparisons  Given the limitations, inherent vaccination rate bias is estimated to

be +8.9% for the Spring 2013 Panel

 Matching may provide information to enhance the design of future

samples using Internet panels

S t it t h h l d ti

• Suggest over-recruit pregnant women who have less education,

Hispanic or multi-racial, older (30-49), or landline only

Abt SRBI | pg 23

• Matching an Internet Panel Sample of Pregnant

Women to a Probability Sample

Thank You!

Andrew Burkey a.burkey@srbi.com

The findings and conclusions in this The findings and conclusions in this presentation are those of the authors and do not necessarily represent the views of Centers for Disease Control and Prevention.

Abt SRBI | pg 24