How Do I Ask Questions? For your convenience, there are two ways to - - PDF document

1

Thank you for joining us today!

If you haven’t dialed into the audio (telephone) portion, please do so now: 1-866-516-5393 Access Code: 33403311 Access Code: 33403311 If you are experiencing technical problems with the GoToWebinar program (visual portion), contact the help desk: 1-800-263-6317 Reference Webinar ID: 309164457 Today’s presentation and handouts are available for y p download at: http://www.cffutures.org/presentations/webinars

1

The webinar will begin shortly. Thank you for your patience.

How Do I Ask Questions?

For your convenience, there are two ways to ask questions two ways to ask questions during this webinar presentation:

• 1. Type and send your questions

through the Question and Answer log located on the bottom half on your panel/dashboard.

2

panel/dashboard.

• 2. There will also be time at the

end of the webinar for you to ask questions via the conference line.

2

Propensity Score Matching Propensity Score Matching Strategies for Evaluating Substance Abuse Strategies for Evaluating Substance Abuse Services for Child Welfare Client, Session II of II Services for Child Welfare Client, Session II of II

I) Welcome and Introductions I) Welcome and Introductions Ken DeCerchio, MSW, CAP II) Optimal Propensity Score Matching (OPSM) Shenyang Guo, PhD III) Example: Optimal Propensity Score Matching Shenyang Guo, PhD IV) Using Propensity Score Matching to Evaluate The Regional Partnership Grant Program Shenyang Guo, PhD V) Discussion/Questions

Part II – Optimal Propensity Score Matching

1 Optimal propensity score matching

• 1. Optimal propensity score matching

(Rosenbaum, 2002)

• 2. Example of OPSM

3 U i it t hi t

• 3. Using propensity score matching to

evaluate the Regional Partnership Grant Program

3

• 1. Optimal propensity score

matching (Rosenbaum, 2002)

Limitations of Greedy Matching (1)

The key characteristic of greedy matching: it divides a large decision problem (i.e., matching) into a series of smaller, simpler decisions each of which is handled optimally; each k th d i i t ti ith t id i makes those decisions one at a time without reconsidering early decisions as later ones are made (Rosenbaum, 2002). As such, the method has two limitations:

• 1. A dilemma between incomplete matching and inaccurate

matching: while trying to maximize exact matches, cases may be excluded due to incomplete matching; or while trying to i i i t t hi t i ll lt maximize cases, more inexact matching typically results (Parsons, 2001).

• 2. Greedy matching is criticized also because it requires a

sizeable common-support region to work. When such region does not exist, greedy matching fails.

4

Limitations of Greedy Matching (2)

To illustrate the locally-optimal nature of greedy matching, consider the following crossword puzzle:

1 2 3 4 5 6 7 8 9 10 11 12 13

Across:

14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64

Across: 58 Guided excursion 62 Wheel holder 65 Feathery wraps Down: 58 PC key 59 Big name in kitchen gadgets

62 63 64 65 66 67

T O U R A X L E B O A S The answer: Guided excursion could also be “TRIP”. This is optimal in terms of 58 across, but does not fit to 59 down “big name in kitchen gadgets”, where R needs to be replaced by O.

Limitations of Greedy Matching (3)

Greedy matching assumes overlapping f ti t d

• f estimated

propensity scores between treated subjects and controls. That is, a sizeable common-support region exists The region exists. The common support region may look like the following chart:

5

Limitations of Greedy Matching (4)

The greedy matching fails for the following dataset – a real dataset used in the illustrating example of OPSM:

.25 .3 .35 .4 .45 ps

Boxplots of Estimated Propensity Scores

20 40 60 Density

boost

20 40 60 Density

boost

Histograms of Estimated Propensity Scores

.2 1

.2 .25 .3 .35 .4 .45 Users .2 .25 .3 .35 .4 .45 Nonusers

Limitations of Greedy Matching (5)

Consider the following matching problem: The task: create two matched pairs from four participants ith th f ll i it 1 5 6 & 9 with the following propensity scores: .1, .5, .6, & .9. A solution from the greedy matching: (.5, .6) and (.9, .1), because the p-score distance is the smallest for the first pair, and the two participants look most similar (i.e., |.5-.6|=.1) among the four. The total distance is : |.5-.6|+|.1-.9|=.9. A solution from the optimal matching: (.1, .5) and (.6, .9). Doing so, none of the two pairs created is better than the first pair created by the greedy matching (|.1-.5|=.4 > .1, and |.6-.9|=.3>.1). However, the total distance is optimized: |.1-.5|+|.6-.9|=.7, which is better than the total distance of the greedy matching (i.e., .9).

6

Optimal Matching (1)

Overview

Although OPSM has a history of only 10 years, the application has grown rapidly and fruitfully for two reasons: pp g p y y the use of network flow theory to optimize matching, and the availability of fast computing software packages R optmath (Hansen, 2007) that makes the implementation feasible. Rosenbaum (2002, pp.302-322) offers a comprehensive review of the theory and application principles of optimal matching. Hansen developed an optmatch that performs optimal matching in R and is available free with R. Haviland, Nagin, and Rosenbaum (2007) provided an excellent application example.

Optimal Matching (2)

The OPSM Method

OPSM is a process to develop S strata (A1, ...As; B1, ...Bs) consisting of S nonempt disjoint participants of A and S consisting of S nonempty, disjoint participants of A and S nonempty, disjoint subsets of B, so that |As|>1, |Bs|>1, As∩As’ =∅ for s≠s’, Bs∩Bs’ =∅ for s≠s’, and . In words OPSM produces S matched sets each of which

, ...

1

A A A

S ⊆

∪ ∪

B B B

S ⊆

∪ ∪...

1

In words, OPSM produces S matched sets, each of which contains |A1| and |B1|, |A2| and |B2|, ...and |AS| and | BS|. By definition, within a stratum or matched set, treated participants are similar to controls in terms of propensity scores.

7

Optimal Matching (3)

The OPSM Method

Depending on the structure (i.e., the ratio of number of treated participants to control participants ithin each strat m) the participants to control participants within each stratum) the analyst imposes on matching, we can classify matching into the following three types:

• 1. Pair matching: Each treated participant matches to a single

control.

• 2. Matching using a variable ratio or variable matching:

g g g Each treated participant matches to, for instance, at least

• ne and at most four controls.
• 3. Full matching: Each treated participant matches to one or

more controls, and similarly each control participant matches to one or more treated participants.

Optimal Matching (4)

The OPSM Method

OPSM is the process of developing matched sets (A1, ...As; B B ) ith si e of ( β) in s ch a a that the total B1, ...Bs) with size of (α,β) in such a way that the total sample distance of propensity scores is minimized. Formally, optimal matching minimizes the total distance Δ defined as

∑

=

= Δ

S s s s s s

B A B A

1

) , ( |) | |, (| δ ω

where is a weight function and δ is the difference between treated and control in terms of their observed covariates, such as their difference on propensity scores or Mahalanobis metrics. There are three ways to define the weight function (see Rosenbaum, 2002).

= s 1

|) | |, (|

s s

B A ω

8

Optimal Matching (5)

The OPSM Method

• How does OPSM accomplish this goal? Suffice it to say, that

this method achieves the goal by using a network flow this method achieves the goal by using a network flow approach (i.e., a topic in operations research) to matching. A primary feature of network flow is that it concerns the cost of using b for a as a match, where a cost is defined as the effect of having the pair of (a, b) on the total distance defined by the total-distance equation.

• From an application perspective, the structure imposed on
• ptimal matching (i.e., whether you want to run a 1-to-1 pair

matching, or a matching with a constant ratio of treated to control participants, or a variable matching with specifications of the minimum and maximum number of controls for each treated participant, or a full matching) affects both the level of bias reduction and efficiency.

Steps of Conducting OPSM

1. Run R optmatch: based on the ratio of sample treated subjects over controls, decide different matching structures; typically run pair matching, full matching, and variable matching with different treated-to-control ratios; 2. Balance checking: merge the matched strata to the original data, and check covariate imbalance using the method suggested by Haviland et al (2007) – one hopes that after matching, the sample treated and controlled subjects are balance on all observed covariates [Guo (2008a) developed Stata program imbalance to apply Haviland et al formula]; Stata program imbalance to apply Haviland et al formula]; 3. Post-matching analysis: based on the matched sample, run

• utcome analysis to determine treatment effectiveness

(there are different methods for post-matching analysis; see the next slide).

9

Post-Matching Analysis

1. For matched sample created by optimal pair matching (i.e., within each pair, you have 1-to-1 treated-to-control subjects), you create difference score between treated and control subjects on outcome variable, and each independent variable, and then run a regression using the difference-

• scores. The intercept of the regression indicates the

treatment effectiveness. 2. For matched sample created by optimal full matching or

• ptimal variable matching, you conduct the Hodges-

Lehmann aligned rank test. Guo (2008b) developed Stata Lehmann aligned rank test. Guo (2008b) developed Stata program hl to conduct this analysis. 3. Method 2 is bivariate. You could do a multivariate analysis by creating Hodges-Lehmann aligned rank score for the

• utcome variable, and for all independent variables; and

then you run a robust regression.

• 2. Example of Optimal

propensity score matching propensity score matching

10

The Example (1)

The research question: Does multi-generational dependence on welfare or poverty affect child academic development? p The problem: all prior studies implicitly assumed a causal effect of poverty on children’s academic

• achievement. However, most such studies used

covariance control methods such as regression or covariance control methods such as regression or regression-typed models without explicitly control for sample selection and confounding covariates. As a result, studies using covariance control may fail to draw valid causal inferences.

The Example (2)

The data: the 1997 Child Development Supplement (CDS) to the Panel Study of Income Dynamics (PSID) and the core PSID annual data from 1968 to 1997 (Hofferth et al., 2001). Dependent variable: academic achievement using the age-normed “letter-word identification” score of the Woodcock–Johnson Revised Tests of Achievement. “Treatment variable”: study child ever used AFDC from birth to 1997 (the variable is created from a careful birth to 1997 (the variable is created from a careful examination of the 30 years annual data of PSID).

11

The Example (3)

Covariates:

• 1. Primary caregiver’s number of years using AFDC in childhood

(ages 6 to 12 years), ranging from 0 to 7 years (the variable is created from a careful examination of the 30 years annual data y

• f PSID);
• 2. current income or poverty status, measured as the ratio of

family income to poverty threshold in 1996;

• 3. caregiver’s education in 1997, which was measured as years
• f schooling;
• 4. child race, which was measured as African American versus

non-African American; ;

• 5. child age in 1997; and
• 6. child gender.

After listwise deletion of missing data, the study sample is comprised of 1,003 children associated with 708 caregivers.

Sample Syntax Running R optmatch

#optmatch set.seed(10) library(foreign) cds <- read.dta("chpt5_2.dta") attach(cds) attach(cds) #logistic regression lcds <- glm(kuse ~ pcg.adc + age97 + mratio96 + pcged97 + black, family = binomial, data=cds) summary(lcds) library(optmatch) #create propensity scores based on the logistic reg. pdist <- pscore.dist(lcds) # f ll t h #run full match fm <- fullmatch(pdist) (fm.d <- matched.distances(fm,pdist,pres=TRUE)) unlist(fm.d,max) mean(unlist(fm.d)) sum(unlist(fm.d)) stratumStructure(fm)

12

Findings (1)

The level of multi-generational dependency

• n welfare shown by the sample

____________________________________________________________________________________ Child's AFDC use from birth Caregiver's AFDC use in childhood (ages 6 to 12) Total to current age in 1997 _____________________________________________ Never used Used ____________________________________________________________________________________ Never used 615 (61.3) 114 (11.4) 729 (72.7) Used 154 (15.4) 120 (12.0) 274 (27.3) Total 769 (76.7) 234 (23.3) 1,003(100) ____________________________________________________________________________________ p<.001, Chi-square test; each percentage (in parenthesis) is obtained by dividing the

• bserved frequency by the sample total 1003

Findings (2)

Sample description and results of t test and regression

_____________________________________________________________________________________________________________ Child AFDC Use from Birth p-value Estimated regression

A t test shows that

Variable to Current Age in 1997 coefficient (robust S.E.) [% or Mean (SD)] using letter-word score _________________________ in 1997 as outcome Never Used _____________________________________________________________________________________________________________ Outcome: letter-word identification score in 1997 103.98 (16.8) 94.16(14.8) .000a Covariate Ratio of family income to povery line in 1996 3.16 (2.76) 1.05 (1.00) .000b 1.13 (.33) ** Caregiver's education in 1997 (years of schooling) 13.2 (1.88) 11.6 (1.55) .000b .91 (.34) ** Caregiver's number of years using AFDC in childhood .48 (1.36) 1.86 (2.59) .000b

• .76 (.31) *

Child race: African American (reference: other) 36.20% 78.10% .000b

• 1.88 (1.23)

Child age in 1997 6.50 (2.78) 7.11 (2.81) .001b .87 (.17) *** Child gender: male (reference: female) 53 50% 51 80% 636b 2 00 ( 99) *

A t-test shows that the mean difference

• f letter-word-

identification score between children who ever used AFDC and children who never used is - 9 82 points

Child gender: male (reference: female) 53.50% 51.80% .636

• 2.00 (.99)

Child AFDC use status: used (reference: never)

• 4.73 (1.39) **

Constant 84.85 (4.5) *** R2 .15 Number of children (number of families) 274 (202) 729 (506) 1,003 (708) _____________________________________________________________________________________________________________

• a. Independent-sample t test, two-tailed
• b. Wilcoxon rank-sum (Mann-Whitney) test

* p<.05, ** p<.01, *** p<.001

9.82 points (p<.0001). An OLS regression estimates the difference as -4.73 points (p<.0001).

13

Findings (3)

The overlapping assumption is violated:

.25 .3 .35 .4 .45 ps

Boxplots of Estimated Propensity Scores

20 40 60 Density

boost

20 40 60 Density

boost

Histograms of Estimated Propensity Scores

.2 1

.2 .25 .3 .35 .4 .45 Users .2 .25 .3 .35 .4 .45 Nonusers

Findings (4)

Results of optimal matching

____________________________________________________________________________________________________________________ Matching Scheme Stratum Structure Total % of Ratio of "Treatmen:Control" Distance Cases Lost (Number of Matched Sets) in Matching ____________________________________________________________________________________________________________________ Full matching 22:1 20:1 15:1 14:1 10:1 8:1 7:1 6:1 5:1 4:1 3:1 2:1 1:1 ( 1 1 1 1 2 2 1 1 1 3 4 8 40 1:2 1:3 1:4 1:5 1:6 1:7 1:8 1:9 1:10 1:12 1:13 1:14 1:18 20 12 6 6 4 3 2 2 2 2 3 1 1 1:19 1:27 1:29 1:245 2 1 1 1) 31831 0.0% Variable matching 1 (at least 1 at most 4) 1:1 1:3 1:4 ( 122 1 151) 241707 0.0% Variable matching 2 (at least 2 at most 4) 1:2 1:3 1:4 ( 182 1 90) 255603 0 0% ( 182 1 90) 255603 0.0% Variable matching 3 (Hansen's equation) 2:1 1:1 1:2 1:4 ( 14 89 1 156) 240840 0.0% Variable matching 4 (at least 2 at most 7) 1:2 1:3 1:7 ( 237 1 36) 228723 0.0% Pair matching 1:1 0:1 ( 274 455) 40405 45.40% ____________________________________________________________________________________________________________________

14

Findings (5)

How does optimal matching remove imbalances? Taking the ratio of family income to poverty li i 1996 l b f

______________________________________________________________________________ Covariate and Matching Scheme dx dxm Before After Matcing Matching ______________________________________________________________________________ Ratio of family income to povery line in 1996 1.02 Full matching 0.04 Variable matching 1 (at least 1 at most 4) 0.79 Variable matching 2 (at least 2 at most 4) 0.94 Variable matching 3 (Hansen's equation) 0.79 Variable matching 4 (at least 2 at most 7) 0.87 Pair matching 0.25 Caregiver's education in 1997 (years of schooling) 0.91 Full matching 0.17 Variable matching 1 (at least 1 at most 4) 0.72 Variable matching 2 (at least 2 at most 4) 0.87

line in 1996 as an example, before matching, the treated and control groups differ on this variable by more than 100% of a standard

• deviation. Whereas, after full

matching, the standard bias is only 4% of a standard deviation. Nearly all matching schemes reduce bias for almost all variables to some

Variable matching 3 (Hansen's equation) 0.76 Variable matching 4 (at least 2 at most 7) 0.81 Pair matching 0.34 Caregiver's number of years using AFDC in childhood 0.67 Full matching 0.01 Variable matching 1 (at least 1 at most 4) 0.56 Variable matching 2 (at least 2 at most 4) 0.65 Variable matching 3 (Hansen's equation) 0.50 Variable matching 4 (at least 2 at most 7) 0.63 Pair matching 0.40 Child race: African American (reference: other) 0.93 Full matching 0.01 Variable matching 1 (at least 1 at most 4) 0.79 Variable matching 2 (at least 2 at most 4) 0.92 Variable matching 3 (Hansen's equation) 0.78 Variable matching 4 (at least 2 at most 7) 0.85 Pair matching 0.44 Child i 1997 0 22

for almost all variables to some extent, but some do more, and some do less. In terms of covariate balancing, the table confirms that full matching worked the best, and pair matching the second best.

Child age in 1997 0.22 Full matching 0.07 Variable matching 1 (at least 1 at most 4) 0.21 Variable matching 2 (at least 2 at most 4) 0.22 Variable matching 3 (Hansen's equation) 0.19 Variable matching 4 (at least 2 at most 7) 0.24 Pair matching 0.14 Child gender: male (reference: female) 0.03 Full matching 0.05 Variable matching 1 (at least 1 at most 4) 0.02 Variable matching 2 (at least 2 at most 4) 0.05 Variable matching 3 (Hansen's equation) 0.05 Variable matching 4 (at least 2 at most 7) 0.03 Pair matching 0.09 ______________________________________________________________________________ Note.

Absolute standardized difference in covariate means, before (dx) and after matching (dxm).

Findings (6)

Estimated average treatment effect on letter-word identification score in 1997 with Hodges-Lehmann aligned rank test (Matching scheme: full matching):

_____________________________________________________________________

Average Treatment Effect Point Estimate of the Hodges Standard Error Z p-value (Effect Size: Cohen's d) Lehmann test statistic

• f the test statistic

_____________________________________________________________________

• 1.97 (.190)
• 7051.766

3247.278

• 2.172

.015 _____________________________________________________________________

) ˆ ( ˆ

s s

W E W −

( )/

) ˆ ( ˆ

s s

W E W − ) ˆ (

s

W Var

) ˆ (

s

W Var

As the table shows, we used full matching and found that children who used AFDC had a letter-word identification score in 1997 that was, on average, 1.97 points lower than those who had never used AFDC; the difference was statistically significant at a .05 level. We used the Hodges-Lehmann test to gauge the statistical significance. The study also detected an effect size of .19, which is a small effect size in terms

• f Cohen’s (1988) criteria.

15

Findings (7)

Regressing difference-score of letter-word score on difference-scores of

____________________________________________________________________________ Estimated Regression Covariate Difference-score Coefficient (Robust S.E.)

difference scores of covariates after pair matching:

____________________________________________________________________________ Constant

• 3.17 (1.73) *

Ratio of family income to povery line in 1996 3.14 (1.67) * Caregiver's education in 1997 (years of schooling) .01 (.51) Child race: African American (reference: other)

• 3.34 (2.33)

Child age in 1997 .23 (.33) Child gender: male (reference: female)

• 1.21 (1.84)

Number of children (number of families) 274 (202) ____________________________________________________________________________ * p< 05 one tailed test

The estimated intercept from this model is -3.17 (p<.05). Thus, using pair matching and regression adjustment, the study found that, on average, children who used AFDC had a letter-word identification score in 1997 that was 3.17 points lower than children who never used AFDC; this finding showed statistical significance.

* p<.05, one-tailed test.

Findings (8)

Comparison of findings across all models

________________________________________________________________ Estimated Average Estimated Average Model Treatment Effect ________________________________________________________________

Independent-sample t test

• 9.82 ***

OLS regression

• 4.73 ***

Optimal matching (full) with Hodges-Lehmann aligned rank test

1 97 **

rank test

• 1.97

Regressing difference-score of outcome on difference- scores of covariates after pair matching

• 3.17 *

________________________________________________________________ Note. * p<.05, ** p<.01, *** p<.001, one-tailed test

16

Conclusions of the Study (1)

Based on the previous table, we find:

1 All models estimated a significant treatment

• 1. All models estimated a significant treatment

effect, but t-test and regression ignore selection bias and tend to produce biased and inconsistent estimates about treatment effects.

• 2. Among all estimates, which are more accurate
• r acceptable? Answer: optimal full matching

using the Hodges-Lehmann test and optimal pair matching using difference-score regression.

Conclusions of the Study (2)

Conclusion and assessing bias:

1 The estimate from optimal full matching is

• 1. The estimate from optimal full matching is
• 1.97 (p < .05), and from pair matching is
• 3.17 (p < .05). Based on these findings, we

may conclude that poverty on average causes a reduction in letter word causes a reduction in letter-word identification score by a range of 2 to 3 points.

Note: we are using the term “cause.” Indeed, it took a long journey before we could finally use the term!

17

Conclusions of the Study (3)

Conclusion and assessing bias:

• 2. The t-test exaggerated the impact by 398%:
• r 210%: .

% 398 100 * 97 . 1 97 . 1 82 . 9 = −

% 210 100 * 17 . 3 17 . 3 82 . 9 = −

• 3. The OLS regression exaggerated the impact

by 140% or 49% .

% 140 100 * 97 . 1 97 . 1 73 . 4 = −

% 49 100 * 17 . 3 17 . 3 73 . 4 = −

Conclusions of OPSM

This session illustrates one of the latest advances in propensity score matching, that is, OPSM. is, OPSM. Our study confirms the advantages of conducting OPSM. To evaluate causality or treatment effectiveness using observational data, researchers simply cannot afford to neglect bi d d b i biases produced by regression or any regression-typed models. The conventional greedy matching should be conducted in conjunction with other methods, such as OPSM.

18

3 Using propensity score

• 3. Using propensity score

matching to evaluate the Regional Partnership Grant Program g The Challenge of the RPG Evaluation

RPG is not a randomized clinical trial. Each Regional Partnership Grantee is an intervention. To evaluate the effectiveness of RPG, , evaluators need to use equivalent county data to find most comparable comparison group.

19

Identify Important Matching Variables

Running the logistic regression to predict i i h k f h propensity scores is the key for the evaluation. It’s important to identify important matching variables (i.e., those that make the RPG and comparison groups comparable). Variables may include those that measure Variables may include those that measure child and caregiver’s social demographic

• characteristics. The challenge is that you

must have same variables available at both the RPG and comparison groups.

Choose An Appropriate Model

• f PSM
• The nature of the outcome variable matters!
• Greedy matching applies to any kind of
• Greedy matching applies to any kind of
• utcome variables: continuous, categorical,

and time-to-event variables. The limitation: it always reduces sample size and may make the final sample differs from the original sample.

• Optimal propensity score matching only works

for continuous outcome variable for continuous outcome variable.

• Propensity score weighting can be used to

analyze categorical and time-to-event variables that does not reduce sample size (Guo & Fraser, 2010: pp.161-162, pp.197-199).

20

CONTACT INFORMATION

Ken DeCerchio, MSW, CAP Program Director Shenyang Guo, PhD Professor of Social Work Program Director Center for Children & Family Futures Phone: (714) 505-3525 Cell: (850) 459-3329 E-mail: kdecerchio@cffutures.org Website: www.ncsacw.samhsa.gov Professor of Social Work University of North Carolina Phone: (919) 843-2455 Email: sguo@email.unc.edu

39

Questions and Discussion Questions and Discussion

40

21

THANK YOU! PLEASE TAKE A BRIEF MOMENT TO COMPLETE OUR EVALUATION. YOU WILL BE RE-DIRECTED TO THE EVALUATION AFTER EXITING THIS WEBINAR.

41