How Do I Ask Questions? For your convenience, there are two ways to - PDF document

Thank you for joining us today! � If you haven’t dialed into the audio (telephone) portion, please do so now: 1-866-516-5393 Access Code: 33403311 Access Code: 33403311 � If you are experiencing technical problems with the GoToWebinar program (visual portion), contact the help desk: 1-800-263-6317 Reference Webinar ID: 309164457 � Today’s presentation and handouts are available for y p download at: http://www.cffutures.org/presentations/webinars The webinar will begin shortly. Thank you for your patience. 1 How Do I Ask Questions? For your convenience, there are two ways to ask questions two ways to ask questions during this webinar presentation: 1. Type and send your questions through the Question and Answer log located on the bottom half on your panel/dashboard. panel/dashboard. 2. There will also be time at the end of the webinar for you to ask questions via the conference line. 2 1

Propensity Score Matching Propensity Score Matching Strategies for Evaluating Substance Abuse Strategies for Evaluating Substance Abuse Services for Child Welfare Client, Session II of II Services for Child Welfare Client, Session II of II I) Welcome and Introductions I) Welcome and Introductions Ken DeCerchio, MSW, CAP II) Optimal Propensity Score Matching (OPSM) Shenyang Guo, PhD III) Example: Optimal Propensity Score Matching Shenyang Guo, PhD IV) Using Propensity Score Matching to Evaluate The Regional Partnership Grant Program Shenyang Guo, PhD V) Discussion/Questions Part II – Optimal Propensity Score Matching 1 1. Optimal propensity score matching Optimal propensity score matching (Rosenbaum, 2002) 2. Example of OPSM 3 3. Using propensity score matching to U i it t hi t evaluate the Regional Partnership Grant Program 2

1. Optimal propensity score matching (Rosenbaum, 2002) Limitations of Greedy Matching (1) � The key characteristic of greedy matching: it divides a large decision problem (i.e., matching) into a series of smaller, simpler decisions each of which is handled optimally; each makes those decisions one at a time without reconsidering k th d i i t ti ith t id i early decisions as later ones are made (Rosenbaum, 2002). � As such, the method has two limitations: 1. A dilemma between incomplete matching and inaccurate matching: while trying to maximize exact matches, cases may be excluded due to incomplete matching; or while trying to maximize cases, more inexact matching typically results i i i t t hi t i ll lt (Parsons, 2001). 2. Greedy matching is criticized also because it requires a sizeable common-support region to work. When such region does not exist, greedy matching fails. 3

Limitations of Greedy Matching (2) To illustrate the locally-optimal nature of greedy matching, consider the following crossword puzzle: Across: Across: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 58 Guided excursion 17 18 19 62 Wheel holder 20 21 22 23 24 25 26 65 Feathery wraps 27 28 29 30 31 32 33 34 35 36 37 38 39 40 Down: 41 42 43 44 45 46 47 58 PC key 48 49 50 51 52 53 54 59 Big name in kitchen gadgets 55 56 57 58 59 60 61 62 62 63 63 64 64 The answer: T O U R 65 66 67 A X L E B O A S Guided excursion could also be “TRIP”. This is optimal in terms of 58 across, but does not fit to 59 down “big name in kitchen gadgets”, where R needs to be replaced by O. Limitations of Greedy Matching (3) Greedy matching assumes overlapping of estimated f ti t d propensity scores between treated subjects and controls. That is, a sizeable common-support region exists. The region exists The common support region may look like the following chart: 4

Limitations of Greedy Matching (4) The greedy matching fails for the following dataset – a real dataset used in the illustrating example of OPSM: Histograms of Estimated Propensity Scores Boxplots of Estimated Propensity Scores boost boost .45 60 60 .4 40 40 .35 Density Density ps .3 20 20 .25 .2 0 0 .2 .25 .3 .35 .4 .45 .2 .25 .3 .35 .4 .45 Users Nonusers 0 1 Limitations of Greedy Matching (5) Consider the following matching problem: � The task: create two matched pairs from four participants with the following propensity scores: .1, .5, .6, & .9 . ith th f ll i it 1 5 6 & 9 � A solution from the greedy matching: (.5, .6) and (.9, .1) , because the p-score distance is the smallest for the first pair, and the two participants look most similar (i.e., |.5-.6|=.1) among the four. The total distance is : |.5-.6|+|.1-.9|=.9. � A solution from the optimal matching: (.1, .5) and (.6, .9) . Doing so, none of the two pairs created is better than the first pair created by the greedy matching (|.1-.5|=.4 > .1, and |.6-.9|=.3>.1). However, the total distance is optimized: |.1-.5|+|.6-.9|=.7, which is better than the total distance of the greedy matching (i.e., .9). 5

Optimal Matching (1) Overview � Although OPSM has a history of only 10 years, the application has grown rapidly and fruitfully for two reasons: pp g p y y the use of network flow theory to optimize matching, and the availability of fast computing software packages R optmath (Hansen, 2007) that makes the implementation feasible. � Rosenbaum (2002, pp.302-322) offers a comprehensive review of the theory and application principles of optimal matching. � Hansen developed an optmatch that performs optimal matching in R and is available free with R . � Haviland, Nagin, and Rosenbaum (2007) provided an excellent application example. Optimal Matching (2) The OPSM Method � OPSM is a process to develop S strata ( A 1 , ...A s ; B 1 , ...B s ) consisting of S nonempty, disjoint participants of A and S consisting of S nonempt disjoint participants of A and S nonempty, disjoint subsets of B , so that | A s |>1, | B s |>1, A s ∩ A s’ = ∅ for s ≠ s’ , B s ∩ B s’ = ∅ for s ≠ s’ , and ∪ ∪ S ⊆ A ... A A , 1 ∪ ... ∪ S ⊆ . B B B 1 � In words OPSM produces S matched sets each of which � In words, OPSM produces S matched sets, each of which contains | A 1 | and | B 1 |, | A 2 | and | B 2 |, ...and | A S | and | B S |. By definition, within a stratum or matched set, treated participants are similar to controls in terms of propensity scores. 6

Optimal Matching (3) The OPSM Method Depending on the structure (i.e., the ratio of number of treated participants to control participants participants to control participants within each stratum) the ithin each strat m) the analyst imposes on matching, we can classify matching into the following three types: 1. Pair matching : Each treated participant matches to a single control. 2. Matching using a variable ratio or variable matching: g g g Each treated participant matches to, for instance, at least one and at most four controls. 3. Full matching: Each treated participant matches to one or more controls, and similarly each control participant matches to one or more treated participants. Optimal Matching (4) The OPSM Method OPSM is the process of developing matched sets ( A 1 , ...A s ; B 1 , ...B s ) with size of ( α , β ) in such a way that the total β ) in s ch a B B ) ith si e of ( a that the total sample distance of propensity scores is minimized . Formally, optimal matching minimizes the total distance Δ defined as S ∑ Δ = ω δ (| |, | |) ( , ) A B A B s s s s = = s 1 1 s where is a weight function and δ is the difference ω (| |, | |) A B s s between treated and control in terms of their observed covariates, such as their difference on propensity scores or Mahalanobis metrics. There are three ways to define the weight function (see Rosenbaum, 2002). 7

Optimal Matching (5) The OPSM Method � How does OPSM accomplish this goal? Suffice it to say, that this method achieves the goal by using a network flow this method achieves the goal by using a network flow approach (i.e., a topic in operations research) to matching. A primary feature of network flow is that it concerns the cost of using b for a as a match, where a cost is defined as the effect of having the pair of ( a, b ) on the total distance defined by the total-distance equation. � From an application perspective, the structure imposed on optimal matching (i.e., whether you want to run a 1-to-1 pair matching, or a matching with a constant ratio of treated to control participants, or a variable matching with specifications of the minimum and maximum number of controls for each treated participant, or a full matching) affects both the level of bias reduction and efficiency. Steps of Conducting OPSM 1. Run R optmatch : based on the ratio of sample treated subjects over controls, decide different matching structures; typically run pair matching, full matching, and variable matching with different treated-to-control ratios; 2. Balance checking: merge the matched strata to the original data, and check covariate imbalance using the method suggested by Haviland et al (2007) – one hopes that after matching, the sample treated and controlled subjects are balance on all observed covariates [Guo (2008a) developed Stata program imbalance to apply Haviland et al formula]; Stata program imbalance to apply Haviland et al formula]; 3. Post-matching analysis: based on the matched sample, run outcome analysis to determine treatment effectiveness (there are different methods for post-matching analysis; see the next slide). 8