Enough: Nonparametric Matching Methods under Treatment Heterogeneity - - PowerPoint PPT Presentation

When Doubly Robust is Not Robust Enough: Nonparametric Matching Methods under Treatment Heterogeneity

Hui Shao PhD, Charles Stoecker PhD, Lizheng Shi PhD Department of Global Health Management and Policy School of Public Health and Tropical Medicine Tulane University

Tulane University, GHMP Population Heterogeneity and Treatment Heterogeneity

• Patient populations are heterogeneous.
• Age, sex, income, disease etiology and severity, presence
• f comorbidities and etc.
• Varying characteristics can potentially confound/modify

treatment effect.

Tulane University, GHMP Population Heterogeneity and Treatment Heterogeneity Treatment Outcome X

Treatment Effect

Treatment Outcome X

Treatment Effect Modifying

Treatment Outcome X

Treatment Effect Modifying

Confounding Modifying Selecting Treatment based on treatment effect

Tulane University, GHMP Treatment Selection

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Treatment Effect Probability of receiving treatment

Treatment Section based on treatment effect Average Treatment Effect on Treated (ATT) Average Treatment Effect (ATE)

• Technological advances have increased our ability to

1) Study treatment heterogeneity 2) Deliver precision-targeted individualized treatment plans.

• Patients potentially receive better treatment effect were more likely to choose the treatment.
• Physicians more likely to recommend the treatment to patients who would benefit more

from it.

Tulane University, GHMP What can we get from OLS?

𝑃𝑣𝑢𝑑𝑝𝑛𝑓𝑗 = 𝛾0 + 𝛾1 ∗ 𝑈𝑠𝑓𝑏𝑢𝑛𝑓𝑜𝑢𝑗 + 𝛿 ∗ 𝑌𝑗 + 𝜁𝑗 1) Modern Analytical framework (e.g. OLS) rarely discuss treatment heterogeneity issue. 2) And 𝛾1 was most likely to be assumed as population average across different individuals. 3) What does 𝛾1 actually represent? ATT or ATE?

Tulane University, GHMP What can we get from OLS? 𝛾1 = ෍

𝐿=1 𝐿

𝐹 Y𝑗 T𝑗 = 1, 𝑌𝑗 = 𝑌𝐿 − 𝐹 Y𝑗 T𝑗 = 0, 𝑌𝑗 = 𝑌𝐿 ∗ 𝑋

𝐿

𝛾1denotes Treatment Effect from OLS K denotes possible combination of 𝑌𝑗(strata) 𝑋

𝐿 denotes weights for strata K

Tulane University, GHMP What can we get from OLS? However 𝑋

𝐿 =

𝑤𝑏𝑠 T𝑗 𝑌𝑗 = 𝑌𝐿 ∗ Pr(𝑌𝑗 = 𝑌𝐿) σ𝐿=1

𝐿

𝑤𝑏𝑠 T𝑗 𝑌𝑗 = 𝑌𝐿 ∗ Pr(𝑌𝑗 = 𝑌𝐿) Acknowledging that 𝑤𝑏𝑠 T𝑗 𝑌𝑗 = 𝑌𝐿 = Pr T𝑗 = 1 𝑌𝑗 = 𝑌𝐿 ∗ (1 − Pr T𝑗 = 1 𝑌𝑗 = 𝑌𝐿 ) That is, individuals with propensity score closer to 0.5 receive higher weights 𝑈ℎ𝑓𝑝𝑠𝑓𝑢𝑗𝑑𝑏𝑚𝑚𝑧, 𝑥𝑓 𝑥𝑝𝑣𝑚𝑒 𝑥𝑏𝑜𝑢 𝑋

𝐿 = Pr(𝑌𝑗 = 𝑌𝐿)

Tulane University, GHMP What can we get from OLS?

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Treatment Effect Probability of receiving treatment

Chose treatment based on treatment effect

High Weights Medium Weights Medium Weights Low Weights Low Weights

OLS regression provides an estimation of treatment effect that neither ATT nor ATE, but a weighted average of treatment effect on variance of treatment assignment in each strata.

Tulane University, GHMP PSM to estimate ATT 1-1: one to one matching K-NN: k nearest neighborhood matching PSW-uncap: propensity score weighting without capping PSW-cap: propensity score with capping on both sides. LLM: local linear matching Kernel: Kernel matching

Tulane University, GHMP How matching handle this bias and provide ATT?

.5 1 1.5 2 .2 .4 .6 .8 1 X Treatment Control

Before Matching After Matching 𝑤𝑏𝑠 T𝑗 𝑌𝑗 = 𝑌𝐿 = 𝑤𝑏𝑠(T𝑗) Then 𝑋

𝐿 =

𝑤𝑏𝑠(T𝑗) ∗ Pr(𝑌𝑗 = 𝑌𝐿) σ𝐿=1

𝐿

𝑤𝑏𝑠(T𝑗) ∗ Pr(𝑌𝑗 = 𝑌𝐿) = Pr(𝑌𝑗 = 𝑌𝐿)

Equal weights for each individual

High Weights

Tulane University, GHMP doubly robust “Doubly robust” method refers to applying OLS regression on matched sample to achieve “optimal estimation accuracy” Use matching to resample the data to two balanced groups (pseudo randomization) Step 1 Step 2 Regression on matched data Evidence shows an accuracy improvement after applying doubly robust method2. However, previous studies never discussed heterogeneity scenarios.

2.Bang H, Robins JM. Doubly robust estimation in missing data and causal inference models. Biometrics 2005;

Tulane University, GHMP doubly robust

Small bandwidth Medium bandwidth Large bandwidth

• When using 1 to 1 matching, or kernel matching with narrow

bandwidth, it is relatively easier to achieve proposed matching pattern

• Kernel matching with large bandwidth is likely to achieve above

matching pattern (right).

• Applying OLS regression on this matched sample will still result in

assigning different weights on different strata based on Var (Ti|Xi=XK).

Low Weights

Low Weights

Equal weights

Tulane University, GHMP Objectives To test if the “doubly robust” method improves the estimation accuracy, compares to a direct mean comparison after matching.

Tulane University, GHMP Generating propensity score Three representative propensity score distributions were designed.

• 1. Johnson, N., "Systems of Frequency Curves Generated by Methods of Translation," Biometrika 36 (1949)

Balanced, most common More Patients in control group More Patients in treatment

Tulane University, GHMP Treatment Heterogeneity For each density design, three treatment heterogeneity scenario were designed

Linear Peak at 0.7 More random

Tulane University, GHMP Monte-Carlo Simulation

• 9 DGPs
• 500 individuals were generated
• 1000 iterations were conducted.

Generate Data

βOLS

1 by 1 KNN PSW Kernel Local Linear Run OLS

Record ATT Record ATT Record ATT Record ATT Record ATT

Tulane University, GHMP Treatment Heterogeneity MSE = σ𝑗=1

𝑗

(𝐵𝑈𝑈𝑛𝑏𝑢𝑑ℎ𝑗𝑜𝑕,𝑗 − 𝐵𝑈𝑈𝑢𝑠𝑣𝑓,𝑗)2 𝑗

• Mean squared error (MSE) was used to measure the bias

for each matching method.

• i denotes each iteration.
• Relative bias is the measurement of estimation accuracy

in this study: 𝑆𝑓𝑚𝑏𝑢𝑗𝑤𝑓 𝐶𝑗𝑏𝑡: MSE𝑛𝑏𝑢𝑑ℎ𝑗𝑜𝑕 MSE𝑃𝑀𝑇

Tulane University, GHMP Results & Discussion

86.2 43.45 71.38 54.14 48.3 32.07 32.03 10 20 30 40 50 60 70 80 90 100

Relative Bias (%)

Density 1 Heterogeneity 1

Bias of OLS

Tulane University, GHMP Results & Discussion

• Red line denotes relative bias from OLS regression.
• One to one matching provides unreliable matching results.
• 5 out of 9 simulated DGPs showed that one by one matching yield higher relative bias than simple OLS regression.

Tulane University, GHMP Results & Discussion

• PSW also provides unreliable estimation.
• 7 out of 9 simulated DGPs showed that PSW yield worse estimation than simple OLS regression.
• Capping provides better estimation.

Tulane University, GHMP Results & Discussion

• KNN matching provides consistent lower relative bias than OLS regression.
• However, the efficacy of this method is sensitive to the number of matching assigned to each individual.

Tulane University, GHMP Results & Discussion

• Kernel matching provides best estimation among all PSM methods.
• And this evidence is consistent in all DGPs.

Tulane University, GHMP Results & Discussion

10 20 30 40 50 60 70 80 90 100 0.0000 0.2000 0.4000 0.6000 0.8000 1.0000 1.2000 1.4000 1.6000 1.8000 2.0000

Rlative Bias to OLS Bandwith

Kernel Matching

Bandwidth 0.29 (Leave-one-out algorithm)

• The efficacy of Kernel matching relies on the choice of bandwidth.
• The leave-one-out algorithm2 is a widely used algorithm to select

bandwidth.

• Our simulation showed that this algorithm provided reliable estimation.
• 3. Frölich, M., Finite-sample properties of propensity-score matching and weighting estimators. Review of

Economics and Statistics, 2004. 86(1): p. 77-90.

Tulane University, GHMP doubly robust

• You can choose certain bandwidth to achieve lowest relative bias.
• Bias from doubly robust method increases as bandwidth increases.

Tulane University, GHMP Discussion Use matching to provide pseudo randomization Step 1 Step 2 Regression on matched data

• We also applied doubly robust method on other PSMs.
• Doubly robust method yields consistent equal or worse

estimation than a direct mean comparison.

Tulane University, GHMP Conclusion

• Treatment selection on expected

treatment effect (treatment heterogeneity) will likely to cause biased estimation from OLS regression.

• PSM method can reduce this bias.

Among all the PSM methods, kernel matching yields consistently best estimation.

• Doubly robust method is not

recommended.