Research Rolling Entry Matching Allison Witman, Ph.D., Christopher - - PowerPoint PPT Presentation

www.rti.org

RTI International is a registered trademark and a trade name of Research Triangle Institute.

Comparison Group Selection with Rolling Entry in Health Services Research

Rolling Entry Matching

Allison Witman, Ph.D., Christopher Beadles, Ph.D., Thomas Hoerger, Ph.D., Yiyan Liu, Ph.D., Nilay Kafali, Ph.D., Sabina Gandhi, Ph.D., Peter Amico, Ph.D., Ann Larsen

1

Acknowledgement and Disclaimer

• This research was supported by the Centers for Medicare and

Medicaid Services under the U.S. Department of Health and Human Services

– Contract number: HHSM-500-2010-00021I

• The contents of this presentation are solely the responsibility of the

authors and do not necessarily represent the official views of the U.S. Department of Health and Human Services or any of its agencies.

2

Rolling Entry Interventions

• Rolling entry: participants start intervention at different times
• Rolling entry presents challenges for comparison group selection

– Small number of entrants in a period hinders propensity score models – Precipitating events prior to entry may be hard to pin down, but generate

changes in dynamic variables (e.g., utilization, spending)

• What is the counterfactual “entry date” for comparison?

3

Time Time

Dynamic Variable

Rolling Entry Matching (REM) Selects Comparison Groups for Rolling Entry Interventions

• Small number of entrants in a period
• Pools all entrants into a single propensity score model
• Dynamic variables that predict treatment change prior to entry
• Comparisons with similar pattern during the same time period are the best match
• Entry date of comparison is chosen based on entry date of best-matched

treatment observation

• REM: An example

– Synthetic intervention – Implementation of REM

1.

Quasi-panel dataset

2.

Apply matching methodology

3.

Matching algorithm

– Results

4

Rolling Entry Matching Synthetic Intervention

• Synthetic population that resembles Medicare claims data from

actual intervention

• 600 participants and 10,000 potential comparisons
• 8 entry quarters

– Quarterly enrollment pattern: 40, 50, 60, 70, 80, 90, 100, 110

• Follow participants for 8 baseline and 8 intervention quarters
• Spending spike

– Participants: quarter prior to entry – Comparisons: Random quarter

• Treatment effect is -$50 slope change in intervention period

5

Implementing Rolling Entry Matching: Step 1

1.

REM quasi-panel dataset

–

1 observation of each participant

–

Potential comparisons appear 8 times, once for each entry quarter

6

ID QUARTER TREAT AGE PREVIOUS VISITS 1 3 1 72 3 2 8 1 73 5 ID QUARTER TREAT AGE PREVIOUS VISITS 1 3 1 72 3 2 8 1 73 5 3 1 71 3 3 2 71 4 3 3 72 3 4 72 2 3 5 72 3 3 6 72 4 3 7 73 1 3 8 73 5

Implementing Rolling Entry Matching: Step 2

2.

Apply matching method

–

Propensity score

–

Mahalanobis distance

–

Coarsened exact matching

• Using synthetic data, we estimate a propensity score model

containing baseline:

–

Demographics

–

Chronic conditions

–

Previous Health care utilization

–

Previous Medicare payments

7

Implementing Rolling Entry Matching: Step 3

3.

REM matching algorithm

–

Match on propensity score

–

1:variable matching within a caliper up to 3 comparisons per participant

–

Match within quarters

–

Comparisons selected for the quarter in which they are the best match

–

Comparisons used with replacement within a quarter

8

ID QUARTER TREAT 1 3 1 2 8 1 3 1 3 2 3 3 3 4 3 5 3 6 3 7 3 8 AGE PREVIOUS VISITS PROPENSITY SCORE 72 3 0.13 73 5 0.15 71 3 0.14 71 4 0.18 72 0.10 72 2 0.11 72 3 0.17 72 4 0.18 73 1 0.14 73 5 0.15

.1 .2 .3 .4 .5 .6 .7 .8 .9 1 IP Admissions in Previous Year IP Admissions in Previous Quarter ED Visits in Previous Year ED Visits in Previous Quarter Number of Dual Eligible Months Number of Chronic Conditions ESRD Disabled Black Hispanic White Male Age Payments in Previous Year Payments in Previous Quarter

Absolute Standardized Differences Before and After Matching

Before Matching After Matching

Results: Treatment and Comparison Balance

9

• All variables reach a standardized difference ≤0.10 after matching

Results: Difference-in-Difference Treatment Effect

• Estimate change in slope using a difference-in-difference framework

– Synthetic treatment effect is -$50

10

Quarterly Spending Intervention Effect

• 50.10*

(28.38) Observations 21,760 R-squared 0.704 Notes: Table presents difference-in-difference linear regression coefficients and standard errors. The regression included controls for age, gender race, disability status, dual status, chronic diseases, ED use, inpatient use, a linear function of time, and indicators for the treatment group and post-intervention period. *** p<0.01, ** p<0.05, * p<0.1

Conclusion

• Implications for policy and practice

– Post ACA, rolling entry interventions utilized to maximize enrollment and

impact

– Evaluation methodology must advance to provide unbiased impact

estimates

• Rolling Entry Matching (REM) is a new methodology for selecting

comparison groups in the presence of rolling intervention entry

• Limitations

– Results illustrated in synthetic data only

• More research is needed to test methodology

– In synthetic intervention, performs similarly to cohort matching

11

Acknowledgements and Contact

• Acknowledgments

– Funded by the Centers for Medicare and Medicaid Services

• Contact information

Allison Witman RTI International 3040 E. Cornwallis Road Research Triangle Park, NC 27709 www.rti.org awitman@rti.org

12

Results: REM versus Cohort Matching

• Propensity score matching can perform poorly when sample size is

small

– Concato et al 1995, Peduzzi et al 1995, Peduzzi et al 1996, Zhao 2004

• Vary treatment group size between 300 and 4,500

– Treatment group cohorts range between 25 and 825 – Comparison group fixed at 10,000

13