Ev Evalua aluation tion Cha Challenges llenges in in a a Time - PowerPoint PPT Presentation

Ev Evalua aluation tion Cha Challenges llenges in in a a Time Time of of Extens Extensiv ive e Inno Innova vations tions Presentation at AcademyHealth Annual Research Conference New Orleans, LA June 27, 2017 Randy Brown

Disclaimer The contents of this presentation are solely the responsibility of the author and do not necessarily represent the official views of the U.S. Department of Health and Human Services or any of its agencies. 2

Description of the Problem 3

How do we measure an intervention’s effect? • Usual goal: To learn how treatment group outcomes differ from what they would have been without the intervention – In the past, this counterfactual was either business as usual or usual care • With many opportunities for participating in new initiatives, it’s difficult to understand the counterfactual – Comparison group members could be in practices participating in other concurrent initiatives – Could conclude “no effects” when in fact it should be “equal effects” of the intervention and alternatives – Treatment group could be in other initiatives as well (CMS can restrict that to some degree) • Problem arises whether RCT or quasi-experimental designs are used 4

Overview of this talk • Concurrent practice transformation interventions • The Health Quality Partners (HQP) example • Design approaches to minimize contamination – The problem with restrictions – Factorial design alternative • Estimation approaches to account for contamination – Comprehensive Primary Care (CPC) initiative example • Where to go from here 5

Complex landscape of concurrent initiatives to improve cost and quality of care delivery Changes in underlying Post-MACRA Medicare Medicare FFS payments alternative payment models (APMs) • Phased implementation of Merit- • CMMI initiatives based Incentive Payment System (MIPS) – Basic APMs, for example: • Original Shared Savings Program, – Increases financial rewards for Million Hearts undertaking “improvement – Advanced APMs (stronger incentives), for activities” example: • CPC+ (two rounds) • Other Medicare Physician Fee • Next Generation ACO Schedule changes, for example: – APMs with advanced APM options, for example: – Comprehensive Care • OCM, SSP, BPCI, comprehensive ESRD Management Fees • Stakeholder-proposed Physician-Focused – Medicare Diabetes Prevention Payment Models Program Expanded Model – PTAC recommended models (April 2017) • Project Sonar • ACS- Brandeis Advanced APM • Many more being planned 6

The disappointing case of Health Quality Partners • Medicare Care Coordination Demonstration participant that had large, significant effects over first 8 years (2002 – 2010) – Only for high-risk patients: CAD, CHF, or COPD, and 1+ hospitalization in year before enrollment – RCT showed HQP reduced hospitalizations by 34% and expenditures by 22% • But during 4-year extension period (2011 – 2014), no effects – Intervention remained the same – Staff changes didn’t explain change in results – Patients slightly sicker than original sample on average, but reweighting showed this did not explain decline in effect • Decline in effect was due to lower hospitalizations for patients in randomized control group compared to pre-extension period (possibly ACO influence; also concurrent national decline) – Outcomes for treatment group patients were similar in initial and extension 7

Current design approaches to minimize contamination • Restrict applicants from participating in other initiatives – Would reduce take-up unless offered incentive • Stratify selection of treatment and comparison groups based on other initiatives engaged in at baseline – But practices still could join other initiatives after being assigned to comparison group – Treatment group practices could drop other initiatives after being accepted • Test different initiatives in different markets – But would need too many sites to eliminate overlap; doesn’t address existing initiatives or other payers’ initiatives – Might not be able to get enough local practices to participate if the initiative offered only in selected sites 8

A new design approach to minimize contamination • Use factorial/orthogonal designs to test model options against each other, rather than against “no intervention” – For example, allow all eligible applicants to participate (no pure control group), but assign them randomly to different selected combinations of incentives and requirements – Promotes quicker, more efficient learning about what works – Assigning all applicants to test some options reduces disincentive of restricting their participation in other initiatives – Randomization nets out effects of pre-existing initiatives • Requires working with practices to ensure all potential assignment combinations are preferred to nonparticipation • See article by Grannemann and Brown, Health Services Research (2017) 9

Estimation approaches to account for contamination • Include binary variables indicating prior and concurrent participation in other initiatives in regression equations • Simulate effects of the program under different assumptions about participation in other initiatives • Problem: coefficients on participation in other initiatives will reflect any selection bias as well as program effects – Group not participating in other initiatives might be least similar to practices participating in model being evaluated 10 10

CPC example of attempt to control for contamination • CPC is large initiative: 500 practices in 7 states/regions – Practices receive PBPM care management fee for attributed patients – Also receive quarterly feedback on performance – Regional learning faculty provide support and consultative services • One-third of matched comparison practices are in Medicare ACOs • Estimated model controlling for Medicare ACO participation – Found non-ACO comparison practices had smaller expenditure increase over time than ACO comparison practices – Thus, ACOs don’t appear to be attenuating CPC effects – ACOs might not locate in areas that already have costs under control, so not a valid measure of ACO effects 11 11

Where to go from here? • Factorial designs offer the best option for rigorous evaluation and rapid learning about optimal combinations of incentive levels, incentive types, and requirements of a specific intervention model – Shifts basic question from “Does this model work?” to “How can we design this model to work optimally?” – Can facilitate a more collaborative approach with providers to establish fair trade- offs that don’t discourage practice participation – Allows testing of MANY variations in a single experiment without adding more participants • Conventional evaluations should document extent of contamination, model it, and conduct sensitivity tests – But modeling is likely to yield biased estimates • Change focus of evaluations from “did it work” to “what changes were associated with improved outcomes” 12 12

For more information • Randy Brown – rbrown@mathematica-mpr.com 13 13