Ev Evalua aluation tion Cha Challenges llenges in in a a Time - - PowerPoint PPT Presentation

Ev Evalua aluation tion Cha Challenges llenges in in a a Time Time of

• f Extens

Extensiv ive e Inno Innova vations tions

Presentation at AcademyHealth Annual Research Conference New Orleans, LA

Randy Brown

June 27, 2017

2

Disclaimer The contents of this presentation are solely the responsibility of the author and do not necessarily represent the official views of the U.S. Department of Health and Human Services

• r any of its agencies.

3

Description of the Problem

4

How do we measure an intervention’s effect?

• Usual goal: To learn how treatment group outcomes differ

from what they would have been without the intervention

– In the past, this counterfactual was either business as usual or usual care

• With many opportunities for participating in new

initiatives, it’s difficult to understand the counterfactual

– Comparison group members could be in practices participating in

• ther concurrent initiatives

– Could conclude “no effects” when in fact it should be “equal effects” of the intervention and alternatives – Treatment group could be in other initiatives as well (CMS can restrict that to some degree)

• Problem arises whether RCT or quasi-experimental

designs are used

5

Overview of this talk

• Concurrent practice transformation interventions
• The Health Quality Partners (HQP) example
• Design approaches to minimize contamination

– The problem with restrictions – Factorial design alternative

• Estimation approaches to account for contamination

– Comprehensive Primary Care (CPC) initiative example

• Where to go from here

6

Complex landscape of concurrent initiatives to improve cost and quality of care delivery

• Phased implementation of Merit-

based Incentive Payment System (MIPS) – Increases financial rewards for undertaking “improvement activities”

• Other Medicare Physician Fee

Schedule changes, for example: – Comprehensive Care Management Fees – Medicare Diabetes Prevention Program Expanded Model

• CMMI initiatives

– Basic APMs, for example:

• Original Shared Savings Program,

Million Hearts – Advanced APMs (stronger incentives), for example:

• CPC+ (two rounds)
• Next Generation ACO

– APMs with advanced APM options, for example:

• OCM, SSP, BPCI, comprehensive ESRD
• Stakeholder-proposed Physician-Focused

Payment Models – PTAC recommended models (April 2017)

• Project Sonar
• ACS- Brandeis Advanced APM
• Many more being planned

Changes in underlying Medicare FFS payments Post-MACRA Medicare alternative payment models (APMs)

7

The disappointing case of Health Quality Partners

• Medicare Care Coordination Demonstration participant that had

large, significant effects over first 8 years (2002–2010)

– Only for high-risk patients: CAD, CHF, or COPD, and 1+ hospitalization in year before enrollment – RCT showed HQP reduced hospitalizations by 34% and expenditures by 22%

• But during 4-year extension period (2011–2014), no effects

– Intervention remained the same – Staff changes didn’t explain change in results – Patients slightly sicker than original sample on average, but reweighting showed this did not explain decline in effect

• Decline in effect was due to lower hospitalizations for patients in

randomized control group compared to pre-extension period (possibly ACO influence; also concurrent national decline)

– Outcomes for treatment group patients were similar in initial and extension

8

Current design approaches to minimize contamination

• Restrict applicants from participating in other initiatives

– Would reduce take-up unless offered incentive

• Stratify selection of treatment and comparison groups

based on other initiatives engaged in at baseline

– But practices still could join other initiatives after being assigned to comparison group – Treatment group practices could drop other initiatives after being accepted

• Test different initiatives in different markets

– But would need too many sites to eliminate overlap; doesn’t address existing initiatives or other payers’ initiatives – Might not be able to get enough local practices to participate if the initiative offered only in selected sites

9

A new design approach to minimize contamination

• Use factorial/orthogonal designs to test model options

against each other, rather than against “no intervention”

– For example, allow all eligible applicants to participate (no pure control group), but assign them randomly to different selected combinations of incentives and requirements – Promotes quicker, more efficient learning about what works – Assigning all applicants to test some options reduces disincentive

• f restricting their participation in other initiatives

– Randomization nets out effects of pre-existing initiatives

• Requires working with practices to ensure all potential

assignment combinations are preferred to nonparticipation

• See article by Grannemann and Brown, Health Services

Research (2017)

10 10

Estimation approaches to account for contamination

• Include binary variables indicating prior and concurrent

participation in other initiatives in regression equations

• Simulate effects of the program under different

assumptions about participation in other initiatives

• Problem: coefficients on participation in other initiatives

will reflect any selection bias as well as program effects

– Group not participating in other initiatives might be least similar to practices participating in model being evaluated

11 11

CPC example of attempt to control for contamination

• CPC is large initiative: 500 practices in 7 states/regions

– Practices receive PBPM care management fee for attributed patients – Also receive quarterly feedback on performance – Regional learning faculty provide support and consultative services

• One-third of matched comparison practices are in Medicare

ACOs

• Estimated model controlling for Medicare ACO participation

– Found non-ACO comparison practices had smaller expenditure increase over time than ACO comparison practices – Thus, ACOs don’t appear to be attenuating CPC effects – ACOs might not locate in areas that already have costs under control, so not a valid measure of ACO effects

12 12

Where to go from here?

• Factorial designs offer the best option for rigorous

evaluation and rapid learning about optimal combinations of incentive levels, incentive types, and requirements of a specific intervention model

– Shifts basic question from “Does this model work?” to “How can we design this model to work optimally?” – Can facilitate a more collaborative approach with providers to establish fair trade-offs that don’t discourage practice participation – Allows testing of MANY variations in a single experiment without adding more participants

• Conventional evaluations should document extent of

contamination, model it, and conduct sensitivity tests

– But modeling is likely to yield biased estimates

• Change focus of evaluations from “did it work” to “what

changes were associated with improved outcomes”

13 13

For more information

• Randy Brown

– rbrown@mathematica-mpr.com