Ev Evalua aluation tion Cha Challenges llenges in in a a Time - - PowerPoint PPT Presentation

ev evalua aluation tion cha challenges llenges in in a a
SMART_READER_LITE
LIVE PREVIEW

Ev Evalua aluation tion Cha Challenges llenges in in a a Time - - PowerPoint PPT Presentation

Ev Evalua aluation tion Cha Challenges llenges in in a a Time Time of of Extens Extensiv ive e Inno Innova vations tions Presentation at AcademyHealth Annual Research Conference New Orleans, LA June 27, 2017 Randy Brown


slide-1
SLIDE 1

Ev Evalua aluation tion Cha Challenges llenges in in a a Time Time of

  • f Extens

Extensiv ive e Inno Innova vations tions

Presentation at AcademyHealth Annual Research Conference New Orleans, LA

Randy Brown

June 27, 2017

slide-2
SLIDE 2

2

Disclaimer The contents of this presentation are solely the responsibility of the author and do not necessarily represent the official views of the U.S. Department of Health and Human Services

  • r any of its agencies.
slide-3
SLIDE 3

3

Description of the Problem

slide-4
SLIDE 4

4

How do we measure an intervention’s effect?

  • Usual goal: To learn how treatment group outcomes differ

from what they would have been without the intervention

– In the past, this counterfactual was either business as usual or usual care

  • With many opportunities for participating in new

initiatives, it’s difficult to understand the counterfactual

– Comparison group members could be in practices participating in

  • ther concurrent initiatives

– Could conclude “no effects” when in fact it should be “equal effects” of the intervention and alternatives – Treatment group could be in other initiatives as well (CMS can restrict that to some degree)

  • Problem arises whether RCT or quasi-experimental

designs are used

slide-5
SLIDE 5

5

Overview of this talk

  • Concurrent practice transformation interventions
  • The Health Quality Partners (HQP) example
  • Design approaches to minimize contamination

– The problem with restrictions – Factorial design alternative

  • Estimation approaches to account for contamination

– Comprehensive Primary Care (CPC) initiative example

  • Where to go from here
slide-6
SLIDE 6

6

Complex landscape of concurrent initiatives to improve cost and quality of care delivery

  • Phased implementation of Merit-

based Incentive Payment System (MIPS) – Increases financial rewards for undertaking “improvement activities”

  • Other Medicare Physician Fee

Schedule changes, for example: – Comprehensive Care Management Fees – Medicare Diabetes Prevention Program Expanded Model

  • CMMI initiatives

– Basic APMs, for example:

  • Original Shared Savings Program,

Million Hearts – Advanced APMs (stronger incentives), for example:

  • CPC+ (two rounds)
  • Next Generation ACO

– APMs with advanced APM options, for example:

  • OCM, SSP, BPCI, comprehensive ESRD
  • Stakeholder-proposed Physician-Focused

Payment Models – PTAC recommended models (April 2017)

  • Project Sonar
  • ACS- Brandeis Advanced APM
  • Many more being planned

Changes in underlying Medicare FFS payments Post-MACRA Medicare alternative payment models (APMs)

slide-7
SLIDE 7

7

The disappointing case of Health Quality Partners

  • Medicare Care Coordination Demonstration participant that had

large, significant effects over first 8 years (2002–2010)

– Only for high-risk patients: CAD, CHF, or COPD, and 1+ hospitalization in year before enrollment – RCT showed HQP reduced hospitalizations by 34% and expenditures by 22%

  • But during 4-year extension period (2011–2014), no effects

– Intervention remained the same – Staff changes didn’t explain change in results – Patients slightly sicker than original sample on average, but reweighting showed this did not explain decline in effect

  • Decline in effect was due to lower hospitalizations for patients in

randomized control group compared to pre-extension period (possibly ACO influence; also concurrent national decline)

– Outcomes for treatment group patients were similar in initial and extension

slide-8
SLIDE 8

8

Current design approaches to minimize contamination

  • Restrict applicants from participating in other initiatives

– Would reduce take-up unless offered incentive

  • Stratify selection of treatment and comparison groups

based on other initiatives engaged in at baseline

– But practices still could join other initiatives after being assigned to comparison group – Treatment group practices could drop other initiatives after being accepted

  • Test different initiatives in different markets

– But would need too many sites to eliminate overlap; doesn’t address existing initiatives or other payers’ initiatives – Might not be able to get enough local practices to participate if the initiative offered only in selected sites

slide-9
SLIDE 9

9

A new design approach to minimize contamination

  • Use factorial/orthogonal designs to test model options

against each other, rather than against “no intervention”

– For example, allow all eligible applicants to participate (no pure control group), but assign them randomly to different selected combinations of incentives and requirements – Promotes quicker, more efficient learning about what works – Assigning all applicants to test some options reduces disincentive

  • f restricting their participation in other initiatives

– Randomization nets out effects of pre-existing initiatives

  • Requires working with practices to ensure all potential

assignment combinations are preferred to nonparticipation

  • See article by Grannemann and Brown, Health Services

Research (2017)

slide-10
SLIDE 10

10 10

Estimation approaches to account for contamination

  • Include binary variables indicating prior and concurrent

participation in other initiatives in regression equations

  • Simulate effects of the program under different

assumptions about participation in other initiatives

  • Problem: coefficients on participation in other initiatives

will reflect any selection bias as well as program effects

– Group not participating in other initiatives might be least similar to practices participating in model being evaluated

slide-11
SLIDE 11

11 11

CPC example of attempt to control for contamination

  • CPC is large initiative: 500 practices in 7 states/regions

– Practices receive PBPM care management fee for attributed patients – Also receive quarterly feedback on performance – Regional learning faculty provide support and consultative services

  • One-third of matched comparison practices are in Medicare

ACOs

  • Estimated model controlling for Medicare ACO participation

– Found non-ACO comparison practices had smaller expenditure increase over time than ACO comparison practices – Thus, ACOs don’t appear to be attenuating CPC effects – ACOs might not locate in areas that already have costs under control, so not a valid measure of ACO effects

slide-12
SLIDE 12

12 12

Where to go from here?

  • Factorial designs offer the best option for rigorous

evaluation and rapid learning about optimal combinations of incentive levels, incentive types, and requirements of a specific intervention model

– Shifts basic question from “Does this model work?” to “How can we design this model to work optimally?” – Can facilitate a more collaborative approach with providers to establish fair trade-offs that don’t discourage practice participation – Allows testing of MANY variations in a single experiment without adding more participants

  • Conventional evaluations should document extent of

contamination, model it, and conduct sensitivity tests

– But modeling is likely to yield biased estimates

  • Change focus of evaluations from “did it work” to “what

changes were associated with improved outcomes”

slide-13
SLIDE 13

13 13

For more information

  • Randy Brown

– rbrown@mathematica-mpr.com