Measuring Year-to-Year Improvement in Risk-Adjusted Outcome - - PowerPoint PPT Presentation

Measuring Year-to-Year Improvement in Risk-Adjusted Outcome Measures: Filling a Methods Gap

Academy Health ARM June 27, 2016

1

Funding and Disclosures

• This work was funded by a contract with the Centers

for Medicare & Medicaid Services, Center for Clinical Standards and Quality

• CMS Government Task Leader: Vinitha Meyyur
• Contract #: HHSM-500-2013-13018I, Task Order

HHSM-500-T0002 – Modification 0002

2

Outline

• Need for improvement measurement

methods

• Goals for methods
• Case Study: ACO admission measure in

Medicare Shared Savings Program (SSP)

– Program design, goals, challenges – Three methods – Results – Evaluation methods against program and technical goals

3

Why Methods for Measuring Improvement?

• Providers want credit for improvement
• Comparing year-over-year risk-adjusted scores

shows change in relative rank

• Programs increasingly incorporating

improvement scores in payment models

4

Goals for Methods

• Fit with program design and goal
• Technically valid and feasible
• Practical to implement
• Usable for program and providers

– Detect meaningful improvement? – Are readily understood? – Support target setting?

5

ACO RISK-STANDARDIZED ACUTE ADMISSION RATES AMONG PATIENTS WITH HEART FAILURE

6

Case Study:

• ACOs earn quality points on a sliding scale based on level of

performance in four quality domains

• Shared savings based on quality points
• ACOs can earn bonus improvement points for statistically

significant improvement from one year to the next:

• Size of improvement does not matter
• ACOs who reach 90th percentile or better earn maximum, regardless of

improvement

• Getting worse on one measure cancels out improvement in another

for measures in same domain => two-sided test

7

Medicare SSP Improvement Bonus

ACO Measure Tested: ACO 37 – Risk-Standardized Acute Admission Rate among Heart Failure Patients

Cohort:

• Non-hospitalized heart failure patients Medicare FFS age 65+
• Assigned to ACO

Outcome

• Number of acute, unplanned hospital admissions per 100 person-years at

risk for hospitalization Data

• Medicare Claims

Risk-adjustment model

• Hierarchical negative binomial model
• Risk-adjustment variables:
• Age
• 21 comorbidities
• Cardiac device variable

8

Measurement Challenges

• An ACO’s patients change from Year 1 to Year 2
• Admission risk of enrollees changes from Year 1 to Year 2
• Natural events may affect admission rate
• Regression to the mean could contribute to year-to-year

change

9

Population Studied

• Patients: Medicare FFS patients 65+
• Year 1 (2012)= 123,626
• Year 2 (2013)= 134,961
• ACOs
• 114 ACOs in SSP in both 2012 and 2013
• ACO volume range: 303 - 9,914 (median: 690)
• ACO risk factor frequencies in Y1, Y2 similar

10

Two-Thirds of Patients Stay in Their ACO

11

Distribution of the percentage of Y1 patients also in Y2 across ACOs (79,942 of 123,626 patients stay in their ACO from Y1 (2012) to Y2 (2013) [64.7%])

12

Variables Overall (N=258,587) Y1 Y2 Age Mean(std) 79.8 (7.8) 79.8 (7.8) Race White 87% 87% Black 9% 9% Male 51% 51% High risk cardiovascular (CV) factors 32.8% 31.9% Low risk CV factors 85.4% 84.1% Arrhythmia 64.4% 63.3% Structural Heart Disease 41.5% 39.9% Advanced cancer 7.8% 7.6% Dementia 22.4% 21.3% Diabetes w/ complications 52.6% 52.2% Dialysis 3.1% 3.1% Disability/Frailty 22.9% 21.8% GI/GU 33% 32% Hematology 17% 15% Infection & immune disorders 6% 7% Kidney disease 39% 39% Liver disease 2% 2% Neurological 45% 44% Psychiatric illness/Substance abuse 37% 36% Pulmonary disease 59% 57% Other advanced organ failure 20% 19% CRT/ICD/Pacemaker 24% 23% Iron deficiency anemia 54% 52% Major organ transplant 0% 0% Other organ transplant 1% 0%

Tailored Objectives for Method

• Fits with program design
• Compares ACO to itself (not used to compare quality among ACOs)
• Accounts for enrollee and risk shifts
• Technically feasible
• Statistically tests for better or worse rate
• Detects improvement
• Readily implemented
• Useable by ACOs
• Provides target rate

13

ACO methods tested

Options

Objectives

Year 1 as Benchmark Pre-Post Test Matched Patients

Fits with program goal: ACO vs. self Addresses risk shifts Addresses regression to mean Identifies statistically significant change Readily implemented Usable

14

Year 1 Benchmark

15

Method

• Fit risk model to each ACO in Year 1
• Apply model coefficients to Year 2 data to calculate expected for

Year 2 Score

• Observed (Year 2) – Expected (Year 2)
• Significance level: 0.05

Results for 114 ACOs

ACO methods tested

Options

Objectives

Year 1 as Benchmark Pre-Post Test Matched Patients

Fits with program goal: ACO vs. self Addresses risk shifts Addresses regression to mean Identifies statistically significant change Readily implemented Usable

16

Option 2: Pre-Post Test

17

Method

• Fit risk model to ACO using Year 1 & Year 2 combined
• Add a year variable (Y2=1; Y1=0)

Score

• Exponentiate year variable to get rate ratio (<1= improvement)
• Significance level: 0.05

Results for 114 ACOs

ACO Methods Tested

Options

Objectives

Year 1 as Benchmark Pre-Post Test Matched Patients

Fits with program goal: ACO vs. self Addresses risk shifts Addresses regression to mean Identifies statistically significant change Readily Implemented Usable

18

Option 3: Matched Patients

19

Method

• Use propensity score matching to match patients from Year 1 and

Year 2 (Mahalanobis distance matching method)

• Fit risk model to subset of matched patients adding a year variable

(Y2=1; Y1=0) Score

• Exponentiate year variable to get rate ratio (<1=improvement)
• Significance level 0.05

Results for 12 ACOs Proportion of Year 2 matched patients: 83%-98.8%

• Tested in 12 ACOs, 3 in each of 4 volume quartiles
• 1 in highest volume quartile significantly improved
• 11 showed no statistically significant change

ACO methods tested

Options

Objectives

Set Year 1 as Benchmark Pre-Post Test Matched Patients

Fits with program goal: ACO vs. self Addresses risk shifts Addresses regression to mean Identifies statistically significant change Readily implemented Usable

20

ACO Methods Tested

Options

Objectives

Year 1 as Benchmark Pre-Post Test Matched Patients

Fits with program goal: ACO vs. self Addresses risk shifts Confident about risk-adjustment Addresses regression to mean More stable model coefficients Identifies statistically significant change Readily Implemented Usable Can set target early in Year 2

21

Assessment Similar for the 114 ACOs: Year 1 Benchmark vs. Pre-Post Test

22

Year 1 Benchmark Pre-Post Test

Option 1 total Significantly better No different Significantly worse Significantly better 17 (14.9%) 13 (11.4%) 0 (0.0%) 30 (26.3%) No different 0 (0.0%) 81 (71.1%) 0 (0.0%) 81 (71.1%) Significantly worse 0 (0.0%) 2 (1.7%) 1 (0.9%) 3 (2.6%) Option 2 total 17 (14.9%) 96 (84.2%) 1 (0.9%) 114 (100%)

*Concordance: 86.9%; Kappa=0.64

And similar for 12 ACOs evaluated with Patient Matching

23

12 ACOs Performance Status

Year 1 Benchmark Pre-Post Test Patient Matching

Volume quartile 1 (low volume): 1 No different No different No different 2 No different No different No different 3 Significantly improved Significantly improved No different Volume quartile 2: 4 No different No different No different 5 No different No different No different 6 No different No different No different Volume quartile 3: 7 No different No different No different 8 No different No different No different 9 No different No different No different Volume quartile 4 (high volume): 10 No different No different No different 11 Significantly improved Significantly improved Significantly improved 12 No different No different No different

Summary and Conclusion

• Criteria for method should be aligned with

program goals

• All methods were technically feasible
• The three methods have different strengths

and weaknesses

• The results were aligned
• The choice should reflect program priorities

24

Acknowledgements

• Craig S. Parzynski, M.S.
• Haikun Bao, Ph.D
• Jeph Herrin, Ph.D.
• Faseeha K. Altaf, M.P.H.
• Hayley J. Dykhoff, B.A.
• Mayur Desai, Ph.D., M.P.H.
• Susannah M. Bernheim, M.D., M.H.S.
• Zhenqiu Lin, Ph.D.

25

QUESTIONS?

26

TEP Members/Organizations

Name Organization (Title) Mimi Huizinga, MD, MPH Premier, Inc. (Vice President, Chief Clinical Officer of PACT Collaborative) David Introcaso, PhD National Association of ACOs ([NAACOS] Vice President, Policy and Operations) John Michael McWilliams, MD, PhD Harvard Medical School (Associate Professor of Health Care Policy and Medicine) David Muhlestein, JD, PhD, MHA, MS Leavitt Partners (Senior Director of Research and Development) Ami Parekh, MD, JD University of California, San Francisco (Assistant Clinical Professor) Denise Prince, MBA, MPH Geisinger Health System (System Vice President, Value Based Care) Keystone Accountable Care Organization, LLC (Chief Administrative Officer) Jeff Stensland, PhD Medicare Payment Advisory Commission ([MedPAC] Principal Policy Analyst) Name Organization (Title) Michael Barrett, BS Universal American/Collaborative Health System (Senior Vice President ACO Southeast Region and National Development) Larry Becker, BS Xerox (Director, Strategic Partnerships, Alliances, and Analytics for Xerox Corporation) Scott Berkowitz, MD, MBA Johns Hopkins Medicine Alliance For Patients, LLC. (Executive Director); Office

• f Johns Hopkins Physicians (Senior

Medical Director, Accountable Care Office); Johns Hopkins Medicine (Assistant Professor) Alex Blum, MD, MPH Evergreen Health Co-op (Chief Medical Officer) Erin Deloreto, MPAP QualCare Alliance Network, Inc. (Assistant Vice President, Operations) Aparna Higgins, MA America's Health Insurance Plans ([AHIP] Senior Vice President, Private Market Innovations)

28