Optimizing Risk Prediction for High Utilizers in a Safety Net System - - PowerPoint PPT Presentation

Optimizing Risk Prediction for High Utilizers in a Safety Net System

Zeyu (Zach) Li, MPH Kathleen Tatem, MPH Spriha Gogia, PhD, MPH Jeremy Ziring * Remle Newton-Dame, MPH Jesse Singer, DO, MPH Dave Chokshi, MD, MSc FACP

All the presenting author and co-authors have no relevant financial relationship to disclose * NYU School of Medicine, New York, NY

INTRODUCTION

2

About New York City Health + Hospitals

• Largest municipal health system in the

country

• Safety-net system: Mandate to care for

the uninsured/underserved in New York City

• 11 hospitals, 6 diagnostic and treatment

centers, 5 long-term care facilities, 70+ ambulatory care centers, correctional health services

• >1 million patients, 6 million visits per

year

STATEN ISLAND MANHATTAN BROOKLYN QUEENS THE BRONX

3

• Define “risk”
• Stratify population into high, med and low

risk with risk score algorithm

• Segment high risk population into

intervenable groups

• Target drivers of high risk in each segment

with effective programming

Risk Stratification Approach

Stratification Segmentation Targeting

4

METHODS

5

Study Population

• N = 833,969 adult patients with encounters at H+H during the measurement year

(Q3 2016 – Q2 2017)

• Excluded:
• Pregnant women
• Actively incarcerated
• Only ancillary care (eg. Radiology visits, blood draws)
• Missing name, date of birth or sex in patient record

6

Outcomes of Interest

• Limited consensus on the definition of “High Risk” in Literature: visits, re-visits, days
• Goal: predict high ED/Inpatient (“acute”) utilization in the prediction year (Q3

2017 – Q2 2018) from data in the measurement year (Q3 2016 – Q2 2017)

• Outcomes tested:
• 10+ acute days: Original algorithm (logistic regression) * outcome (top 1%)
• 5+ acute days : To get at medium/high risk populations (top 5%)
• No. of acute days: Continuous outcome

7 *https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5910357/

Data Prep and Flow

Data Sources Candidate Predictors: 70+

Modeling Dataset

Final Dataset

Clinical Utilization Social Determinants Demographics

8

Candidate Predictors

Demographics

• Continuous Binned Age
• Age Category (18-44, 45-64,

65-80, 81+)

• Sex
• Marital status
• Race/ethnicity
• Preferred spoken language
• Payer group
• Medicare
• Medicaid
• Self Pay
• Commercial
• Other

Clinical

• Count of chronic conditions

(Elixhauser)

• Individual chronic conditions

from Elixhauser, including:

• HTN Complicated
• HTN Uncomp
• DM Complicated
• DM Uncomp
• Substance Use
• Alcohol Use
• Solid Tumor
• Met. Cancer
• Obesity
• Renal Failure
• Liver Disease
• Congestive Heart Failure
• 31 in total
• Sickle Cell (CCS grouper)
• Frailty indicator
• Antipsychotic Rx
• Anticoagulant Rx

Utilization

• No. of ED visits
• No. of inpatient (IP) visits
• No. of ED/IP days
• 10+ ED/IP days
• 90+ ED/IP days
• No. of outpatient visits
• No. of primary care visits
• 1+ non-emergent ED visits
• 1+ primary care-treatable ED

visits

Social Determinants

• No. of zip code changes
• Neighborhood poverty level
• No. of payer changes
• Missed visits
• Homelessness
• History of incarceration

9

Model Development & Validation Strategy

Training Set Validation Set Data Train Test Test Test Test Test Tuned Candidate Models Final Model

70% 30%

5-fold validation for all models

10

RESULTS

11

Evaluating Model Performance

Software

SAS R

Algorithm

Logistic LASSO CART

Outcome

10+ Days 10+ Days 5+ Days

• No. of Days
• No. of Days
• No. of

Variables

33 30 34 17 18

Overall Model Performance AUROC

0.86 0.83 0.79 N/A N/A

RMSE

N/A N/A N/A 3.32 3.33

Top 1% Model Statistics PPV

44.6% 44.6% 44.8% 47.6% 48.4%

Sensitivity

16.2% 16.2% 16.2% 17.3% 15.0%

12

Final Model Coefficients

LASSO Model Predicting No. of Acute Days

Demographics Utilization Medicare Pt 0.06

• No. of ED Visits

0.35 Self-pay Pt

• 0.04
• No. of IP Visits

0.36 Other Payer Pt

• 0.08
• No. of ED/IP days (<30)

0.08 Pt with avoidable ER visit

• 0.08

Clinical Indicators Social Determinants Alcohol Use Dx 0.19

• No. of Zip changes

0.08 Psychoses Dx 1.17

• No. of Payer changes

0.004 Substance Use Dx 0.38 Pt w/Missed visits (2+) 0.04 Congestive Heart Failure Dx 0.05 Homeless 0.28 Renal Failure Dx 0.02 History of Incarceration 0.47

• No. of Elixhauser (count chronic

conditions) 0.21 Pt with antipsychotic Rx 0.40

Note: continuous variables are more influential than they appear

13

From Data to Patient Care: Risk Stratification

Predicted Acute Days

Predicted vs. Observed Avg. Acute Days, Q3 2017-Q2 2018 Predicted acute days vs. average observed days: near-linear relationship

14

DISCUSSION

15

From Data to Patient Care – Model Deployment

• High risk patient lists
• List of patients predicted to be high risk are shared with facilities

quarterly

• Epic integration
• High risk flag integrated in Epic to promote care coordination of

patients before/during their visit

• Model Deployment:
• High Risk definition in our system
• Top 5% of scored patients (translates to 3.6 days)
• 57.5% of high risk patients return for an ED or inpatient visit

16

From Data to Patient Care - Segmentation

17

THANK YOU !

Contact us: PopHealthHighRisk@nychhc.org

18

APPENDIX

19

Risk Scoring at NYC H+H: A Brief History

• In 2015: Algorithm to identify high risk ACO (Medicare) patients (Risk Score 1.0)
• 2016/2017: Risk Score 2.0, a payer agnostic algorithm to predict high utilization (10+

acute days) at NYC Health + Hospitals

• Prioritizes utilization, behavioral diagnoses like schizophrenia and alcohol

diagnoses

• Quarterly lists of top 1% high risk patients sent to facilities
• Algorithm published in JGIM earlier this year*
• 2017/2018: Distribution of high risk patient lists and dashboard with action segments
• This includes summary information of utilization, diagnoses, payer and

segments of high risk patients at NYC H+H overall and by facility.

• 2018/2019: Risk Score 3.0 Development
• Risk score re-optimization with augmented social determinants and machine

learning methods

20 *https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5910357/

Risk Score 3.0: Literature Review

n=31 n=24 n=9

n=9

EPIC EHR INTEGRATION

• A few organizations have successfully

integrated home-grown risk scores into Epic

• Epic Healthy Planet can be leveraged

for Risk Stratification

RISING RISK

• Difficult to predict
• Various approaches:
• Bloomers and Persisters
• Binary Outcomes
• Population Segmentation
• Preventable Costs
• Impactability

SEGMENTATION

• Mix of statistical methods available for

segmentation:

• CART
• K-means clustering
• Latent class analysis (MPLUS needed)
• Expert led segment development

81+ Articles Reviewed

RISK SCORE CREATION

• Common risk prediction modeling

methods:

• Multivariable regression
• LASSO
• Random forest
• CART
• Different outcomes of interest:
• Service utilization, clinical,

mortality, and hospitalization

• Prediction on general utilization

is rare

• Numerous predictors (see next slide)

21

Homelessness Identification

• Patients identified as homeless based on structured data

documentation:

• Demographics (Registration)
• Homeless/Undomiciled/Shelter listed in address field
• Hospital or Shelter address listed in address field
• ‘Person is homeless’ box checked in registration system
• 10+ zip code changes within previous 12 months
• Clinical diagnoses (Medical Record)
• ICD-10 homelessness code (Z59.0) on problem list
• Billing data (Bill / Claim)
• ICD-10 homelessness code (Z59.0) in billing data
• This does not include any free text information, such as Social Work

psychosocial assessments

Cohort Characteristics

Characteristics Development Cohort N=583 778,

• No. (%)

Validation Cohort N=250 191,

• No. (%)

23

Cohort Characteristics (Continued)

Characteristics Development Cohort N=583 778, No. (%) Validation Cohort N=250 191, No. (%)

• No. of Chronic Conditions

24