How to Predict Overdose Death with PDMP Data and Advanced Analytics - - PowerPoint PPT Presentation

How to Predict Overdose Death with PDMP Data and Advanced Analytics

Live Webinar Thursday, March 16, 2017

Sp Spon

• nsored by:

y:

Live Webinar 3/16/17 | 1:00 p.m. ET

Live Webinar 3/16/17 | 1:00 p.m. ET

Q+A – Submit a question, located below the slides Resources List – Access website links and download slides Help – Submit any technical issues, located below the slides

Twitter

Join the discussion on Twitter! Live tweet using the hashtag #RXLiveWebinar

Live Webinar 3/16/17 | 1:00 p.m. ET

How to Predict Overdose Death with PDMP Data and Advanced Analytics

A cooperative effort between OARRS and Appriss Health

Speakers

Chad Garner, MS

• OARRS Director

Jim Huizenga, MD

• Emergency Physician, BCEM
• Chief Clinical Officer for Appriss Health

David Speights, PhD

• Ph.D. Biostatistics
• Chief Data Scientist for Appriss

Disclaimer

• Dr. Huizenga and Dr. Speights are both employees of Appriss Health.
• Mr. Garner is the PMP director for the Ohio Automated Rx Reporting

Service (OARRS).

• He certifies that he has NO affiliations with or involvement in any organization or entity with

any financial interest (such as honoraria; educational grants; participation in speakers’ bureaus; membership, employment, consultancies, stock ownership, or other equity interest; and expert testimony or patent-licensing arrangements), or non-financial interest (such as personal or professional relationships, affiliations, knowledge or beliefs) in the subject matter or materials discussed in this manuscript.

Objectives

• Explain how PDMP data and advanced analytics can impact detection of

unintentional overdose deaths

• Identify comprehensive data results, from a variety of complex data patterns, for

early detection of overdose risk

• Review the early identification process of prevention and management of

substance use disorders in the U.S. and OARRS

Overview

• Data Overview
• Initial Analysis
• Secondary Analysis
• Advanced Data Analysis
• Summary and Future Directions

Perspective “Treatment without prevention is simply unsustainable”

Bill Gates

Data Overview

OARRS Operational Overview

• Operational since October 2, 2006
• Collects information regarding all Schedule II-V controlled substances dispensed by

Ohio-licensed pharmacies

• 25 million prescriptions, 4 million patients per year
• Patient-identifiable data kept for 3 years
• Patient-deidentified data kept indefinitely

Data Overview - Decedents

2014 Death Data

• Data Sample
• 2,482 unintentional overdose deaths
• Inclusion Criteria
• Department of Health determined death due to unintentional overdose between

Jan 1 – Dec 31, 2014

• Matching ID within OARRS
• Overdose death data not available until June 2015
• OARRS keeps 3 years of patient-identifiable data
• Selection results
• 1,687 decedents

Initial OARRS Analysis

Microsoft SQL Server Analysis Services was used to create predictive data models across 12 data measures using 4 different algorithms.

1. Microsoft Decision Trees* 2. Microsoft Clustering 3. Microsoft Naïve Bayes 4. Microsoft Logistics

Output was used to trim the data model to 4 data measures showing strong association with overdose death.

Initial Analysis

Four risk factors strongly associated with OD death

Risk Factor Odds Ratio Pharmacies ≥ 4 3.7 Benzo / Opioid overlap ≥ 35 days 2.4 Max MED ≥ 100 2.3 Cash Payment ≥ 1 2.2 All 4 present 10.3

Odds and Odds Ratios

What are odds?

• Chance of an event occurring divided by the chance that the event won’t
• ccur
• If the chance of something is small then it is approximately equal to the

probability What are odds ratios?

• Odds ratios are a ratio which compares one group to another group and is

used to express relative risk

Odds and Odds Ratios - Example

Example Odds Ratio of group 2 compared to group 1 1.52 / 0.07 = 21.71 Group 2 is more than 20 times as likely to suffer the outcome as Group 1

Group Chance Of Outcome Odds 1 0.07% 0.07% / 99.93% = .07% 2 1.50% 1.50% / 98.50% = 1.52%

Initial Analysis

Four risk factors strongly associated with OD death

Risk Factor Odds Ratio Pharmacies ≥ 4 3.7 Benzo / Opioid overlap ≥ 35 days 2.4 Max MED ≥ 100 2.3 Cash Payment ≥ 1 2.2 All 4 present 10.3

Secondary Analysis

Using Narx Scores as a predictor of overdose death

• Type specific use indicators for narcotics, sedatives and stimulants
• Range from 000-999
• As the score increases, so does the presence of:
• Providers
• Pharmacies
• MME
• Overlaps

Narx Scores in vivo

Narx Scores in vivo

Narx Scores In Workflow

Narx Scores

Narx Score Distribution

Scores

< 200 200-499 500-650 >650

Narx Scores Summary

Narx Scores are

• Numerical representations of PDMP data that capture “use”.
• Information at a glance.

Narx Scores are not

• Rules (they are tools).
• Are not synonymous with abuse.

Secondary Analysis

Methodology – Narx Scores as a predictor of overdose risk

• 100:1 case / control study
• Determined the highest narcotic score in the year prior to death for each decedent and for

100 date-matched living controls

• Calculated Odds Ratios

Secondary Analysis

Results

Narcotic Score Odds Ratio 0-199 1 200-299 6.4 300-399 7.4 400-499 10.2 500-599 15.5 600-699 23.3 700-799 29.8 800-899 37.7 900-999 63.4

Primary and Secondary Analysis Summary

Using traditional techniques that include both red flags and a composite use indicator, we were able to determine significant associations with unintentional

• verdose death
• Initial analysis identified 4 red flags strongly associated with overdose death risk
• Secondary analysis strongly associated Narx Scores with overdose death risk

Machine Learning Approach

Data Overview Used the same data from the secondary analysis

• 1:100 case to controls
• Artificial resultant 1% incidence of disease

Applied machine learning and other predictive techniques to develop a 3-digit score similar to Narx Scores, termed an Overdose Risk Score

• Range from 000-999
• Risk doubles approximately every 100 points
• Similar distribution to Narx Scores

Advanced Data Analysis – General Method

Drug Dispensation History and Trends Types of Narcotics/Sedatives Prescriber/Pharmacy Visit History and Acceleration Narcotics to Morphine Equivalencies Data Inputs Literature Defined Red Flags High Risk Behavioral Patterns

Inputs processed through predictive models to determine the composite risk

Machine Learning Models Optimized by Simulated Annealing

Decision Engine

Overdose Risk Score (000 to 999)

Machine Learning Approach

Variable Derivation

Variable Determination

• Hypothesize a variable and the expected effect
• Develop variable for case and controls
• Determine independent predictive ability

More than 70 variables were evaluated using this approach Examples

• Amount of narcotics (in MME) used in the prior 365 days
• Amount of sedatives (in MME) used in the prior 60 days

From the 70 variables, approximately one dozen chosen for final model

• Some that are used for Narx Scores
• Some that are used for Red Flags
• Some new variables that look at change over time

Model Validation During Development

• For each decedent and matched control 4 random dates were chosen in the one

year prior to the date of death for the decedent producing 4 separate modeling sets to use in model fitting and evaluation.

• Each Set was further split into a 75% training sample and 25% validation sample

Final Model Validation

• After model completion, we used death data from 2013 and 2015 to validate

the final model and compare to the 2014 results

Machine Learning Approach

Variable Derivation

Machine Learning Approach

Model Evaluation with the KS Statistic

• Kolmogorov-Smirnov (KS) Statistic measures the maximum difference between

the cumulative percentage of two populations (Non-Decedents vs Decedents) by score.

• Standard metric used in statistics to evaluate models.

0% 100% Non-Decedents Decedents

KS Low High

SCORE CUMULATIVE %

0% 5% 10% 15% 20% 25% 30% 35% 40% 0 to 50 51 to 100 101 to 150 151 to 200 201 to 250 251 to 300 301 to 350 351 to 400 401 to 450 451 to 500 501 to 549 550 to 599 600 to 649 650 to 699 700 to 749 750 to 799 800 to 849 850 to 899 900 to 949 950 to 999

Percent of Population

Overdose Model Score

Model Scores for Decedents and Non-Decedents

Non-Decedents Decedents

Model Score Distribution for Decedents and Non-Decedents

Decedents Mean: 505 Median: 507 95th %tile: 835 99th %tile: 938 Non-decedents Mean: 209 Median: 209 95th %tile: 569 99th %tile: 730

Decedents have higher risk scores

Model KS Sample 1 KS Sample 2 KS Sample 3 KS Sample 4

• Avg. KS

Overdose Risk Score 75%/25% train/test 47.32 48.80 46.23 47.87 47.56 Overdose Risk Score 100%/100% train/test 48.34 49.62 47.57 48.27 48.45

• KS was evaluated on all four test samples (25% holdout group)
• During final testing, models were fit/tested against the full sample
• Many commercial models have KS scores in the 35 to 50+ range

KS Score was 49.62

Cumulative Percent of Population KS Plot from Sample 2

Machine Learning Approach

Model Evaluation with the KS Statistic

Machine Learning Approach

Some Key Model Variables

The chance of overdose is 48 times higher when visiting 7+ pharmacies compared to no pharmacies

Group visiting 7+ pharmacies has 9% decedents (compared to average of 1%)

Advanced Data Analysis

The chance of overdose is 44 times higher when visiting 11+ providers compared to no providers (in the last 2 years)

Group visiting 11+ providers has 9% decedents (compared to average of 1%)

0% 1% 2% 3% 4% 5% 6% 7% 0 to 483 mg 484 to 966 mg 967 to 1449 mg 1450 to 1932 mg 1933 mg +

Percent Decedents

mg of Sedative Prescribed in the Past Year

Some Key Model Variables

Group having 1933+ mg of sedatives in the last year has 6% decedents (compared to average of 1%)

Machine Learning Approach

Final Score Construction

• Core model output is converted to a scaled score from 0 to 1000
• Low values = low risk
• High values = high risk
• Risk of overdose death doubles approximately every 100 points

0% 5% 10% 15% 20% 25% 0-99 100-199 200-299 300-399 400-499 500-599 600-699 700-799 800-899 900-999

Percent Decedents

Overdose Risk Score

Predictive Power of Overdose Risk Score

Overdose risk doubles every 100 points (approx) Individuals in the last bin are 329 times more likely to die of overdose than individuals who score 0-199

Comparison of Odds Ratio for Drug Overdose Death

329 times more likely to die due to

drug overdose than people with score < 200

OD Risk PERCENT OF DECEDENTS

(1:100 Ratio Overall)

ODDS RATIO 0-199 0.1% 1 200-299 0.7% 10 300-399 0.9% 12 400-499 1.9% 25 500-599 3.3% 44 600-699 6.3% 85 700-799 10.0% 141 800-999 13.2% 194 900-999 20.5% 329

50 100 150 200 250 300 350

Odds Ratio

Score

Overdose Risk Model Score strongly associated with risk

• f overdose death

Machine Learning Approach

Validation of Overdose Risk Model Scores

Testing on years not used in the modeling process shows stability of the model

NarxCare Example

Future Directions

• 1. Include More Data Types
• Demographics
• Non-fatal overdose
• Criminal justice (CJ) data where appropriate
• Claims data
• CCD
• Mandatory reporting (NAS)
• 2. Include More Data
• Invite more states

Future Directions

• 3. Study more outcomes
• Non-fatal overdose
• Misuse and substance use disorders
• Arrest, Injury, etc.

Goal is to eventually detect risk at the earliest possible point to provide an optimal intervention point.

Questions?

Chad Garner, MS

• Email: chad.garner@pharmacy.ohio.gov

Jim Huizenga, MD

• Email: jhuizenga@apprisshealth.com

David Speights, PhD

• Email: dspeights@appriss.com

Contact In Information

General Webinar Information: webinars@vendomegrp.com