Predictive Modeling for Suicide Risk Robert Bossarte, PhD VISN 2 - - PowerPoint PPT Presentation

Predictive Modeling for Suicide Risk

Robert Bossarte, PhD VISN 2 Center of Excellence for Suicide Prevention West Virginia University

March 10, 2015

VETERANS HEALTH ADMINISTRATION

Why Should We Invest in Modeling Risk?

• Evidence that VA’s approach to suicide prevention is

effective comes from findings that suicide rates in VA male patients have, in general, decreased relative to rates in the rest of American men.

• Nevertheless, the finding that rates remain high

represents a call to supplement VA’s current strategy with new approaches to identifying patients at risk, and new methods for enhancing their care.

VETERANS HEALTH ADMINISTRATION

Excess Risk for Suicide among Males Who Used VHA Services, 2001 – 2011

3

VETERANS HEALTH ADMINISTRATION

Basic Strategy – A Foundation for Risk Stratified Care

• Going beyond intercepting people on the trajectory towards

suicide

• Identifying people whose care should be enhanced

– One target group may be those at highest predicted risk – Another includes those at more moderate risk, who account for a substantial proportion of the total burden of suicide

VETERANS HEALTH ADMINISTRATION

How?

• Generate a data base for patient-months using data for FY09-11

– Include all VHA users who died from suicide, by month, and 1% of VHA users who survived the month – Create split samples for model development and validation – Consider demographics and variables known to be risk factors for suicide in VA and/or

• ther populations

– Include specific events as lag variables – Include interactions known to be important

• Develop a logistic regression model using the development sample

– Sort and rank patients by tiers of model-predicted risk

• Evaluate model using a separate validation sample
• Test the extent to which it predicts suicide during a single subsequent years for all

VHA patients who were alive at the start of the year

• Characterize patients in high risk strata, to inform intervention development

VETERANS HEALTH ADMINISTRATION

VETERANS HEALTH ADMINISTRATION

VETERANS HEALTH ADMINISTRATION

Prediction Sample: Percentage of Suicide Deaths

Tier of Predicted Months Probability, % Patients 1 3 6 9 12 0.01 596 1.4 0.9 0.5 0.4 0.3 0.10 5,969 4.3 2.9 2.0 1.7 1.6 1.00 59,696 10.4 9.4 9.0 8.1 8.2 5.00 298,493 23.2 23.9 25.0 23.6 23.7 10.00 596,966 38.4 38.2 37.1 35.8 35.5 50.00 2,984,831 83.9 83.5 81.5 79.9 80.7 100.00 5,969,662 100.0 100.0 100.0 100.0 100.0

VETERANS HEALTH ADMINISTRATION

Prediction: Trajectories

VETERANS HEALTH ADMINISTRATION

Who are the high risk patients?

In general, most patients in the high risk strata are Veterans with known mental health conditions at ongoing use of mental health services

VETERANS HEALTH ADMINISTRATION

Why Should We Worry About New Models?

• Original project sought to identify VHA patients with the

greatest suicide risk concentration.

• The long term goals of this project was to provide a foundation for the

continued development and evaluation of improved models for predicting suicide risk.

• Evaluation of the preliminary model suggested instability

associated with overfitting and variable selection.

• Long term sustainability and operationalization of predictive

models would be enhanced by development of processes requiring fewer data points/system resources.

11

VETERANS HEALTH ADMINISTRATION

Observations from Initial Exercise and Next Steps

• Administrative data can be used to predict suicide risk within the next 30

days.

• Increased risk concentration remains over longer periods for high risk

groups.

• There are associations between calculated risk strata and high risk flags

– The proportion of patients with flags increases with calculated risk – Only a minority of patients calculated to be at high risk are flagged

• The large number of variables increases possibility of overfitting.
• Extensive data and analytic requirements restrict opportunities for rapid

updates to the analytic model or “real time” calculation of risk scores.

12

VETERANS HEALTH ADMINISTRATION

• Begin with the same data.
• Utilize a 3-Fold cross-validation strategy to identify the number of variables needed to

achieve optimal risk concentration

– logistic regression (weighted) w/ forward selection – Identify optimal number of variables (approximate)

• Apply machine learning algorhithms.

– Glmnet R package: fits a generalized linear model via penalized maximum likelihood.

• Mixing parameter used to estimate models with penalties varying between lasso <-> elastic
• Variable ‘stop’ parameter used to set approximate max variable number determined from CV logistic
• Application of optimal Glmnet model to original validation and prediction cohorts to

assess model stability/ “out of sample performance”.

Applying Methods of Machine Learning to VA’s Original Model

VETERANS HEALTH ADMINISTRATION

Comparison of Model Fit (AUC)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Development Validation FY11 Prediction POC RM

14

VETERANS HEALTH ADMINISTRATION

Comparison of Risk Concentration (Top 5%)

5 10 15 20 25 30 35 40 Development Validation FY11 Prediction POC RM

15

VETERANS HEALTH ADMINISTRATION

Contact Information

Robert.Bossarte@va.gov rbossarte@hsc.wvu.edu

16