Applying Machine Learning Methods to Predicting Reliance on VA - - PowerPoint PPT Presentation

▶
applying machine learning methods to predicting reliance
SMART_READER_LITE
LIVE PREVIEW

Applying Machine Learning Methods to Predicting Reliance on VA - - PowerPoint PPT Presentation

Oct 28, 2022 158 likes •590 views

Applying Machine Learning Methods to Predicting Reliance on VA Primary Care Edwin S. Wong, PhD VA Puget Sound Health Care System Department of Health Services, University of Washington 2018 AcademyHealth Annual Research Meeting June 25, 2018

slide-1
SLIDE 1

Applying Machine Learning Methods to Predicting Reliance on VA Primary Care

Edwin S. Wong, PhD VA Puget Sound Health Care System Department of Health Services, University of Washington 2018 AcademyHealth Annual Research Meeting June 25, 2018

slide-2
SLIDE 2

VETERANS HEALTH ADMINISTRATION

Acknowledgements

  • Sources of funding

– VA Career Development Award (Wong, CDA 13-024)

  • Disclosure:

– Dr. Wong reports ownership of common stock in UnitedHealth Group Inc. totaling less than $15,000 in market value

2

slide-3
SLIDE 3

VETERANS HEALTH ADMINISTRATION

Why Machine Learning?

  • Increased capabilities not possible with traditional methods

– Broader range of health care applications – Support analysis of data of greater size and complexity – Ability to develop models of greater complexity – Offer richer insights – Improvement in model performance

slide-4
SLIDE 4

VETERANS HEALTH ADMINISTRATION

Example Machine Learning Applications in Health Services Research

  • Predicting health and health care outcomes
  • Detecting outliers

– High cost patients

  • Classification
  • Subgroup analysis

– Phenotyping, risk stratification

  • Measuring heterogenous treatment effects
slide-5
SLIDE 5

VETERANS HEALTH ADMINISTRATION

Example Machine Learning Applications in Health Services Research

  • Predicting health and health care outcomes
  • Detecting outliers

– High cost patients

  • Classification
  • Subgroup analysis

– Phenotyping, risk stratification

  • Measuring heterogenous treatment effects
slide-6
SLIDE 6

VETERANS HEALTH ADMINISTRATION

Application: Dual Use of VA and Non-VA Health Care

  • Veterans Affairs Health Care System (VA)

– Large, nationally integrated health system – 8.4 million Veteran enrollees in FY2016

  • VA enrollees are not precluded from obtaining care through

non-VA sources, independent of VA

– ~80% have at least one other non-VA health insurance source – Nearly all age 65+ dually enrolled in Medicare

slide-7
SLIDE 7

VETERANS HEALTH ADMINISTRATION

Research Objective

  • To examine how to best predict which VA enrollees will be

mostly reliant on VA primary care next year using predictor variables in the current year

  • Policy Relevance:

– VA reliance is an input to projection models used to inform VA health care budget requests submitted to Congress – Better predictions of reliance may improve accuracy of these requests

slide-8
SLIDE 8

VETERANS HEALTH ADMINISTRATION

Data Sources

  • VA Corporate Data Warehouse

– Comprehensive administrative data on all users of VA health care

  • Medicare Claims

– Utilization of outpatient services through fee-for-service Medicare

  • 2012 VA Survey of Healthcare Experiences of Patients

– Random sample of Veterans receiving care at VA outpatient facilities

  • Area Health Resource File

– Characteristics in Veterans’ residence county

slide-9
SLIDE 9

VETERANS HEALTH ADMINISTRATION

Population Studied

  • Sample of 83,825 VA patients responding to the 2012 VA SHEP

– Dually enrolled in fee-for-service Medicare in FY2012 and FY2013 – Alive at end of FY2013 – Weighted to population of 4.6 million VA patients

slide-10
SLIDE 10

VETERANS HEALTH ADMINISTRATION

Definition of VA Reliance

  • Counts of face-to-face office visits in primary care
  • VA Reliance = Proportion of all visits from VA

– # visits in VA ÷ (# visits in VA + # visits via Medicare)

  • Dichotomous measure denoting whether Veterans were

mostly reliant on VA

– VA reliance ≥ 0.5

1Burgess JF, et al. (2011). Health Econ 20(2).

slide-11
SLIDE 11

VETERANS HEALTH ADMINISTRATION

Predictor Variables (Features)

Group Example Variables

Demographics Age, gender, marital status, race/ethnicity Access to Care Distance to nearest VA, copayment exemption Comorbidities Heart failure, hypertension, diabetes, liver disease Patient-Reported Experiences Provider rating, ability to receive immediate care, parking availability, office cleanliness Local Area Factors Poverty rate, unemployment rate, hospital beds per 1,000

  • 59 features in 5 categories
slide-12
SLIDE 12

VETERANS HEALTH ADMINISTRATION

Machine Learning Framework for Classification

  • Analytic Objective: Learn a target classifier function C that

best assigns input variables X to an output variable y:

– y <- C(X) – Binary classification: y = 0 (not VA reliant), 1 (VA reliant) – X = Matrix of predictor variables, or features

  • Policy Goal: Make accurate predictions of Veterans’ future

reliance classification given observed features in the present

slide-13
SLIDE 13

VETERANS HEALTH ADMINISTRATION

Machine Learning Objective

  • Goal: Assessing properties of model “out-of-sample”

– How would model perform in practice? – Causality deemphasized – Focus on performance and fit

  • Use training sample to estimate model
  • Assess model performance on separate validation sample
  • Consider multiple algorithms or models

– “Best” model will depend on research question and analytical data – No single model is always superior

slide-14
SLIDE 14

VETERANS HEALTH ADMINISTRATION

Road Map for Classifying VA Reliance

  • Model set-up

– Pre-processing of data (cleaning and transforming) – Identify performance metric (loss function) – Resampling methods (validation set generation methods) – Identify candidate algorithms

slide-15
SLIDE 15

VETERANS HEALTH ADMINISTRATION

Road Map for Classifying VA Reliance

  • Build Models

– Estimate model parameters – Determine best value of tuning parameters – Assessing model fit – Calculating the performance of the final model

  • Identify “best” of candidate models
slide-16
SLIDE 16

VETERANS HEALTH ADMINISTRATION

Visual Roadmap

16 Predicted values Raw Data Cleaned Data Test Train/ Validation Train Validation Model(s) Train Validation Model Parameters Performance Metrics (loss function) Test Best Model(s) Train/Val

Gathering data and preprocessing Train, Test, Validation split Build and tweak candidate models Compare best models against test data

Best Model Out of sample data

slide-17
SLIDE 17

VETERANS HEALTH ADMINISTRATION

Preprocessing

  • Data may have irregularities that may influence model

stability and performance

  • Differing assumptions and requirements of models
  • Common preprocessing tasks:

– Correcting inconsistent data – Addressing missing data – Centering and scaling – Transformations of individual predictors or groups of predictors – Discretizing continuous predictors

slide-18
SLIDE 18

VETERANS HEALTH ADMINISTRATION

Variable Selection

  • More parsimonious model may be preferred

– More complex models may achieve a high performance at the cost of

  • verfit

– Computational limitations – Easier to interpret

  • Assess gain in performance from complex model against a

simpler, lower variance model

18

slide-19
SLIDE 19

VETERANS HEALTH ADMINISTRATION

Performance Metrics for Classification Models

Metric Description Accuracy Proportion correctly classified by model Kappa Statistic Inter-rater agreement; performance adjusting for agreement due to random chance Sensitivity True positive (TP) rate: TP / [TP + FN] Specificity True negative (TN) rate: TN / [TN + FP] Area Under ROC Curve (AUROC) Average value of sensitivity for all the possible values of specificity

  • Several common metrics to assess performance:

FP=false positive, FN=false negative

slide-20
SLIDE 20

VETERANS HEALTH ADMINISTRATION

Performance Metrics for Classification Models

Metric Description Accuracy Proportion correctly classified by model Kappa Statistic Inter-rater agreement; performance adjusting for agreement due to random chance Sensitivity True positive (TP) rate: TP / [TP + FN] Specificity True negative (TN) rate: TN / [TN + FP] Area Under ROC Curve (AUROC) Average value of sensitivity for all the possible values of specificity

  • Several common metrics to assess performance:

FP=false positive, FN=false negative

slide-21
SLIDE 21

VETERANS HEALTH ADMINISTRATION

Resampling Methods

  • Facilitate estimation of model performance on data “unseen”

in training process

  • Resampling allows for assessing variability and stability of

model

  • Helps protect against model overfitting

– Overfit models will “memorize” data – Good performance using training data does not necessarily generalize “out-of-sample”

slide-22
SLIDE 22

VETERANS HEALTH ADMINISTRATION

Resampling Methods

  • Repeat for a given number of resampling iterations

– Construct validation sample by holding out observations – Fit model on remaining observations (i.e. training sample) – Predict on validation sample – Calculate performance using specified metric

  • Assess model performance across all iterations
slide-23
SLIDE 23

VETERANS HEALTH ADMINISTRATION

Resampling Methods

Method Brief Description Simple Cross Validation Partition data into training and test sample K-fold Cross Validation Split data into K equally sized blocks Repeated K-fold Cross Validation Create multiple versions of K-folds Leave Group Out Define random proportion of data to train model and repeat multiple times Bootstrapping Construct random sample with replacement

  • f same size as original data set
  • Several commonly applied methods to define validation

samples

slide-24
SLIDE 24

VETERANS HEALTH ADMINISTRATION

Resampling Methods

Method Brief Description Simple Cross Validation Partition data into training and test sample K-fold Cross Validation Split data into K equally sized blocks Repeated K-fold Cross Validation Create multiple versions of K-folds Leave Group Out Define random proportion of data to train model and repeat multiple times Bootstrapping Construct random sample with replacement

  • f same size as original data set
  • Several commonly applied methods to define validation

samples

slide-25
SLIDE 25

VETERANS HEALTH ADMINISTRATION

Unified Prediction Framework in r

  • caret - classification and regression training
  • Streamlined workflow
  • R package supporting modeling and prediction across 200+ models
  • Model building using train function and specify:

– Resampling – Pre-processing functions – Performance metric – Defining grid of tuning parameters

  • Assess model performance out-of-sample using predict function
  • Collecting and visualizing resampling results
  • Supports parallel processing
slide-26
SLIDE 26

VETERANS HEALTH ADMINISTRATION

Selecting Candidate Algorithms

  • >200 machine learning algorithms and models supported by

Caret

  • Documentation on Github:

– https://topepo.github.io/caret/available-models.html

  • Considerations:

– Computational Limitations – Training time – Predictive performance – Model assumptions – Number of features – Scalability

slide-27
SLIDE 27

VETERANS HEALTH ADMINISTRATION

Candidate Models

Model Brief Description

Logistic Regression Parametric regression for binary outcomes Lasso Regularization method, dropping some coefficients Single Tree Repeated partitioning of predictor space into subspaces producing biggest performance improvement

slide-28
SLIDE 28

VETERANS HEALTH ADMINISTRATION

Candidate Models

Model Brief Description

Random Forest

Ensemble method: construct multiple regression trees using bootstrapped training sets, splitting over subset of predictors and averaging predictions across trees

Gradient Boosting Machine

Sequentially construct regression tree using information from previously grown trees, prediction a weighted average of predictions across trees

Artificial Neural Network

Directed graph comprising of a network of nodes connected using weights that are updated using predefined rules

slide-29
SLIDE 29

VETERANS HEALTH ADMINISTRATION

Steps in caret

  • Set up parameters for training models

cv10Ctrl <- trainControl(method="cv", number=10, classProbs=TRUE, summaryFunction = twoClassSummary)

  • Specify custom tuning parameters

parmGrid <- expand.grid(n.trees=(2:10)*1000, interaction.depth=c(1,3,5), shrinkage=0.001, n.minobsinnode=10)

slide-30
SLIDE 30

VETERANS HEALTH ADMINISTRATION

Steps in caret (cont’d)

  • Train Models

gbm_tune <- train(y ~ age + sex + educ_cat + …, data=trainData, method="gbm", distribution="bernoulli", metric = "ROC", preProcs = c(“center”), tuneGrid=paramGrid, trControl=cv10_ROC))

  • Predict out-of-sample

phat <- predict(gbm_tune, newdata=newData[,covars], type="prob")

slide-31
SLIDE 31

VETERANS HEALTH ADMINISTRATION

Predictive Performance of Candidate Models

Model

AUROC Sensitivity Specificity Logistic Regression 0.740 0.912 0.321 Lasso 0.741 0.914 0.318 Single Tree 0.709 0.894 0.355 Random Forest 0.756 0.934 0.298 Gradient Boosting Machine 0.759 0.918 0.339 Neural Network 0.718 0.872 0.362

slide-32
SLIDE 32

VETERANS HEALTH ADMINISTRATION

Predictive Performance of Candidate Models

Model

AUROC Sensitivity Specificity Logistic Regression 0.740 0.912 0.321 Lasso 0.741 0.914 0.318 Single Tree 0.709 0.894 0.355 Random Forest 0.756 0.934 0.298 Gradient Boosting Machine 0.759 0.918 0.339 Neural Network 0.718 0.872 0.362

slide-33
SLIDE 33

VETERANS HEALTH ADMINISTRATION

Predictive Performance of Candidate Models

Model

AUROC Sensitivity Specificity Logistic Regression 0.740 0.912 0.321 Lasso 0.741 0.914 0.318 Single Tree 0.709 0.894 0.355 Random Forest 0.756 0.934 0.298 Gradient Boosting Machine 0.759 0.918 0.339 Neural Network 0.718 0.872 0.362

slide-34
SLIDE 34

VETERANS HEALTH ADMINISTRATION

Predictive Performance of Candidate Models

34

slide-35
SLIDE 35

VETERANS HEALTH ADMINISTRATION

Visualizing Resample Results

35

slide-36
SLIDE 36

VETERANS HEALTH ADMINISTRATION

Visualizing Resample Results

36

slide-37
SLIDE 37

VETERANS HEALTH ADMINISTRATION

Model Performance vs. Tuning Parameters

slide-38
SLIDE 38

VETERANS HEALTH ADMINISTRATION

Variable Importance

slide-39
SLIDE 39

VETERANS HEALTH ADMINISTRATION

Summary and Concluding Remarks

  • Example of machine learning methods for classification within

a unified framework applied to a policy relevant VA metric

  • Performance across 6 candidate machine learning models

produced differing results

  • Best performing model (gradient boosting machine) exhibited

performance considered “fair” based on prior thresholds

  • Economic theory suggests other features (e.g. attributes of

non-VA options) may further improve predictive performance

slide-40
SLIDE 40

VETERANS HEALTH ADMINISTRATION

References

  • Machine Learning

– Hastie T, Tibshirani R, Friedman J (2008). The Elements of Statistical Learning: Data Mining, Inference and Prediction. Springer – Kuhn M, Johnson K (2013). Applied Predictive Modeling. Springer – Wu X, Kumar V, Quinlan JR, et al. (2008). “Top 10 Algorithms in Data Mining.” Knowledge Information Systems; 14: 1-37.

slide-41
SLIDE 41

VETERANS HEALTH ADMINISTRATION

References

  • Predictive Modeling Using Caret

– Kuhn, M (2008). Building Predictive Models in R Using the caret Package. Journal of Statistical Software; 28(5). – Kuhn, M (2013). Predictive Modeling with R and the caret

  • Package. https://www.r-project.org/conferences/useR-

2013/Tutorials/kuhn/user_caret_2up.pdf – Kuhn, M (2018). The caret Package. http://topepo.github.io/caret/index.html – Caret – Available Models. https://topepo.github.io/caret/available-models.html

slide-42
SLIDE 42

VETERANS HEALTH ADMINISTRATION

References

  • Health Service Applications

– Rose (2014). “A Machine Learning Framework for Plan Payment Risk Adjustment.” Health Services Research; 51(6 Part 1): 2358-2374 – Park and Basu (2018). “Alternative Evaluation Metrics for Risk Adjustment Methods.” Health Economics; 27(6); 984- 1010 – Weng, Reps, Kai, et al. (2017). “Can machine-learning improve cardiovascular risk prediction using routine clinical data?” PLOS One; doi: 10.1371/journal.pone.0174944

slide-43
SLIDE 43

VETERANS HEALTH ADMINISTRATION

Drawbacks of Machine Learning Approaches

  • Often impossible to interpret
  • No single algorithm performs the best across different

applications

  • Primarily designed for cross sectional designs (though this

is changing)

43