TRIPODS Workshop: Models & Machine Learning for Causal Inference - - PowerPoint PPT Presentation

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy

TRIPODS Workshop: Models & Machine Learning for Causal Inference & Decision Making

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD

Department of Biostatistics Brown University School of Public Health

Jan 14-18, 2019 Providence, RI, USA

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy

Outline

Microsimulation Models Definition MSM for lung cancer Description The MILC model Calibration Comparative Analysis Results Predictive accuracy Simulation Study Results

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Definition

MSMs in medical decision making

Micro-simulation models (MSMs): Complex predictive models aimed at simulating individual disease trajectories using Markov Chain Monte Carlo methods. ◮ Incorporate combined information from several sources ◮ Predict outcomes of interest under different medical interventions ◮ Compare findings to decide on the best practice (cost-effectiveness analysis) ◮ Inform health policies

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Description The MILC model

The MIcrosimulation Lung Cancer (MILC) Model :

an MSM describing the natural history of lung cancer

Chrysanthopoulou SA (2017). MILC: A Microsimulation Model of the Natural History of Lung Cancer. International Journal of Microsimulation. 10(3):5-26

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Description The MILC model

Micro-Simulation Model for lung cancer

◮ Streamlined MSM model ◮ Continuous time ◮ Dynamic ◮ Natural history of lung cancer (no screening component) ◮ Covariates

➜ Age ➜ Gender ➜ Smoking:

⊲ Status (current, former, never) ⊲ Start and quit smoking age ⊲ Intensity (cigarettes/day)

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Description The MILC model

Markov State Diagram

Figure: Markov State Diagram of the continuous time MSM for lung cancer

S0: Disease-free state S1: Local state

(onset of the 1st malignant cell)

S2: Regional state

(lumph nodes)

S3: Distant state

(distant metastases)

S4: Death state hij: the hazard of moving

from state i to j

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Description The MILC model

Components

• Onset of the 1st malignant cell

→ TSCE carcinogenesis model (Moolgavkar 1990)

• Tumor growth

→ Gompertz function (Laird 1964)

• Disease progression

→ log-Normal distribution (Spratt 1964)

• Survival

→ competing risks (CIF estimates) → NHIS and SEER data

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Description The MILC model

Structure of the MILC model

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Description The MILC model

• R package ”MILC”: http://CRAN.R-project.org/package=MILC

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Comparative Analysis Results

Calibration methods in MSMs: a comparative analysis

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Comparative Analysis Results

Calibration vs Estimation in statistical theory

Statistical modeling: ”calibration” ≡ ”estimation” ≡ ”model fitting” Here: ”calibration” ≡ ”model tuning” Calibration pertains to the specification of those sets of values for the model parameters that can result in predictions close to the

• bserved, pre-specified target quantities.

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Comparative Analysis Results

Why calibration?

◮ MSM’s complexity

→ no closed form of the model’s outputs (e.g., likelihoods, hazard rates, transition probabilities, etc.)

◮ Latent variables + high dimensionality

→ identifiability problems

◮ Multiple sets of parameter values

→ underlying correlation structures → parameter uncertainty

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Comparative Analysis Results

Calibration methods for MSMs (overview)

◮ Directed → optimum set (Nelder-Mead, simulation annealing, etc)

◮ Undirected → ”acceptable” sets (exhaustive or sampling design based grid search) ◮ Bayesian → joint posterior distribution

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Comparative Analysis Results

Comparative analysis

Objective: Comparative analysis of two calibration methods ◮ Bayesian: MCMC Calibration (Rutter et at, 2009) ◮ Empirical: Grid Search using Latin Hypercube Sampling (GSLHS) Implementation: MILC model Comparative analysis: Quantitative and qualitative

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Comparative Analysis Results

Methods

Method

Bayesian Empirical

(Rutter et al. 2009) (Grid Search using LHS design) Goal

draw values from the draw sets of ”acceptable” joint posterior h(θ| Y)

values for θ Specifics ֒ → Gibbs sampler ֒ → Latin Hypercube Sampling ֒ → appr. MH algorithm ֒ → LR test Results Distribution of (multiple draws for) θ

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Comparative Analysis Results

Comparative Analysis Study Design I

• 1. Parameters to calibrate:

◮ tumor growth (θ1=m) ◮ disease progression (θ2=mdiagn, θ3=mreg, θ4=mdist)

• 2. Input data:

◮ sample (smpl.CN, N=5000) from 1980 US population of males, current smokers

• 3. Calibration targets, Yclbr =[Y<60 Y60−80 Y>80]T:

◮ lung cancer incidence rates by age group (2006 SEER data)

• 4. Size (total number of micro-simulations):

→ MBayes = 2 · 1010 (!!!) (sequential) → MEmp = 5 · 109 (!!!) (parallel)

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Comparative Analysis Results

Comparative Analysis Study Design II

Terms of comparison:

• Overlap

◮ calibrated values for θ ◮ predictions

• Model validation (predictions

Y vs Yclbr): ◮ Internal

• Y =

M50(Θ, smpl.C5000)

• ◮ External
• Y =

M50(Θ, smpl.V5000)

• Efficiency

◮ computational time

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Comparative Analysis Results

Comparative Analysis Study Design III

Tools:

• Graphical ways (density, contour, and box plots)
• Discrepancy measures

◮ Mean Absolute Deviations (MAD) MAD = 1 V

V

• v=1

|˜ yvj − yj| yj (1) ◮ Mean Squared Deviations (MSD) MSD = 1 V

V

• v=1

˜ yvj − yj yj 2 (2) ◮ Euclidean & Mahalanobis distances DM =

• (

Y − Yclbr)T · S−1 · ( Y − Yclbr) (3)

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Comparative Analysis Results

Density plots: marginal distributions of the calibrated MSM parameters.

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Comparative Analysis Results

Contour plots: bivariate distributions of the calibrated parameters

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Comparative Analysis Results Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Comparative Analysis Results

Discrepancy measures: predictions vs calibration targets

Internal Validation Bayesian Calibration Empirical Calibration <60 yrs 60-80 yrs >80 yrs Overall <60 yrs 60-80 yrs >80 yrs Overall Yclbr MAD 0.040 0.139 0.022 0.060 0.215 0.046 0.026 0.096 MSD 0.051 0.031 0.008 0.030 0.072 0.005 0.004 0.027 External Validation Bayesian Calibration Empirical Calibration <60 yrs 60-80 yrs >80 yrs Overall <60 yrs 60-80 yrs >80 yrs Overall Yclbr MAD 0.056 0.158 0.022 0.078 0.195 0.066 0.004 0.088 MSD 0.052 0.036 0.008 0.032 0.063 0.007 0.003 0.024

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Comparative Analysis Results

High Performance Computing (HPC) using R I

Total micro-simulations: MBayes = 2 · 1010, MEmp = 5 · 109

MSM Calibration ⇒ Embarrassingly Parallel Computations

Solutions: ◮ Code profiling → “Rprof ” library ◮ Parallel computing → “snow” & “Rmpi” libraries ◮ Code for parallel processing: for each θ = [θ1, θ2, θ3, θ4]T update,

→ Bayesian method: m=4×50,000 (sequential) → Empirical method: m=50,000 (simultaneous checks)

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Comparative Analysis Results

High Performance Computing (HPC) using R II

Algorithm efficiency improvement (m=50,000) Profiling Parallel Nodes Type Time ( % impr.) computing (secs) × × 1

• 5532.77
• √

× 1

• 2121.05

161.0 √ √ 10 SOCK 112.08 1792.0 √ √ 32 SOCK 15.55 620.0 √ √ 64 SOCK 15.74

• 1.2

√ √ 64 MPI 4.38 259.0 √ √ 128 MPI 2.79 57.0 √ √ 256 MPI 1.98 41.0 Overall Improvement: 5532.77/1.98 = 2794.328 faster (!!!)

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Comparative Analysis Results

Discussion

• Overlapping distributions of the calibrated parameters (ΘBAYES vs ΘEMP):

◮ univariate (density plots, MAD, MSD) ◮ multivariate (contour plots, Mahalanobis distances)

• Predictions (

Y BAYES vs Y EMP): ◮ overall comparable ◮ Y EMP less dispersed ◮ Y BAYES better for rare events

• Computational burden:

◮ Emprical(parallel) more efficient than Bayesian(sequential) methods ◮ undirected(parallel) more efficient than directed(sequential) methods ◮ Calibration in R ⇒ HPC techniques (!!!)

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Simulation Study Results

Assessing the predictive accuracy of MSMs

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Simulation Study Results

Predictive accuracy

• Performance of predictive models

◮ Explained variation (R2 statistics) ◮ Calibration (GoF statistics) ◮ Discrimination (C-statistics)

• Implementation: type of predictions (e.g., continuous, ordinal,

nominal, survival data)

• Predictive accuracy: How close individual predictions are to
• bserved data
• Predictive accuracy of MSMs: Important though not explored yet

◮ incorporation of individual level characteristics ◮ between individuals variability

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Simulation Study Results

Assessing predictive accuracy of MSMs

• Output of interest: survival time (event or censoring)

◮ common survival models (Cox-PH, AFT, etc.): predicted risk VS observed survival time ◮ MSMs (special type of survival predictive models): predicted survival time VS observed survival time

• Proposed methods:

◮ Concordance statistics: correct classification, given a set of covariates, based on the predictions (enough?) → Harrell’s index (Harrell et al 1996) → Uno’s index (Uno et al 2011) ◮ Hypothesis testing: Predicted vs observed survival function → log-rank → Renyi type statistics → Cramer von Mises statistics

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Simulation Study Results

Simulation Study

◮ Objective: Compare the proposed methods for the assessment of the predictive accuracy of MSMs ◮ Input sample (N=5000) of males, current smokers (US 1980 population) ◮ Simulate “observed” lung cancer incidence (Gompertz distribution) ◮ Predict lung cancer incidence (calibrated MILC model)

• Y BAYES =

M(ΘBAYES, smpl.15000), Y EMP = M(ΘEMP, smpl.15000) for V={200, 400, 600, 800, 1000} vectors for θ ◮ Apply C-statistics and Hypothesis testing methods for the predictive accuracy of the two MSMs (predicted vs ”observed” survival data) ◮ Evaluate the performance of the proposed methods

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Simulation Study Results Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Simulation Study Results

Calibrated MSM Method Bayesian Empirically C-statistic Harrell 0.7946±0.0079 0.7927±0.0081 Uno 0.7295±0.0306 0.7300±0.0323 Hypothesis Test Log-Rank 26.00 3.50 (non-rejection Renyi 24.25 8.00 rate %) C-M (Q1) 25.25 15.75 C-M (Q2) 58.00 27.75

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Simulation Study Results

Discussion

Conclusions: ◮ Runs of the MSM for V=400 vectors for θ (randomly selected from Θ) adequate for assessing the predictive accuracy of the model ◮ C-statistics: ◮ almost identical results (estimates & conclusions) ◮ cannot capture differences in the accuracy of the individual predictions ◮ Hypothesis testing ◮ similar results (analogous estimates & same conclusions) ◮ log-rank & Renyi type tests more sensitive in detecting differences between observed and predicted individual survival data ◮ preferable for the assessment of the predictive accuracy of an MSM

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Simulation Study Results

Acknowledgments

Collaborators ◮ Constantine Gatsonis, PhD (Brown University) ◮ Carolyn Rutter, PhD (RAND - CISNET, colorectal cancer group) ◮ Matthew Harrison, PhD (Brown University) ◮ Joseph Hogan, PhD (Brown University)

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference

Microsimulation Models MSM for lung cancer Calibration Predictive accuracy Simulation Study Results

References

◮ Gatsonis, C., et al (2011), “The National Lung Screening Trial: Overview and Study Design,” Radiology, 258, 243-253. ◮ Laird, A. K. (1964), “Dynamics of Tumor Growth” British Journal of Cancer, 18, 490-502 ◮ McMahon, P. M. (2005), Policy assessment of medical imaging utilization: methods and applications, [doctoral thesis] ◮ Meza, R., Hazelton, W. D., Colditz, G. A., and Moolgavkar, S. H. (2008), “Analysis of lung cancer incidence in the nurses’ health and the health professionals follow-up studies using a multistage carcinogenesis model”, Cancer Causes & Control, 19, 317-328. ◮ Moolgavkar, S. H. and Luebeck, G. (1990), “Two-Event Model for Carcinogenesis: Biological, Mathematical, and Statistical Considerations”, Risk Analysis, 10, 323-341. ◮ Rutter CM, Miglioretti DL, Savarino JE., (2009), “Bayesian calibration of microsimulation models”, JASA;104(488):1338-50 ◮ Rutter CM, Zaslavsky AM, Feuer EJ, (2010), “Dynamic Microsimulation models for health outcomes: a review”, MDM ◮ Spratt, J. S. and Spratt, T. L. (1964), “Rates of Growth of Pulmonary Metastases and Host Survival” Annals of Surgery, 159, 161-171. ◮ Steyerberg, E. W., et al (2010), “Assessing the Performance of Prediction Models A Framework for Traditional and Novel Measures”, Epidemiology, 21, 128-138. ◮ Stout N.K., et al (2009), “Calibration Methods used in Cancer Simulation Models and Suggested Reported Guidlines”, Pharmacoeconomics, 27, 533-545. ◮ Vanni T., et al (2011), “Calibrating models in economic evaluation: a seven-step approach”, Pharmacoeconomics, 29, 35-49

Microsimulation Models in Medical Decision Making : Calibration and Predictive Accuracy text Stavroula Chrysanthopoulou, PhD TRIPODS Workshop: Models & Machine Learning for Causal Inference