SLIDE 1
AISC, 2012
1
Collaborator
Patrick Heagerty University of Washington, Seattle
Biostat, Duke University
- P. Saha Chaudhuri
Dynamic Thresholds and a Summary ROC Curve: Assessing the Prognostic - - PowerPoint PPT Presentation
Dynamic Thresholds and a Summary ROC Curve: Assessing the Prognostic - - PowerPoint PPT Presentation
Dynamic Thresholds and a Summary ROC Curve: Assessing the Prognostic Accuracy of Longitudinal Markers Paramita Saha Chaudhuri Department of Biostatistics and Bioinformatics Duke University School of Medicine October 6, 2012 1 AISC, 2012
SLIDE 2
SLIDE 3
AISC, 2012
2
Overview
- Background
⊲ Example - serial MR-proADM, respiratory tract infection ⊲ Time-dependent ROC
- Summary ROC
⊲ Estimation ⊲ Simulation ⊲ Example
- Comments
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 4
Background
SLIDE 5
AISC, 2012
3
Example: Adverse Outcome and MR-proADM among lower RTI
Prediction of adverse events
- N = 1359 patients with severe RTI from 6 tertiary care hospitals in
Switzerland
- Endpoint: composite adverse outcomes (death, ICU admission, other
complications, or recurrent infection Rx antibiotic)
- Predictive measurements:
⊲ Cardiovascular biomarker midregion proadrenomedullin (MR-proADM) ⊲ Single MR-proADM measurement on admission ⊲ Serial MR-proADM measurements
- Goal: validate (added) utility of longitudinal measurements
- Hartmann et al. (2012)
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 6
AISC, 2012
4
Adverse Outcome and MR-proADM - contd.
❚ ✁✂ ✄ ☎ ✂ ✆ ✂ ✝ ☎ ✂ ✝ ✞ ❈ ✟✠ r ✂ ✡ r ✂ ☛☛ ✟✝ ☞ ❙ ✂ r ✌ ✍ ✁✂ ✌ ☛ ♠ r ✂ ✁ ✂ ✝ ✞ ✟♦ ✞ t ✂ ❝ ✌ r ☎ ✟ ✎ ✌ ☛❝ ♠ ✍ ✌ r ❜- ✟✁
- ✁
- ✟✝
- ✝
- ✞
- r ✌
- ✝ ♦
- ✟✝
- ✺
- ❵ ✸
- ✻
- ✁
- ø
- û
- û
- ♣
- ✁
- ♣
- ✁
- ♠÷ ✄
- ✁
- ÷
- ❏
- ❏ ❑
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 7
AISC, 2012
5
Adverse Outcome and MR-proADM - contd.
▼ ✁✂ ✄☎ ✆ ✝▼ ✇ ✞ ✟ t ✠✡ ✟ t ✄☎ s☛✡ ✟ t ✐ s☞ ✐ ✌✐ ☞ ✍ ✞ ✎ ❜ ✐ ☎ ✏ ✞ ✄ ✑ ✡ ✄ ✞t ☞ ✞ ❞ ✵ ❢ ☎ ✄ ✂ ✄ ✡ ☞ ✐ ♣ t ✐ s ☛ ☞ ✡ ✞t ✠ ✇ ✐ t ✠✐ s ✸ ✵ ☞ ✞ ❞✟ ✭ ✒ ✷ ✼✓✔ ✕✖ ✗ ✘ ✵✔ ✵✵✵✙ ✖ ❈ ✐ s☞ ✡ ✚ ❂ ✵✔ ✼✼ ✛ ✖ ☎ t ✠✡ ✄ ✄ ✡ ✟✍ ✎ t ✟ s ☎ t ✟ ✠ ☎ ✇ s ✜ ✔ ■ t ✂ ✄☎ ✌✐ ☞ ✡ ☞ ✞☞ ☞ ✡ ☞ ✌ ✞ ✎ ✍ ✡ t ☎ ❜ ☎ t ✠ t ✠✡ ❈ ✢- ✣
- ✧
- ✁✂
- ✣
- ✁✂
- ▲ ✜
- ▲
- ✁✂
- ▲
- ▲
- ✠
- ✁✂
- ✎ ☎ ✇
- ✁
- ✁✂
- ▲
- ✎ ☎ ✇
- ✠
- ▲
- ✠
- ✎
- ✁✂
- ✁
- ▲
- ☞
- ✙✵
- ❄
(a) Survival plot for single measurement
▼ ✁✂ ✄☎ ✆ ✝▼ ✇ ✞ ✟ t ✠✡ ✟ t ✄☎ s☛✡ ✟ t ✐ s☞ ✐ ✌✐ ☞ ✍ ✞ ✎ ❜ ✐ ☎ ✏ ✞ ✄ ✑ ✡ ✄ ✞t ☞ ✞ ❞ ✵ ❢ ☎ ✄ ✂ ✄ ✡ ☞ ✐ ♣ t ✐ s ☛ ☞ ✡ ✞t ✠ ✇ ✐ t ✠✐ s ✸ ✵ ☞ ✞ ❞✟ ✭ ✒ ✷ ✼✓✔ ✕✖ ✗ ✘ ✵✔ ✵✵✵✙ ✖ ❈ ✐ s☞ ✡ ✚ ❂ ✵✔ ✼✼ ✛ ✖ ☎ t ✠✡ ✄ ✄ ✡ ✟✍ ✎ t ✟ s ☎ t ✟ ✠ ☎ ✇ s ✜ ✔ ■ t ✂ ✄☎ ✌✐ ☞ ✡ ☞ ✞☞ ☞ ✡ ☞ ✌ ✞ ✎ ✍ ✡ t ☎ ❜ ☎ t ✠ t ✠✡ ❈ ✢- ✣
- ✧
- ✁✂
- ✣
- ✁✂
- ▲ ✜
- ▲
- ✁✂
- ▲
- ▲
- ✠
- ✁✂
- ✎ ☎ ✇
- ✁
- ✁✂
- ▲
- ✎ ☎ ✇
- ✠
- ▲
- ✠
- ✎
- ✁✂
- ✁
- ▲
- ☞
- ✙✵
- ❄
(b) Survival plot for serial measurements
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 8
AISC, 2012
6
Background
Notation
- Ti, Mi(t) - survival time and marker for ith subject
⊲ Higher marker values ⇒ poor survival
- Ci - (independent) censoring time
- We observe
⊲ Zi = min(Ti, Ci), ⊲ δi = 1 1 (Ti ≤ Ci)
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 9
AISC, 2012
7
Background
Time-dependent Predictive Accuracy
- Predictive accuracy of a marker for time-to-event outcome
- Application of binary diagnostic accuracy concepts like sensitivity, specificity,
ROC curve, AUC, etc. to a time-to-event outcome
- Dichotomize time-to-event T
⊲ Cases: T = t or T ≤ t ⊲ Controls: T > t or T > t∗
- Goal - How well can the marker M distinguish between cases and controls
(at t)?
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 10
AISC, 2012
8
Background
Incident/dynamic definition
- Incident case - subject experienced event at t (T = t)
- Dynamic control - subject did not experience event by t (T > t)
- TP(c, t) = P(M > c | T = t)
- FP(c, t) = P(M > c | T > t)
- At each time t, divide the subjects in the riskset as either a case or a control
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 11
AISC, 2012
9
Background
Summary measures
- ROC curve - Incident/dynamic
Let cp denotes the threshold that yields a FP fraction of p FP(cp, t) = P(M > cp|T > t) = p ROCI/D
t
(p) = TP(cp, t) = P(M > cp|T = t)
- Area under the ROC curve (AUC)
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 12
Summary ROC
SLIDE 13
AISC, 2012
10
Summary ROC
Goal: Characterize Prediction Accuracy of Longitudinal Marker
- Time-dependent marker M(t) (lagged)
- Dynamic threshold c(t)
- TP[c(t), t] = P(M(t) > c(t) | T = t)
- FP[c(t), t] = P(M(t) > c(t) | T > t)
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 14
AISC, 2012
11
Summary ROC
Dynamic False Positive criterion
cp(t) : p = P[M(t) > cp(t)|T > t] Goal: Overall predictive accuracy when this dynamic FP criterion is used repeatedly over time for a time-dependent marker. Scientific question: If this dynamic criteria is used repeatedly:
- How many cases will test positive at the appropriate times?
- How many cases will be missed?
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 15
AISC, 2012
12
Summary ROC
Dynamic False Positive criterion – contd.
- Test-positive threshold cp(t) to produce fixed FP proportion p
- At t:
⊲ Proportion of the cases: P(T = t) ⊲ Proportion of cases testing positive: TP[cp(t), t] ⊲ Proportion of the subjects that will correctly test positive: TP[cp(t), t] × P(T = t)
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 16
AISC, 2012
13
Summary ROC
Total True Positive and Summary ROC
- Overall proportion: sum over (integrate) the failure times
⊲ Total True Positive (TTP) ⊲ TTP at FP = p: TTP(p) =
- t
TP[cp(t), t] P(T = t) dt =
- t
P[M(t) > cp(t)|T = t] P(T = t) dt.
- Summary (Survival) ROC: plot of (FP, TTP) for different FP thresholds
p ∈ (0, 1)
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 17
Estimation
SLIDE 18
AISC, 2012
14
Summary ROC
Estimation
TTP(p) =
- t
P[M(t) > cp(t)|T = t] P(T = t) dt.
- Involves both time and marker
⊲ Dynamic test-positive threshold: cp(t) ⊲ True Positive: TP[cp(t), t]
- Involves only time: P(T = t)
- Numerically integrate to get Summary ROC
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 19
AISC, 2012
15
Estimation
Dynamic Test-positive Threshold
cp(t) : p = P[M(t) > cp(t)|T > t]
- Estimated using empirical quantiles among controls at t
- P[M(t) > c|T > t]
=
- i
1 1{Mi(t) > c, Ti > t}
- i
1 1{Ti > t} = SM(t)|T >t(c)
- cp(t)
=
- S−1
M(t)|T >t(p) = inf{x :
SM(t)|T >t(x) ≥ p}
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 20
AISC, 2012
16
Estimation
True Positive Proportion
TP[c, t] = P(M(t) > c | T = t)
- Estimated using Cox model and riskset reweighting (Heagerty and Zheng,
2005)
- TP(c, t)
=
- i∈R(t)
1 1{Mi(t) > c} × exp [Mi(t). γ(t)] W(t) ⊲ γ(t): log hazard ratio associated with the marker: λ(t|M(t)) = λ0(t)exp [M(t) × γ(t)] ⊲ W(t) =
- i∈R(t)
exp [Mi(t). γ(t)]: normalizing constant
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 21
AISC, 2012
17
Estimation
Survival Time Distribution
- P(T = t) = S(t) − S(t+)
- Estimated non-parametrically using
⊲ Locally weighted KM estimator conditional on the marker ⊲ Nearest neighbor estimation of Akritas (1994) ⊲ Empirical marker distribution ⊲ Numerically integrated over marker
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 22
Simulation
SLIDE 23
AISC, 2012
18
Simulation
FP TTP Estimate Relative Bias (%) MCSD Coverage a 0.01 0.053 0.052
- 0.026
0.005 91.0 0.05 0.180 0.181 0.006 0.012 94.4 0.10 0.294 0.297 0.009 0.016 94.0 0.20 0.463 0.467 0.008 0.019 94.0 0.50 0.779 0.781 0.003 0.015 93.6 0.80 0.948 0.948 0.000 0.006 93.8 0.90 0.981 0.981 0.000 0.003 93.6 AUC 0.704 0.706 0.006 0.011 93.2
aNominal: 95.0
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 24
Example
SLIDE 25
AISC, 2012
19
Competing Risks ROC
Example
Multicenter AIDS Cohort Study
- N = 438 sero-convert patients
- Endpoints - Any major event
⊲ AIDS (n = 176) ⊲ Death before AIDS (n = 38)
- Time-dependent composite marker
⊲ Sum and difference of (−1)·CD4 measurements ⊲ Sum and difference of (−1)·CD8 measurements ⊲ from last two visits
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 26
20 40 60 80 100 120 −7 −6 −5 −4 −3 −2 −1
Time (months) Marker
Control
20 40 60 80 100 120 −7 −6 −5 −4 −3 −2 −1
Time (months) Marker
AIDS or Death
10th (Time−dependent) Test + Test − 10th (Overall) Test + Test −
SLIDE 27
AISC, 2012
21
Summary ROC for MACS Study - CD4
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 FP TTP Summary ROC 1−Year ROC 10−Year ROC AUC 0.756 0.921 0.616
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 28
AISC, 2012
22
Summary ROC for MACS Study - Comparison
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 FP TTP Summary ROC − CD4/CD8 Summary ROC − CD4 Summary ROC − CD8 AUC 0.756 0.628 0.751
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 29
Comments
SLIDE 30
AISC, 2012
23
Comments
- Summary ROC: a new summary for time-dependent ROC curves
- Plot of (FP, TTP) for all FP = p ∈ (0, 1)
- To characterize overall predictive accuracy of marker or riskscore
- Naturally accommodates longitudinal marker to provide overall accuracy
- Total True Positive at p: total yield over time for a dynamic threshold with
fixed FP
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 31
AISC, 2012
24
Comments - contd.
- Independent censoring for TP estimator
- Alternate estimators can be used
- Total False Positive can be defined similarly
- Saha-Chaudhuri and Heagerty (2012)
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 32
AISC, 2012
25
Thank you!!
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 33
AISC, 2012
26
References
Akritas, M. G. (1994). Nearest neighbor estimation of a bivariate distribution under random censoring. Annals of Statistics, 22 : 1299 – 1327. Hartmann, O., Schuetz, P., Albrich, W. C., Anker, S. D., Mueller, B., and Schmidt, T. (2012). Time-dependent cox regression: Serial measurement of the cardiovascular biomarker proadrenomedullin improves survival prediction in patients with lower respiratory tract infection. International Journal of Cardiology. Heagerty, P. J. and Zheng, Y. (2005). Survival model predictive accuracy and ROC
- curves. Biometrics, 61 : 92 – 105.
Saha-Chaudhuri, P. and Heagerty, P. J. (2012). Dynamic thresholds and a summary roc curve: Assessing the prognostic accuracy of longitudinal markers. To be submitted.
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 34
AISC, 2012
27
Comments
Why dynamic threshold?
”We suggest that the fixed-threshold approach used to define high risk categories may be sub-optimal, especially in the case of risk algorithms that depend heavily
- n age.”
– Michael Pencina, The Framingham Heart Study
- Fixed marker threshold is unable to capture time-dependent nature of marker
- Increasing longitudinal trend is preserved by a time-dependent threshold
- Fixed threshold may still be useful in some situations (esp. shorter duration)
Biostat, Duke University
- P. Saha Chaudhuri
SLIDE 35
AISC, 2012
28
Comments
Alternate Estimators
- Model-based marker instead of raw marker
- Joint models
- Alternate measures of quantile
- Alternate measure of TP
- Alternate measure of survival time distribution (smoothed or not)
Biostat, Duke University
- P. Saha Chaudhuri