[PPT] - Session 5 of Module 8: Methods for Assessing Immunological PowerPoint Presentation

SLIDE 1

Session 5 of Module 8: Methods for Assessing Immunological Correlates of Risk and Optimal Surrogate Endpoints

Peter Gilbert

Summer Institute in Statistics and Modeling in Infectious Diseases

U of W July 18–20, 2016

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 1 / 73

SLIDE 2

Outline of Module 8: Evaluating Vaccine Efficacy

‐

Session 1 (Halloran) Introduction to Study Designs for Evaluating VE Session 2 (Follmann) Introduction to Vaccinology Assays and Immune Response Session 3 (Gilbert) Introduction to Frameworks for Assessing Surrogate Endpoints/Immunological Correlates of VE Session 4 (Follmann) Additional Study Designs for Evaluating VE Session 5 (Gilbert) Methods for Assessing Immunological Correlates of Risk and Optimal Surrogate Endpoints Session 6 (Gilbert) Effect Modifier Methods for Assessing Immunological Correlates of VE (Part I) Session 7 (Gabriel) Effect Modifier Methods for Assessing Immunological Correlates of VE (Part II) Session 8 (Sachs) Tutorial for the R Package pseval for Effect Modifier Methods for Assessing Immunological Correlates of VE Session 9 (Gilbert) Introduction to Sieve Analysis of Pathogen Sequences, for Assessing How VE Depends on Pathogen Genomics Session 10 (Follmann) Methods for VE and Sieve Analysis Accounting for Multiple Founders

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 2 / 73

SLIDE 3

Outline of Session 5

1 Traditional CoR methods: Inverse probability weighted Cox model 2 Key issues

Marker sampling design
Marker measurement error

3 Improved CoR methods (Breslow et al., 2009; Rose and van der Laan,

2011)

4 Estimated optimal surrogate (van der Laan, Price, Gilbert, 2016)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 3 / 73

SLIDE 4

Prospective Cohort Study Sub-Sampling Design Nomenclature

Terms used: case-cohort, nested case-control, 2-phase sampling
Case-cohort sampling originally meant taking a Bernoulli random

sample of subjects at study entry for marker measurements (the “sub-cohort”), and also measuring the markers in all disease cases (Prentice, 1986, Biometrika)

Nested case-control sampling is Bernoulli or without replacement

sampling done separately within disease cases and controls (retrospective sampling)

2-phase sampling is the generalization of nested case-control sampling

that samples within discrete levels of a covariate as well as within case and control strata (Breslow et al., 2009, AJE, Stat Biosciences)

Source of confusion: Some papers allow case-cohort to include

retrospective sampling

We restrict case-cohort to its original meaning
P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 4 / 73

SLIDE 5

The Cox Model with a Sub-Sampling Design

Cox proportional hazards model

λ(t|Z) = λ0(t)exp

βT

0 Z(t)

λ(t|Z) = conditional failure hazard given covariate history until time t
β0 = unknown vector-valued parameter
λ0(t) = λ(t|0) = unspecified baseline hazard function
Z are “expensive” covariates only measured on failures and subjects in

a random sub-sample

i.e., Z = immune response biomarkers, measured at fixed time τ

post-randomization or at longitudinal visits

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 5 / 73

SLIDE 6

Notation and Set-Up (Matches Kulich and Lin, 2004, JASA)

T = failure time (e.g., time to HIV infection diagnosis)
C = censoring time
X = min(T, C), ∆ = I(T ≤ C)
N(t) = I(X ≤ t, ∆ = 1)
Y (t) = I(X ≥ t)
Cases are subjects with ∆ = 1
Controls are subjects with ∆ = 0
P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 6 / 73

SLIDE 7

Notation and Set-Up (Matches Kulich and Lin, 2004, JASA)

Consider a prospective cohort of N subjects, who are stratified by a

variable V with K categories

ǫ = indicator of whether a subject is selected for measurement of

immune responses Z (and they are measured)

αk = Pr(ǫ = 1|V = k), where αk > 0
(Xki, ∆ki, Zki(t), 0 ≤ t ≤ τ, Vki, ǫki ≡ 1) observed for all marker

subcohort subjects

At least (Xki, ∆ki ≡ 1, Zki(Xki)) observed for all cases
P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 7 / 73

SLIDE 8

Estimation of β0

With full data, β0 may be estimated by the MPLE, defined as the

root of the score function UF(β) =

n

i=1

τ

Zi(t) − ¯

ZF(t, β)

dNi(t),

(1) where ¯ ZF(t, β) = S(1)

F (t, β)/S(0) F (t, β);

S(1)

F (t, β)

= n−1

n

i=1

Zi(t)exp

βTZi(t)
Yi(t)

S(0)

F (t, β)

= n−1

n

i=1

exp

βTZi(t)
Yi(t)
P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 8 / 73

SLIDE 9

Estimation of β0

Due to missing data (1) cannot be calculated under the sub-sampling

design

Most estimators are based on pseudoscores parallel to (1), with

¯ ZF(t, β) replaced with an approximation ¯ ZC(t, β) UC(β) =

K

k=1

nk

i=1

τ

Zki(t) − ¯

ZC(t, β)

dNki(t)
The double indices k, i reflect the stratification
P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 9 / 73

SLIDE 10

Estimation of β0

The marker sampled cohort at-risk average is defined as

¯ ZC(t, β) ≡ S(1)

C (t, β)/S(0) C (t, β),

where S(1)

C (t, β)

= n−1

K

k=1

nk

i=1

ρki(t)Zki(t)exp

βTZki(t)
Yki(t)

S(0)

C (t, β)

= n−1

K

k=1

nk

i=1

ρki(t)exp

βTZki(t)
Yki(t)
P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 10 / 73

SLIDE 11

Estimation of β0

ρki(t) is set to zero for subjects with incomplete data, eliminating

them from the estimation

Cases and subjects in the marker subcohort have ρki(t) > 0
Usually ρki(t) is set as the inverse estimated sampling probability

(Using the same idea as the weighted GEE methods of Robins, Rotnitzky, and Zhao, 1994, 1995)

Different estimators are formed by different choices of weights ρki(t)
Two classes of estimators (case-cohort and 2-phase)
P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 11 / 73

SLIDE 12

Example CoR Analysis: RV144 HIV-1 VE Trial

Haynes et al. (2012, NEJM) assessed in vaccine recipients the association

f 6 immune response biomarkers measured at Week 26 with HIV-1

infection through 3.5 years

2-phase sampling design: Measured Week 26 responses from all

HIV-1 infected cases (n = 41) and from a stratified random sample of controls (n = 205 by gender ×# vaccinations × per-protocol) Immune Response Variable

Est. HR (95% CI)

2-Sided P-value IgA Magnitude-Breadth to Env 1.58 (1.07–2.32) 0.02 Avidity to A244 Strain 0.90 (0.55–1.46) 0.66 ADCC to 92TH023 Strain 0.92 (0.62–1.37) 0.67 Neutralization M-B to Env 1.46 (0.87–2.47) 0.15 IgG to gp70-V1V2 Env 0.57 (0.37–0.90) 0.014 CD4 T cell Magn to 92TH023 1.17 (0.83–1.65) 0.37 Borgan et al. (2000, Lifetime Data Analysis) Cox model estimator II

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 12 / 73

SLIDE 13

Case-cohort Estimators (Called N-estimators in Kulich and Lin, 2004)

The subcohort is considered a sample from all study subjects

regardless of failure status

The whole covariate history Z(t) is used for all subcohort subjects
For cases not in the subcohort, only Z(Ti) (the covariate at the failure

time) is used

Prentice (1986, Biometrika): ρi(t) = ǫi/α for t < Ti and

ρi(Ti) = 1/α

Self and Prentice (1988, Ann Stat): ρi(t) = ǫi/α for all t
P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 13 / 73

SLIDE 14

Case-cohort N-estimators

General stratified N-estimator
ρki(t) = ǫi/

αk(t) for t < Tki and ρki(Tki) = 1

αk(t) is a possibly time-varying estimator of αk

αk is known by design, but nonetheless estimating αk provides greater

efficiency for estimating β0 (Robins, Rotnitzky, Zhao,1994)

A time-varying weight can be obtained by calculating the fraction of

the sampled subjects among those at risk at a given time point (Barlow, 1994; Borgan et al., 2000, Estimator I)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 14 / 73

SLIDE 15

Two-phase Sampling Estimators (Called D-estimators in Kulich and Lin, 2004)

Weight cases by 1 throughout their entire at-risk period
D-estimators treat cases and controls completely separately
αk apply to controls only, so that αk should be estimated using data
nly from controls
Nested case-control estimators are the special case with one covariate

sampling stratum K = 1

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 15 / 73

SLIDE 16

Two-phase Sampling D-estimators

General D-estimator

ρki(t) = ∆ki + (1 − ∆ki)ǫki/ αk(t)

Borgan et al. (2000, Estimator II) obtained by setting
αk(t) =

n

i

ǫki(1 − ∆ki)Yki(t)/

n

i

(1 − ∆ki)Yki(t), i.e., the proportion of the sampled controls among those who remain at risk at time t

the cch package in R (by Thomas Lumley and Norm Breslow)

implements the Cox model for case-cohort (N-estimators) and 2-phase sampling (D-estimators) (code for using cch to analyze a data set is provided at http://faculty.washington.edu/peterg/SISMID2016.html)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 16 / 73

SLIDE 17

Main Distinctions Between N- and D- Estimators

D-estimators require data on the complete covariate histories of cases
N-estimators only require data at the failure time for cases
E.g., for the Vax004 HIV VE trial, the immune responses in cases were
nly measured at the visit prior to infection, so N-estimators are valid

while D-estimators are not valid

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 17 / 73

SLIDE 18

Main Distinctions Between N- and D- Estimators

For N-estimators, the sampling design is specified in advance,

whereas for D-estimators, it can be specified after the trial (retrospectively)

D-estimators more flexible
P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 18 / 73

SLIDE 19

Gaps of Both N- and D- Estimators

Does Not Need Allows Outcome- Full Covariate Dependent Estimator Histories in Cases Sampling N (Prosp. case-cohort) Yes No D (Retrosp. 2-phase) No Yes

For time-dependent correlates, none of the partial-likelihood based

methods are flexible on both points

All of the methods require full covariate histories in controls
P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 19 / 73

SLIDE 20

Outline

1 Traditional CoR methods: Inverse probability weighted Cox model 2 Key issues

Marker sampling design
Marker measurement error

3 Improved CoR methods (Breslow et al., 2009; Rose and van der Laan,

2011)

4 Estimated optimal surrogate (van der Laan, Price, Gilbert, 2016)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 20 / 73

SLIDE 21

Some Marker Sampling Questions to Consider Further

Prospective or retrospective sampling?
How much of the cohort to sample?
Sampling design: Which subjects to sample?
P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 21 / 73

SLIDE 22

Prospective or Retrospective Sampling?

Prospective case-cohort sampling: Select a random sample for immunogenicity measurement at baseline

Advantages of prospective sampling
Can estimate case incidence for groups with certain immune responses
Can study correlations of immune response with multiple study

endpoints

Straightforward to descriptively study the distribution of the immune

responses in the whole study population at-risk when the immune responses are measured

Practicality: The lab will know what subjects to sample as early as

possible, and there is one simple subcohort list

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 22 / 73

SLIDE 23

Prospective or Retrospective Sampling?

Retrospective 2-phase sampling: At or after the final analysis, select a random sample of control subjects for immunogenicity measurement

Advantages of retrospective sampling
Can match controls to cases to obtain balance on important covariates
E.g., balanced sampling on a prognostic factor gains efficiency

(balanced sampling = equal number of subjects sampled within each level of the prognostic factor for cases and controls)

Can flexibly adapt the sampling design in response to the results of the

trial

E.g., Suppose the results indicate effect modification, with VE >> 0 in

a subgroup and VE ≈ 0% in other subgroups. Could over-sample controls in the ‘interesting’ subgroup.

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 23 / 73

SLIDE 24

Prospective or Retrospective Sampling?

For cases where there is one primary endpoint and it is not of major

interest to estimate absolute case incidence, retrospective sampling may be typically referred

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 24 / 73

SLIDE 25

How Many Controls to Sample?

In prevention trials, for which the clinical event rate is low, it is very

expensive and unnecessary to sample all of the controls

Vax004 trial vaccine recipients: 225 HIV infected cases; ≈ 3000

controls

RV144 trial vaccine recipients: 41 HIV infected cases; ≈ 7000 controls
Rule of thumb: Under the null hypothesis, a K : 1 Control:Case ratio

achieves relative efficiency of 1 −

1 1+K compared to complete sampling

K Relative Efficiency 1 0.50 2 0.67 3 0.75 4 0.80 5 0.83 10 0.91

Simulations useful for studying the trade-offs of different K under

alternative CoR hypotheses

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 25 / 73

SLIDE 26

Which Controls to Sample?

Two-Phase Sampling

Phase I: All N trial participants are classified into K strata on the

basis of information known for everyone: Nk in stratum k; N = K

k=1 Nk

Phase II: For each k, nk ≤ Nk subjects are sampled at random, and

the ‘expensive’ immune response biomarkers Z are measured for the resulting n = K

k=1 nk subjects

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 26 / 73

SLIDE 27

Which Controls to Sample?

Principle: Well-powered CoR evaluation requires broad variability in the biomarker and in the risk of the clinical endpoint

Can improve efficiency by over-sampling the “most informative”

subjects

Disease cases (usually sampled at 100%)
Rare or unusual immune responses; or rare covariate patterns believed

to affect immune response (e.g., HLA subgroups)

Auxiliary Phase I variables measured in everyone are most valuable

when they predict the missing data (i.e., the biomarker of interest)

In general, optimal sampling obtained with sampling probabilities

proportional to the cost-adjusted square-root variance of the efficient influence function (Gilbert, Yu, Rotnitzky, 2014, Stat Med)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 27 / 73

SLIDE 28

Outline

1 Traditional CoR methods: Inverse probability weighted Cox model 2 Key issues

Marker sampling design
Marker measurement error

3 Improved CoR methods (Breslow et al., 2009; Rose and van der Laan,

2011)

4 Estimated optimal surrogate (van der Laan, Price, Gilbert, 2016)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 28 / 73

SLIDE 29

Measurement Error Reduces Power to Detect a CoR

Illustrative Example

‘True’ CoR

S∗ ∼ N(0, 1)

‘Measured CoR’

S = S∗ + ǫ, ǫ ∼ N(0, σ2)

Infection status Y generated from Φ(α + βS∗)

with α set to give P(Y = 1|S∗ = 0) = 0.20 and β set to give P(Y = 1|S∗ = 1) = 0.15 σ2 ranges from 0 to 2 (no-to-large measurement error)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 29 / 73

SLIDE 30

Measurement Error Reduces Power to Detect a CoR

Simple Simulation Study

Consider a study with n = 500 participants
Consider power of a logistic regression model to detect an association

between S and Y

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 30 / 73

SLIDE 31

Measurement Error Reduces Power to Detect a CoR

0.0 0.5 1.0 1.5 2.0 0.4 0.5 0.6 0.7 0.8

Measurement Error Sigma2 Power Deterioration of Power to Detect a CoR with Increasing Measurement Error

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 31 / 73

SLIDE 32

Power Calculations for Assessing CoRs

Ideally, the power/sample size calculations should explicitly account

for measurement error in the assay

E.g., Gilbert, Janes, Huang (2016, Stat Med), implemented in the R

package CoRpower posted at http://faculty.washington.edu/peterg/programs.html

E.g., specify ρ ≡ σ2/σ2
bs, the proportion of inter-vaccinee variability of

the biomarker that is biologically relevant

Rule of thumb: ρ =relative efficiency for estimating a CoR odds ratio

for the underlying perfect biomarker compared to the observed biomarker (McKeown-Eyssen, Tibshirani, 1994, AJE)

‘Noise’ components of σ2
bs may be estimated, especially from

laboratory assay validation studies

Within-vaccinee variability of replicates
Between-vaccinee variability due to variability in the time from the last

immunization to marker sampling

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 32 / 73

SLIDE 33

Power to Detect a CoR of HIV Infection in Vaccinees in HVTN 505 (α = 0.05)

06/03/2014 •

Method: 2-phase logistic regression (Holubkov and Breslow, 1997)

V2 Benchm hmark ark V2 = magnitu nitude de of

bserved

ed primar ary gp70-V1V2 binding ding Ab Ab Inver erse e CoR in RV144 (Haynes et al., , 2012 12) rho = biolo logi gicallly allly relevan ant propor

rti

tion

n of va

varianc ance e of the biomark marker er

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 33 / 73

SLIDE 34

Outline

1 Traditional CoR methods: Inverse probability weighted Cox model 2 Key issues

Marker sampling design
Marker measurement error

3 Improved CoR methods (Breslow et al., 2009; Rose and van der Laan,

2011)

4 Estimated optimal surrogate (van der Laan, Price, Gilbert, 2016)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 34 / 73

SLIDE 35

Typical Correlates Assessments are Inefficient

Broadly in epidemiology studies, biomarker-disease associations are

commonly assessed ignoring much data collected in the study

That is, only subjects with the biomarker measured are included in

the analysis

Standard analyses use inverse probability weighting of the biomarker

sampled subcohort, including all of the methods discussed so far

These ubiquitously-used methods are implemented in the R package

cch (Breslow and Lumley)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 35 / 73

SLIDE 36

Typical Correlates Assessments are Inefficient

Breslow et al.∗ urge statisticians/epidemiologists to consider using the

whole cohort in the analysis of case-cohort/2-phase sampling data

Baseline data on demographics and potential confounders are typically

collected in all subjects (the Phase I data measured in everyone)

These Phase I data are most valuable when they predict “missing”

data

∗Breslow, Lumley et al. (2009, AJE, Stat Biosciences)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 36 / 73

SLIDE 37

How to Leverage All of the Data?

Question: How can we use the Phase I data to improve the

assessment of CoRs?

One Answer: One approach adjusts the sampling weights used in the

standard analyses described above to obtain approximately efficient estimators (e.g., Breslow et al., 2009, AJE, Stat Biosciences)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 37 / 73

SLIDE 38

Some Lessons Learned from Breslow et al. (2009)

1 Obtain ‘worthwhile’ efficiency gain for the CoR assessment if baseline

covariates can explain at least 40% of the variation in the immunological biomarker (R2 ≥ 0.40)

2 If interested in interactions (evaluation of whether a baseline covariate

measured in everyone modifies the association of the biomarker and the clinical endpoint), can obtain worthwhile efficiency gain with a lower R2

3 Even if no gain for the CoR assessment, will usually dramatically

improve efficiency for assessing the associations of the Phase I covariates with outcome

4 Therefore it may often be the preferred method, and all practicioners

should have methods accounting for all of the data in their analytic toolkit

5 Additional research needed to make these more-efficient methods

work well for multivariate markers and for time-dependent markers

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 38 / 73

SLIDE 39

How to Leverage All of the Data?

Question: How can we use the Phase I data to improve the

assessment of CoRs?

Another Answer: Use an efficient and double-robust method:

Inverse probability of censoring weighted targeted minimum loss based estimation (IPCW-TMLE) (Rose and Van der Laan, 2011, Int J Biost)

Right-Censored Data Structure for Fixed Follow-up Time t

V = Phase I information: Covariates (Z, V0), ˜

T = min(T, C), ∆ = I(T ≤ C), Y ∗ = I( ˜ T ≤ t)∆, Phase II sampling probability ǫ

S = (A, W ) = Phase II information: Immune response biomarkers

measured at τ

Focus on the marker A of interest; W = all other markers
Repeat the analysis taking each element of W as A
P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 39 / 73

SLIDE 40

IPCW-TMLE: Target Parameters for Inference (Binary Marker)

Full data structure X = (V , S) = (Z, V0, ˜ T, ∆, Y ∗, A, W )

General target parameters

PX,0 = true probability distribution of X
MF = statistical model for PX,0
ΨF : MF → Rd = target parameter of the full-data distribution
ψF

0 = ΨF(PX,0) = target parameter of the true probability

distribution of X

Causal risk target parameters for a binary marker A

ψF

RD,0

= EX,0 [EX,0(Y |A = 1, W ) − EX,0(Y |A = 0, W )] (2) ψF

RR,0

= EX,0 [EX,0(Y |A = 1, W )] EX,0 [EX,0(Y |A = 0, W )] in each case with MF nonparametric

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 40 / 73

SLIDE 41

IPCW-TMLE: Target Parameters for A Quantitative

Idea from Alex Luedtke
Make inferences about

maxl<uΨF

RD(PX,0; l, u) = maxl<u{E[Y |A ≥ u] − E[Y |A ≤ l]}

and maxl<uΨF

RR(PX,0; l, u) = maxl<u

E[Y |A ≤ l] E[Y |A ≥ u]

subject to a constraint on l and u such as that mentioned above
Assesses whether any trichotomization of the marker A yields a

significant CoR, with the inference formally accounting for the searching for the best-discriminating cut-points

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 41 / 73

SLIDE 42

IPCW-TMLE: Data-Adaptive Target Parameters for A Quantitative

Following Van der Laan, Hubbard, and Pajouh (2013), define

data-adaptive causal contrasts using K-fold cross-validation

Based on the first K − 1 data pieces, define two cut-points l1 < u1 for

A that maximize the IPCW-TMLE of |ΨF

RD(PX,0; l1, u1)|, under a

constraint such as ≥ 5% cases with A < l1 and with A > u1

Obtain IPCW-TMLE of ΨF

RD(PX,0; l1, u1) from withheld K th piece

Repeat for each set of K − 1 data pieces with the K th piece withheld,

yielding K maximizing cutpoints (l1, u1) · · · (lK, uK) and K corresponding ICPW-TMLE estimators on withheld data sets

Define the data-adaptive causal risk difference parameter as

K

k=1

ΨF

RD(PX,0; lk, uk),

estimated by the average of the IPCW-TMLE estimates

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 42 / 73

SLIDE 43

Implementation to Obtain the IPCW-TMLE of E[Y |A]

Observed data i.i.d. copies of O = (Z, V0, ˜ T, ∆, Y ∗, ǫ, ǫA, ǫW )

If the full data X were available, then existing TMLE procedures

could be used

True target parameter PX,0 defined wrt a specified full-data loss

function LF(PX)(X): PX,0 = argminPX ∈MF E0L(PX)(X)

TMLE Step 1: Construct an initial estimator P0

X,n of PX,0

TMLE Step 2: Bias-correct P0

X,n through an iterative algorithm to

yield P∗

X,n, making the empirical average of the full-data efficient

influence curve at P∗

X,n equaling zero, hence yielding an efficient

estimator

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 43 / 73

SLIDE 44

Implementation to Obtain the IPCW-TMLE of E[Y |A]

IPCW-TMLE proceeds in the same way, except in each step the

following IPCW-loss function is used in place of the full-data loss function, where Πn(V ) is a nonparametric or TMLE estimator of the marker sampling probability Π0(V ) = P(ǫ = 1|V ) L(PX)(O) ≡ ǫ Πn(V )LF(PX)(X)

Step 1 (initial estimation of PX,0) can be maximally flexible and

robust by using 2 or 3 superlearners for each element of PX,0

1 Sampling probability estimator Πn(V ) 2 Conditional risk EX,0(Y |A, W ) 3 “Exposure mechanism” g0(a|W ) ≡ PX,0(A = a|W )

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 44 / 73

SLIDE 45

Implementation to Obtain the IPCW-TMLE of E[Y |A]

Properties of IPCW-TMLE

Πn(V ) guaranteed consistent for Π0(V ) if all the marker missingness

is by design

Double-robustness property: IPCW-TMLE is consistent even if the

superlearner inconsistently estimates one (but not both) of EX,0(Y |A, W ) and g0(a|W )

Consistent estimation of both terms implies the IPCW-TMLE is

asymptotically efficient

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 45 / 73

SLIDE 46

Implementation to Obtain the IPCW-TMLE of E[Y |A]

Ordinary superlearner (van der Laan, Polley, and Hubbard, 2007) may

be used to estimate each piece Π0(V ), EX,0(Y |A, W ), g0(a|W ), e.g., implemented with the Superlearner R package, using learners that allow specification of subject-specific weights ǫi/Πn(Vi)

If there is substantial happenstance missingness of markers, then the

missing at random assumption may fail

In this setting the superlearner for Πn(V ) may be helpful
Neugebauer et al. (2013) demonstrated in marginal structural models

that replacing a standard strategy of logistic regression modeling of the propensity score with superlearner reduced bias and improved efficiency

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 46 / 73

SLIDE 47

Outline

1 Traditional CoR methods: Inverse probability weighted Cox model 2 Key issues

Marker sampling design
Marker measurement error

3 Improved CoR methods (Breslow et al., 2009; Rose and van der Laan,

2011)

4 Estimated optimal surrogate (van der Laan, Price, Gilbert, 2016)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 47 / 73

SLIDE 48

Introduction to an Optimal Surrogate

Goal: Determine a surrogate outcome for a long-term outcome so

that future randomized or observational studies can restrict themselves to only collecting the surrogate outcome

Data from a clinical trial for developing a surrogate: n iid
bservations of O = (W , Z, S, Y )
W = Vector of baseline covariates
Z = Treatment assignment (e.g., 1=vaccine; 0=placebo)
S = Vector of response variables/markers at an intermediate time

point τ

Y = Outcome of interest at a final time point after τ (binary or

quantitative)

Assume Z is randomized conditional on W
P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 48 / 73

SLIDE 49

Introduction to an Optimal Surrogate

Define an optimal surrogate for the current trial as the function of

the data (W , Z, S) collected by the intermediate time point τ that

ptimally predicts the final outcome Y
A true parameter that we estimate with a targeted super-learner
Goal: Use the estimated optimal surrogate in future clinical trials

for estimation and testing of a mean contrast treatment effect on Y

Tackles the transportability problem of inferring the causal treatment

effect in a new trial without measuring clinical endpoints Y (e.g., addressed by Pearl and Bareinboim, 2011, 2012)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 49 / 73

SLIDE 50

Optimal Surrogate Framework vs. Other Frameworks

vs. controlled/natural effects and VE curve frameworks:

Departs by being based on average causal effects identified from standard assumptions in randomized trials

vs. Prentice/valid replacement endpoint framework: Aligns in

that the optimal surrogate satisfies the Prentice definition

Partially aligns with the Prentice criteria
The best optimal surrogate will have treatment and candidate

surrogate highly predictive of Y , similar to Prentice criteria 1 and 2

The framework posits a conditional mean version of Prentice criterion 3

for licensing correct inferences on Y in a new trial

It handles equally well the general case where S varies or is constant in

the placebo group

vs. meta-analysis framework: Aligns in its objective of inference on

the clinical treatment effect in a future study without collecting Y in that study (Gail et al., 2000, Biostatistics)

Departs in being based on a single (or few) trials and different

transportability assumptions

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 50 / 73

SLIDE 51

Optimal Surrogate Framework

Departs from all previous frameworks by defining the optimal

surrogate as an unknown target parameter

Predicted values from the estimated optimal surrogate are used as the

actual surrogate endpoint

In large samples this resulting surrogate must satisfy the Prentice

definition (under the standard assumptions of an RCT)

New approach in treating the surrogate endpoint problem as a

supervised statistical learning problem

Previous methods evaluate a pre-selected univariable or

low-dimensional vector candidate surrogate

The optimal surrogate approach is robust in that asymptotically

consistent hypothesis tests and confidence intervals for the clinical treatment effects in the current and future trials are obtained without parametric modeling assumptions

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 51 / 73

SLIDE 52

Statistical Formulation of Estimation of an Optimal Surrogate

Observed data: iid copies O = (W , Z, S, Y ) ∼ P0

W = vector of baseline covariates
Z = binary treatment assigned at baseline
S = vector of intermediate outcomes measured at a fixed time point τ
Y = final univariate outcome measured at a later final time point
Potential outcomes (S1, S0) and (Y1, Y0) under treatment assignment

Z = 1 and Z = 0

Treatment Z is randomized conditional on W
P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 52 / 73

SLIDE 53

A Nonparametric Approach

X = (W , S0, S1, Y0, Y1) = full-data structure with distribution PX,0
O = (W , Z, S, Y ) = observed data with distribution P0 determined

by PX,0 and g0(z | X) = g0(z | W )

The statistical model M for P0 makes at most some assumptions

about g0

Known in a randomized trial
M puts no assumptions on the marginal distribution of W nor on the

conditional distribution of (S, Y ) given A, W

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 53 / 73

SLIDE 54

Candidate Surrogate Outcomes

Any real-valued function (W , A, S) → ψ(W , A, S) ∈ I

R is a candidate surrogate, representing a measurement one can collect by time τ

Question: How to define a good surrogate in terms of the true data

distribution P0?

Starting point: We would like the surrogate Sψ ≡ ψ(W , A, S) to be

valid in the actual study, according to the Prentice definition: E0(Y1 − Y0) = 0 if and only if E0(Sψ

1 − Sψ 0 ) = 0,

where Sψ

z = ψ(W , z, Sz), for z ∈ {0, 1}

Guarantees that an α-level test for Hψ

0 : E0(Sψ 1 − Sψ 0 ) = 0 is also an

α-level test for H0 : E0(Y1 − Y0) = 0

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 54 / 73

SLIDE 55

Optimal Surrogate Outcome

Criterion for ranking valid surrogates and defining a P0-optimal

surrogate: full-data mean squared error ψ → MSEPX,0(ψ) ≡

z

EPX,0

g0(z | W )(Yz − ψ(W , z, Sz))2
Goal is to minimize the weighted mean square prediction error for

predicting Yz across z ∈ {0, 1}

Given a class Ψ of possible surrogate functions ψ(), the P0-optimal

surrogate in this class is defined as ψF

0 = arg min ψ∈Ψ MSEPX,0(ψ)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 55 / 73

SLIDE 56

Optimal Surrogate Outcome

Theorem 1. The minimizer of ψ → MSEPX,0(ψ) over all functions (W , A, S) → ψ(W , A, S) is: ¯ S0 = ψ0(W , Z, S) ≡ E0(Y | W , Z, S) Potential outcomes of this P0-optimal surrogate: ¯ S0,z = E0(Yz | W , Sz), z ∈ {0, 1} and EP0(¯ S0,z | W ) = EP0(Yz | W )

Implication: Under P0, a 95% confidence interval for the causal

effect of treatment on the P0-optimal surrogate is also a 95% confidence interval for the causal effect of treatment on Y

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 56 / 73

SLIDE 57

Conditions for a New Study P Under Which the P0-Optimal Surrogate is also the P-Optimal Surrogate

Theorem 2. Consider a new study with iid observations O∗ = (W ∗, Z ∗, S∗, Y ∗) with distribution P, where Z ∗ is randomized conditional on W ∗

Transportability assumption:

E[Y ∗|W ∗ = w, Z ∗ = z, S∗ = s] = E[Y |W = w, Z = z, S = s] for all (w, z, s) in a support of (W ∗, Z ∗, S∗)

Support assumption: A support of (W ∗, Z ∗, S∗) is contained in a

support of (W , Z, S) Result: The P0-optimal surrogate equals the P-optimal surrogate: for all (w, z, s) in a support of (W ∗, Z ∗, S∗), EP(Y ∗ | W ∗ = w, Z ∗ = z, S∗ = s) = EP0(Y | W = w, Z = z, S = s) and EP(Y ∗ | W ∗ = w, Z ∗ = z, S∗ = s) = EP(Y ∗

z | W ∗ = w, S∗ z = s)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 57 / 73

SLIDE 58

Transportability Theorem Under a Prentice Criterion 3: Application to a New Treatment Z ∗ = Z

If the new study considers a new treatment Z ∗ = Z, then generally

the transportability theorem will not apply, because E[Y ∗|W ∗ = w, Z ∗ = z, S∗ = s] = E[Y |W = w, Z = z, S = s] Theorem 3.

Transportability and Support assumptions: Same as in Theorem 2
Prentice criterion 3 assumption for both settings:

E[Y ∗|W ∗, Z ∗, S∗] = E[Y ∗|W ∗, S∗] and E[Y |W , Z, S] = E[Y |W , S] Result: The P-optimal surrogate equals the P0-optimal surrogate: EP(Y ∗ | W ∗ = w, Z ∗ = z, S∗ = s) = EP0(Y | W = w, Z = z, S = s) EP(Y ∗ | W ∗ = w, Z ∗ = z, S∗ = s) = EP(Y ∗

z | W ∗ = w, S∗ z = s)

EP0(Yz | W = w, Sz = s) & EP(Y ∗

z | W ∗ = w, S∗ z = s) constant in z

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 58 / 73

SLIDE 59

Super-learning of the P0-optimal surrogate

Estimation of the P0-optimal surrogate is a standard prediction

problem

Estimate E0(Y | W , Z, S) by a minimizer of the risk of a loss
E.g., for Y binary, use log-likelihood loss

L(ψ)(O) = −{Y log ψ(W , A, S) + (1 − Y ) log(1 − ψ(W , A, S))}

Loss-based super-learning∗: yields an optimal estimator among any

given class of candidate estimators

Oracle inequality for the cross-validation selector: the estimator is

asymptotically at least as good as any candidate in the set of candidate estimators

∗van der Laan, Polley, and Hubbard (2007); van der Laan and Rose (2011)

textbook

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 59 / 73

SLIDE 60

Dengue Phase 3 Trial Example

Two randomized, double-blinded, placebo-controlled, multicenter,

Phase 3 trials of a recombinant, live, attenuated, tetravalent dengue vaccine (CYD-TDV)

CYD14: Asia-Pacific region (Capeding, et al., 2014, The Lancet)
CYD15: Latin America (Villar et al, 2015, NEJM)

Trial Designs

2:1 randomization to vaccine:placebo
Immunizations at months 0, 6, 12
Primary follow-up from Month 13 to Month 25 (active phase of

follow-up)

Primary endpoint: Symptomatic, virologically confirmed dengue

(VCD)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 60 / 73

SLIDE 61

Results on Vaccine Efficacy (Estimates from a Proportional Hazards Model)

CYD14: VE = 56.5% (95% CI 43.8–66.4) CYD15: VE = 64.7% (95% CI 58.7–69.8)

CYD15 Trial (Villar et al., 2015, NEJM) CYD14 Trial (Capeding et al., 2014, The Lancet)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 61 / 73

SLIDE 62

Illustration of Estimated Optimal Surrogate Approach

Analysis carried out by Brenda Price

Based on pseudo CYD14 and CDY15 simulated data sets
Treat CYD14 as the current trial; CYD15 as the future trial

Notation and Variables

Z = Vaccination status (1=vaccine; 0=placebo)
Y = Disease outcome (1=VCD endpoint between Month 13 and 25;

0 = no VCD endpoint by Month 25)

W = Baseline covariates: age, sex, baseline PRNT50 neutralization

titers to the 4 serotypes in the vaccine

S = Month 13 PRNT50 neutralization titers to the 4 serotypes in the

vaccine

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 62 / 73

SLIDE 63

Illustration of Estimated Optimal Surrogate Approach

Objectives

1 Estimate the P0-optimal surrogate via targeted super-learner in

CYD14, yielding ψTMLE

n

(W , A, S)

2 Estimate VE ∗ in CYD15 based on the estimated optimal surrogate

from CYD14 without using the CYD15 outcome data Y

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 63 / 73

SLIDE 64

Estimates of Dengue Risks and VEs in CYD14 and CYD15

1 Estimate the P0-optimal surrogate via targeted super-learner in

CYD14, yielding ψTMLE

n

(W , A, S). Obtain:

E[Y |Z = z]

= 1 nz

nz

i=1

I(Zi = z)ψTMLE

n

(Wi, Zi = z, Si), z = 0, 1

VE

= 1 − E[Y |Z = 1]/ E[Y |Z = 0]

2 Estimate VE ∗ in CYD15 based on the estimated optimal surrogate

from CYD14 without using the CYD15 outcome data Y

E[Y ∗|Z ∗ = z]

= 1 n∗

z n∗

z

i=1

I(Z ∗

i = z)ψTMLE n

(W ∗

i , Z ∗ i = z, S∗ i ), z = 0, 1

VE

∗

= 1 − E[Y ∗|Z ∗ = 1]/ E[Y ∗|Z ∗ = 0] Wald 95% CIs based on influence functions and the delta method

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 64 / 73

SLIDE 65

Super-learner to Estimate the Optimal Surrogate

Use the MSE loss function for the superlearner cross-validation

selector (matched to the optimality criterion for a surrogate)

Table: Input Variables for the Learning Algorithms

Input Variables W : Baseline demographics age (range 2–14 years), sex W : Baseline titers to the 4 serotypes inside the CYD-TDV vaccine, min and max of the 4 titers, interactions with age S: Month 13 titers to the 4 serotypes inside the CYD-TDV vaccine, min and max of the 4 titers, interactions with age

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 65 / 73

SLIDE 66

Super-learner to Estimate the Optimal Surrogate

Table: Learning Algorithms Employed

Learners mean: E(Y |Z = z, W , S) = βz for z ∈ {0, 1} LR: Logistic regression with all input variables step LR: Best LR model by AIC through a step-wise search gam2: generalized additive modela with 2 degrees of freedom gam3: generalized additive model with 3 degrees of freedom gam4: generalized additive model with 4 degrees of freedom discrete SLb super-learnerb

a Hastie and Tibshirani (1990) textbook b van der Laan, Polley, and Hubbard (2007); van der Laan and Rose

(2011) textbook

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 66 / 73

SLIDE 67

Cross-validated Mean-Squared Errors (CV-MSEs): CYD14

mean glm step gam gam3 SuperLearner Discrete SL gam4 0.013 0.015 0.017 0.019 0.021

10−fold CV−MSE Estimate Method

CV−MSE Vaccine

mean glm step gam gam3 SuperLearner Discrete SL gam4 0.030 0.035 0.040

10−fold CV−MSE Estimate

CV−MSE Placebo

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 67 / 73

SLIDE 68

Empirical RCDFs for the Estimated Optimal Surrogate Values: CYD14

Vaccine Case (5th pctl 0.624) Placebo Case (5th pctl 0.617) Vaccine Control (5th pctl 0.069) Placebo Control (5th pctl (0.105)

0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 Estimated Optimal Surrogate Reverse Cumulative Distribution

MSE Loss: CYD14

Reverse Cumulative Distribution

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 68 / 73

SLIDE 69

Estimated Optimal Surrogate TMLEs of E[Y |Z = 1], E[Y |Z = 0], and VE: CYD14

Parameter TMLE MLE using Y E[Y |Z = 1] 1.8% (95% CI 1.5–2.1) 1.7% (95% CI 1.4–2.1) E[Y |Z = 0] 3.7% (95% CI 3.1–4.4) 3.7% (95% CI 3.1–4.4) VE = 1 − E[Y |Z=1]

E[Y |Z=0]

52% (95% CI 41–66) 55% (95% CI 43–68)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 69 / 73

SLIDE 70

Using the Estimated Optimal Surrogate in CYD15

Calculate the estimated optimal surrogate endpoint ψTMLE

n

(W ∗, Z ∗, S∗) (built in CYD14) for all CYD15 participants– How well does it predict Y ∗?

Vaccine Case (5th pctl 0.193) Placebo Case (5th pctl 0.24) Vaccine Control (5th pctl 0.072) Placebo Control (5th pctl (0.099)

0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 Estimated Optimal Surrogate Reverse Cumulative Distribution

AUC Loss: CYD15

UC for the estimated optimal surrogate was 0.78 (95% CI

Reduced classification accuracy for the new setting
P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 70 / 73

SLIDE 71

How Well Does the Surrogate Estimate VE∗ in CYD15?

Table: Estimation in CYD15 based on the estimated optimal surrogate ψTMLE

n

(W ∗, Z ∗, S∗) built in CYD14 (not using outcome data Y ∗ in CYD15) vs. estimation using Y ∗ in CYD15

Parameter ψTMLE

n

MLE using Y ∗ E[Y ∗|Z = 1] 1.5% (95% CI 1.4-1.6) 1.8% (95% CI 1.4–1.9) E[Y ∗|Z = 0] 3.3% (95% CI 3.2–3.4) 3.7% (95% CI 3.6–4.5) VE ∗ = 1 − E[Y ∗|Z=1]

E[Y ∗|Z=0]

54% (95% CI 44–67) 59% (95% CI 51–65)

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 71 / 73

SLIDE 72

Compare Predictive Ability of Input Variable Sets

Table: Cross Validated AUCs∗ with 95% CIs

Input Set CYD14 Vaccine CYD14 Placebo CYD15 Vaccine CYD15 Placebo (1) Demographics 0.61 (0.57, 0.66) 0.6 (0.55, 0.65) 0.54 (0.5, 0.58) 0.5 (0.47, 0.54) (2) All baseline 0.89 (0.86, 0.92) 0.79 (0.76, 0.83) 0.58 (0.54, 0.61) 0.55 (0.51, 0.58) (3) Month 13 titers 0.71 (0.67, 0.75) 0.63 (0.58, 0.68) 0.65 (0.62, 0.69) 0.57 (0.54, 0.61) (4) All data 0.89 (0.86, 0.91) 0.76 (0.72, 0.8) 0.78 (0.76, 0.8) 0.6 (0.57, 0.64)

∗Cross-valided area under the ROC-curves (Van der Laan, Hubbard, and

Pajouh, 2013)

The user can judge the tradeoff of accuracy and simplicity of the

estimated optimal surrogate

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 72 / 73

SLIDE 73

Checking Assumptions of the Transportability Theorem

Transportability Assumptions

1 A∗ is randomized conditional on W ∗ 2 E[Y ∗|W ∗ = w, A∗ = a, S∗ = s] = E[Y |W = w, A = a, S = s] for all

(w,a,s) in a support of (W ∗, A∗, S∗)

3 A support of (W ∗, A∗, S∗) is contained in a support of (W , A, S)

Condition 1 is met by the design of CYD15: both CYD14 and CYD15

randomized treatment

Condition 2 could be examined by comparing estimates of

E[Y ∗|W ∗ = w, A∗ = a, S∗ = s] = E[Y |W = w, A = a, S = s]

Condition 3
CYD14 age range 2–14; CYD15 9–16 (assumption fails)
All titer variables had the same minimum values
Maximum titers also similar except Month 13 serotype 3 maximum

titers 14% higher for CYD15 and baseline serotype 1 (4) maximum titers 18% (2%) greater for CYD15

P. Gilbert (U of W)

Session 5: Evaluating CoRs and Optimal Surrogates 07/2016 73 / 73