Evaluation & Implementation Challenges with Genomic Signatures - - PowerPoint PPT Presentation

April 3-4, 2006

Evaluation & Implementation Challenges with Genomic Signatures in Clinical Drug Development

• V. Devanarayan, Ph.D.

Exploratory Statistics, Pharmaceutical R&D Abbott EMA Workshop on Pharmacogenomics: From Science to Clinical Care European Medicines Agency, London, UK October 8-9, 2012

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

2

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

I have the following financial relationships to disclose:

• I am a minor stockholder in Abbott Laboratories
• I am an Employee of Abbott Laboratories, and
• I will not discuss off label use in my presentation.

Abbott funded all work related to preparation of this presentation.

• V. Devanarayan

October 8, 2012

Disclosure Information

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

3

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

Typical uses of biomarkers in drug development

• Predict responders & non-responders to a drug.
• Predict safety events such as liver and kidney injury.
• Patient-selection for clinical trial.

– Better specificity in disease diagnosis (e.g., AD vs. FTD vs. VD) – Identify which patients are likely to progress in disease

• Reduce variability, placebo response, etc.
• Dose selection (PK-PD modeling)
• Proof of Mechanism & Concept in early drug development

– Pharmacodynamic, Target engagement (receptor occupancy), etc.

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

4

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

Some Practical Challenges

• 1. Variability (Analytical + Biological)
• 2. Biological Relevance
• 3. Biomarker performance evaluation
• Internal & External Verification
• Predictive Accuracy (disease progression, adverse events, …)
• P-values (patient response/non-response), treatment differentiation, …)
• 4. Robustness
• 5. Translation
• Animals to Humans, between human subpopulations (gender, race/region,

age, disease severity and subtypes, etc.)

I will now briefly review some of these topics via illustrations.

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

5

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

Biomarker performance drops greatly when a different assay is used!

• Marker X with 15% CV is a key

predictor from the multi-analyte panel.

• Prediction Accuracy ~ 85%

8 9 10 11 12 13 Canonical2 C D 12 13 14 15 16 17 18 19 20 Canonical1

+ Disease

• Normal
• Same Marker X in the panel from

another lab has 35% CV

• Prediction Accuracy ~ 65%

7 8 9 10 11 12 Canonical2 C D 2 3 4 5 6 7 8 Canonical1

+ Disease

• Normal

Discriminant Analysis

Analytical + Biological Variability  Biomarker Performance: Example 2

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

6

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

Variability artificially added to the original data in increasing increments. (via simulation). Biomarker performance decreases with increasing variability.

50% 60% 70% 80% 90% 100% 0% 20% 40% 60% 80% 100%

Total Variability (CV) Prediction Accuracy

Original data 15% CV Original data plus noise

Analytical + Biological Variability  Biomarker Performance: Generalization

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

7

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

Assay quality impacts biomarker utility in Clinical Proof-of-Concept study

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 0.1 1 10 100 1000

Concentration OD

Expt-1 Expt-3 Expt-4

• ELISA calibration curve data from

some experiments for measuring a critical PD marker.

• Significant lower plateau in most

calibration curves.

• Need to evaluate where the study

samples fall on the curve.

Calibration Curve (Unknowns Overlaid)

0.25 0.35 0.45 0.55 0.65 0.75 0.85 0.95 1.05 0.01 0.1 1 10 100 1000

Calibrator Concentration OD

• Calib. Curve

Study Samples

Vehicle Treatment

• Most samples fall on the lower

plateau of the curve.

• High variability!
• Need to re-optimize this assay

to improve sensitivity.

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

8

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

Assay quality impacts biomarker use in Clinical Proof-of-Concept study (contd.)

20 40 60 80 100 120 1.25 1.5 1.75 2 2.25 2.5 2.75 3 3.25 3.5 3.75

Treatment Effect (Fold Change) % Power

CV = 73% CV = 40%

Original assay Better assay

Power Analysis

Improving assay sensitivity & reducing CV to 40% enables 2- fold change to be detected with 80% power. Biomarker is now ready for use in the Clinical PoC study. Poor assay sensitivity results in 73% CV.  fold-change > 3.25 can be detected with 80% power. But expected fold-change is 2-fold. So this biomarker is not suitable for PoC study.

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

9

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

Analytical batch-effect impacts biomarker confirmation: Example

+ non-responders (training) x responders (training)

• non-responders (test)
• responders (test)

Before Normalization Before normalization, all “responders” are incorrectly predicted. After Normalization Normalization results in significant improvement, although far from perfect.

• Due to other issues (more heterogeneity in external set).

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

10

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

x Healthy (training) + Disease (training)

• Healthy (test)

x Disease (test)

Whole Gx PD Healthy Internal 96% 92% External 100% 100%

Optimal signature derived from the entire genomic array.

Biological relevance, assay availability, etc. Example

Targeted signature performs almost as well (in this example), and is more likely to be accepted for routine implementation.

Biomarker signatures from the whole genome may include genes that are not in the biological pathway, or sensitive assays may not be available.

Targeted PD Healthy Internal 94% 99% External 90% 100%

Signature derived from only a subset of genes in the biological pathway and for which sensitive assays were available

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

11

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

• Using same data to identify and evaluate a biomarker signature

will inflate the performance metrics (e.g., ROC AUC).

• Cross-Validation/Resampling methods help reduce the bias.
• k-fold cross-validation (CV):

– Original data divided randomly into k equal parts

• If N=100, k=5, obtain 5 random subsets of 20 each.

– Leave first part out, “train” on the remaining, “test” on the left-out. – Repeat this for each of the other parts; – Aggregate predictions from all left-out parts. – Calculate performance (e.g., sensitivity/specificity, p-value, …) – Repeat this procedure 25 times. Report Mean & SD of the metrics.

Biomarker Performance Evaluation Internal Validation

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

12

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

• Example of Questionable results:

– Dave et al. "Prediction of survival in follicular lymphoma based on molecular features of tumor infiltrating cells". NEJM, Nov. 18, 2004

• vol. 35set 2:2159-2169

– Reasons are explained and illustrated at:

• http://www-stat.stanford.edu/~tibs/FL/report/index.html

Unfortunately, poor cross-validation is quite common in biomarker publications. Can’t take publication/literature claims for granted.

Biomarker Performance Evaluation Internal Validation (contd.)

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

13

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

Biomarker Performance Evaluation External Validation

• After rigorous internal cross-validation, test the signatures in

independent external cohorts.

– Should adequately represent the target population with respect to several features (gender, race, age, disease severity, …)

• Samples in training & external sets are seldom run together.
• So batch-effect normalization may be necessary.
• 1. Normalize the training & external data.

– A method that works well in my experience: Eigen-Strat.

• 2. Apply previously derived signature on the normalized training set.
• 3. Use this model on normalized external data to predict the response.

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

14

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

Biomarker performance biased by improper Cross-Validation

6-marker proteomic multiplex signature for possible use in selecting patients for a Clinical Trial Predictive Accuracy:

• Internal Cross-Validation:
• No CV: 84%
• Partial CV: 72%
• Full CV: 65%
• External Validation (new study): 63%

7 8 9 10 11 12 Canonical2 C D 2 3 4 5 6 7 8 Canonical1

+ Disease Progression

• No Progression

Example 1: Evaluation of Biomarker Performance

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

15

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

Example 2: Evaluation of Biomarker Performance

4-SNP Genotype Signature for Predicting Patient Response to Treatment

• Derived from a large genotype array (100s of SNPs) via a Statistical Algorithm

Signature Positive: SNP-1 ≠ WT,SNP-2 ≠ WT, SNP-3 = WT, SNP-4 ≠ WT

• Patients in this Signature Positive group are expected to respond better.

p-value of Treatment Effect in Signature Positive vs. Negative:

• Internal Validation:
• No Cross-Validation: p < 0.0001
• 10-fold Cross-Val: p = 0.06
• External Validation (independent

clinical study): p =0.1

200 400 600 800 1000 0.0 0.2 0.4 0.6 0.8 1.0

Signature Positive Signature Negative

Time to Death (days) Survival Probability

Improper Cross-Validation exaggerates biomarker performance.

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

16

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

Robustness

Frequency 0.0 0.2 0.4 0.6 0.8 1.0 200 600 Frequency 0.0 0.2 0.4 0.6 0.8 1.0 200 400

Example: 5-marker Signature for identifying patients more likely to respond to

• treatment. Robustness of this signature is evaluated via Simulations.

15% CV & 30% random noise are artificially added to the original data. Distribution of p-values for Treatment Effect evaluated via 1000 iterations. During a study, additional variability can be introduced (unavoidable factors)

• changes in reagents, instruments, operators, sample collection/storage, …
• This is typically not accounted for during biomarker validation/evaluation.

Additional 15% CV

p-value

p-value

Median p-value = 0.009

p-value

p-value

Median p-value = 0.103

Additional 30% CV

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

17

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

Translation

Biomarker Signature derived & evaluated in male cancer patients Confirmed via external validation on same population Same Biomarker Signature does not perform well when tested in a different study (females, older age group, more severe cancer)

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Surviving 100 200 300 400 500 600 700 800 900 PFS_DUR

Time to Death (days) Survival Probability Signature Positive Signature Negative

p = 0.1

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Surviving 100 200 300 400 500 600 700 800 900 1100 OS_DUR

Time to Death (days) Survival Probability Signature Positive Signature Negative

p < 0.001

This gets more challenging between animals & humans!

• V. Devanarayan, Ph.D.

Exploratory Statistics, Abbott GPRD

18

EMA Workshop on Pharmacogenomics – From Science to Clinical Care October 8-9, 2012

Summary

• For most diseases & treatments, biomarkers are critical

for clinical drug development.

– Non-responders, disease progression, safety monitoring, …

• Some practical challenges:
• 1. Variability (Analytical & Biological)
• 2. Biological Relevance, Assay availability, etc.
• 3. Predictive performance evaluation
• Internal Validation (cross-validation methods)
• External Verification
• 4. Robustness, Reproducibility, etc.
• 5. Translation (species, demographics, disease subtypes, etc.)
• Consideration of these & other challenges is critical for

successful biomarker strategy.