Outline Uses of biomarkers Methods for biomarker identification and - - PowerPoint PPT Presentation

Methods for biomarker identification and model interrogation and statistical approaches for model comparisons

Lee Lancashire Bioinformatics Group Leader: Compandia Ltd. Visiting Scholar: Nottingham Trent University Stratified Medince: Diagnostic, Prognostic and Predictive Biomarkers in Clinical Practice University of Birmingham 30th June 2010

Outline

• Introduction into biomarkers and problems associated

with their identification

• Outline of current solutions being developed by

Compandia to overcome these issues

– Case study 1

• Introduction into statistical approaches for comparing

diagnostic models

– Case study 2

Uses of biomarkers

• Early detection screening
• Diagnosis
• Outcome risk‐ prognostic
• Treatment selection‐ predictive
• 1. Classification using biomarkers
• Binary classification

– (Instances, Class labels): (x1, y1), (x2, y2), ..., (xn, yn) – yi {0,1} ‐ valued – Classifier: provides class prediction Ŷ for an instance

• Outcomes for a prediction:

1 1 True positive (TP) False positive (FP) False negative (FP) True negative (TN)

Predicted class True class

Problems with biomarker identification

• Dimensionality

– Particularly in genomic and proteomic studies – Thousands of genes, proteins or peptides representing the profile

• f an individual
• Complexity

– Genes and proteins relate to phenotype with non‐linear relationships

Biomarker Distiller

• An advanced algorithm based on ANNs.

– Predict classes or continuous variables. – Models the outcome of the question being asked. E.g. Responder or non‐responder, patient or control. – Can cope with noise, complexity and non‐linearity found in biological data

• Comprehensive and robust data‐mining.

– For a typical gene array dataset‐ searches through 50 million model combinations for an

• ptimum solution

– Every model developed is optimised for performance on an unseen data set.

• Models predict well for new blind cases.

– Provide decision tools that are applicable to all cases that could present

• Finds an optimised solution.

– E.g. 9 genes compared with 70+ genes (comparison with other, recursive methods)

• We can gain information on a system by interrogation of this optimised model.

– Assess performance measures e.g. ROC curves, sensitivity and specificity – Ranking of cases and population structure – A probability visualisation for all cases – Response curves and surfaces for each parameter in the model. – Performance and probabilities for any new or blind cases available

Case study 1

vant Veer Data Set

70 Gene signature 83% accuracy 85% sensitivity 81% specificity Compandia

9 Gene signature 98% median accuracy 99% sensitivity 97% specificity

• Compandia’s reanalysis of this classic 24,000 gene array study delivered a

signature with far fewer genes delivering greater levels of sensitivity and specificity

• Few genes identified in Compandia’s 9 gene signature were in common with

vant Veer study

Model Performance

• Identified 9 gene signature ‐ v’ant veer 70 genes
• Predicts metastatic risk to median accuracies of 98% for blind data ‐

v’ant veer 80%

• Sensitivity: 99% ‐v’ant veer 90%
• Specificity: 97% ‐v’ant veer 65%
• Additional 19 Nature samples 100% correct.
• Secondary data NEJM 295 cases:
• Signature was independent predictor of metastases free and overall

survival in the presence of original 70 gene signature and other factors.

• Immunohistochemical validation of prinicpal prognosticator selected

aggressive subgroup of patients with poor prognosis

Lancashire et al. Breast Cancer Res Treat 2009.

Pathways Distiller

• An advanced network inference algorithm based on ANNs.
• Application of Distiller methods to systems biology.
• Turn the ANN in on itself, uses markers defining biological states to

predict other interacting markers

– ANN models predict target marker from multiple markers. – Repeated for every marker in the set of interest.

• ANN models analysed to determine strength, sign and direction of

interaction.

– E.g. Strong positive interaction occurring in both directions

• Moves beyond simple predictive genes to address the relationship

(pathway) between genes in the context of a given problem eg PPG v GPG

• Filter at high level scrutiny to reveal key nodes and interactions

Compandia – Gene Array Study

vant Veer Data Set

70 Gene signature 83% median accuracy 85% sensitivity 81% specificity Compandia

9 Gene signature 98% median accuracy 99% sensitivity 97% specificity

Gene Signature Model Top 100 Genes Pathways Distiller

Application of Pathways Distiller:­ Low Level Filter Compandia Pathways Distiller:‐ High Level Filter

• 2. Comparison of Diagnostic Models
• Often: Improvement in measure X  measure Y becomes worse
• Idea: Visualise trade‐off in a two‐dimensional plot
• Output: continuous

(instead of actual class prediction)

• Discretise by choosing

a cut‐off – f(x) ≥ c  class 1 – f(x) < c  class 0

• Trade‐off visualisations:

cutoff‐parameterised curves

Receiver Operating Characteristic Curves

• “…the probability that given two subjects, one who will develop an event and one who

will not, the model will assign a higher probability of an event to the former”.

• Trade off between true positives (sensitivity) and false positives (1‐ specificity).
• Area under the ROC curve (AUC, or c‐statistic) is an established measure of model

discrimination for binary outcomes

• High utility of a biomarker corresponds to having high (close to 1) PPV and NPV,

• Note that only event vs. non‐event comparisons are made.

Discrimination

• A model with good discrimination has an ability to

characterise or separate two or more classes of objects or events.

AUC as a poor estimator

• Discrimination and calibration are established methods for single

model assessment.

• Pencina et al and Cook allude to ROC AUC problems:

– Does not involve the original measurement scale for the biomarker – A model that predicts all events as 0.51 and all non‐events as 0.49 would have perfect discrimination

• Better measures of performance of prediction models needed?
• How do we quantify improvement in model performance introduced

by adding new biomarkers to existing models?

Comparing addition of new biomarkers

• Increase in AUC

– Not as useful as AUC itself. – No intuitive interpretation. – Very small in magnitude if powerful markers are already in the model.

• Hanley Comparison

– Calculates p value for whether addition of a biomarker leads to a statistically significant improvement.

Solution? Pencina method?

• Tests for whether predictor X1 is more concordant than predictor X2.
• For binary responses this provides several assessments of whether one set
• f predicted probabilities is better than another.
• Said to be a distinct improvement over comparing ROC areas, sensitivity,
• r specificity.
• NRI (Net Reclassification Improvement)

– Quantifies the correct movement in categories with the addition of a biomarker (upwards for events; downwards for non-events)

NRI example

• Analysis of proportions of subjects with improvement in

predicted probability

75 20 25 80 60 30 40 70

Actual 0 1 Model 1 0 Sensitivity = 70% Specificity = 60%

= 80% = 75%

Advantages of Pencina Approach

• Focuses on the distribution of scores that are of greatest

interest to clinicians i.e. those at the critical decision thresholds.

• Test whether a new biomarker moves a patient from a

zone of lower risk to a zone of higher risk‐ thus crossing the treatment threshold.

• Shows us NET improvement in discrimination by focusing
• n the treatment thresholds to better discriminate events

from non events.

• These category thresholds can be the ones deemed to

matter most clinically and if changes are not large enough to alter practices, new predictors may not have clinical impact.

Predictive Biomarker Case Study Example

• Development of Multi‐Dimensional Models of Serum IGFs using

Artificial Neural Networks as “Composite Biomarkers” to Predict Colorectal Cancer

• Aims: Can we discriminate between colorectal cancer and healthy

controls using these “biomarkers”??

• 7 Variables: Base model of Age, Gender, BMI

Additional biomarkers: Serum IGF‐I, IGF‐II, IGFBP‐2 and IGFBP‐3

• ROCs used to assess model performance
• Results as sensitivity, specificity & AUC

Results

• Prediction of colorectal cancer

Model Interrogation

• 3‐D response surfaces

– Non‐linear decision boundaries

– Determines the role the biomarkers have in sample classification – Modifies the biomarkers in a pair‐wise fashion and monitors the changing model output – Strength of this response provides an insight into how the biomarkers govern sample classification, i.e. whether they are increased or decreased with respect to sample class

Non‐linear decision boundaries

• Use response surfaces to visualise the class separation by

the ANN model….

Lancashire et al. Int J Cancer 2010.

Summary

• Important to identify optimal combinations of biomarkers:

– Increase our predictive capabilities – Eliminate noise and redundancy from a dataset – Avoids wasting resources

• Methods for assessing biomarker combinations have been developed.

– Artificial neural networks for biomarker ID – Statistical approaches for model comparisons

• Methods for interrogation and further down stream analysis have been

developed

– Response surface analysis – Biomarker interaction modelling – systems biology

Acknowledgements

• Compandia

– Dr Graham Ball – Dr Andy Sutton – Prof. Bob Rees

• Paterson Institute for Cancer Research

– Dr Andrew Renehan Lee.Lancashire@compandia.co.uk

Questions?