CIHC 2010 Computational methods for evidence-based decision support - - PowerPoint PPT Presentation

CIHC 2010 Computational methods for evidence-based decision support

in pharmaceutical decision making Tommi Tervonen

Econometric Institute, Erasmus University Rotterdam

Eindhoven, 20/10/2010

Outline

1 Introduction to regulatory drug benefit-risk analysis

Evidence-based medicine Benefit-risk analysis

2 Evidence synthesis methods for clinical trials

Meta-analysis Network meta-analysis

3 Decision aiding for pharmaceutical decisions

Utility theory Value functions MAVT SMAA

4 Aggregate Data Drug Information System (ADDIS)

Case study

A simple example of evidence-based medicine

Q: Should we advise parents to administer over the counter cough medicines for acute cough? Aims: To determine the effectiveness of over the counter (OTC) cough medicines for acute cough in children (...) Methods: Systematic review of randomised controlled trials (RCTs) (...) Results: Six trials involving 438 children met all inclusion

• criteria. Antitussives, antihistamine–decongestant

combinations, other fixed drug combinations, and antihistamines were no more effective than placebo in relieving symptoms of acute cough (...) Most drugs appeared to be well tolerated with a low incidence of mostly minor adverse effects. Conclusion: OTC cough medicines do not appear more effective than placebo in relieving symptoms of acute cough (...)

Schroeder & Fahey, BMJ, 2002

Randomized Controlled Trial (RCT)

Properly designed RCT’s provide the highest quality evidence

• n the treatments’ effects

Evidence-Based Medicine (EBM)

Evidence-based medicine aims to apply the best available evidence gained from scientific research to medical decision making A large share of decisions made by health care professionals are informed by evidence-based medicine, e.g. prescription, regulatory- and reimbursement policy decisions Although the scientific evidence is transparent and achieved with methodological rigour, the actual decisions are often unstructured, ad hoc and lack transparency as the treatment benefit-risk valuation is not explicit

Application of EBM in regulatory drug benefit-risk analysis

For a drug to be granted marketing authorization, it must be proven efficant, safe, and have a sufficient benefit-risk (BR) profile compared to other drugs already in the market

Knowledg e Tim e S ev erity of condition/unm et m edica l need a c b Knowledge s urfa ce Eichler & al., Nature Drug Disc, 2008

Drug benefit-risk analysis

BR analysis should include all relevant evidence, and therefore apply (network) meta-analysis

Drug benefit-risk analysis

Problems

1 Inclusion of all relevant

evidence in the meta-analysis is not guaranteed

2 The BR analysis is

unstructured and non-transparent

Approach taken in ADDIS

Separate clinical data (measurements) from the value judgements (MCDA) Include all data present in the original analysis (imprecise measurements) Provide metrics for decision uncertainty Enable model generation for re-applicability

Outline

1 Introduction to regulatory drug benefit-risk analysis

Evidence-based medicine Benefit-risk analysis

2 Evidence synthesis methods for clinical trials

Meta-analysis Network meta-analysis

3 Decision aiding for pharmaceutical decisions

Utility theory Value functions MAVT SMAA

4 Aggregate Data Drug Information System (ADDIS)

Case study

Meta-analyses

Often clinical trials are under-powered to reach statistical significance Sometimes different trials give different results (e.g. due to heterogenous population) → need to synthesize, that is, to meta-analyze the existing evidence base

Example meta-analysis

Hansen et al. Ann Intern Med 2005;143:415-426

Fixed effect meta-analysis

A fixed effect model is based on assumption that every study is evaluating a common treatment effect. That means the effect of treatment, allowing for the play of chance, was the same in all studies. Another way of explaining this is to imagine that if all the studies were infinitely large they’d give identical results. The resulting summary treatment effect estimate is this one ’true’ or ’fixed’ treatment effect, and the confidence interval describes how uncertain we are about the estimate.

From fixed to random effects

Sometimes this underlying assumption of a fixed effect meta-analysis (i.e. that diverse studies can be estimating a single effect) is too simplistic The alternative approaches to meta-analysis are (i) to try to explain the variation or (ii) to use a random effects model A “group” effect is random if we can think of the levels we

• bserve in that group to be samples from a large population

(e.g. when collecting data from different medical centers, the “center” can be thought of as random)

Random effects meta-analysis

Assumes that the true treatment effects in the individual studies may be different from each other. That means there is no single number to estimate in the meta-analysis, but a distribution of numbers. The most common random effects model assumes that these different true effects are normally distributed. The meta-analysis therefore estimates the mean and standard deviation of the different effects. In inverse variance random effects model, the weight of each study is proportional to the inverse of its variance

Fixed- and random effects models

Meta-analysis limits

Hansen et al. (2005) systematic review: 46 studies comparing n = 10 second-generation AD In total, 20 comparisons are available Out of n(n−1)

2

= 45 possible comparisons 3 meta-analyses are performed

Meta-analysis limits

Fluoxetine Paroxetine (8) Sertraline (5) Venlafaxine (6)

Hansen et al. Ann Intern Med 2005;143:415-426

Meta-analysis limits

Paroxetine Bupropion (1) Duloxetine (1) Mirtazapine (2) Venlafaxine (2) Sertraline (3) (1) Escitalopram (2) Fluoxetine (8) (2) (1) (1) (7) Fluvoxamine (2) (6) Citalopram (1) (3) (1) (2) (1) (2)

Meta-analysis with more than 2 treatments?

Fluoxetine (baseline) 1.00 (1.00 - 1.00) Paroxetine 1.09 (0.97 - 1.21) Sertraline 1.10 (1.01 - 1.20) Venlafaxine 1.12 (1.02 - 1.23) ... ... ... How likely is it that Venlafaxine has the greatest efficacy? What happens if we choose another baseline?

Other studies included → possibly different results

Not all drugs can be included (escitalopram) We’re “double counting” multi-arm trials

Paroxetine Bupropion (1) Duloxetine (1) Mirtazapine (2) Venlafaxine (2) Sertraline (3) (1) Escitalopram (2) Fluoxetine (8) (2) (1) (1) (7) Fluvoxamine (2) (6) Citalopram (1) (3) (1) (2) (1) (2)

Network meta-analysis

Paroxetine Bupropion (1) Duloxetine (1) Mirtazapine (2) Venlafaxine (2) Sertraline (3) (1) Escitalopram (2) Fluoxetine (8) (2) (1) (1) (7) Fluvoxamine (2) (6) Citalopram (1) (3) (1) (2) (1) (2)

Include all evidence in one analysis

Network meta-analysis

Is an extension of normal meta-analysis Allows comparison of ≥ 2 alternatives

Integrating direct and indirect evidence While checking for (in-)consistencies

A.K.A.: Mixed/Multiple Treatment Comparison (MTC)

Network meta-analysis models

Is a Bayesian hierarchical model That can be estimated with a Markov Chain Monte Carlo software (BUGS, JAGS) ADDIS (http://drugis.org/) can do it for you

Network meta-analysis: model

Main components are:

1 Individual effect estimates 2 Linkage of the effect estimates (relative effects) 3 Random effects for the relative effects 4 Priors 5 Linkage of the relative effects

Level 1 (likelyhood)

Modeling the effect rik/nik, for treatment k of study i, as a binomial proccess with success probability pik: rik ∼ Bin(pik, nik) Using MCMC simulation, the model converges to maximum joint likelyhood estimate of these pik given the data on rik and nik.

Linkage to relative effects

Apply a transformation to obtain normally distributed variable: θik = logit(pik) = log pik 1 − pik ; pik = logit−1(θik) And choose a baseline b(i) and define θik (the log odds) in terms

• f a baseline µi and relative effect δib(i)k (the log odds-ratio):

θik = µi + δib(i)k

Level 2 (random effects)

We assume the relative effects (LORs) to be normally distributed: δixy = N(dxy, σ2

xy)

And, for a three-arm trial i with treatments x, y, z, b(i) = x: δixy δixz

• ∼ N

dxy dxz

• ,
• σ2

xy

ρxy,xzσxyσxz ρxy,xzσxyσxz σ2

xz

Level 2 (random effects)

We assume the relative effects (LORs) to be normally distributed: δixy = N(dxy, σ2) And, for a three-arm trial i with treatments x, y, z, b(i) = x: δixy δixz

• ∼ N

dxy dxz

• ,
• σ2

σ2/2 σ2/2 σ2

• Under the assumption of equal variances

Level 3 (Priors)

For each of the parameters of interest, µi, djk and σ, specify priors: µi ∼ N(0, 1000) djk ∼ N(0, 1000) σ ∼ U(0, 2)

Overview

So far, this is just a random effects meta-analysis!

Consistency assumption

dyz = dxy − dxz ‘Borrow strength’ from indirect evidence. The left-hand term (dyz) is a functional parameter The right-hand terms (dxy, dxz) are basic parameters Only basic parameters are stochastic

Inconsistency Factors

Introduce an ‘inconsistency factor’ for each functional parameter: dyz = dxy − dxz + wxyz After the model has converged, we test wxyz = 0 Comparing inconsistency and consistency models on model fit is also useful Correctly specifying the model is tricky (but possible with 15 definitions, 5 lemmas, 3 corollaries and 4 theorems)

Example: evidence network

Cipriani & al., Lancet 2009

Example: results (LOR)

Example: results (rank probabilities)

Outline

1 Introduction to regulatory drug benefit-risk analysis

Evidence-based medicine Benefit-risk analysis

2 Evidence synthesis methods for clinical trials

Meta-analysis Network meta-analysis

3 Decision aiding for pharmaceutical decisions

Utility theory Value functions MAVT SMAA

4 Aggregate Data Drug Information System (ADDIS)

Case study

What is a decision?

In our context: a choice of the “best” treatment from a set of competing ones There exists uncertainty about the outcomes ... and how can we formally support such decisions?

• St. Petersburg paradox

Consider the following game

We flip a coin. If head comes up, the game ends. If tail comes, we flip again. You begin the game with 1 euro. After each head, we double it. On head, you get the current pot.

How much are you willing to pay to play this game? A poor person might be unwilling to pay more than a euro, the minimum possible pay-off of the game.

• St. Petersburg paradox

Expected value: E = 1 2 · 1 + 1 4 · 2 + . . . =

∞

• k=1

1 2 = ∞ But still, no-one would be willing to stake e.g. 10000000e Why don’t people behave as to maximize expected wealth?

Utility theory

Utility is dependent on the particular circumstances of the person making the estimate. ∆U ∝ ∆X X (U = utility, X = total current wealth) Individuals (should) behave as to maximize expected utility Utilities elicited with lotteries:

Utility functions

Figure: Risk averse Figure: Risk seeking

Utility functions in healthcare

HRQOL, CUI, ... Quality Adjusted Life Year (QALY) the most common measure, 1.0 = a year of full health, 0.0 = dead. E.g. a life year in wheel chair could have a value of 0.7. The QALY model requires utility independent, risk neutral, and constant proportional tradeoff behaviour QALY is often used as a scale that is translatable to monetary terms, which can be incorrect!

Problems with utility theory

Entire risk profile cannot be captured with a single number (expected utility) Utile has no meaning to most people No natural utility functions People violate axioms (e.g. framing / prospect theory)

Back from utility to value functions

Utility is a measure of preference under uncertainty Value is a measure of preference under certainty

Multi-attribute value theory (MAVT)

Functional representation F(x1, . . . , xn) = f (v1(x1), . . . , vn(xn)) In practice we usually apply an additive form F =

n

• i=1

vi(xi)wi Applicable when all attributes are mutually preferentially independent

Preferential independence

Attribute specific value functions are meaningful only if preferences for performance levels in one attribute do not depend on the performance levels of other attributes This should not to be confused with a possible correlation of alternatives’ scores! Is a property of preferences, not that of alternatives

Weights

v(x) =

n

• i=1

vi(xi)wi The weights in MAVT represent (ONLY) the importance of scale swings “How many times more important is to increase salary from 1000 to 2000 euros than to have office in 16th floor instead of 2nd floor?” As only the ratio is meaningful, they can be normalized to sum to e.g. 1

Existing BR models

O’Brien & Lynd’s benefit-risk plane for 2-criteria stochastic

• utcomes

Mussen & al’s regulatory BR assessment with MAVT Felli & al’s with binning of continuous data to categories + MAVT Tervonen & al’s with data in original format and ordinal preferences

Drug BR model with clinical data

Evaluate a set of alternative treatments with respect to

• utcome measures in the trials

Benefits = clinically relevant endpoints, Risks = adverse drug reactions (side effects) Measurements directly from trial results:

Single-study BR model: beta-distributed incidence rate Network BR model: Log-normal distributed relative effects

For odds-ratios we need to estimate/assume baseline, as the scale swing is meaningless Preferences with ordinal swing weighting (“would you rather decrease the incidence rate of diarrhea from 0.6 to 0.1 or increase HAM-D responder rate from 0.2 to 0.5?”)

SMAA - an inverse approach to MAVT

Figure: Traditional MCDA

Lahdelma & Salminen, Springer, 2010

SMAA - an inverse approach to MAVT

Figure: SMAA

Lahdelma & Salminen, Springer, 2010

Weight space

Total lack of preference information is represented by a uniform joint probability distribution of the weight space If some preference information is available, the weight space can be restricted with linear constraints

Rank acceptability index

Describes the share of different weights and criteria measurements ranking an alternative on a certain rank br

i =

• ξ∈χ

fχ(ξ)

• w∈W r

i (ξ)

fW (w) dw dξ

Central weight vector & confidence factor

Central weight vector describes the preferences of a typical DM supporting this alternative with the assumed preference model

CW’s are used for inverse approach: instead of asking preferences and giving results, answers the question “which preferences support an alternative to be the most preferred

• ne?”

Confidence factor is the probability for an alternative to be the preferred one with the preferences expressed by its central weight vector

CF measures whether the criteria measurements are accurate enough to discern the efficient alternatives

Computation

Analytical techniques based on discretizing the integrals with respect to each dimension are infeasible, so the integrals are estimated through Monte Carlo simulation 10000 simulations provide sufficient accuracy for the indices Algorithm has less-than squared mean complexity and is very fast in practice

Tervonen & Lahdelma, EJOR, 2007

Outline

1 Introduction to regulatory drug benefit-risk analysis

Evidence-based medicine Benefit-risk analysis

2 Evidence synthesis methods for clinical trials

Meta-analysis Network meta-analysis

3 Decision aiding for pharmaceutical decisions

Utility theory Value functions MAVT SMAA

4 Aggregate Data Drug Information System (ADDIS)

Case study

What is ADDIS?

Cross platform open source software for managing and analyzing aggregate clinical trial results Implements random effects- and network meta-analyses as well as benefit-risk analyses Current version 1.0, new releases every few months. Been developed for 1,5 years, and is still under development at RuG/UMCG

Problems in clinical data models

Same clinically relevant endpoint can be measured in different times (e.g. 3 or 6 weeks) and on different scales (e.g. HAM-D

• r MADRS)

Standardization of the data structures haven’t aimed for model generation → need for a minimal data model

ADDIS data model

Indication Na m e : String (Ke y) Study Na m e : String (Ke y) StudyCharacte ristic Na m e : String (Ke y) ArmCharacte ristic Na m e : String (Ke y) Outcome Me asure De finition: String (Ke y), Be nDir: { H, L} , Type : { R, C} StudyOutcome Me asure Me asure me nt Sa m ple Size : Na tura l Rate Me asure me nt Count: Na tura l ContinuousMe asure me nt Me a n: Re a l, StdDe v: Re a l Arm Tre atme nt De finition: String (Ke y) ActualArm Characte ristic Va lue : String

ActualStudyCharacte ristic

Va lue : String

⊗

ADDIS data model: example instantiation

Indication

Nam e De pre ssion Type 2 Dia be te s

Study

Nam e Indication Cole m a n ... NCT00296517 De pre ssion De pre ssion

StudyCharacte ristic

Nam e Study size Group a lloca tion Tre a tm e nt blinding Pa tie nt e ligibility crite ria

ArmCharacte ristic

Nam e Arm size Dosing Ge nde r distribution

Outcome Me asure

De finition BDir Type Re sponde rs ... Cha nge from ... He a da che Na use a Che st pa in H ... L L L R C R R R

StudyOutcome Me asure

Outcom e Me asure Study

Me asure me nt

S.O.M. Arm Sam ple Size ... ... 98

Rate Me asure me nt

Me asure m e nt Count

ContinuousMe asure me nt

Me asure m e nt Me an StdDe v

Arm

Study Tre atm e nt

Tre atme nt

De finition Pla ce bo Fluoxe tine Rosiglita zone

ActualArmCharacte ristic

Arm Arm

• Char. Value

... Dosing 20 m g/da y

ActualStudyCharacte ristic

Study

• StudyChar. Value

Cole m a n... Cole m a n... Group ... Tre a tm e nt... Ra ndom ize d Double

• blind

⊗

ADDIS Case study

Hansen & al. (Ann Intern Med, 2005) assessed safety and efficacy of four second generation antidepressants and concluded that there are “no significant differences among the drugs” In general, the assessment of antidepressants is hard; placebo effect is always present causing high uncertainty on the results Q’s:

1 How can the benefit-risk assessment of second-generation

antidepressant be structured based on evidence from the clinical trials?

2 Can we come up with something better than “no significant

differences”?

Time for ADDIS!

Conclusions

There is a current need to support drug regulatory decision making with benefit-risk models For truly evidence-based pharmacological decision support, we need to synthesize evidence with network meta-analysis Stochastic multi-criteria decision models can be generated from clinical trial contents ADDIS is the first (and only) software supporting generation and computation of network meta-analysis- and benefit-risk models Thank you for attention!