Overview of Outcomes Research Methods for Imagers Stella Kang, MD, - - PowerPoint PPT Presentation

Stella Kang, MD, MSc

Director, Comparative Effectiveness & Outcomes Research Assistant Professor Department of Radiology Department of Population Health NYU Langone Health

Overview of Outcomes Research Methods for Imagers

What are we trying to accomplish with imaging?

How do we assign value to a test?

• Value = health outcomes achieved per dollar spent.
• Requires dedicated analyses of the qualitative and

quantitative changes in health outcomes and/or efficiency.

• Why do we want to look at the outcomes of the test?
• Measures have evolved
• e.g. “What do we gain from the 6th stool guaiac?”

(Neuhauser & Lewicki, NEJM 1975)

Porter ME. What is value in health care? NEJM 2010

84% sensitivity, 90% specificity 91% sensitivity, 94% specificity 93% sensitivity, 95% specificity Value?

• 1. Use standardized measures of studying value.
• Test performance is only one factor that contributes

to value .

Test Sensitivity Specificity Cancers Found (out of 50) Total Cost Cost per Cancer 1 88% 80% 44 $60,680 $1379 2 93% 82% 46.5 $106,75 $2295 3 95% 88% 47.5 $194,83 $4101

Not bad, right?...

Test Sensitivity Specificity Cancers Found (out of 50) Total Cost Difference in Cost Difference in Cancers Incremental Cost per Additional Cancer

1 88% 80% 44 $60,680

• $1379

2 93% 82% 46.5 $106,750 $46,070 2.5 3 95% 88% 47.5 $194,830 $88,080 1.0

Test Sensitivity Specificity Cancers Found (out of 50) Total Cost Difference in Cost Difference in Cancers Incremental Cost per Additional Cancer

1 88% 80% 44 $60,680

• $1379

2 93% 82% 46.5 $106,750 $46,070 2.5 $18,428 3 95% 88% 47.5 $194,830 $88,080 1.0

Test Sensitivity Specificity Cancers Found (out of 50) Total Cost Difference in Cost Difference in Cancers Incremental Cost per Additional Cancer

1 88% 80% 44 $60,680

• $1379

2 93% 82% 46.5 $106,750 $46,070 2.5 $18,428 3 95% 88% 47.5 $194,830 $88,080 1.0 $88,020

Goal: Avoid “Flat Goal: Avoid “Flat Goal: Avoid “Flat Goal: Avoid “Flat of the

• f the
• f the
• f the curve” medicine

curve” medicine curve” medicine curve” medicine QALY Cost

MRI ++ MRI + MRI CT $

QALYs QALYs

$ $/QALY good $/QALY poor

10

Purposes of CEA for healthcare

• Guide public health practice
• Guide clinical practice
• Inform funding decisions or reimbursement rate for

interventions

• Determine how to allocate scarce resources

SMDM-ISPOR task force

• 2. Assess the test’s impact on outcomes: compare the

diagnostic and treatment options.

Cost Effectiveness Plane

Cost-effectiveness ratio calculated Cost- effectiveness ratio calculated Treatment is dominated Treatment dominates

• ther options

(-) Difference in Costs (+) (-) Difference in Effectiveness (+)

Routine care: the “old way” or status quo

Goal: Avoid “Flat Goal: Avoid “Flat Goal: Avoid “Flat Goal: Avoid “Flat of the

• f the
• f the
• f the curve” medicine

curve” medicine curve” medicine curve” medicine QALY Cost

MRI ++ MRI + MRI CT $

QALYs QALYs

$ $/QALY favorable $/QALY unfavorable

14

• 2. Assess the test’s impact on outcomes:
• Life expectancy (LE)
• Quality adjusted life expectancy (QALE)
• Costs (test + all downstream costs)

Measuring Economic Consequences

Changes in health status

Numerator “Costs” Denominator “Health Effects”

Costs depending upon perspective

Diagnostic Test/ Intervention Changes in health status

Cumulative $$$

• If the threshold of test positivity changes, the

result can be a difference in patient outcomes.

• Determinants of the optimal criterion for a

positive test result:

• Pre-test probability of disease
• The benefit of a correct diagnosis (true

positive)

• The harms associated with false-positive

results

Weigh Trade-offs

A Worked Example: Decision Analysis

• 10-15% U.S. adults with gallstones; $6 billion in annual costs.
• MRCP: excellent sensitivity and specificity, comparable with

Endoscopic Ultrasound detection of choledocholithiasis.

• MRCP may spare patients without choledocholithiasis an

unnecessary endoscopy (and potential complications).

• MRI can also evaluate other potential causes of biliary
• bstruction.

Adapted from Maple GIE 2010

Clinical Decision Rule: Is it good enough?

• Clinical decision rule exists for diagnostic triage in

acute biliary obstruction. But emergency, surgical, medicine services do not follow the algorithm.

• When is broad recommendation for MRI cost-

effective, and when is it better to risk-stratify the diagnostic evaluation?

Diagnostic Testing for Bile Ducts

• Formulate the question:

What is the decision? What are the trade-offs of each choice?

• Quantify comparative costs and effectiveness of ≥2

diagnostic or treatment strategies.

Decision Analytic Modeling

MRCP vs. Risk-stratified Testing for Suspected Acute Biliary Obstruction

What is the cost effectiveness of the ASGE risk stratification guidelines

• vs. MRCP-based management of patients with suspected acute biliary
• bstruction?
• Should everyone get MRCP if acute biliary obstruction is suspected?
• Clinically risk-stratified diagnostic testing?
• Contrast v. non-contrast MRI/MRCP?
• Downside of risk-stratified approach:
• Low risk: missed choledocholithiasis, biliary strictures or cancer;
• High risk: unnecessary ERCP

Kang SK et al. Radiology. 2017.

Model Population:

• Base case analysis: 50-year-old men with symptomatic

gallstones and possibly acute biliary obstruction.

• Men at 40 and 65 years of age; women at 40, 50, 65 years of

age.

• No known malignancy, chronic pain, or painless jaundice.

CEA: Acute biliary obstruction

• Formulate the question:
• What is the decision? What happens with each choice?
• Quantify comparative costs and effectiveness of ≥2

diagnostic or treatment strategies.

• 1) Construct decision analytic model
• 2) Enter parameter values
• 3) Test the model and obtain results

CEA: Acute biliary obstruction

CEA: Decision Analytic Model

1) Decision tree and transition states

Test No Test Disease + Disease -

TP TN FN FP

… …

Schematic of a Decision Tree

Kang SK et al. Radiology. 2017.

• Pancreatitis
• Acute

cholecystitis

Decision Analytic Model

Death

(all-cause, cancer- specific, surgical mortality, other causes)

Sick Well Test or Intervention

• Life expectancy
• r Quality-adjusted life

expectancy

• Costs
• Number lives saved

Transition States

Decision Analytic Model

Death

(all-cause, cancer- specific, surgical mortality, other causes)

Suspected acute biliary

• bstruction

Post- treatment state

• Life expectancy
• r Quality-adjusted life

expectancy

• Costs
• Number lives saved

Post- endoscopic complication

Decision Analytic Model

Death

(all-cause, cancer- specific, surgical mortality, other causes)

Suspected acute biliary

• bstruction

Post- treatment state

• Life expectancy
• r Quality-adjusted life

expectancy

• Costs
• Number lives saved

Post- endoscopic complication

Decision Analytic Model

Death

(all-cause, cancer- specific, surgical mortality, other causes)

Suspected acute biliary

• bstruction

Post- treatment state

• Life expectancy
• r Quality-adjusted life

expectancy

• Costs
• Number lives saved

Post- endoscopic complication

Decision Analytic Model

Death

(all-cause, cancer- specific, surgical mortality, other causes)

Suspected acute biliary

• bstruction

Post- treatment state

• Life expectancy
• r Quality-adjusted life

expectancy

• Costs
• Number lives saved

Post- endoscopic complication

Parameter values: probabilities, diagnostic accuracy, costs, utilities Sources?

• Trials
• Systematic

review/Meta- analysis

• Observational

Studies

• Assess

applicability, quality of studies

Patient/Strategy LE (years) Δ LE (years) QALY (years) Δ QALY (years) Lifetime Costs ($) ICER ($/QALY) 50-year-old man ASGE-Based Management 27.302

• 16.361
• 171,014
• Non-Contrast

MRCP 27.314 0.012 16.542 0.181 172,884 10,311 Contrast- Enhanced MRI/MRCP 27.314 0.012

a

16.544 0.183 173,082 117,418

b Kang SK et al. Radiology. 2017.

Results

Results

• Model can also provide intermediate outcomes
• The increase in missed cancers was more than twofold

with use of the clinical decision rule:

• 26% of malignancies missed with clinical decision rule
• 9.6% missed cancers with use of non-contrast MRCP
• 8.7% with contrast-enhanced MRI/MRCP.

• 3. Assess different test techniques, uses, populations to

identify applications with greatest impact.

Guide the research

• Sensitivity analysis: tells us what causes model results to

change the most.

• Vary the patient characteristics, disease progression risk, test

performance etc across clinically plausible ranges.

• Again, knowing the literature helps to understand plausible

ranges.

Kang SK, Huang WC, Elkin E, et al. Radiology 2019

Example: small kidney tumors

Results Sensitivity Analysis

Kang SK, Huang WC, Elkin E, et al. Radiology 2019

Sensitivity Analysis for MRCP Study

• Model results were sensitive to age, specificity of MRCP,

probability of biliary stricture, probability of malignancy.

• Contrast-enhanced MRI if risk profile meant patient had >

15% probability of stricture.

• MRCP specificity < 91% for choledocholithiasis favors

clinical decision rule.

• Contrast-enhanced MRI > 91% sensitivity for cancer or

cost < $440 makes this test uniformly favorable.

Kang SK et al. Radiology 2017

When the imaging “is not cost effective…”Value a

• Think critically:
• From whose perspective?
• Right comparator?
• What were the flaws?
• What are the solutions?
• Value of Information
• Challenge region of the ROC
• What studies are needed?

MRCP: Room for Improvement

• Specificity is key: if less than that reported, then ASGE

recommendations are more effective.

• Sensitivity is not as important, given the disease

consequences and availability of EUS/ERCP.

• Randomized controlled trial: MRCP-first approach

versus an ERCP-first for suspected biliary obstruction

• MRCP-first arm underwent a greater number of

subsequent pancreaticobiliary procedures over 12 months vs. ERCP-first arm.

Chen YI et al. GIE 2017

Feedback Loop of Evidence

• Can also consider patient

studies if there is a clear need.

• Consider intermediate
• utcomes.

• Goal: translate imaging procedure performance to

measures of value.

• 1) Evidence synthesis: summarize test performance and

potential benefits/harms, identify research needs.

• 2) Decision science: compare tests with other procedures

and medical interventions to show incremental difference.

• 3) Compare different techniques, case mix, populations in

which the test might yield greatest clinical impact or priorities for research.

• 4) Consider other CER study design/patient study if there is

a clear need and way to do it.

Conclusions

Stella Kang, MD, MSc stella.kang@nyulangone.org