Overview of Outcomes Research Methods for Imagers Stella Kang, MD, MSc Director, Comparative Effectiveness & Outcomes Research Assistant Professor Department of Radiology Department of Population Health NYU Langone Health
What are we trying to accomplish with imaging?
How do we assign value to a test? • Value = health outcomes achieved per dollar spent. • Requires dedicated analyses of the qualitative and quantitative changes in health outcomes and/or efficiency . • Why do we want to look at the outcomes of the test? • Measures have evolved • e.g. “What do we gain from the 6th stool guaiac?” (Neuhauser & Lewicki, NEJM 1975) Porter ME. What is value in health care? NEJM 2010
84% sensitivity, 90% specificity 91% sensitivity, 94% specificity 93% sensitivity, 95% specificity Value?
1. Use standardized measures of studying value. • Test performance is only one factor that contributes to value .
Test Sensitivity Specificity Cancers Found Total Cost (out of 50) Cost per Cancer 1 88% 80% 44 $60,680 $1379 2 93% 82% 46.5 $106,75 $2295 0 3 95% 88% 47.5 $194,83 $4101 0 Not bad, right?...
Test Sensitivity Specificity Cancers Total Cost Difference Difference Incremental Found in Cost in Cancers Cost per (out of Additional 50) Cancer 1 88% 80% 44 $60,680 -- -- $1379 2 93% 82% 46.5 $106,750 $46,070 2.5 3 95% 88% 47.5 $194,830 $88,080 1.0
Test Sensitivity Specificity Cancers Total Cost Difference Difference Incremental Found in Cost in Cancers Cost per (out of Additional 50) Cancer 1 88% 80% 44 $60,680 -- -- $1379 2 93% 82% 46.5 $106,750 $46,070 2.5 $18,428 3 95% 88% 47.5 $194,830 $88,080 1.0
Test Sensitivity Specificity Cancers Total Cost Difference Difference Incremental Found in Cost in Cancers Cost per (out of Additional 50) Cancer 1 88% 80% 44 $60,680 -- -- $1379 2 93% 82% 46.5 $106,750 $46,070 2.5 $18,428 3 95% 88% 47.5 $194,830 $88,080 1.0 $88,020
Goal: Avoid “Flat Goal: Avoid “Flat Goal: Avoid “Flat of the Goal: Avoid “Flat of the of the of the curve” medicine curve” medicine curve” medicine curve” medicine MRI ++ MRI + QALYs MRI $ QALYs QALY $/QALY poor $ CT $/QALY good Cost 10
Purposes of CEA for healthcare • Guide public health practice • Guide clinical practice • Inform funding decisions or reimbursement rate for interventions • Determine how to allocate scarce resources SMDM-ISPOR task force
2. Assess the test’s impact on outcomes: compare the diagnostic and treatment options.
Cost Effectiveness Plane Cost-effectiveness Treatment is dominated ratio calculated (-) Difference in Costs (+) Routine care: the “old way” or status quo Cost- effectiveness ratio calculated Treatment dominates other options (-) Difference in Effectiveness (+)
Goal: Avoid “Flat Goal: Avoid “Flat Goal: Avoid “Flat of the Goal: Avoid “Flat of the of the of the curve” medicine curve” medicine curve” medicine curve” medicine MRI ++ MRI + QALYs MRI $ QALYs QALY $/QALY unfavorable $ CT $/QALY favorable Cost 14
2. Assess the test’s impact on outcomes: - Life expectancy (LE) - Quality adjusted life expectancy (QALE) - Costs (test + all downstream costs)
Measuring Economic Consequences Numerator “Costs” Cumulative $$$ Costs depending upon perspective Diagnostic Test/ Intervention Changes in health status Denominator Changes in health status “Health Effects”
Weigh Trade-offs • If the threshold of test positivity changes, the result can be a difference in patient outcomes . • Determinants of the optimal criterion for a positive test result: • Pre-test probability of disease • The benefit of a correct diagnosis (true positive) • The harms associated with false-positive results
A Worked Example: Decision Analysis • 10-15% U.S. adults with gallstones; $6 billion in annual costs. • MRCP: excellent sensitivity and specificity, comparable with Endoscopic Ultrasound detection of choledocholithiasis. • MRCP may spare patients without choledocholithiasis an unnecessary endoscopy (and potential complications). • MRI can also evaluate other potential causes of biliary obstruction.
Clinical Decision Rule: Is it good enough? Adapted from Maple GIE 2010
Diagnostic Testing for Bile Ducts • Clinical decision rule exists for diagnostic triage in acute biliary obstruction. But emergency, surgical, medicine services do not follow the algorithm. • When is broad recommendation for MRI cost- effective, and when is it better to risk-stratify the diagnostic evaluation?
Decision Analytic Modeling • Formulate the question: What is the decision? What are the trade-offs of each choice? • Quantify comparative costs and effectiveness of ≥2 diagnostic or treatment strategies.
MRCP vs. Risk-stratified Testing for Suspected Acute Biliary Obstruction What is the cost effectiveness of the ASGE risk stratification guidelines vs. MRCP-based management of patients with suspected acute biliary obstruction? • Should everyone get MRCP if acute biliary obstruction is suspected? - Clinically risk-stratified diagnostic testing? - Contrast v. non-contrast MRI/MRCP? • Downside of risk-stratified approach: - Low risk: missed choledocholithiasis, biliary strictures or cancer; - High risk: unnecessary ERCP Kang SK et al. Radiology. 2017.
CEA: Acute biliary obstruction Model Population: • Base case analysis: 50-year-old men with symptomatic gallstones and possibly acute biliary obstruction. • Men at 40 and 65 years of age; women at 40, 50, 65 years of age. • No known malignancy, chronic pain, or painless jaundice.
CEA: Acute biliary obstruction • Formulate the question: • What is the decision? What happens with each choice? • Quantify comparative costs and effectiveness of ≥2 diagnostic or treatment strategies. • 1) Construct decision analytic model • 2) Enter parameter values • 3) Test the model and obtain results
CEA: Decision Analytic Model 1) Decision tree and transition states TP … Disease + FN Test TN … Disease - FP No Test
Schematic of a Decision Tree - Pancreatitis - Acute cholecystitis Kang SK et al. Radiology. 2017.
Decision Analytic Model Transition States Well Sick -Life expectancy Death or Quality-adjusted life (all-cause, cancer- Test or specific, surgical expectancy Intervention mortality, other causes) -Costs -Number lives saved
Decision Analytic Model Post- Suspected endoscopic acute biliary obstruction complication -Life expectancy Death Post- or Quality-adjusted life (all-cause, cancer- treatment specific, surgical expectancy state mortality, other causes) -Costs -Number lives saved
Decision Analytic Model Post- Suspected endoscopic acute biliary obstruction complication -Life expectancy Death Post- or Quality-adjusted life (all-cause, cancer- treatment specific, surgical expectancy state mortality, other causes) -Costs -Number lives saved
Decision Analytic Model Post- Suspected endoscopic acute biliary obstruction complication -Life expectancy Death Post- or Quality-adjusted life (all-cause, cancer- treatment specific, surgical expectancy state mortality, other causes) -Costs -Number lives saved
Decision Analytic Model Post- Suspected endoscopic acute biliary obstruction complication -Life expectancy Death Post- or Quality-adjusted life (all-cause, cancer- treatment specific, surgical expectancy state mortality, other causes) -Costs -Number lives saved
Parameter values: probabilities, diagnostic accuracy, costs, utilities Sources? - Trials -Systematic review/Meta- analysis -Observational Studies -Assess applicability , quality of studies
Results Patient/Strategy LE Δ LE QALY Δ QALY Lifetime ICER (years) (years) (years) (years) Costs ($) ($/QALY) 50-year-old man ASGE-Based 27.302 -- 16.361 -- 171,014 -- Management Non-Contrast 27.314 0.012 16.542 0.181 172,884 10,311 MRCP a b Contrast- 27.314 0.012 16.544 0.183 173,082 117,418 Enhanced MRI/MRCP Kang SK et al. Radiology. 2017.
Results • Model can also provide intermediate outcomes • The increase in missed cancers was more than twofold with use of the clinical decision rule: • 26% of malignancies missed with clinical decision rule • 9.6% missed cancers with use of non-contrast MRCP • 8.7% with contrast-enhanced MRI/MRCP.
3. Assess different test techniques, uses, populations to identify applications with greatest impact.
Guide the research • Sensitivity analysis: tells us what causes model results to change the most. • Vary the patient characteristics, disease progression risk, test performance etc across clinically plausible ranges. • Again, knowing the literature helps to understand plausible ranges.
Example: small kidney tumors Kang SK, Huang WC, Elkin E, et al. Radiology 2019
Results Sensitivity Analysis Kang SK, Huang WC, Elkin E, et al. Radiology 2019
Recommend
More recommend
