Plan GRADE background two steps confidence in estimates (quality - - PowerPoint PPT Presentation

Plan

• GRADE background
• two steps

– confidence in estimates (quality of evidence) – strength of recommendation

• quality and strength can differ
• profiles and summary of findings
• exercises in applying GRADE

• any experience participating in

guideline panels?

• Is grading recommendations a good

idea?

• Why?
• experience with grading

– systems used?

Grading good idea, but which grading system to use?

• many available

– Australian National and MRC – Oxford Center for Evidence-based Medicine – Scottish Intercollegiate Guidelines (SIGN) – US Preventative Services Task Force – American professional organizations

• AHA/ACC, ACCP, AAP, Endocrine society, etc....
• cause of confusion, dismay

Common international grading system?

• GRADE (Grades of recommendation,

assessment, development and evaluation)

• international group

– Australian NMRC, SIGN, USPSTF, WHO, NICE, Oxford CEBM, CDC, CC

• ~ 30 meetings over last 13 years
• (~10 – 70 attendants)

Grading system – for what?

• interventions

– management strategy 1 versus 2

• what grade is not about

– individual studies (body of evidence)

What GRADE is not primarily about

• diagnostic accuracy questions

– in lung cancer, what is the accuracy of CT scanning of the mediastinum

• what it is about: diagnostic impact

– does use of CT scanning improve outcomes

• prognosis

70+ Organizations

9

2005 2006 2007 2008 2009 2010 2011

GRADE uptake

What are we grading?

• two components
• confidence in estimate of effect

adequate to support decision (quality

• f body of evidence)
• high, moderate, low, very low
• strength of recommendation
• strong and weak

Confidence in estimate (quality of evidence)

no confidence totally confident High Moderate Low Very Low

Structured question

• patients:

– women considering breast cancer screening – 50 to 74 – no  risk genetic mutation chest radiation

• intervention

– film mammography

• alternative

– no screening

Need to define all patient-important outcomes and evaluate their importance

• desirable consequences

– reduction in breast cancer mortality

• undesirable consequences

– false positive screening results - anxiety – invasive procedures from positive results – complications of invasive procedures – unnecessary diagnosis and treatment

Determinants of confidence

• RCTs start high
• observational studies start low
• what can lower confidence?
• risk of bias
• inconsistency
• indirectness
• imprecision
• publication bias

Risk of Bias

• well established

– concealment – intention to treat principle observed – blinding – completeness of follow-up

• more recent

– selective outcome reporting bias – stopping early for benefit

Consistency of results

• if inconsistency, look for explanation

– patients, intervention, outcome, methods

• judgment of consistency

– variation in size of effect – overlap in confidence intervals – statistical significance of heterogeneity – I2

Relative Risk with 95% CI for Vitamin D Non-vertebral Fractures

Relative Risk with 95% CI for Vitamin D (Non-Vertebral Fractures, Dose >400)

Relative Risk with 95% CI for Vitamin D (Non-Vertebral Fractures, Dose = 400)

Confidence judgments: Directness

• populations

– older, sicker or more co-morbidity

• interventions

– warfarin in trials vs clinical practice

• outcomes

– important versus surrogate outcomes – glucose control versus CV events

Flatulence

Figure 6: Hierarchy of outcomes according to their patient-importance to assess the effect of phosphate lowering drugs in patients with renal failure and hyperphophatemia

Importance

• f endpoints

Critical for decision making Important, but not critical for decision making Of low patient- importance

2 5 Pain due to soft tissue Calcification / function 6 Fractures 7 Myocardial infarction 8 Mortality 9 3 4 1

Coronary calcification Ca2+/P- Product

Surrogates of declining importance

Bone density Ca2+/P- Product Soft tissue calcification Ca2+/P- Product Lower by one level for indirectness Lower by two levels for indirectness

Alendronate Risedronate Placebo

Directness

interested in A versus B available data A vs C, B vs C

Imprecision

• small sample size

– small number of events

• wide confidence intervals

– uncertainty about magnitude of effect

• how do you decide what is too wide?
• primary criterion:

– would decisions differ at ends of CI

Precision

• atrial fib at risk of stroke
• warfarin increases serious gi bleeding

– 3% per year

• 1,000 patients 1 less stroke

– 30 more bleeds for each stroke prevented

• 1,000 patients 100 less strokes

– 3 strokes prevented for each bleed

• where is your threshold?

– how many strokes in 100 with 3% bleeding?

1.0%

1.0%

1.0%

1.0%

0.5% 1.0%

Example: clopidogrel or ASA?

• pts with threatened stroke
• RCT of clopidogrel vs ASA

– 19,185 patients

• ischaemic stroke, MI, or vascular death

compared

– 939 events (5·32%) clopidogrel – 1021 events (5·83%) with aspirin

• RR 0.91 (95% CI 0.83 – 0.99) (p=0·043)
• rate down for precision?

1.0%

Clopidogrel or ASA for threatened vascular events RCT 19,185 patients 1.7% - 0.9 – 0.1% RR 0.91 (95% CI 0.83 – 0.99)

Non-inferiority

Non-inferiority

Non-inferiority

Publication bias

• high likelihood could lower quality
• when to suspect
• number of small studies
• industry sponsored

Funnel Plot

Fish oil on mortality

What can raise confidence?

• large magnitude can rate up one level

– very large two levels

• common criteria

– everyone used to do badly – almost everyone does well – quick action

• hip replacement for hip osteoarthritis

Dose-response gradient

• childhood lymphoblastic leukemia
• risk for CNS malignancies 15 years after

cranial irradiation

• no radiation: 1% (95% CI 0% to 2.1%)
• 12 Gy: 1.6% (95% CI 0% to 3.4%)
• 18 Gy: 3.3% (95% CI 0.9% to 5.6%).

Confidence assessment criteria

Quality Assessment Summary of Findings Quality Relative Effect (95% CI) Absolute risk difference Outcome Number of participants (studies) Risk of Bias Consistency Directness Precision

Publication Bias

Myocardial infarction 10,125 (9) No serious limitations No serious imitations No serious limitations No serious limitations Not detected High 0.71 (0.57 to 0.86) 1.5% fewer (0.7% fewer to 2.1% fewer) Mortality 10,205 (7) No serious limitations No serious limiations No serious limitations Imprecise Not detected Moderate 1.23 (0.98 – 1.55) 0.5% more (0.1% fewer to 1.3% more) Stroke 10,889 (5) No serious limitaions No serious limitations No serious limitations No serious limitations Not detected High 2.21 (1.37 – 3.55) 0.5% more (0.2% more to 1.3% more0

Beta blockers in non-cardiac surgery

Population

• No. of

participants (trials) † Higher PEEP Lower PEEP Adjusted Relative Risk (95% CI; P-value) ‡ Adjusted Absolute Risk Difference (95% CI) Quality Patients with ARDS 1892 (3) 324/951 (34.1%) 368/941 (39.1%) 0.90 (0.81 to 1.00; 0.049)

• 3.9% (-7.4% to -0.04%)

High Patients without ARDS 404 (3) 50/184 (27.2%) 41/220 (18.6%) 1.37 (0.98 to 1.92; 0.065) 6.9% (-0.4% to 17.1%) Moderate (imprecision)

High versus low PEEP in ALI and ARDS

Overall level of evidence

• most systems just use evidence about

primary benefit outcome

• but what about others (risk)?
• what to do?
• options

– ignore all but primary – weakest of any outcome – some blended approach – weakest of critical outcomes

Strength of Recommendation

• strong recommendation

– benefits clearly outweigh risks/hassle/cost – risk/hassle/cost clearly outweighs benefit

• what can downgrade strength?
• low confidence in estimates
• close balance between up and downsides

Risk/Benefit tradeoff

• aspirin after myocardial infarction

– 25% reduction in relative risk – side effects minimal, cost minimal – benefit obviously much greater than risk/cost

• warfarin in low risk atrial fibrillation

– warfarin reduces stroke vs ASA by 50% – but if risk only 1% per year, ARR 0.5% – increased bleeds by 1% per year

Strength of Recommendations

Aspirin after MI – do it Warfarin rather than ASA in Afib

• - probably do it
• - probably don’t do it

Significance of strong vs weak

• variability in patient preference

– strong, almost all same choice (> 90%) – weak, choice varies appreciably

• interaction with patient

– strong, just inform patient – weak, ensure choice reflects values

• use of decision aid

– strong, don’t bother – weak, use the aid

• quality of care criterion

– strong, consider – weak, don’t consider

1

LQE in a life-threatening situation Fresh frozen plasma and intracranial bleed

2

LQoE benefit and HQoE suggests harm Head-to-toe CT/MRI screening for cancer.

3

LQoE suggests equivalence, HQoE less harm for one alternative Helicobacter pylori eradication early stage gastric MALT lymphoma

4

HQoE suggests equivalence, LQoE suggests harm in one alternative ACEI in hypertension in women planning conception and in pregnancy.

5

HQoE suggests benefit in one

• utcome, LQoE suggests harm

in more highly valued

• utcome

Testosterone in males with or at risk of prostate cancer

1

LQE in a life-threatening situation Fresh frozen plasma and intracranial bleed

2

LQoE benefit and HQoE suggests harm Head-to-toe CT/MRI screening for cancer.

3

LQoE suggests equivalence, HQoE less harm for one alternative Helicobacter pylori eradication early stage gastric MALT lymphoma

4

HQoE suggests equivalence, LQoE suggests harm in one alternative ACEI in hypertension in women planning conception and in pregnancy.

5

HQoE suggests benefit in one

• utcome, LQoE suggests harm

in more highly valued

• utcome

Testosterone in males with or at risk of prostate cancer

1

LQE in a life-threatening situation Fresh frozen plasma and intracranial bleed

2

LQoE benefit and HQoE suggests harm Head-to-toe CT/MRI screening for cancer.

3

LQoE suggests equivalence, HQoE less harm for one alternative Helicobacter pylori eradication early stage gastric MALT lymphoma

4

HQoE suggests equivalence, LQoE suggests harm in one alternative ACEI in hypertension in women planning conception and in pregnancy.

5

HQoE suggests benefit in one

• utcome, LQoE suggests harm

in more highly valued

• utcome

Testosterone in males with or at risk of prostate cancer

1

LQE in a life-threatening situation Fresh frozen plasma and intracranial bleed

2

LQoE benefit and HQoE suggests harm Head-to-toe CT/MRI screening for cancer.

3

LQoE suggests equivalence, HQoE less harm for one alternative Helicobacter pylori eradication early stage gastric MALT lymphoma

4

HQoE suggests equivalence, LQoE suggests harm in one alternative ACEI in hypertension in women planning conception and in pregnancy.

5

HQoE suggests benefit in one

• utcome, LQoE suggests harm

in more highly valued

• utcome

Testosterone in males with or at risk of prostate cancer

1

LQE in a life-threatening situation Fresh frozen plasma and intracranial bleed

2

LQoE benefit and HQoE suggests harm Head-to-toe CT/MRI screening for cancer.

3

LQoE suggests equivalence, HQoE less harm for one alternative Helicobacter pylori eradication early stage gastric MALT lymphoma

4

HQoE suggests equivalence, LQoE suggests harm in one alternative ACEI in hypertension in women planning conception and in pregnancy.

5

HQoE suggests benefit in one

• utcome, LQoE suggests harm

in more highly valued

• utcome

Testosterone in males with or at risk of prostate cancer

Flavanoids for Hemorrhoids

• venotonic agents

– mechanism unclear, increase venous return

• popularity

– 90 venotonics commercialized in France – none in Sweden and Norway – France 70% of world market

• possibilities

– French misguided – rest of world missing out

Systematic Review

• 14 trials, 1432 patients
• key outcome

– risk not improving/persistent symptoms – 11 studies, 1002 patients, 375 events – RR 0.4, 95% CI 0.29 to 0.57

• minimal side effects
• is France right?
• what is the quality of evidence?

What can lower confidence?

• risk of bias

– lack of detail re concealment – questionnaires not validated

• indirectness – no problem
• inconsistency, need to look at the

results

Review : Phlebotonics for hemorrhoids Comparison: 01 Venotonics vs placebp Outcome: 08 Overall improvement: no improvement/some improvement Study RR (random) Weight RR (random)

• r sub-category

log[RR] (SE) 95% CI % 95% CI 01 Up to seven days Chauvenet

• 0.8916 (0.2376)

12.67 0.41 [0.26, 0.65] Cospite

• 2.2073 (0.6117)

5.51 0.11 [0.03, 0.36] Thanapongsathorn

• 0.4308 (0.2985)

11.18 0.65 [0.36, 1.17] Subtotal (95% CI) 29.36 0.37 [0.18, 0.77 Test for heterogeneity: Chi² = 6.92, df = 2 (P = 0.03), I² = 71.1% Test for overall effect: Z = 2.67 (P = 0.008) 02 Up to four w eeks Annoni F

• 1.6094 (0.7073)

4.50 0.20 [0.05, 0.80] Clyne MB

• 0.9943 (0.3983)

8.94 0.37 [0.17, 0.81] Pirard J

• 1.1712 (0.3086)

10.94 0.31 [0.17, 0.57] Thanapongsathorn

• 1.1087 (1.1098)

2.18 0.33 [0.04, 2.91] Thorp 0.2624 (0.3291) 10.46 1.30 [0.68, 2.48] Titapan

• 0.8916 (0.3691)

9.56 0.41 [0.20, 0.85] Wijayanegara

• 0.5978 (0.1375)

14.97 0.55 [0.42, 0.72] Subtotal (95% CI) 61.54 0.48 [0.32, 0.72 Test for heterogeneity: Chi² = 13.87, df = 6 (P = 0.03), I² = 56.7% Test for overall effect: Z = 3.57 (P = 0.0004) 03 Further than four w eeks Godeberg

• 1.7719 (0.3906)

9.10 0.17 [0.08, 0.37] Subtotal (95% CI) 9.10 0.17 [0.08, 0.37 Test for heterogeneity: not applicable Test for overall effect: Z = 4.54 (P < 0.00001) Total (95% CI) 100.00 0.40 [0.29, 0.57 Test for heterogeneity: Chi² = 28.66, df = 10 (P = 0.001), I² = 65.1% Test for overall effect: Z = 5.14 (P < 0.00001) 0.001 0.01 0.1 1 10 100 1000 Favours treatment Favours control

Publication bias?

• size of studies

– 40 to 234 patients, most around 100

• all industry sponsored

Review : Phlebotonics for hemorrhoids Comparison: 01 Venotonics vs placebp Outcome: 08 Overall improvement: no improvement/some improvement 0.001 0.01 0.1 1 10 100 1000 0.0 0.4 0.8 1.2 1.6 RR (fixed)

What can lower confidence?

• detailed design and execution

– lack of detail re concealment – questionnaires not validated

• inconsistency

– almost all show positive effect, trend – heterogeneity p < 0.001; I2 65.1%

• indirectness
• imprecision

– RR 0.4, 95% CI 0.29 to 0.57

• publication bias

– 40 to 234 patients, most around 100

Is France right?

• recommendation

– yes – no against use

• strength

– strong – weak

TRIUMPH: Best Practices in Inpatient Glucose Monitoring UCLA Clinicians Use IT to Facilitate Innovations in Hyperglycemia Care

Hyperglycemia in the ICU

A landmark clinical trial performed in 2001 changed clinicians' views about stress hyperglycemia in the inpatient setting. The authors of that New England Journal of Medicine study, concluded that "Intensive insulin therapy to maintain blood glucose at

• r below 110 mg per deciliter reduces

morbidity and mortality among critically ill patients in the surgical intensive care unit" (Van Den Berghe, et al., 2001).

Van den Berghe, NEJM, 2001

• landmark clinical trial changed practice
• 1548 patients surgical ICU, ventilated

– intensive insulin therapy vs conventional

• interim analysis at three month intervals
• p < 0.01 (“designed to allow early

termination”)

• stopped after 4th interim analysis

– 98 deaths

Van den Berghe, NEJM, 2001

• ICU mortality

– 35 of 744 (4.6%) in intensive insulin – 63 of 765 (8.0%) in conventional

• RR 0.58 (95% CI 0.38 to 0.78)
• hypoglycemia < 40 mg/dl (2.2 mmol/l)

– RR 6.65 (95% CI 2.83 – 15.62)

GRADE assessment

• precision

– confidence intervals look OK but... – OIS 6,838 vs 1,548

• inconsistency

– no problem

GRADE assessment

• risk of bias

– unblinded – conducted by enthusiasts – no documentation of co-intervention – stopped early

Five vs Four Courses of Therapy for Acute Myeloid Leukemia

Wheatley K, Clayton D. Controlled Clinical Trials 2003;24:66

Five vs Four Courses of Therapy for Acute Myeloid Leukemia

Wheatley K, Clayton D. Controlled Clinical Trials 2003;24:66

Five vs Four Courses of Therapy for Acute Myeloid Leukemia

Wheatley K, Clayton D. Controlled Clinical Trials 2003;24:66

True beneficial effect No effect Stopping boundary

True beneficial effect No effect Stopping boundary Look after every patient or event stop

True beneficial effect No effect Stopping boundary Interim analyses every q patients or events stop stop stop

GRADE assessment

• indirectness
• single centre enthusiasts

– will this be replicable?

• publication bias

– undetected

Confidence in 42% mortality 

no confidence totally confident High Moderate Low Very Low risk of bias: no blinding, co-intervention, stopped early imprecision: well below optimal information size indirectness: single center of enthusiasts

NICE-SUGAR

• 6,104 patients in mixed medical/surgical

ICUs

• 80 to 108 (4.5 to 6.0) vs 180 (10.0) or

less

• mortality 27.5% in IIT and 24.9% in

control

– RR 1.14, 95% CI 1.02 to 128, p = 0.02

• hypoglycemia < 40 (2.2) in 6.8% vs 0.5%

We recommend a blood glucose target of 140 to 180 mg/dL (7.7 to 10 mmol/L), rather than a more stringent target (eg, 80 to 110 mg/dL [4.4 to 6.1 mmol/L]) (Grade 1A). We also suggest a blood glucose target of 140 to 180 mg/dL (7.7 to 10 mmol/L), rather than a more liberal target (eg, 180 to 200 mg/dL [10 to 11.1 mmol/L]) (Grade 2C).

Conclusion

• clinicians, policy makers need summaries

– confidence in estimates – strength of recommendations

• explicit rules

– transparent, informative

• GRADE

– simple, transparent, systematic – increasing wide adoption