Assessing the Quality of Observational studies in ILCOR Eddy Lang - - PowerPoint PPT Presentation

Assessing the Quality of Observational studies in ILCOR

Eddy Lang Associate Professor University of Calgary Michael Sayre Professor, Emergency Medicine University of Washington Peter Morley Associate Professor University of Melbourne Russell Griffin Science and Medicine Advisor American Heart Association

Overview

• Observational studies (non-RCTs) in

evidence hierarchy

• Assessing risk of bias in observational

studies

• GRADE risk of bias tool
• Pooling data from observational studies

Evidence pyramid

4

Quality assessment criteria

Lower if… Quality of evidence

High Moderate Low Very low Study limitations (design and execution) Inconsistency Indirectness Imprecision Publication bias

Higher if… Study design

RCTs  Observational studies  Large effect (e.g., RR 0.5) Very large effect (e.g., RR 0.2) Evidence of dose-response gradient All plausible confounding… …would reduce a demonstrated effect …would suggest a spurious effect when results show no effect

It’s not fair!

• Observational studies may be best

available

• RCTs not feasible ? ethical
• Large observational studies > RCTs

6

“Categories” of quality (1)

       

Further research is very unlikely to change our confidence in the estimate of effect High

    

Low Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

    

Moderate Further research is likely to have an important impact on

• ur confidence in the estimate of effect and may change

the estimate

      

Very low Any estimate of effect is very uncertain

7

Conceptualizing quality (2)

       

We are very confident that the true effect lies close to that of the estimate of the effect. High

    

Low Our confidence in the effect is limited: The true effect may be substantially different from the estimate of the effect.

    

Moderate We are moderately confident in the estimate of effect: The true effect is likely to be close to the estimate of effect , but possibility to be substantially different.

      

Very low We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect.

Systematic review Guideline development

P I C O

Outcome Outcome Outcome Outcome Critical Important Critical Not Summary of findings & estimate of effect for each outcome Rate

• verall quality of evidence

across outcomes based on lowest quality

• f critical outcomes

RCT start high,

• bs. data start low
• 1. Risk of bias
• 2. Inconsistency
• 3. Indirectness
• 4. Imprecision
• 5. Publication

bias Grade down Grade up

• 1. Large effect
• 2. Dose

response

• 3. Confounders

Very low Low Moderate High Formulate recommendations:

• For or against (direction)
• Strong or weak (strength)

By considering:  Quality of evidence  Balance benefits/harms  Values and preferences Revise if necessary by considering:  Resource use (cost)

• “We recommend using…”
• “We suggest using…”
• “We recommend against using…”
• “We suggest against using…”

Newcastle-Ottawa Quality Assessment Scale: Cohort Studies

• Selection (4)
• Comparability (1)
• Outcome (3)

– A study can be awarded a maximum of one star for each numbered item within the Selection and outcome categories. A maximum of two stars can be given for Comparability

Selection

• 1. Representativeness of the exposed cohort

a) truly representative of the average ___________ (describe) in the community  b) somewhat representative of the average ___________ in the community  c) selected group of users eg nurses, volunteers d) no description of the derivation of the cohort

• 2. Selection of the non exposed cohort

a) drawn from the same community as the exposed cohort  b) drawn from a different source c) no description of the derivation of the non exposed cohort

• 3. Ascertainment of exposure

a) secure record (eg surgical records)  b) structured interview  c) written self report d) no description

• 4. Demonstration that outcome of interest was not present at start of study

a) yes  b) no

In the case of mortality studies, outcome of interest is still the presence

• f a disease/ incident,

rather than death; that is a statement of no history of disease or incident earns a star

Comparability

• 1. Comparability of cohorts on the basis of the

design or analysis a) study controls for ___________ (select the most important factor)  b) study controls for any additional factor (This criteria could be modified to indicate specific control for a second important factor.) 

Outcome

• 1. Assessment of outcome

a) independent blind assessment  b) record linkage  c) self report d) no description

• 2. Was follow up long enough for outcomes to occur

a) yes (select an adequate follow up period for outcome of interest)  b) no

• 3. Adequacy of follow up of cohorts

a) complete follow up - all subjects accounted for  b) subjects lost to follow up unlikely to introduce bias - small number lost - > ___ % (select an adequate %) follow up, or description of those lost)  c) follow up rate < ___% (select an adequate %) and no description of those lost d) no statement

Adjusted Effect Estimates for Coronary Heart Disease (All Events) (HRT: Estrogen Ever Use) Cohort Studies

Selection Comparability Outcome

Lauritzen / 83 Wilson / 85 Petitti / 87 Henderson / 91 Lafferty / 94 Folsom / 95 Ettinger / 96 Wolf / 96

0.01 0.1 1 10

The GRADE approach to RoB

Key Questions for Body of Obs Studies

• Do I preserve the LoE at low for this
• utcome?
• Do I downgrade to very low?
• Are upgrade criteria present?

RoB in Observational Studies

• 1. Failure to develop and apply

appropriate eligibility criteria (inclusion of control population)

• Under- or overmatching in case-control

studies

• Selection of exposed and unexposed in

cohort studies from different populations

RoB in Observational

• 2. Flawed measurement of both exposure

and outcome

• Differences in measurement of exposure

(e.g., recall bias in case-control studies)

• Differential surveillance for outcome in

exposed and unexposed in cohort studies

RoB in observational

• 3. Failure to adequately control

confounding

• Failure of accurate measurement of all

known prognostic factors

• Failure to match for prognostic factors

and/or lack of adjustment instatistical analysis

• 4. Incomplete follow-up

Reasons to upgrade?

• Large effect size

– OR > 2.0 or < 0.5 = increase to moderate – OR > 4.0 or < 0.2 = increase to high

• Dose – response ? Time factor
• All possible confounding supports

conclusions

RoB incons Large effect

Pooling observational studies

• Included studies
• Similar PICOs
• RevMan

Questions?