A public health approach to appraising evidence: Meta-tool for - - PowerPoint PPT Presentation

A public health approach to appraising evidence: Meta-tool for Quality Appraisal (MetaQAT)

Laura Rosella Carolyn Bowman Beata Pach June 16, 2016

PublicHealthOntario.ca

Agenda

• Discuss quality appraisal and evidence in public health
• Development of PHO MetaQAT
• Overview of the meta-tool
• Applied exercise

2

PublicHealthOntario.ca

Appraisal within review process

3

• Steps to systematically review evidence
• Different types of reviews
• All follow same core process
• Level of detail and rigour of the different steps

differ between types

• Examples:
• Systematic review, literature review, scoping

review, rapid review

Idea or request Review question Search Select Appraise Extract and analyze data Synthesize Report Some iteration is required

PublicHealthOntario.ca

“Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context.”

• To be blunt – judiciously assessing quality of evidence is challenging
• Methods and content expertise required
• Informs public health decisions – requires a judicious and transparent

process

• The “why” is often as important as the outcome

Quality (critical) appraisal

Burls A. What is critical appraisal? 2nd ed. [Internet]. Newmarket, UK: Hayward Group; 2009 [cited 2016 Jun 14]. Available from: http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/what_is_critical_appraisal.pdf

4

PublicHealthOntario.ca

Don’t be so negative!

• Appraisal should be about

understanding

• Both strengths and weaknesses
• We need to make decisions,

important to use evidence to fullest extent possible

• Understand how evidence can be

used appropriately

5

PublicHealthOntario.ca

Subjective is not a bad word

• Judgments are made based on appraisal principles
• Using the same principles, similar issues will be identified
• No one right answer, that doesn’t mean anything goes!
• “Document the information on which judgements are based”

6

Sterne J, Higgins J, Reeves B, editors. A Cochrane risk of bias assessment tool: for non-randomized studies of interventions (ACROBAT-NRSI). Version 1.0.0 [Internet]. London: Cochrane Collaboration; 2014 [cited 2016 Jun 14]. Available from: http://www.riskofbias.info Sterne J, Hernan M, Reeves B, Savovic J, Berkman N, Viswanathan M, et al. The risk of bias in non-randomized studies – of interventions (ROBINS-I) assessment tool [Internet]. London: Cochrane Collaboration; 2016 [cited 2016 Jun 14]. Available from: http://www.riskofbias.info

PublicHealthOntario.ca

Transparency

• Importance of documentation – if others know what you

appraised and why, they can ‘appraise your appraisal’

• Transparency is key: another person should be able to

understand why you made your assessment

• Think of it as a way that one can audit your analysis of the

evidence – akin to auditing code following a statistical analysis

7

PublicHealthOntario.ca

Skills and knowledge needed

These traits can be covered by a team

8

Quality Appraisal

Topic Methods Setting Application Validity concepts Documentation

PublicHealthOntario.ca

Skills and knowledge needed

These traits can be covered by a team

9

Quality Appraisal

Topic Methods Setting Application Validity concepts Documentation

PublicHealthOntario.ca

Internal Validity

• Elwood: measure of how

easily a difference in an

• utcome between two

groups can be attributed to the effects of an exposure or intervention

• L. Green: “the essence of

rigor” External Validity

• The process of generalizing the

findings of the study to the population from which the sample was drawn (or even beyond that population to a more universal statement)

• Without internal validity a study

cannot have external validity

• L. Green “ the essence of

relevance”

10

Internal and external validity

Ellwood M. Critical appraisal of epidemiological studies and clinical trials. New York: Oxford University Press; 2007. Green LW, Glasgow RE. Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. Eval Health Prof. 2006;29(1):126-153.

PublicHealthOntario.ca

Internal validity can be influenced by...

• Study design
• Measurement properties of the variables
• Study recruitment/response rates
• Selection pressures
• Sampling strategy
• Setting
• Investigators/funding
• Among others factors…

11

PublicHealthOntario.ca

Current approach to assessing risk of bias - RCTs

12

Higgins J, Altman D, Gotzsche P, Juni P, Moher D, Oxman A, et al; Cochrane Bias Methods Group, Cochrane Statistical Methods Group. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. Available from: http://www.bmj.com/content/bmj/343/bmj.d5928.full.pdf Figure 1 used under license terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).

Principles for assessing risk of bias: 1. Do not use quality scales 2. Focus on internal validity 3. Assess the risk of bias in trial results, not reporting or other issues 4. Assessments of bias require judgment 5. Choose domains to be assessed based on both theoretical and empirical considerations 6. Focus on risk of bias in data as presented in the review rather than as originally reported 7. Report outcome specific evaluations of risk of bias

PublicHealthOntario.ca 13

Sterne J, Higgins J, Reeves B, editors. A Cochrane risk of bias assessment tool: for non-randomized studies of interventions (ACROBAT-NRSI). Version 1.0.0 [Internet]. London: Cochrane Collaboration; 2014 [cited 2016 Jun 14]. Available from: http://www.riskofbias.info Sterne J, Hernan M, Reeves B, Savovic J, Berkman N, Viswanathan M, et al. The risk of bias in non-randomized studies – of interventions (ROBINS-I) assessment tool [Internet]. London: Cochrane Collaboration; 2016 [cited 2016 Jun 14]. Available from: http://www.riskofbias.info

Current approach to assessing risk

• f bias – non-randomized studies

PublicHealthOntario.ca

Assessing risk of bias says nothing about the intended use of the evidence

• Application of the evidence is important in public health
• Context matters – where the evidence was generated, where

it is going to be applied

• Importance of different factors when appraising evidence

should be guided by the intended application

14

PublicHealthOntario.ca

Evidence for Public Health Questions

• Evidence-based medicine evidence-based public health
• What is the best evidence to answer your question?
• Best available evidence

15

Application to new areas Different types of questions and context Rethink what is appraised and why

Harder T, Takla A, Rehfuess E, Sanchez-Vivar A, MtysiakKlose D, Echmanns T, et al. Evidence-based decision-making in infectious diseases epidemiology, prevention and control: matching research questions to study designs and quality appraisal tools. BMC Med Res Methodol. 2014;14:69. Available from: http://www.biomedcentral.com/content/pdf/1471-2288-14-69.pdf

PublicHealthOntario.ca

Beware the evidence pyramid

• Important concept in

evidence-based medicine

• Pyramid developed to

approach evidence for effectiveness of clinical interventions – based on internal validity of study designs

• Describes best evidence to

answer a particular kind of question

SUNY Downstate Medical Center, Medical Research Library of Brooklyn. Guide to research methods: the evidence pyramid [Internet]. Brooklyn, NY: SUNY Downstate Medical Center; 2004 [cited 2016 Jun 14]. Available from: http://library.downstate.edu/EBM2/2100.htm

16

PublicHealthOntario.ca

What’s better?

• A large and rigorous prospective cohort study or a small poorly

conducted randomized control trial?

• A systematic review of dozens of well-conducted case control

studies or one large randomized control trial?

• Large and rigorous randomized controlled trial done in Japan or a

time-series analysis in Calgary? (assuming you live in Toronto)

• A well conducted study that’s ‘lower’ on the evidence hierarchy may

actually be better...every study needs to be interpreted and assessed in its own right

Critical appraisal and the evidence hierarchy

17

PublicHealthOntario.ca

1. Quality appraisal should be a transparent process 2. Appraisal involves making considered judgments 3. Overall quality appraisal ≠ internal validity only 4. The question and the context should guide your judgments

Points to remember

18

PublicHealthOntario.ca

TOOLS, TOOLS, EVERYWHERE

19

PublicHealthOntario.ca

Tools to guide appraisal

• Many tools exist
• Challenges:
• Most not validated
• Most from evidence-based medicine

perspective – clinical effectiveness focus

• Many use numeric scales or scores – not

valid, not recommended by Cochrane Collaboration

• Many tools appropriate for specific

contexts

• One topic, project, or study design

20

PublicHealthOntario.ca

A systematic review of 121 published critical appraisal tools

• 87% of critical appraisal tools were specific to a research

design with most tools having been developed for experimental studies

• 49% of the critical appraisal tools summarized the quality

appraisal into a numeric summary score

• Few critical appraisal tools had documented evidence of

validity of their items, or reliability of use

• “We found no gold standard critical appraisal tool for any

type of study design”

Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA. A systematic review of the content of critical appraisal tools. BMC Med Res

• Methodol. 2004;4:22. Available from: http://www.biomedcentral.com/content/pdf/1471-2288-4-22.pdf

21

Tools to guide appraisal

PublicHealthOntario.ca

A review of use of tools in systematic reviews published in core medical journals

• 28% did no quality appraisal
• Most commonly used tool used in 26% of reviews
• Cochrane risk of bias tool
• 54 combinations of 47 individual reported tools
• 9% reported completing quality appraisal but no tool reported

22

Seehra J, Pandis N, Koletsi D, Fleming P. Use of quality assessment tools in systematic reviews was varied and inconsistent. J Clin Epidemiol. 2016;69:179-184.

Tools to guide appraisal

PublicHealthOntario.ca

Suggested benefits of appraisal tool frameworks

• Checklists that ensure that we don’t overlook important

considerations (i.e. Aide-mémoire)

• BUT then limit what we do look for depending what’s in the tool
• Standardize the approaching to reviewing the evidence
• Some judgments are inherently subjective – more important to know

why and how judgment was made

• “we found that discrepancies more often resulted from interpretation of

the tool rather than different information being identified and recorded for the same study”

Hartling L, Hamm M, Milne A, Vandermeer B, Santaguida P, et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol 2013;66(9):973-81.

23

Tools to guide appraisal

PublicHealthOntario.ca

A comparison of tools Voss and Rehfuess

QUALITY APPRAISAL IN SYSTEMATIC REVIEWS OF PUBLIC HEALTH INTERVENTIONS: AN EMPIRICAL STUDY ON THE IMPACT OF CHOICE OF TOOL ON META-ANALYSIS Recommendations: 1) Testing of a broader set of QATs on a more up-to-date systematic review of a public health intervention covering a wide range of epidemiological study designs 2) Research into the development of a reliable QAT with broad applicability across study designs.

Voss PH, Rehfuess EA. Quality appraisal in systematic reviews of public health interventions: an empirical study on the impact of choice of tool on meta-analysis. J Epidemiol Community Health. 2013;67(1):98-104.

24

PublicHealthOntario.ca

Features of a Gold Standard QAT Voss and Rehfuess

1) Instead a set of companion QATs comprising common components and study design-specific components could initially be developed for standard designs and then expanded to address more complex designs. 2) An overall summary score may lead to information loss and oversimplification. Instead, the authors propose a careful quantification of domains (rather than individual questions) in a weighted checklist. 3) Should address both internal and external validity with a clear distinction between these. 4) Much more effort should be invested into the development and testing of appropriate and specific questions and answer categories; a manual can help with their correct interpretation. 5) Subject-specific adaptation of selected questions makes quality appraisal more precise and reliable than a one size fits all QAT.

25

PublicHealthOntario.ca

DEVELOPMENT AND VALIDATION

26

PublicHealthOntario.ca

MetaQAT project stages

Stage 1: Gather information Stage 2: Put information together Stage 3a: Draft the meta-tool Stage 3b: Refine draft meta-tool Stage 4: Pilot testing Stage 5: Validation

27

PublicHealthOntario.ca

Project initiation

28

• Many projects at PHO involve

literature synthesis and quality appraisal

• Our knowledge products are rooted in

evidence, needs to be appraised

• Variation in practice – consistent

approach will ensure rigour, transparency

• Library task: search and find an

appropriate tool to fit our diverse

• rganizational needs

PublicHealthOntario.ca

Stage 1: Gather information

Inquiry from senior leadership – find an appropriate tool

29

• Database

searching

• Systematic review

identified

• Katrak et al 2004
• Reference

searching

• Medline search –

subsequent to 2004 SR

• Grey literature

searching

• NCCMT, OPHLA,

CADTH, JBI, NICE, ECDC

• University Libraries
• Consultation with

experts and colleagues

PublicHealthOntario.ca

No single tool meets all of PHO’s needs

• Outcome of stage 1: not a critical appraisal tool (too limited)

30

Stage 1: Gather information

Current state:

Many tools appropriate for specific contexts (topic, project, study design), however:

• Most not validated
• Most from evidence-based

medicine perspective

• Internal validity (risk of bias) focus
• Many use numeric scales or scores
• Not valid, not recommended by

Cochrane Collaboration

PHO requirements:

• Compatible with all of the types of

evidence we use

• Compatible with all of the types of

projects we do (mixed evidence)

• Consistent with current best

practice

• No numeric scores; acceptable to

field

• Document a transparent process

VS

PublicHealthOntario.ca

Stage 2: putting the information together

• Group tools by study design
• Compare the questions between tools
• Similarities across tools within each design
• Compare these similarities across study designs
• Core process is similar across tools
• Coverage of process matches OPHLA guide
• Generic, applies to many evidence types
• Generic framework emerges
• All questions can be grouped into four categories
• Still need study specific detail for high level of rigour
• Not a critical appraisal tool, a quality assessment process

31

PublicHealthOntario.ca

Stage 3a: develop draft meta-tool

• Couple generic framework with companion tools to create a

process guide

• Expands the scope of appraisal beyond internal validity
• Incorporates existing critical appraisal tools (CATs) for assessing internal

validity of specific designs

• Companion tools – pragmatic approach
• Widespread use in public health literature
• Preference for validated tools
• Compromise – reporting tools widely used as CATs
• Ease of use
• Revised meta-tool based on scientific advice

32

PublicHealthOntario.ca

Stage 3b: refine draft meta-tool

• Feedback sessions
• Staff selected for feedback sessions
• Sample article assessed, interpretations of questions
• Revisions made for clarity, reorganization
• Revised version presented to third group
• Revisions made
• Tool approved for pilot testing and further development and

evaluation

33

PublicHealthOntario.ca

Stage 4: pilot testing

• Introduced to teams for trial use in specific projects
• Variety of topics and types of projects
• Training methods developed
• Introduction to tool as project reached appraisal stage
• Evaluation
• Fit of tool for projects and staff, requirements for full implementation

34

PublicHealthOntario.ca

Selected pilot testing outcomes

INTEGRATION OF DIVERSE GROUPS OF EVIDENCE

• Applied to projects with a heterogeneous mix of evidence
• The flexibility of MetaQAT allowed for the integrated

assessment of evidence across methods and designs

• Facilitated screening decisions
• Broadened scope of appraisal promoted appropriate

consideration of context versus internal validity alone

• Maintains consistent process of appraisal across a group of

evidence that includes grey literature and surveillance literature

35

PublicHealthOntario.ca

Stage 5: Validation

• Two groups of appraisers
• one group used MetaQAT, other group appraised strengths and

weaknesses of several studies

• Appraisals compared between groups
• Content analysis
• Both groups identified similar issues
• MetaQAT group identified public health relevance

36

PublicHealthOntario.ca

SIMILARITIES

• Main issues similar across groups for all three studies – two

examples:

• Reliability and accuracy of and use of validated measures
• Significant concern of impact of unmeasured confounders

DIFFERENCES

• Application and public health context covered by MetaQAT

group – two examples

• “severe TBI affect only small fraction of population. Focus on TBI in

general/any sports TBI may be more relevant to public health”

• “not all that useful in the public health context. Difficult to do; a lot more

about reasons for maternal and child hospitalizations. If this was done, effect on obesity would be small”

Validation

37

PublicHealthOntario.ca

Publication

• Development and

validation detailed in publication

38

PublicHealthOntario.ca

Meta-tool

• Quality appraisal framework that includes companion tools

from existing literature

• Use for many different types of evidence, review questions,

application contexts

• Documentation – encourages recording of key quality

information and rationale to make the appraisal process transparent

39

PublicHealthOntario.ca

MetaQAT structure

40

Relevancy Companion tools

AMSTAR CASP TREND CONSORT Mixed Methods AGREE II Navigation Guide PHO Guide for Grey Literature

Reliability Validity Applicability Appraisal Framework

Rosella L, Bowman C, Pach B, Morgan S, Fitzpatrick T, Goel V. The development and validation of a meta-tool for quality appraisal

• f public health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health. 2016;136:57-65. Available from:

http://www.sciencedirect.com/science/article/pii/S0033350615004370

PublicHealthOntario.ca

MetaQAT domains

Domain Idea

Relevancy

• Determine if the item being appraised is sufficiently related to your topic
• r research question
• In a structured systematic review, relevancy can also be considered in the

inclusion and exclusion criteria

Reliability

• Evaluate the completeness of reporting; complete reporting is necessary in
• rder to conduct a thorough examination of quality
• Lack of detail regarding the conduct of the study or report may be

indicative of lower quality information

Validity

• The likelihood and magnitude of error or bias in a study
• “Risk of Bias” or “Internal Validity”

Applicability

• Consider how the evidence might be applied to public health practice
• Also can consider more broadly what can be learned from the evidence

and how you can apply that knowledge to public health decision making

41

The form uses the term “study” to refer to the work being appraised and is meant to include all study types, including research published as grey literature, as syntheses, and any other type of evidence being considered.

Rosella L, Bowman C, Pach B, Morgan S, Fitzpatrick T, Goel V. The development and validation of a meta-tool for quality appraisal of public health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health. 2016;136:57-65. Available from: http://www.sciencedirect.com/science/article/pii/S0033350615004370

PublicHealthOntario.ca

MetaQAT questions

42

Domain Questions

• 1. Relevancy
• a. Does the study address a topic(s) relevant to the issue under

investigation?

• 2. Reliability
• a. Is the study presented clearly?
• b. Are the research methodology and results clearly described?

c. Are ethics procedures described?

• 3. Validity
• a. Is the study methodology appropriate for the scope of

research?

• b. Is the research methodology free from bias?

c. Are the authors’ conclusions explicit and transparent?

• d. Can I be confident about the findings?
• 4. Applicability
• a. Can the results be applied within the scope public health?

Rosella L, Bowman C, Pach B, Morgan S, Fitzpatrick T, Goel V. The development and validation of a meta-tool for quality appraisal of public health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health. 2016;136:57-65. Available from: http://www.sciencedirect.com/science/article/pii/S0033350615004370

PublicHealthOntario.ca

MetaQAT prompts

Domains

Relevancy Reliability Validity Applicability

Main Questions

Is the study design appropriate? Is the study free from bias? Are the conclusions justified? Are you confident in the findings?

Prompts

Is the research question congruent with the study design? Does the methodology match the theory or the conceptual model? Are appropriate controls considered if applicable? Are the statistical/analytic methods appropriate for the design and/or the question? Are important theoretical factors accounted for in the analysis?

43

PublicHealthOntario.ca

MetaQAT questions

44

Domain Questions

• 1. Relevancy
• a. Does the study address a topic(s) relevant to the issue under

investigation?

• 2. Reliability
• a. Is the study presented clearly?
• b. Are the research methodology and results clearly described?

c. Are ethics procedures described?

• 3. Validity
• a. Is the study methodology appropriate for the scope of

research?

• b. Is the research methodology free from bias?

c. Are the authors’ conclusions explicit and transparent?

• d. Can I be confident about the findings?
• 4. Applicability
• a. Can the results be applied within the scope public health?

Rosella L, Bowman C, Pach B, Morgan S, Fitzpatrick T, Goel V. The development and validation of a meta-tool for quality appraisal of public health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health. 2016;136:57-65. Available from: http://www.sciencedirect.com/science/article/pii/S0033350615004370

PublicHealthOntario.ca

Answer format

• Written answer format
• Summarize the important issues in words
• Makes the appraisal transparent
• The statements are often used later in the review process
• This is what makes the appraisal process transparent and therefore

credible

45

PublicHealthOntario.ca

Pdf format

46

PublicHealthOntario.ca

Excel format

47

PublicHealthOntario.ca

Summarizing the appraisal of an item

• Textual summary of key strengths and weaknesses by domain

and question

• This information will provide a good understanding of the quality of the

item

• E.g. which is more helpful? 8/10 or “the study group was small but

characteristics were very similar to our local population”

• “The use of scales for assessing quality or risk of bias is

explicitly discouraged in Cochrane reviews”

Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions. Version 5.1.0 [Internet]. London: Cochrane Collaboration; 2011 [cited 2016 Jun 14]. Section 8.3.3, Quality scales and Cochrane reviews. Available from: http://handbook.cochrane.org Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282(11)1054-1060.

48

PublicHealthOntario.ca

Summarizing appraisal - example

49

Bornbaum C, Kornas K, Peirson L, Rosella L. Exploring the function and effectiveness of knowledge brokers as facilitators of knowledge translation in health-related settings: a systematic review and thematic analysis. Implementation Science. 2015;10(162). Available from: http://www.implementationscience.com/content/10/1/162 . Additional file 4 used under license terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).

PublicHealthOntario.ca

Summarizing appraisal - example

• Journal article - Maryland alcohol sales tax and sexually transmitted infections a natural

experiment

50

Relevancy

• STIs relevant to Ontario
• Alcohol tax structure is different

Reliability

• Sufficient detail is provided to understand the study

Validity

• The authors make causal conclusions, which cannot be supported

because of the observational design

• Long causal chain assumed between alcohol sales tax and STI

incidence – opportunity for confounding (education, current events e.g. high profile HIV case, decreased spending but consistent consumption via lower quality product)

• Inconsistent results – change in gonorrhea but not chlamydia
• Discard conclusions, discuss results with lower confidence

Applicability

• Tax structure is different, so any application would have to be adapted
• Difference in access to alcohol – although this is changing e.g. grocery

stores

Staras S, Livingston M, Wagenaar A. Maryland alcohol sales tax and sexually transmitted infections a natural experiment. Am J Prev Med 2016;50(3):73- 80.

PublicHealthOntario.ca

Using the appraisal - example

• Integrate with results
• Authors found a decrease in gonorrhea rates after the alcohol sales tax
• increase. No change was found for the incidence rate of chlamydia.

Confidence in these results is lowered by many uncontrolled confounding factors. Alcohol taxation is also different in the study setting from that in Ontario. This evidence should still be considered when looking at alcohol policy in Ontario, however more robust and applicable evidence should be used if available.

51

Staras S, Livingston M, Wagenaar A. Maryland alcohol sales tax and sexually transmitted infections a natural experiment. Am J Prev Med 2016;50(3):73- 80.

PublicHealthOntario.ca

Hard work is hard

• We need to be clear that the process of systematically

appraising evidence and documenting this process is difficult and can be time consuming but these challenges may exist regardless of the tool/framework

52

Quality Appraisal

Topic Methods Setting Application Validity concepts Documentation

PublicHealthOntario.ca

References

• Bornbaum C, Kornas K, Peirson L, Rosella L. Exploring the function and effectiveness of knowledge brokers as facilitators of knowledge

translation in health-related settings: a systematic review and thematic analysis. Implementation Science. 2015;10(162). Available from: http://www.implementationscience.com/content/10/1/162 .

• Burls A. What is critical appraisal? 2nd ed. [Internet]. Newmarket, UK: Hayward Group; 2009 [cited 2016 Jun 14]. Available from:

http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/what_is_critical_appraisal.pdf

• Ellwood M. Critical appraisal of epidemiological studies and clinical trials. New York: Oxford University Press; 2007.
• Green LW, Glasgow RE. Evaluating the relevance, generalization, and applicability of research: issues in external validation and

translation methodology. Eval Health Prof. 2006;29(1):126-153.

• Harder T, Takla A, Rehfuess E, Sanchez-Vivar A, MtysiakKlose D, Echmanns T, et al. Evidence-based decision-making in infectious diseases

epidemiology, prevention and control: matching research questions to study designs and quality appraisal tools. BMC Med Res

• Methodol. 2014;14:69. Available from: http://www.biomedcentral.com/content/pdf/1471-2288-14-69.pdf
• Hartling L, Hamm M, Milne A, Vandermeer B, Santaguida P, et al. Testing the risk of bias tool showed low reliability between individual

reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol 2013;66(9):973-81.

• Higgins J, Altman D, Gotzsche P, Juni P, Moher D, Oxman A, et al; Cochrane Bias Methods Group, Cochrane Statistical Methods Group.

The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. Available from: http://www.bmj.com/content/bmj/343/bmj.d5928.full.pdf

• Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions. Version 5.1.0 [Internet]. London: Cochrane

Collaboration; 2011 [cited 2016 Jun 14]. Section 8.3.3, Quality scales and Cochrane reviews. Available from: http://handbook.cochrane.org 53

PublicHealthOntario.ca

References

• Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282(11)1054-1060.
• Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA. A systematic review of the content of critical appraisal tools.

BMC Med Res Methodol. 2004;4:22. Available from: http://www.biomedcentral.com/content/pdf/1471-2288-4-22.pdf

• Rosella L, Bowman C, Pach B, Morgan S, Fitzpatrick T, Goel V. The development and validation of a meta-tool for quality appraisal
• f public health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health. 2016;136:57-65. Available from:

http://www.sciencedirect.com/science/article/pii/S0033350615004370

• Seehra J, Pandis N, Koletsi D, Fleming P. Use of quality assessment tools in systematic reviews was varied and inconsistent. J Clin
• Epidemiol. 2016;69:179-184.
• Staras S, Livingston M, Wagenaar A. Maryland alcohol sales tax and sexually transmitted infections a natural experiment. Am J Prev Med

2016;50(3):73-80.

• Sterne J, Hernan M, Reeves B, Savovic J, Berkman N, Viswanathan M, et al. The risk of bias in non-randomized studies – of interventions

(ROBINS-I) assessment tool [Internet]. London: Cochrane Collaboration; 2016 [cited 2016 Jun 14]. Available from: http://www.riskofbias.info

• Sterne J, Higgins J, Reeves B, editors. A Cochrane risk of bias assessment tool: for non-randomized studies of interventions (ACROBAT-

NRSI). Version 1.0.0 [Internet]. London: Cochrane Collaboration; 2014 [cited 2016 Jun 14]. Available from: http://www.riskofbias.info

• Voss PH, Rehfuess EA. Quality appraisal in systematic reviews of public health interventions: an empirical study on the impact of choice
• f tool on meta-analysis. J Epidemiol Community Health. 2013;67(1):98-104.

54