DISCLOSURES There are no conflicts of interest or relevant financial - - PDF document

1

Selda Secginli, PhD, Associate Professor

Istanbul University, Florence Nightingale Nursing Faculty, Public Health Nursing Department Istanbul, Turkey

1 Istanbul University

Assessing the quality of methods used in standardized terminology studies

Second International Conference on Research Methods for Standard Terminologies

DISCLOSURES

There are no conflicts of interest or relevant financial interests that have been disclosed by this presenter or the rest of the planners and presenters of this activity that apply to this learning session.

At the end of this session, the learner will be better able to:

• define quality assessment
• discuss the importance of use of the quality assessment

tools

• get an overview of how these tools are used in a

systematic review study based on Omaha System, one of the standardized terminology used in Turkey

3

Objectives

2

• A structured language consisting of terms,

definitions, and codes that clinicians use to guide and document practice. i.e. Omaha System

• They are important for higher quality care, enhanced

care coordination, improved documentation and patient outcomes, reduced costs.

4

Standardized Terminologies

• Selection of standard terminology; implications for

design of forms, orders, decision support, storage; workflow analysis for context and organizing the data for use in reports and clinical decision making is critical.

• The use of structured clinical records with standardized

data increases the likelihood of good data quality.

• Good data quality enhanced study quality.
• Lack of standardization of terminology is an important

barrier to performing high-quality research.

5

Standardized Terminologies-cont.

• The confidence that the trial design, conduct, and

analysis has minimized or avoided biases in its treatment comparisons.

• Quality assessment of research involves appraisal of a

study's internal validity, i.e. the degree to which its design, conduct and analysis have minimized biases or

• errors. For practical reasons, study quality assessment in

reviews often covers both internal and external validity.

• A broader notion of research quality should help

researchers and research users to feel confident about the use of evidence in policy and practice.

6

What Is Quality Assessment?

3

• Pose a significant, important question
• Apply methods that best address the research
• Complete coverage of the relevant literature
• Provide the necessary information to reproduce or replicate the

study

• Ensure the study design, methods, and procedures
• Provide sufficient description of the sample, the intervention.
• Use appropriate and reliable conceptualization and measurement of

variables

• Evaluate alternative explanations for any findings
• Assess the possible impact of systematic bias
• Adhere to quality standards for reporting

7

Standards for assessing the quality of research Key Elements of a Systematic Review

• Studies may be biased due to inadequate

randomization, unsuitable comparison interventions, lack of blind outcome assessment, inadequate follow-up, inability to define and assess outcomes, unreliable measurement techniques, and inappropriate statistical analyses.

• With these tools included a number of quality items,

which are scored numerically provide a quantitative estimate of overall study quality.

9

Quality Assessment Tools

4

• There are many tools for assessing the

methodological quality of the studies.

• Some of these tools were developed for specific

study designs (eg. RCTs, cohort studies, descriptive studies), while others were developed for a range of study designs.

• Mostly, these tools incorporate characteristics that

are associated with bias. However, there are also many tools that contain items related to reporting and these are not related with bias.

10

Quality Assessment Tools-cont.

• Was the assignment to the treatment groups really

random?

• Was the treatment allocation concealed?
• Were the groups similar at baseline in terms of prognostic

factors?

• Were the eligibility criteria specified?
• Were outcome assessors blinded to the treatment

allocation?

• Was the care provider blinded? Was the patient blinded?
• Did the analyses include an intention to treat analysis?

11

Quality criteria for assessing experimental studies

• Some examples should be: The Cochrane ROB tool was

designed for RCTs and is the instrument recommended by The Cochrane Collaboration for use in systematic reviews of RCTs. The Newcastle-Ottawa Scale is commonly used for nonrandomized studies, specifically cohort and case-control studies. JBI critical appraisal tools can be used for RCTs, quasi-experimental, case- control/cohort studies and descriptive/case series studies.

12

Quality Assessment Tools-cont.

5

• These checklists use a series of criteria that can be

scored as being met, not met or unclear or if deemed appropriate not applicable (N/A) to that particular study.

• The decision as to whether or not to include a study can

be made based on meeting a predetermined proportion

• f all criteria, or on certain criteria being met.
• Each items were answered dichotomously, where "yes"

was allocated with one point and "no”, "unclear” and “not applicable” with zero.

• The cut-off score for inclusion of studies after

methodological appraisal was set at 4/10 (evaluating criteria)

13

The Joanna Briggs Institute (JBI) Quality Assessment Tools

14

JBI Critical Appraisal Tools

• To describe and analyze recent literature about

Omaha System in Turkey

• To identify the Omaha System study design,

methodologies and areas for future research, in Turkey (1) studies published between 2000 and 2012, (2) studies published both in English and Turkish, (3) full text articles that were published in a peer- reviewed journal, thesis and dissertations, (4) conducted in Turkey,

15

6

• The electronic databases Ovid MEDLINE, PUB MED, Cochrane CENTRAL

Register of Controlled Trials, CINAHL, PsycINFO, Web of Knowledge, Scopus, Google Scholar, ULAKBIM Turkish Medical Database and Council of Higher Education Thesis Center were

• searched. Additionally, the reference lists of

included studies from the Turkish databases were hand searched.

• In the study, papers that met the inclusion criteria

were considered to be applicable to the review topic and retrieved for further assessment of quality.

16

• The main object is to

assess the methodological quality

• f a study (the

possibility of bias in its design, conduct and analysis) and exclude studies that are of low quality and identify the strengths and limitations of the included studies.

17

• 5. Assessment of

methodological quality

Turkish JBI Quality assessment Tools

• Cohen’s kappa: .74

(experimental studies) .67 (descriptive studies)

• Test-retest reliability: .87

(experimental studies); .91 (descriptive studies)

• Kuder richardson: .68

(experimental studies); .64 (descriptive studies)

• Content validity index: .90

(experimental studies); .87 (descriptive studies)

18

Deneysel ve Yarı Deneysel Araştırmalar İçin MAStARI Kritik Değerlendirme Formu Değerlendiren: Tarih: Yazar: Yıl: Kayıt no: Evet Hayır Belirtil- memiş Uygun değil

• 1. Katılımcılar girişim/tedavi gruplarına gerçekten rastgele mi

atanmıştır?

• 2. Katılımcılar girişim/tedavi gruplarına kör teknikle mi atanmıştır?

(katılımcıların yapılan girişimi/tedaviyi bilmemesi)

• 3. Katılımcıları çalışma gruplarına dağıtan kişi, katılımcıların hangi

tedaviyi aldığından habersiz miydi?

• 4. Çalışmadan ayrılan katılımcıların sonuçları verilmiş ve analize dahil

edilmiş midir?

• 5. Sonuçları değerlendirenler katılımcıların hangi grupta olduklarından

habersiz miydi?

• 6. Çalışmanın başlangıcında deney ve kontrol grupları temel özellikler

yönünden benzer miydi?

• 7. Gruplar uygulanan girişim(ler) dışında aynı şekilde takip edilmiş

miydi?

• 8. Sonuçlar tüm gruplarda aynı şekilde mi ölçülmüştü?
• 9. Sonuçlar güvenilir şekilde ölçülmüş müdür?
• 10. Uygun istatistiksel analizler kullanılmış mıdır?

Genel değerlendirme: ( ) Kabul et ( ) Kabul etme ( ) Daha fazla bilgi araştır Açıklama (Kabul etmeme nedenleri):

7

Methodological Quality of the Included Omaha System Studies

19

17 papers that met the criteria for inclusion in the review were critically appraised for the methodological quality by JBI tools. 5 studies used quasi-experimental design (mostly non-randomized

• ne-group pre-test and post-test). The appraisal scores of five

studies were six (6/10 points). 12 studies used descriptive design. The appraisal scores of five studies were mostly 6 (6/9 points), two studies were 7 (7/9 points), and one study was 5 (5/9 points).

The most common issues:

• participants were not blinded to

treatment allocation,

• allocation to treatment groups

was not concealed from the allocator,

• the outcomes of people who

withdrew were not described and included in the analysis,

• those assessing outcomes were

not blind to the treatment allocation in the studies (items 2,3,4 and 5 in the original tool).

20

Methodological quality of studies related with quasi- experimental studies Methodological quality of studies related with descriptive studies

The most common issues:

• confounding factors were not

identified and strategies to deal with them were not stated ,

• there was not sufficient

descriptions of the groups,

• the outcomes of people who

withdrew were not described and included in the analysis in the studies (items 3,5 and 7 in the original tool).

• Data was extracted from the studies regarding

study reference, sample, setting, purpose, design and findings.

21

• 6. Extract data according to a template

Analyze clinical process (N=5) Analyze client problems (N=3) Analyze client

• utcomes

(N=3)

Reports on unpublished master’s and doctoral dissertation s (N=5)

Advanced classificatio n research (N=1)

Categories of the included Omaha System studies

8

22

• All studies were completed between 2000 and 2014.
• Studies had a wide range of study samples ranges from 30 to 598

and the settings of the studies varies.

• Only 5 studies reported that they used OS coded data from EHRs.
• In all the studies, the reliable and valid OS tools were used for

data collection.

• All articles were authored by nurses ( first author), and only 1

study included interdisciplinary co-authors.

• 5 citations of thesis and doctoral dissertations were used OS.
• The most common research design was descriptive and few were

quasi-experimental design using one group pre-post test

• measurements. No studies were randomized controlled

experimental design.

• 7. Analyze, synthesize, and summarize data

Preliminary review results

• 1. Using standardized terminologies as a standardized

instrument should improve the quality of studies, especially when used for data collection in randomized controlled trials.

• 2. Other methods to ensure quality are needed if

standardized terminologies are used for documentation in EHRs. (Engineering solutions to support

quality documentation, consistent protocols for documentation, fidelity to documentation protocols, and replication to show consistent results from multiple datasets)

23

What are the implications for the future?

• Enhance data quality

(1) data accuracy (2) data completeness

• Foster a data quality culture-improving data quality is a collaborative

effort; provide educational sessions to data providers on data requirements and quality issues; use systems that automatically check for data quality issues and providing feedback to data suppliers

• Use assessment tools to improve study quality.
• Research design and terminology used must be carefully examined
• Conduct well designed randomized controlled trials for counting good

quality

• Provide research exemplars when terminologies are implemented and

used effectively (closer collaboration of researchers and professionals)

• Identify resources (evidence based health care centers) for global

collaboration to create nursing data for big data research

24

What do we need to do to improve quality?

9

• Hartling L, Hamm M, Milne A, Vandermeer B, Santaguida PL, Ansari M, Tsertsvadze A, Hempel S,

Shekelle P, Dryden DM. Validity and inter-rater reliability testing of quality assessment

• instruments. (Prepared by the University of Alberta Evidence-based Practice Center under

Contract No. 290-2007-10021-I.) AHRQ Publication No. 12-EHC039-EF. Rockville, MD: Agency for Healthcare Research and Quality. March 2012.

• Agency for Healthcare Research and Quality, Prospects for Care Coordination Measurement

Using Electronic Data Sources, AHRQ Publication No. 12-0014-EF, March 2012

• The Joanna Briggs Institute. Joanna Briggs Institute Reviewers’ Manual: 2014 edition. Adelaide:

The Joanna Briggs Institute; 2014.

• BIEBNM, 2000 Appraising Systematic Reviews, Changing Practice Sup. 1, [Online, accessed

date]URL: http://www. joannabriggs. edu. au/CP1.pdf.

• Munn Z, Moola S, Riitano D, Lisy K. The development of a critical appraisal tool for use in

systematic reviews addressing questions of prevalence. Int J Health Policy Manag 2014; 3: 123– 128.

• Porritt K, Gomersall J, Craig L. BI's Systematic Reviews: Study Selection and Critical Appraisal.

American Journal of Nursing 2014; 14(6): 47-52.

• Higgins JPT, Green S (editors).(2011). Cochrane Handbook for Systematic Reviews of

Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.

• Topaz M, Golfenshtein N, Bowles KH. The Omaha System: a systematic review of the recent

literature . J Am Med Inform Assoc 2014;21:163–170.

25

References

26

Thank you for your time. Email: secginli@istanbul.edu.tr

Istanbul University