DISCLOSURES There are no conflicts of interest or relevant financial - PDF document

Istanbul University Assessing the quality of methods used in standardized terminology studies Second International Conference on Research Methods for Standard Terminologies Selda Secginli, PhD, Associate Professor Istanbul University, Florence Nightingale Nursing Faculty, Public Health Nursing Department Istanbul, Turkey 1 DISCLOSURES There are no conflicts of interest or relevant financial interests that have been disclosed by this presenter or the rest of the planners and presenters of this activity that apply to this learning session. Objectives At the end of this session, the learner will be better able to: • define quality assessment • discuss the importance of use of the quality assessment tools • get an overview of how these tools are used in a systematic review study based on Omaha System, one of the standardized terminology used in Turkey 3 1

Standardized Terminologies • A structured language consisting of terms, definitions, and codes that clinicians use to guide and document practice. i.e. Omaha System • They are important for higher quality care, enhanced care coordination, improved documentation and patient outcomes, reduced costs. 4 Standardized Terminologies-cont. • Selection of standard terminology; implications for design of forms, orders, decision support, storage; workflow analysis for context and organizing the data for use in reports and clinical decision making is critical. • The use of structured clinical records with standardized data increases the likelihood of good data quality. • Good data quality enhanced study quality. • Lack of standardization of terminology is an important barrier to performing high- quality research . 5 What Is Quality Assessment? • The confidence that the trial design, conduct, and analysis has minimized or avoided biases in its treatment comparisons. • Quality assessment of research involves appraisal of a study's internal validity, i.e. the degree to which its design, conduct and analysis have minimized biases or errors. For practical reasons, study quality assessment in reviews often covers both internal and external validity. • A broader notion of research quality should help researchers and research users to feel confident about the use of evidence in policy and practice. 6 2

Standards for assessing the quality of research • Pose a significant, important question • Apply methods that best address the research • Complete coverage of the relevant literature • Provide the necessary information to reproduce or replicate the study • Ensure the study design, methods, and procedures • Provide sufficient description of the sample, the intervention. • Use appropriate and reliable conceptualization and measurement of variables • Evaluate alternative explanations for any findings • Assess the possible impact of systematic bias • Adhere to quality standards for reporting 7 Key Elements of a Systematic Review Quality Assessment Tools • Studies may be biased due to inadequate randomization, unsuitable comparison interventions, lack of blind outcome assessment, inadequate follow-up, inability to define and assess outcomes, unreliable measurement techniques, and inappropriate statistical analyses. • With these tools included a number of quality items, which are scored numerically provide a quantitative estimate of overall study quality. 9 3

Quality Assessment Tools-cont. • There are many tools for assessing the methodological quality of the studies. • Some of these tools were developed for specific study designs (eg. RCTs, cohort studies, descriptive studies), while others were developed for a range of study designs. • Mostly, these tools incorporate characteristics that are associated with bias. However, there are also many tools that contain items related to reporting and these are not related with bias. 10 Quality criteria for assessing experimental studies • Was the assignment to the treatment groups really random? • Was the treatment allocation concealed? • Were the groups similar at baseline in terms of prognostic factors? • Were the eligibility criteria specified? • Were outcome assessors blinded to the treatment allocation? • Was the care provider blinded? Was the patient blinded? • Did the analyses include an intention to treat analysis? 11 Quality Assessment Tools-cont. • Some examples should be: The Cochrane ROB tool was designed for RCTs and is the instrument recommended by The Cochrane Collaboration for use in systematic reviews of RCTs. The Newcastle-Ottawa Scale is commonly used for nonrandomized studies, specifically cohort and case-control studies. JBI critical appraisal tools can be used for RCTs, quasi-experimental, case- control/cohort studies and descriptive/case series studies. 12 4

The Joanna Briggs Institute (JBI) Quality Assessment Tools • These checklists use a series of criteria that can be scored as being met, not met or unclear or if deemed appropriate not applicable (N/A) to that particular study. • The decision as to whether or not to include a study can be made based on meeting a predetermined proportion of all criteria, or on certain criteria being met. • Each items were answered dichotomously, where "yes" was allocated with one point and "no”, "unclear” and “not applicable” with zero. • The cut-off score for inclusion of studies after methodological appraisal was set at 4/10 (evaluating criteria) 13 JBI Critical Appraisal Tools 14 • To describe and analyze recent literature about Omaha System in Turkey • To identify the Omaha System study design, methodologies and areas for future research, in Turkey (1) studies published between 2000 and 2012, (2) studies published both in English and Turkish, (3) full text articles that were published in a peer- reviewed journal, thesis and dissertations, (4) conducted in Turkey, 15 5

• The electronic databases Ovid MEDLINE, PUB MED, Cochrane CENTRAL Register of Controlled Trials, CINAHL, PsycINFO, Web of Knowledge, Scopus, Google Scholar, ULAKBIM Turkish Medical Database and Council of Higher Education Thesis Center were searched. Additionally, the reference lists of included studies from the Turkish databases were hand searched. • In the study, papers that met the inclusion criteria were considered to be applicable to the review topic and retrieved for further assessment of quality. 16 5. Assessment of Turkish JBI methodological quality Quality assessment Tools • The main object is to • Cohen’s kappa: .74 (experimental studies) .67 assess the (descriptive studies) methodological quality • Test-retest reliability: .87 of a study (the (experimental studies); .91 possibility of bias in its (descriptive studies) design, conduct and • Kuder richardson: .68 analysis) and exclude (experimental studies); .64 studies that are of low (descriptive studies) • Content validity index: .90 quality and identify the (experimental studies); .87 strengths and (descriptive studies) limitations of the included studies. 17 Deneysel v e Yarı Deneysel Ara ştı rmalar İçin MAStARI Kritik De ğ erlendirme Formu De ğ erlendiren : Tarih : Yazar : Yı l : Kay ı t no : Evet Hayır Belirtil- Uygun memi ş de ğ il 1. Kat ı l ı m cı lar giri ş im/tedavi grupla rı na ge rçe kten rastgele mi atanm ış t ı r? 2. Kat ı l ı m cı lar giri ş im/tedavi grupla rı na kö r teknikle mi atanm ış t ı r? (kat ı l ı m cı la rı n yap ı lan giri ş imi/tedaviyi bilmemesi) 3. Kat ı l ı m cı la rı ça l ış ma grupla rı na da ğı tan ki ş i, kat ı l ı m cı la rı n hangi tedaviyi ald ığı ndan habersiz miydi? 4. Ç al ış madan ay rı lan kat ı l ı m cı la rı n sonu çl a rı verilmi ş ve analize dahil edilmi ş midir? 5. Sonu çl a rı de ğ erlendirenler kat ı l ı m cı la rı n hangi grupta oldukla rı ndan habersiz miydi? 6. Ç al ış man ı n ba ş lang ıcı nda deney ve kontrol grupla rı temel ö zellikler y ö n ü nden benzer miydi? 7. Gruplar uygulanan giri ş im(ler) d ışı nda ayn ı ş ekilde takip edilmi ş miydi? 8. Sonu çl ar t ü m gruplarda ayn ı ş ekilde mi ö l çü lm üş t ü ? 9. Sonu çl ar g ü venilir ş ekilde ö l çü lm üş m ü d ü r? 10. Uygun istatistiksel analizler kullan ı lm ış m ı d ı r? Genel de ğ erlendirme : ( ) Kabul et ( ) Kabul etme ( ) Daha fazla bilgi ara ş t ı r A çı klama (Kabul etmeme nedenleri): 18 6