through a source data verification audit Lauren Houston, Dr Yasmine - PowerPoint PPT Presentation

Measuring data quality through a source data verification audit Lauren Houston¹, Dr Yasmine Probst¹, Dr Allison Humphries¹ ¹School of Medicine, Faculty of Science, Medicine and Health, University of Wollongong lah993@uowmail.edu.au @Lauren_Houston

Background  Health data has long been scrutinised 1,2  A large proportion of errors are from transcribing data 3,4  No “gold standard” method exists to measure data quality error rates. 1. Y.W. Lee et al . ( 2006) 3. M. Mealer et al. (2013) @Lauren_Houston 2. M.N. Zozus et al. (2014) 4. M.L. Nahm et al. (2008)

What is source data verification? Case report form Source data Electronic record @Lauren_Houston

What are the gaps in the knowledge?  ICH GCP guidelines are non specific to amount, timing and frequency of monitoring 5  Cost-effectiveness of SDV 6  No single definition to define data quality or universally accepted method to measure error rates 7,8  Audits may be published but not for public viewing 8 5. ICH GCP (1996) 7. R.Rostami et al. (2009) @Lauren_Houston 6. C. Baigent et al. (2008) 8. R.V. Gómez-Rioja et al. (2013)

Aim To monitor data quality by developing and conducting source data verification audits to achieve quality assurance. @Lauren_Houston

Study Background  UOW and IHMRI  Healthy-lifestyle blinded-RCT  12 months  5 clinical Accredited Practising Dietitians  To limit bias the audit was blinded @Lauren_Houston

Method  100% SDV on the 10% random sample  Quality assurance rule developed whereby if,  >5% of data variables were incorrect a second 10% random sample was extracted  Manual verification checks conducted  Calculation of error rate @Lauren_Houston

Procedure of data audit 10% 100% random SDV sample Second 10% Total of 820 If >41 (5%) Participants Random random data points data points sample sample of from 21 are original n=210 n=21 participants incorrect participants Total of 685 Participants Random If >34 (5%) 100% SDV data points sample data points of all data from 21 n=21 are incorrect points n=210 participants @Lauren_Houston

Audit 1 Source data verification Source documents Anthropometric Coded Electronic Physiological electronic spreadsheet spreadsheet record record Medications @Lauren_Houston

Audit 2 Source data verification Source documents Anthropometric Coded Electronic electronic spreadsheet spreadsheet record record Medications @Lauren_Houston

Coded electronic spreadsheet @Lauren_Houston

Statistical Analysis  Total error = (code 2+3+4) / (code 1+2+3+4)  Data “not entered” (code 5) – excluded  Chi square, p<0.05  Post-hoc; adjusted standardised residuals and z test of column proportions @Lauren_Houston

Results Major error Minor error % % Audit 1 Audit 2 3 Audit 1 Audit 2 16 14 2.5 12 2 10 1.5 8 6 1 4 0.5 2 0 0 Anthropometric Physiological Medication Anthropometric Physiological Medications @Lauren_Houston

Results Not recorded data Not entered data % % 60 30 Audit 1 Audit 2 Audit 1 Audit 2 50 25 40 20 30 15 20 10 10 5 0 0 Anthropometric Physiological Medication Anthropometric Physiological Medication @Lauren_Houston

Correct and Incorrect % Correct Incorrect 100 90 80 70 60 50 40 30 20 10 0 Anthropometric Anthropometric Medication Medication Total Total audit 1 audit 2 audit 1 audit 2 audit 1 audit 2 @Lauren_Houston

Results  Chi square had a significant difference; χ2 (4, 1293) = 672.405, p = 0.000  Adjusted standardised residuals determined audit sections were significantly different  From the z test of column proportions anthropometric audit 1 and 2; medications audit 1 and 2 do not differ.  All other sections differed from each other. @Lauren_Houston

Summary  Audit 1 physiological section <5% error  Average total error anthropometric (9%), medications (76%) and overall (34.5%)  Proportion of error trended upward as length of study increased  “Not recorded” (code 4) data had the greatest contribution to total error @Lauren_Houston

Discussion  Importance of identifying errors and determining solutions  If >10% of a clinical dataset is erroneous the data may be considered unreliable  Developed a 5% quality assurance rule  Data quality variations @Lauren_Houston

@Lauren_Houston Limitations Source documents considered the “gold standard” Source document-to-electronic spreadsheet Audits cannot guarantee 100% free from error Clinical research setting and trial design Did not determine the impact of audit findings

Recommendations  Examine the 10% snapshot model with a 5% quality assurance error rate  Standardise a SDV audit process  Assess the frequency and cost-effectiveness  Overcome barriers and increase awareness @Lauren_Houston

How many errors are too many? @Lauren_Houston

Acknowledgements  Dr. Yasmine Probst  Dr. Allison Humphries  Sr/Prof. Linda Tapsell  A/Prof. Marijka Batterham  Illawarra Health and Medical Research Institute  Smart Foods Centre  All participants and staff involved @Lauren_Houston