through a source data verification audit Lauren Houston, Dr Yasmine - - PowerPoint PPT Presentation

@Lauren_Houston

Lauren Houston¹, Dr Yasmine Probst¹, Dr Allison Humphries¹

¹School of Medicine, Faculty of Science, Medicine and Health, University of Wollongong lah993@uowmail.edu.au

Measuring data quality through a source data verification audit

@Lauren_Houston

Health data has long been

scrutinised1,2

A large proportion of errors

are from transcribing data3,4

No “gold standard” method exists

to measure data quality error rates.

Background

• 1. Y.W. Lee et al. (2006)
• 3. M. Mealer et al. (2013)
• 2. M.N. Zozus et al. (2014)
• 4. M.L. Nahm et al. (2008)

@Lauren_Houston

What is source data verification?

Source data Electronic record Case report form

@Lauren_Houston

What are the gaps in the knowledge?

ICH GCP guidelines are non specific to amount, timing

and frequency of monitoring5

Cost-effectiveness of SDV6 No single definition to define data quality or universally

accepted method to measure error rates7,8

Audits may be published but not for public viewing8

• 5. ICH GCP (1996)
• 7. R.Rostami et al. (2009)
• 6. C. Baigent et al. (2008)
• 8. R.V. Gómez-Rioja et al. (2013)

@Lauren_Houston

Aim

To monitor data quality by developing and conducting source data verification audits to achieve quality assurance.

@Lauren_Houston

Study Background

UOW and IHMRI Healthy-lifestyle blinded-RCT  12 months 5 clinical Accredited Practising Dietitians To limit bias the audit was blinded

@Lauren_Houston

100% SDV on the 10% random sample Quality assurance rule developed whereby if,

 >5% of data variables were incorrect a second 10% random sample was extracted

Manual verification checks conducted Calculation of error rate

Method

@Lauren_Houston

Participants n=210 Random sample n=21 Total of 820 data points from 21 participants If >41 (5%) data points are incorrect Second 10% random sample of

• riginal

participants

Procedure of data audit

Participants n=210 Random sample n=21 Total of 685 data points from 21 participants If >34 (5%) data points are incorrect 100% SDV

• f all data

points

10% random sample 100% SDV

@Lauren_Houston

Audit 1

Source data verification Anthropometric Physiological Medications Electronic spreadsheet record Coded electronic spreadsheet record Source documents

@Lauren_Houston

Audit 2

Source data verification Anthropometric Medications Electronic spreadsheet record Coded electronic spreadsheet record Source documents

@Lauren_Houston

Coded electronic spreadsheet

@Lauren_Houston

Statistical Analysis

Total error = (code 2+3+4) / (code 1+2+3+4) Data “not entered” (code 5) – excluded Chi square, p<0.05 Post-hoc; adjusted standardised

residuals and z test of column proportions

@Lauren_Houston

0.5 1 1.5 2 2.5 3 Anthropometric Physiological Medications %

Minor error

Audit 1 Audit 2 2 4 6 8 10 12 14 16 Anthropometric Physiological Medication %

Major error

Audit 1 Audit 2

Results

@Lauren_Houston

10 20 30 40 50 60 Anthropometric Physiological Medication %

Not recorded data

Audit 1 Audit 2 5 10 15 20 25 30 Anthropometric Physiological Medication %

Not entered data

Audit 1 Audit 2

Results

@Lauren_Houston

10 20 30 40 50 60 70 80 90 100

Anthropometric audit 1 Anthropometric audit 2 Medication audit 1 Medication audit 2 Total audit 1 Total audit 2

%

Correct Incorrect

Correct and Incorrect

@Lauren_Houston

Results

Chi square had a significant difference;

χ2 (4, 1293) = 672.405, p = 0.000

Adjusted standardised residuals determined audit

sections were significantly different

From the z test of column proportions

anthropometric audit 1 and 2; medications audit 1 and 2 do not differ.

All other sections differed from each other.

@Lauren_Houston

Summary

Audit 1 physiological section <5% error Average total error anthropometric (9%), medications

(76%) and overall (34.5%)

Proportion of error trended upward as length of study

increased

“Not recorded” (code 4) data had the greatest

contribution to total error

@Lauren_Houston

Discussion

Importance of identifying errors and determining

solutions

If >10% of a clinical dataset is erroneous the data

may be considered unreliable

Developed a 5% quality assurance rule Data quality variations

@Lauren_Houston

Source documents considered the “gold standard” Source document-to-electronic spreadsheet Audits cannot guarantee 100% free from error Clinical research setting and trial design Did not determine the impact of audit findings

Limitations

@Lauren_Houston

Examine the 10% snapshot model with a 5% quality

assurance error rate

Standardise a SDV audit process Assess the frequency and

cost-effectiveness

Overcome barriers and increase awareness

Recommendations

@Lauren_Houston

How many errors are too many?

@Lauren_Houston

Acknowledgements

Dr. Yasmine Probst Dr. Allison Humphries Sr/Prof. Linda Tapsell A/Prof. Marijka Batterham Illawarra Health and Medical

Research Institute

Smart Foods Centre All participants and staff

involved