External Quality Assessment Automated Laboratory Joseph Murray - - PowerPoint PPT Presentation

External Quality Assessment

Automated Laboratory

Joseph Murray

What is External Quality Assessment?

EQA – “is a program in which multiple samples are periodically sent to members of a group of laboratories for analysis; whereby each laboratory’s results are compared with those of other laboratories in the group and/or with an assigned value, and reported to the participating laboratories and others.”

Some key points...

• Analyte value unknown to participants prior to analysis
• EQA specimens should be analysed using exactly the same process as

patient specimens

• Allows for inter-laboratory comparison of reagents, methods,

instruments

• The data is kept confidentially from other participants ensuring

anonymity

• Unlike Internal Quality Control (IQC) which provides an indication of how

an assay is performing in real-time the results of EQA are retrospective

• EQA schemes are available for most of the tests available across all

disciplines in pathology

• Provided by a number of different organisations
• In Clinical Chemistry mainly focus on the analytical process and method

performance –rather than pre- and post-analytical phases

• Participation in EQA schemes is a requirement for UKAS accreditation

An overview of the EQA Process...

External Quality Assessment Scheme USER External Quality Assessment Scheme PROVIDER

• 5. RESULTS from all

participating USER laboratories analyzed and a report indicating the performance of an individual laboratory's performance in relation the performance of all participating laboratories

• 1. Clinical material dispatched

to the USER laboratory

• 2. Clinical Material received

by the USER laboratory

• 4. Examination results

returned to the External Quality Assessment scheme PROVIDER

• 3. Clinical material examined

by the USER laboratory and the results recorded

• 6. Report indicating the

performance of an individual USER laboratory's performance in relation to the performance of all participating laboratories

• 7. USER laboratory reviews its

performance in relation the to the performance of all participating laboratories and takes action to remedy any problems

Why do it?

QUALITY ASSURANCE!!!!!!

EQA is one facet of QUALITY ASSURACE. In addition to IQC, health & safety, clinical governance , training & competency and audit it helps us as a service to achieve our QUALITY goals and form part of the laboratory’s Quality Management System (QMS). Further info can be found in the laboratory ‘Quality Manual’ - PROC-G-QualMan - on iPassport!

How does EQA fit into the overall Quality framework in UK Pathology?

EQA schemes should be accredited by UKAS to ISO standard 17043 National Quality Assurance Advisory Panels (NQAAPs) (discipline specific experts)

• versee EQA schemes and set

performance criteria. Joint Working Group for Quality Assessment (JWGQA) (multidisciplinary group within RCPath), ultimately responsible for EQA schemes

So what happens when performance is poor?

Performance criteria differs between EQA providers – performance standards not harmonised

If issue fails to get resolved passed to JWGQA JWGQA (no mandatory powers) contact relevant hospital CEOs and reports to Care Quality Commission (CQC) and UKAS!!! When performance poor EQA provider works with lab to improve Lab reported to relevant NQAAP if fails to engage with EQA provider/improve performance

Automation EQA Schemes...

Provider Scheme Analyte Distribution Frequency

NEQAS AFP, CEA & hcG AFP, CEA, HCG Monthly Antibiotic Assays Gentamicin, Vancomycin, Amikacin, Tobramycin Monthly Clinical Chemistry Albumin, ALK, ALT, Amylase, AST, Bicarbonate, Calcium, CK, Cl, Creatinine, GGT, Glucose, Iron, K, LDH, Lithium, Mg, Na, Osmolality, Phosphate, Tbil, TP, Urate, Urea Bi-weekly CRP CRP Monthly Fructosamine Fructosamine Monthly Peptide Hormones FSH, LH, Prolactin Monthly Pregnancy Testing HCG-Urine PTH, ACTH, hCT PTH Bi-Monthly Salicylate, Paracetamol, Ethanol Carbamazepine, Digoxin, Ethanol, Lithium, Paracetamol, Phenobarbitone, Phenytoin, Salicylate, Theophylline, Valporate Monthly Specific Proteins A1 antitrypsin, Caeruloplasmin, Haptoglobin, IGA, IGG, IGM, Orosomucoid, Transferin, Cortisol, DHAS, Female Testosterone, Male Testosterone, Oestradiol, Prgesterone, SHBG Monthly Thyroid Hormones FT3, FT4, TSH Monthly Urine Chemistries ACR, Albumin, Amylase, Calcium, Cl, Creatinine, Glucose, K, Mg, Na, Osmolality, PCR, Phospate, TP, Urate, Urea Monthly eFGR Creatinine, eGFR Monthly NEQAS – Cardiac Marker Cardiac Marker Troponin T Monthly NEQAS -IMMQAS CSF Biocehmistry Glucose, Lactate, TP Bi—Monthly PSA PSA Monthly Tumour Markers CA125, CA153, CA199 Bi-Monthly WEQAS Ammonia Ammonia Monthly Bile Acids Bile Acids Monthly Bilirubin Dbil, Tbil Monthly Lipids Cholesterol, HDL, LDL, Trigs, LPA Monthly NT PRO BNP BNP Bi-Monthly Serum ACE ACE Bi-Montlhy

The Process – Booking To Submission

Booking In

• EQA specimen labelled (primary/secondary tube and paperwork) and booked

in by trained member of staff in specimen reception

• Specimen treated like external laboratory requests – ‘EQA’ entered as

‘Requesting Source’

• Each scheme allocated unique patient registration number with demographics

providing additional info. E.g.

• Under ‘Specimen Comments’ distribution number and Sample number

entered

• A number of test super-sets have been created for each scheme

Procedure outlined in SOP ‘Booking in External Quality Assurance Samples’ - PROC-CB-H5.3/Booking - on iPassport!

Sample Processing

• Once labelled and booked Specimen Reception pass specimens (with

paperwork) to Automated laboratory

• Important to analyse EQA samples on day of receipt – if not possible should

be stored appropriately and delay in analysis recorded on paperwork

• Any preparation instructions provided should be adhered to ‘exactly.’

Especially important when reconstituting lyophilised samples e.g.

Reconstitution instructions for Troponin EQA TROPONIN...

Sample Processing

• Once labelled and booked Specimen Reception pass specimens (with

secondary tubes and paperwork) to Automated laboratory

• Important to analyse EQA samples on day of receipt – if not possible should

be stored appropriately and delay in analysis recorded on paperwork

• Any preparation instructions provided should be adhered to ‘exactly.’

Especially important when reconstituting lyophilised samples e.g. Troponin

• Once prepared samples transferred to labelled (bar-coded) secondary tube

for analysis – transfer just enough for analysis so that remainder in primary tube can be stored

• Ensure specimens placed on correct analyser – special consideration should

be made when analysing NEQAS General Chemistry distributions...

CITY General Chemistry EQA:

Two sets of specimens received for NEQAS General Chemistry Distributions per site Need to examine test requests on each sheet and decide most appropriate analyser for analysis!!!! Ideally each set should be analysed on SAME analyser every distribution – this allows performance issues to be correctly attributed to specific analysers Any discrepancies (i.e. moving samples between analysers to complete requests) should be raised with Section

• Senior. Set repertoire can be updated with NEQAS for

subsequent distributions Best practice to record on distribution sheet which analyser(s) was (were) used for analysis Which Analyser Would You Use To Analyse These Specimens? CITY General Chemistry EQA:

Post Analysis

• Check the specimen in Telepath. Ensure all requests have been completed...
• Check the EQA distribution sheet against what has been booked in on
• Telepath. Ensure no requests have been missed!!!! Better to resolve problems

now, rather than having to locate specimens later!

Transcription of Results

• Once analysed results manually transcribed onto distribution sheet
• Transcribe all results from Telepath system to sheet!!! Complete ALL

relevant fields e.g. Date of receipt/analysis, storage conditions, kit/calibrator lot number

Transcription of Results

• DON’T transcribe directly from analyser screen - units and decimal

places may be different!!!

Retention of Specimens & Paperwork

• Completed samples stored in ‘Complete EQA Draw’ on bottom shelf of ‘QC

Fridge’ at Sandwell and ‘EQA draw’ in Freezer in old Haemoglobinopathy lab at City

• Ensure when placing specimen into specimen bags that lids are tightly in place

to prevent spillage!!!

• Once reports have been returned from Scheme Provider EQA specimens at

Sandwell are moved to Freezer in Manuals lab. At City EQA specimens are moved from Freezer in old Immunology Automation lab on a monthly basis and retained in appropriate rack for period of three months

• The paperwork is placed in the ‘EQA tray’ mounted on the wall at Sandwell

and in the tray on the cabinet bellow the Communication Screen at City ready for submission

• Paperwork should NOT be transported across site so as not to go astray!!!

EQA SCHEME PROVIDER WEB ADDRESS NEQAS www.birminghamquality.org.uk WEQAS http://www.weqas.com/ IMMQAS www.immqas.org.uk NEQAS - Cardiac Markers www.ukneqas-cm.org.uk

Result Submission

• The Results for Automation EQA schemes are submitted manually via the

appropriate scheme Providers website – details of which are provided in the table below... Preparation, Analysis, Transcription and Result Submission covered in SOP ‘External Quality Assurance Preparation and Reporting’ - PROC-CB-H5.3/Reporting - on iPassport!

EQA Providers – Performance Criteria & Reports

Performance & Reports

• Performance criteria differs between EQA providers e.g. UK NEQAS

& WEQAS

• Performance criteria also differs between different divisions of same

provider e.g. UK NEQAS ‘Birmingham Quality Clinical Chemistry’, ‘Edinburgh Peptide Hormones’ and ‘Immunology, Immunochemistry & Allergy’

• Report format very different between providers
• Although different reports are designed to be easily and quickly

interpreted whilst performance scoring systems have similar underlying principles

• NQAAP is attempting to harmonise performance criteria between

providers using Minimum Analytical Performance Standards (MAPS)

UK NEQAS – UK National External Quality Assessment Scheme

Birmingham & Edinburgh Performance of each analyte measured based on an A, B, C scoring system...

• A is for Accuracy (total error)
• B is for Bias
• C is for Consistency of bias

Each score is calculate using all specimens within a rolling time window – usually six

• distributions. As new data is added the oldest is lost

Calculated as follows....

• B score is the trimmed mean of % bias for each specimen within time window

% bias = ((result-target)/target)*100

• C score is the trimmed SD of the B Score
• A score is the trimmed mean of the Specimen Accuracy Indices
• Specimen Accuracy Index calculated for each specimen using transformed bias
• Transformed bias = specimen % bias/degree of difficulty
• Degree of difficult is assigned to each specimen by NEQAS based on the ‘difficulty’ in
• btaining the target value
• Modulus of the transformed bias gives the Specimen Accuracy Index

REPORT - Feedback Page

• Provides any information NEQAS

wants to convey to users

• This includes any comments

made by the lab at the time of result submission (e.g. analytical issues) or any performance issues NEQAS wants highlighted

Performance Summary Icons

• Gives and indication of how each

analyte is performing at a glance – ideal for quickly reviewing reports

• Penalty box plots based on NEQAS

scoring system used to illustrate labs performance in relation to peers – a traffic light system helps distinguish between GOOD, ACCEPTABLE & POOR performance.

• Based on data from last six

distributions!!!

Distribution Summary

• Summarises the date from the current

distribution for each analyte

Distribution Summary

• Summarises the date from the current

distribution for each analyte

Method Summary

• List the methods (including

underlying principle) used by the laboratory for each analyte

Method Summary

• List the methods (including

underlying principle) used by the laboratory for each analyte

• The labs A score for each analyte

tabulated alongside the median A scores for ‘Method’ and ‘All Lab’. Allows comparison with peers

• Data represented as box & whisker plots

Analyte Specific Pages

• ‘Specimen level’ statistics - detailed

breakdown of how each analyte performed for each specimen in current distribution

• ‘Rolling time window level

statistics’ for each analyte

‘Specimen Level’ Stats

Analytes A, B and C scores with limits – rolling time window data, not just current distribution Labs method info – key relates to histograms Specimen composition – usefully for identifying possible matrix effects Comm0n terms:

• ALTM – All Laboratory Trimmed Mean
• GLTM – Group Laboratory Trimmed

Mean

• MLTM – Method Laboratory Trimmed

Mean

• % Bias Vs. Transformed Bias

‘Specimen Level’ Stats

Table shows mean, SD & CV for All, Group & Method Histogram displays graphically data from table Current distribution Box Whisker Plots of B & C scores for All, Group & Method

‘Rolling Time Window’ Stats

‘Rolling Time Window’ Stats

a. b. c. d.

• a. Specimen Accuracy Index
• b. A score
• c. Specimen % Bias
• d. B score

Plotted over a number of distributions – allows observation of Trends

a. b. c. d.

• a. Difference (result-target) Vs. Concentration
• b. C Score
• c. Specimen % Bias Vs. Concentration
• d. Penalty Box Plot of B Score against C Score
• a. and c. allows observation of concentration

dependent bias. b. Provides trend data for C

• score. d. allows between lab comparison – are

we in agreement?

‘Rolling Time Window’ Stats

‘Rolling Time Window’ Stats

a. b.

• a. Specimen Accuracy Index Vs. Concentration

– enables observation of concentration dependent inaccuracy

• b. Between-laboratory agreement by

concentration

UK NEQAS For Immunology, Immunochemistry & Allergy (IMMQAS)

Performance of each analyte measured based on a Variance Index Scoring System (VIS). Performance expressed in terms of...

• MRVIS - Mean Running Variance Index Score
• MRBIS - Mean Running Bias Index Score
• SDBIS - Standard Deviation of Bias Index Score

MRVIS gives an indication of the degree of imprecision. MRBIS gives an indication of any consistent tendency (positive or negative) SDBIS gives an indication of the variability of the bias index (hence imprecision) Data is collected within a rolling time window and ‘trimmed’ or ‘smoothed’ to remove erratic outliers An arbitrary scaling factor, Chosen Coefficient of Variation (CCV), is applied to each analyte to reflect current ‘state of the art’ and provide a common means of comparison between analytes

IMMQAS Report

Very similar to other UK NEQAS reports in terms of layout which uses...

IMMQAS Report

Very similar to other UK NEQAS report in terms of layout which uses... A Table of mean, SD & CV for All, Group & Method

IMMQAS Report

Very similar to other UK NEQAS report in terms of layout which uses... A Histogram to display graphically data from table

IMMQAS Report

Very similar to other UK NEQAS report in terms of layout which uses... Graphs to evaluated trend

IMMQAS Report

Very similar to other UK NEQAS report in terms of layout which uses... Scatter or Penalty box plot to displays inter-lab consensus

IMMQAS Report

Very similar to other UK NEQAS report in terms of layout!!!! Exception being the scoring system!!!

WEQAS - Wales External Quality Assessment Scheme

Performance of each analyte measured based on an Standard Deviation Index (SDI) scoring system used in conjunction with a WEQAS SD SDI is a Total Error score and is calculated as: SDI = (Participant result – target value) / WEQAS SD WEQAS SD is fixed for a given analyte and reflects current ‘state of the art’ and provides a common means of comparison between analytes

WEQAS Report – Performance Summary

Table referring to current distribution Performance...

• Overall Lab SDI is score for all

specimens (inc. multi-analyte) ran by lab on current distribution

• Median All Laboratory is

median SDI score for all labs.

• 97.5th percentile is SDI score

within which 97.5% of participants fell

WEQAS Report – Performance Summary

Performance criteria Graph shows Overall Lab SDI score

• ver a number of distributions –

allows observation of trend

WEQAS Report – Performance Summary

WEQAS Report – Performance Summary

Table of SDI scores including individual analyte SDIs for current distribution

Analyte Specific Pages

Analyte Specific Pages

Table shows mean, SD & uncertainty for lab, Method & Instrument SDI given for each specimen as well as average analyte SDI

Analyte Specific Pages

Graph shows analyte SDI scores over a number of distributions

Analyte Specific Pages

Bias Plots (Bland-Altman):

• Left hand plot represents Current

distribution – includes lab, method and instrument performance

• Right hand plot shows Previous

distribution results, including current distribution X-axis target value, y axis target +/-3SD

Analyte Specific Pages

Precision and Accuracy indicators

Review of EQA Reports

• EQA reports reviewed by Pervaz - EQA Officer for Clinical Chemistry
• Each section has a responsibility to monitor their own performance
• An EQA Evaluation Sheet is completed by the EQA Officer when

performance issues arise

• Performance issues are brought to the attention of Senior staff in the

relevant section and urgent action initiated to investigate/resolve the issue

• Biochemistry hold monthly EQA meetings where different sections

report their performance to the group

• EQA reports and evaluation sheets are stored in the EQA folder on

the ClinChem drive Interpretation and review of EQA reports is covered in SOP ‘Review and Interpretation of External Quality Assurance results’ - PROC-CB-H5.3/Interpretation - on iPassport!

EQA – Providing Quality Assurance

Consistent Poor Performance EQA Provider Intervention Improvement (Lab changed reagent supplier)

In-House Report Summaries

Review of EQA reports and monthly summarisation of data – to be displayed on EQA board to keep staff informed!!!

END