CLINICAL SAFETY INCIDENT TAXONOMY PERFORMANCE ON J48 AND RANDOM - PowerPoint PPT Presentation

CLINICAL SAFETY INCIDENT TAXONOMY PERFORMANCE ON J48 AND RANDOM FOREST J.GUPTA, J.PATRICK, S.PHOON SCHOOL OF COMPUTER SCIENCES SYDNEY UNIVERSITY HEALTH INFORMATICS CONFERENCE 2019 MELBOURNE

MOTIVATIONS • Clinical safety incident (CSI) reports are text documents about what caused or could have caused harm to the patient in the process of receiving care in a health care organization. • A patient received vancomycin tablets instead of prescribed 24 hours vancomycin infusion. • CSI class Medication •

MOTIVATIONS • CSI rate – 1:10 patient admitted in hospital for treatment • Approaches to Improve patient safety and quality of service • Record CSI and learn – improve organizational memory • Robust taxonomy to classify CSI • Robust risk classification system • Automation and alerting system • Urgent need for innovation in classifying IIMS datasets

OBJECTIVES • To test and improve - CSI taxonomies • Identify core concepts that determine the uniqueness of CSI classes TOOLS • Backbone of AI is Machine Learning • ML – Supervised, Unsupervised and Reinforcement • Techniques – Classification, Regression, Clustering

OVERVIEW - SOFTWARE ARCHITECTURE & DATA WAREHOUSE Incident Information Management System (IIMS) 1 AA 2 AV 3 BHP 4 BBP 5 CM 6 DOC 7 FALL 12 Classes 8 HAI 13 Classes 9 MED 10 NUT 11 PATH 12 PC 13 PU Data pool: 7 Hospitals datasets Period:2004 - 2012 Approx: 25,000 CIT

O1: Classification and Validation O2: Testing high performing CSI of CSI taxonomy in IIMS taxonomies Research Questions Research Questions RQ1 – IIMS for classification RQ6 – Classifiers effect RQ2 – Amenable field for SLC RQ7 – Advance features effect RQ3 – Field types for SLC RQ8 – Concept numbers effect RQ4 – Categorical and/or text field RQ9 – CSI labeller effect RQ5 – Optimal CSI for SLC RQ10 – Taxonomy effect overall * NB and MNB RQ11 – Taxonomy effect on * J48 and Random Forest classes and * SVM confusion matrix

J48 SIMPLIFIED ADV AND DISADV Advantages of Decision Tree: • Easy to interpret • Handles both continuous and categorical targets attributes. • Performs well on large data sets • Requires minimum data cleaning Disadvantages of Decision Tree: • Prone to over-fitting

RANDOM FOREST ADV AND DISADV Advantages of Random Forest algorithm: High predictive accuracy Efficient on large data sets Ability to handle multiple input features without need for feature deletion Feature selection is possible Disadvantages of Random Forest algorithm: Interpretation is complex.

EXPERIMENT DESIGN

Taxonomies > GRM WHO WHO_I Classes Clinician Expert Classes Expert Classes Expert Aggression Aggressor (AA) 300 376 Behavior 848 BEH 848 (BEH) Aggression Victim (AV) 300 249 Blood and Blood Product (BBP) 300 289 BBP 271 BBP 271 Pathology Lab (PATH) 300 297 PATH 401 Clinical 300 270 Processes 728 Behavior & Human Performance (BHP) CLP 327 (CLP) Clinical Management (CM) 300 310 Documentation (DOC) 300 299 DOC 206 DOC 206 Falls (FALL) 300 305 FALL 325 FALL 325 Hospital Associated Infection (HAI) 300 294 HAI 298 HAI 298 Medication (MED) 300 308 MED 324 MED 324 Nutrition (NUT) 300 295 NUT 300 NUT 300 Pressure Ulcers (PU) 300 302 UC/PU 300 PU 294 Unclassified Class (UC) 6 UC 6 CSIs Total 3600 3600 3600 3600

RESULTS: EXPERIMENTS PERFORMANCE ON CLASSIFIERS J48 AND RANDOM FOREST (RF) ON TAXONOMIES, GENERIC REFERENCE MODEL (GRM), WORLD HEALTH ORGANIZATION (WHO) AND IMPROVED (WHO-I) TAXONOMY FOR CSI CLASSIFICATION. Kappa Processing Exp. No Taxonomy Classifier Accuracy Recall F Measure RMSE AUC Score time (sec.) 1 J48 0.62 0.62 0.62 0.59 0.21 0.56 23.92 GRM 2 RF 0.72 0.72 0.71 0.70 0.18 0.95 22.07 3 J48 0.72 0.72 0.72 0.67 0.23 0.65 19.90 WHO 4 RF 0.81 0.82 0.82 0.78 0.19 0.87 16.62 5 J48 0.74 0.74 0.73 0.71 0.20 0.88 14.78 WHO-I 6 RF 0.83 0.83 0.83 0.81 0.17 0.98 10.24

0.97 0.88 0.89 0.79 0.93 0.80 0.86 0.84 CLASSIFIER J48 AND RANDOM FOREST ON GENERIC REFERENCE MODEL (G), WORLD 0.87 0.80 0.82 0.74 RESULTS:CLINICAL SAFETY INCIDENT CLASSES ACCURACY (ACU) RATE APPLYING 0.81 0.63 0.76 0.65 0.83 0.69 0.89 0.78 HEALTH ORGANIZATION (W) AND IMPROVED WHO (I) TAXONOMIES 0.88 0.77 0.90 0.80 0.81 0.80 0.79 0.72 0.90 0.76 0.53 0.49 0.54 0.36 RF_ACU 0.74 0.64 0.83 0.83 0.78 0.78 J48_ACU 0.91 0.91 0.91 0.74 0.72 0.62 0.74 0.64 0.36 0.34 0.41 0.39 0.88 0.49 0.72 0.62 0.36 0.34 0.41 0.39 0.96 0.83 0.92 0.80 0.44 0.38 0.70 0.50 1.10 1.00 0.90 0.80 0.70 0.60 0.50 0.40 0.30

RESULT:PERCENTAGE NUMBER OF CLINICAL SAFETY INCIDENT CLASSES WITH HIGH ACCURACY AND RECALL RATES APPLYING CLASSIFIER J48 AND RANDOM FOREST ON GENERIC REFERENCE MODEL (G), WORLD HEALTH ORGANIZATION (W) AND IMPROVED WHO (I) TAXONOMIES 120% 100% 80% 60% 40% 20% 0% J48 RF J48 RF J48 RF GRM WHO WHO-I Acurracy (>0.75) Precision (>0.75)

KEY FINDINGS & WHAT NEXT • Using Classifiers for Multiclass datasets like Clinical Incident Types in IIMS is achievable • Confusion matrix is useful in improving the classifiers performance • Standard measures of performance are adequate to determine changes • RF classifier works better with Clinical safety Incident Types • Black box – how is the decision made • Open the box – using set theoretics configurational methodology