CLINICAL SAFETY INCIDENT TAXONOMY PERFORMANCE ON J48 AND RANDOM - - PowerPoint PPT Presentation

CLINICAL SAFETY INCIDENT TAXONOMY PERFORMANCE ON J48 AND RANDOM FOREST

J.GUPTA, J.PATRICK, S.PHOON SCHOOL OF COMPUTER SCIENCES SYDNEY UNIVERSITY HEALTH INFORMATICS CONFERENCE 2019 MELBOURNE

MOTIVATIONS

• Clinical safety incident (CSI) reports are text documents about

what caused or could have caused harm to the patient in the process of receiving care in a health care organization.

• A patient received vancomycin tablets instead of prescribed

24 hours vancomycin infusion.

• CSI class Medication

MOTIVATIONS

• CSI rate – 1:10 patient admitted in hospital for treatment
• Approaches to Improve patient safety and quality of service
• Record CSI and learn – improve organizational memory
• Robust taxonomy to classify CSI
• Robust risk classification system
• Automation and alerting system
• Urgent need for innovation in classifying IIMS datasets

OBJECTIVES

• To test and improve - CSI taxonomies
• Identify core concepts that determine the uniqueness of CSI classes
• Backbone of AI is Machine Learning
• ML – Supervised, Unsupervised and Reinforcement
• Techniques – Classification, Regression, Clustering

TOOLS

OVERVIEW - SOFTWARE ARCHITECTURE & DATA WAREHOUSE

Incident Information Management System (IIMS) 1 AA 2 AV 3 BHP 4 BBP 5 CM 6 DOC 7 FALL 12 Classes 8 HAI 13

Classes

9 MED 10 NUT 11 PATH 12 PC 13 PU Data pool: 7 Hospitals datasets Period:2004 - 2012 Approx: 25,000 CIT

RQ1 – IIMS for classification RQ2 – Amenable field for SLC RQ3 – Field types for SLC RQ4 – Categorical and/or text field RQ5 – Optimal CSI for SLC * NB and MNB * J48 and Random Forest * SVM RQ6 – Classifiers effect RQ7 – Advance features effect RQ8 – Concept numbers effect RQ9 – CSI labeller effect RQ10 – Taxonomy effect overall RQ11 – Taxonomy effect on classes and confusion matrix

O1: Classification and Validation

• f CSI taxonomy in IIMS

Research Questions O2: Testing high performing CSI taxonomies Research Questions

J48 SIMPLIFIED ADV AND DISADV

Advantages of Decision Tree:

• Easy to interpret
• Handles both continuous and

categorical targets attributes.

• Performs well on large data sets
• Requires minimum data cleaning

Disadvantages of Decision Tree:

• Prone to over-fitting

RANDOM FOREST ADV AND DISADV

Advantages of Random Forest algorithm: High predictive accuracy Efficient on large data sets Ability to handle multiple input features without need for feature deletion Feature selection is possible Disadvantages of Random Forest algorithm: Interpretation is complex.

EXPERIMENT DESIGN

Taxonomies > GRM WHO WHO_I

Classes

Clinician Expert Classes Expert Classes Expert Aggression Aggressor (AA) 300 376 Behavior (BEH) 848 BEH 848 Aggression Victim (AV) 300 249 Blood and Blood Product (BBP) 300 289 BBP 271 BBP 271 Pathology Lab (PATH) 300 297 Clinical Processes (CLP) 728 PATH 401 Behavior & Human Performance (BHP) 300 270 CLP 327 Clinical Management (CM) 300 310 Documentation (DOC) 300 299 DOC 206 DOC 206 Falls (FALL) 300 305 FALL 325 FALL 325 Hospital Associated Infection (HAI) 300 294 HAI 298 HAI 298 Medication (MED) 300 308 MED 324 MED 324 Nutrition (NUT) 300 295 NUT 300 NUT 300 Pressure Ulcers (PU) 300 302 UC/PU 300 PU 294 Unclassified Class (UC) 6 UC 6

CSIs Total

3600 3600 3600 3600

RESULTS: EXPERIMENTS PERFORMANCE ON CLASSIFIERS J48 AND RANDOM FOREST (RF) ON TAXONOMIES, GENERIC REFERENCE MODEL (GRM), WORLD HEALTH ORGANIZATION (WHO) AND IMPROVED (WHO-I) TAXONOMY FOR CSI CLASSIFICATION.

• Exp. No

Taxonomy Classifier Accuracy Recall F Measure Kappa Score RMSE AUC

Processing time (sec.)

1

GRM

J48 0.62 0.62 0.62 0.59 0.21 0.56 23.92 2 RF 0.72 0.72 0.71 0.70 0.18 0.95 22.07 3

WHO

J48 0.72 0.72 0.72 0.67 0.23 0.65 19.90 4 RF 0.81 0.82 0.82 0.78 0.19 0.87 16.62 5

WHO-I

J48 0.74 0.74 0.73 0.71 0.20 0.88 14.78 6 RF 0.83 0.83 0.83 0.81 0.17 0.98 10.24

RESULTS:CLINICAL SAFETY INCIDENT CLASSES ACCURACY (ACU) RATE APPLYING CLASSIFIER J48 AND RANDOM FOREST ON GENERIC REFERENCE MODEL (G), WORLD HEALTH ORGANIZATION (W) AND IMPROVED WHO (I) TAXONOMIES

0.50 0.38 0.80 0.83 0.39 0.34 0.62 0.49 0.39 0.34 0.64 0.62 0.74 0.91 0.78 0.83 0.64 0.36 0.49 0.76 0.72 0.80 0.80 0.77 0.78 0.69 0.65 0.63 0.74 0.80 0.84 0.80 0.79 0.88 0.70 0.44 0.92 0.96 0.41 0.36 0.72 0.88 0.41 0.36 0.74 0.72 0.91 0.91 0.78 0.83 0.74 0.54 0.53 0.90 0.79 0.81 0.90 0.88 0.89 0.83 0.76 0.81 0.82 0.87 0.86 0.93 0.89 0.97 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 1.10 J48_ACU RF_ACU

RESULT:PERCENTAGE NUMBER OF CLINICAL SAFETY INCIDENT CLASSES WITH HIGH ACCURACY AND RECALL RATES APPLYING CLASSIFIER J48 AND RANDOM FOREST ON GENERIC REFERENCE MODEL (G), WORLD HEALTH ORGANIZATION (W) AND IMPROVED WHO (I) TAXONOMIES 0% 20% 40% 60% 80% 100% 120% J48 RF J48 RF J48 RF GRM WHO WHO-I Acurracy (>0.75) Precision (>0.75)

KEY FINDINGS & WHAT NEXT

• Using Classifiers for Multiclass datasets like Clinical Incident Types in IIMS is

achievable

• Confusion matrix is useful in improving the classifiers performance
• Standard measures of performance are adequate to determine changes
• RF classifier works better with Clinical safety Incident Types
• Black box – how is the decision made
• Open the box – using set theoretics configurational methodology