Objectives Recognize variation in clinical skill among experienced - PDF document

Simulation-based Mastery Learning Research as Translational Science Northwestern University Feinberg School of Medicine Jeffrey H. Barsuk, MD, MS, SFHM Associate Professor of Medicine Division of Hospital Medicine Conflict of Interest: MERCI: Educational Grant Objectives • Recognize variation in clinical skill among experienced professionals and need for simulation • Describe how simulation-based education qualifies as translational science • Explain the advantages of mastery learning with deliberate practice over traditional educational strategies • List studies that use deliberate practice and mastery learning to improve patient outcomes 1

¡ Recognize variation in clinical skill among experienced professionals and need for simulation Traditional Training ≠ Competence Lumbar Puncture Skills Barsuk et al. Neurology. 2012. Traditional Training ≠ Competence Thoracentesis Skills Wayne DB, et al. J Hosp Med. 2008;3:48. 2

Traditional Training ≠ Competence Central Line Insertion Skills Barsuk JH, et al. Am J Kidney Dis. 2009;54:70 . Ranking the Effectiveness of Error- Reduction Strategies Most Effective – Forcing functions and constraints – Automation and computerization – Standardization and protocols – Checklists and double- check systems – Rules and policies – Education and information Least Effective – Exhortation: “Be careful” Gosbee, Gosbee Joint Commission Resources 2005 Recommended Hierarchy of Actions: Stronger actions Architectural/physical plant changes New device with usability testing before purchasing Engineering control or interlock (forcing functions) Simplify the process and remove unnecessary steps Standardize on equipment or process or caremaps Tangible involvement and action by leadership Intermediate Actions Increase in staffing/decrease in workload Software enhancements/modifications Eliminate/reduce distractions Checklist/cognitive aid Eliminate look and sound alikes Read back Enhanced documentation/communication Redundancy Weaker Actions Double checks Warnings and labels New procedure/memorandum/policy Training Additional study/analysis http://www.patientsafety.gov 3

Why Simulation Works “ Tell me, and I will forget. Show me, and I may remember. Involve me, and I will understand. ” -Confucius, 450 BC Meta analysis of simulation based education vs. traditional medical education Statistics for Each Study Correlation and 95% CI Lower Upper Study Correlation Limit Limit p-Value Randomized Trials 0.81 0.70 0.88 0.000 1. Wayne, et al, 2005 0.80 0.56 0.91 0.000 2. Ahlberg, et al, 2007 0.67 0.40 0.84 0.000 3. Andreatta, et al, 2006 0.62 0.29 0.82 0.001 4. Korndorffer, et al, 2005 0.52 0.17 0.75 0.006 5. Korndorffer, et al, 2005 0.51 0.17 0.74 0.005 6. Van Sickle, et al, 2008 Cohort Studies 0.78 0.73 0.82 0.000 7. Issenberg, et al, 2002 0.61 0.29 0.81 0.001 8. Barsuk, et al, 2009 0.59 0.47 0.69 0.000 9. Butter, et al 2010 Case-Control Studies 0.51 0.29 0.68 0.000 10. Wayne, et al, 2008 Pre-Post Baseline Studies 0.80 0.72 0.86 0.000 11. Wayne, et al, 2008 0.79 0.70 0.86 0.000 12. Barsuk, et al, 2009 0.77 0.71 0.82 0.000 13. Barsuk, et al, 2009 0.71 0.55 0.83 0.000 14. Stefanidis, et al, 2006 -1.00 -0.50 0.00 0.50 1.00 Overall Effect 0.71 0.65 0.76 0.000 Favors Favors SBME Size Traditional Describe how simulation-based education qualifies as translational science 4

Extending the Endpoint: Medical Education as Translational Science Kirkpatrick ¡Level ¡ 1. Reac1on ¡ 3. ¡Behavior ¡ 4. ¡Results ¡ 2. Learning ¡ McGa Gaghi hie W WC. Scien enceT eTranslationalMed edicine . . 2010 20 0 Study Skill Outcome Andreatta et al. Laparoscopic Skills T1 Skills on porcine model Seymour et al. Lap Chole T2 Skills, burning wrong tissue, faster Cohen et al. Colonoscopy T2 Ability to reach cecum, identify abnormal Sedlack et al. Colonoscopy T2 Mucosal visualization, depth inserted Mayo et al. Airway T2 Checklist performance actual emergency Blum et al. Bronchoscopy T2 Visual cutes and subjective thoroughness Draycott et al. Shoulder dystocia T2, T3 Neonatal injury Zendejas et al. Lap Hernia Repair T3 Complications Britt et al. Central Line Insertion T2, T3 Overall complications Kessler et al. Infant Lumbar Puncture T2 Procedure success Explain the advantages of mastery learning with deliberate practice over traditional educational strategies 5

Typical Educational Outcomes P<.001 All Medical Education is Not the Same • Mastery Learning • Deliberate Practice Choosing the Mastery Learning Method to Teach 1. Minimum passing mastery standard (MPS) 2. Time varies, outcomes are uniform Old Bell Curve New “ J ” Curve 6

Deliberate Practice (DP) • Identify well-defined task • Focused, repetitive practice • Informative feedback • Opportunity to correct errors • Goal: skill improvement Ericsson Acad Med. 2004; McGaghie et al., Chest 2009 20 Design and sequencing of training activities A. Ericsson 2007 7

Mastery Learning Educational Outcomes ¡ Academic Medicine 2013 “mastery learning SBME is superior to non-mastery instruction but takes more time” [limited data- 80 studies] List studies that use deliberate practice and mastery learning to improve patient outcomes ¡ 8

Central Venous Catheter Insertion (CVC) Simulation-based Mastery Learning Educa1onal ¡and ¡ Clinical ¡Outcomes ¡ 9

10

CVC Training • Internal medicine and emergency medicine residents • Pre- and post-testing 27-item skills checklist on the simulator • Two 2-hour education sessions • A minimal passing score of 79% was set by an expert panel Barsuk JH, et al. Crit Care Med . 2009 CVC Insertion Checklist T1: CVC Insertion Skills 100 90 80 70 60 % Correct Pre 97.4 50 95.9 (3.5) (5.1) 40 Post 30 50.6 48.3 MPS (23.4) (26.8) 20 10 0 Internal Jugular Subclavian p <.005 Barsuk et al. Crit Care Med 2009 11

T4: ¡CVC ¡Skill: ¡Long-­‑term ¡Reten5on ¡ Barsuk JH, et al. Acad Med 2010 T2/T3: Patient Outcomes Barsuk et al. Crit Care Med 2009 T3: CLABSI Rates 85% REDUCTION ( p =.001) Barsuk et al. Arch Intern Med . 2009 12

• A common concern: – We cannot afford the supplies, and the salary support to do this in our institution. T3 ¡OUTCOMES: ¡BENEFITS ¡TO ¡SOCIETY ¡ T4 Outcomes: Cost Effectiveness • The total annual estimated savings were approximately $820,000, 139 patient hospital days, and 120 MICU days • When compared with the cost of our intervention ($112,000), the net savings was approximately $708,000 Cohen et al. Simulation in Healthcare 2010 T3 ¡OUTCOMES: ¡BENEFITS ¡TO ¡SOCIETY ¡ T4 Outcomes: Cost Effectiveness • The total annual estimated savings were approximately $820,000, 139 patient hospital days, and 120 MICU days • When compared with the cost of our intervention ($112,000), the net savings was approximately $708,000 Cohen et al. Simulation in Healthcare 2010 13

Infection Rate per 1000 Catheter Days, No. Infection Rate per 1000 Catheter Days, No. 10 12 10 12 Oct-08 0 2 4 6 8 Oct-08 0 2 4 6 8 T4: Small Community Hospital Can it work at your hospital? Dec-08 Dec-08 Small Community Hospital Feb-09 Feb-09 MSICU MSICU 74% REDUCTION! Apr-09 Apr-09 Jun-09 Jun-09 Aug-09 Aug-09 CLABSI Rates CLABSI Rates Oct-09 Oct-09 Dec-09 Dec-09 Feb-10 Feb-10 Apr-10 Apr-10 Jun-10 Jun-10 Aug-10 Aug-10 Oct-10 Oct-10 residents enter the unit Simulator-trained residents enter the unit Simulator-trained Dec-10 Dec-10 Feb-11 Feb-11 Apr-11 Apr-11 Jun-11 Jun-11 Aug-11 Aug-11 Oct-11 Oct-11 Dec-11 Dec-11 Feb-12 Feb-12 Apr-12 Apr-12 14

2 ¡cohorts ¡of ¡chief ¡residents ¡ T4: ¡SBML ¡Training ¡Collateral ¡Effects ¡ 40% ¡ 35% ¡ 30% ¡ Pretest ¡ ¡Pass ¡Rate ¡ 25% ¡ 20% ¡ 38% ¡ 38% ¡ 15% ¡ 10% ¡ 19% ¡ 16% ¡ 11% ¡ 5% ¡ 7% ¡ 0% ¡ 2007 ¡ 2008 ¡ 2009 ¡ Internal ¡Jugular ¡ Subclavian ¡ p =.004 p =.028 Barsuk et al. Acad Med 2011 Paracentesis ¡ 15

Paracentesis Skills ¡ Simulation-based Mastery Learning Methods • First year internal medicine residents • N=58 • Pretest • 3 hours education (Deliberate practice) • Posttest (Minimum passing score) Barsuk et al. J Grad Med Educ . 2012 16

T1 Paracentesis Simulated Skills Performance T2/T3 SBML Paracentesis Outcomes Barsuk et al. Am J Med . 2013 T4: Cost of IR and SBML procedures Attribute Interventional Interventional Bedside Bedside Radiology Radiology Procedures Cost Procedures Cost Blood 100 units $100,125.00 40 units $34,698.00 products Number of 331 cases $119.467.83 433 cases -- cases Simulation -- $0 -- $23,434.25 costs Total -- $219,592.83 -- $58,132.25 $134.01 * Cost/case -- $663.42 -- * Cost of simulation training per case based on average of 124 bedside procedures per year (Other cost variables such as differences in physician billing and reimbursement or equipment for IR and bedside procedures were equal between interventional radiology and bedside procedures at our institution. Therefore, they are excluded from the table.) Barsuk et al. Sim in Healthc. Submitted 17