Objectives Recognize variation in clinical skill among experienced - - PDF document

1

Northwestern University Feinberg School of Medicine

Simulation-based Mastery Learning Research as Translational Science

Jeffrey H. Barsuk, MD, MS, SFHM Associate Professor of Medicine Division of Hospital Medicine

Conflict of Interest: MERCI: Educational Grant

Objectives

• Recognize variation in clinical skill among experienced

professionals and need for simulation

• Describe how simulation-based education qualifies as

translational science

• Explain the advantages of mastery learning with

deliberate practice over traditional educational strategies

• List studies that use deliberate practice and mastery

learning to improve patient outcomes

2

¡ Recognize variation in clinical skill among experienced professionals and need for simulation

Traditional Training ≠ Competence Lumbar Puncture Skills

Barsuk et al. Neurology. 2012.

Traditional Training ≠ Competence Thoracentesis Skills

Wayne DB, et al. J Hosp Med. 2008;3:48.

3

Traditional Training ≠ Competence Central Line Insertion Skills

Barsuk JH, et al. Am J Kidney Dis. 2009;54:70.

Ranking the Effectiveness of Error- Reduction Strategies

– Forcing functions and constraints – Automation and computerization – Standardization and protocols – Checklists and double- check systems – Rules and policies – Education and information – Exhortation: “Be careful”

Most Effective Least Effective

Gosbee, Gosbee Joint Commission Resources 2005

Recommended Hierarchy of Actions:

Stronger actions

Architectural/physical plant changes New device with usability testing before purchasing Engineering control or interlock (forcing functions) Simplify the process and remove unnecessary steps Standardize on equipment or process or caremaps Tangible involvement and action by leadership

Intermediate Actions

Increase in staffing/decrease in workload Software enhancements/modifications Eliminate/reduce distractions Checklist/cognitive aid Eliminate look and sound alikes Read back Enhanced documentation/communication Redundancy

Weaker Actions

Double checks Warnings and labels New procedure/memorandum/policy

Training

Additional study/analysis http://www.patientsafety.gov

4

Why Simulation Works

“Tell me, and I will forget. Show me, and I may remember. Involve me, and I will understand.”

• Confucius, 450 BC

Study Statistics for Each Study Correlation and 95% CI Lower Upper Correlation Limit Limit p-Value 0.81 0.70 0.88 0.000 0.80 0.56 0.91 0.000 0.67 0.40 0.84 0.000 0.62 0.29 0.82 0.001 0.52 0.17 0.75 0.006 0.51 0.17 0.74 0.005 0.78 0.73 0.82 0.000 0.61 0.29 0.81 0.001 0.59 0.47 0.69 0.000 0.51 0.29 0.68 0.000 0.80 0.72 0.86 0.000 0.79 0.70 0.86 0.000 0.77 0.71 0.82 0.000 0.71 0.55 0.83 0.000

0.71 0.65 0.76 0.000

Favors Traditional Favors SBME

• 1.00 -0.50

0.00 0.50 1.00

Overall Effect Size

Randomized Trials

• 1. Wayne, et al, 2005
• 2. Ahlberg, et al, 2007
• 3. Andreatta, et al, 2006
• 4. Korndorffer, et al, 2005
• 5. Korndorffer, et al, 2005
• 6. Van Sickle, et al, 2008

Cohort Studies

• 7. Issenberg, et al, 2002
• 8. Barsuk, et al, 2009
• 9. Butter, et al 2010

Case-Control Studies

• 10. Wayne, et al, 2008

Pre-Post Baseline Studies

• 11. Wayne, et al, 2008
• 12. Barsuk, et al, 2009
• 13. Barsuk, et al, 2009
• 14. Stefanidis, et al, 2006

Meta analysis of simulation based education vs. traditional medical education

Describe how simulation-based education qualifies as translational science

5 Extending the Endpoint: Medical Education as Translational Science

20 2010

Kirkpatrick ¡Level ¡

• 1. Reac1on ¡
• 2. Learning ¡
• 3. ¡Behavior ¡
• 4. ¡Results ¡

McGa Gaghi hie W

• WC. Scien

enceT eTranslationalMed edicine. .

Study Skill Outcome

Andreatta et al. Laparoscopic Skills T1 Skills on porcine model Seymour et al. Lap Chole T2 Skills, burning wrong tissue, faster Cohen et al. Colonoscopy T2 Ability to reach cecum, identify abnormal Sedlack et al. Colonoscopy T2 Mucosal visualization, depth inserted Mayo et al. Airway T2 Checklist performance actual emergency Blum et al. Bronchoscopy T2 Visual cutes and subjective thoroughness Draycott et al. Shoulder dystocia T2, T3 Neonatal injury Zendejas et al. Lap Hernia Repair T3 Complications Britt et al. Central Line Insertion T2, T3 Overall complications Kessler et al. Infant Lumbar Puncture T2 Procedure success

Explain the advantages of mastery learning with deliberate practice over traditional educational strategies

6 Typical Educational Outcomes

P<.001

All Medical Education is Not the Same

• Mastery Learning
• Deliberate Practice

Choosing the Mastery Learning Method to Teach

• 1. Minimum passing mastery standard (MPS)
• 2. Time varies, outcomes are uniform

Old Bell Curve New “J” Curve

7

Deliberate Practice (DP)

• Identify well-defined task
• Focused, repetitive practice
• Informative feedback
• Opportunity to correct errors
• Goal: skill improvement

Ericsson Acad Med. 2004; McGaghie et al., Chest 2009

20

Design and sequencing of training activities

• A. Ericsson 2007

8

Mastery Learning Educational Outcomes

¡ Academic Medicine 2013 “mastery learning SBME is superior to non-mastery instruction but takes more time” [limited data- 80 studies]

List studies that use deliberate practice and mastery learning to improve patient

• utcomes

¡

9 Central Venous Catheter Insertion (CVC)

Simulation-based Mastery Learning

Educa1onal ¡and ¡ Clinical ¡Outcomes ¡

10

11

CVC Training

• Internal medicine and emergency

medicine residents

• Pre- and post-testing 27-item skills

checklist on the simulator

• Two 2-hour education sessions
• A minimal passing score of 79% was set

by an expert panel

Barsuk JH, et al. Crit Care Med. 2009

CVC Insertion Checklist

T1: CVC Insertion Skills

Barsuk et al. Crit Care Med 2009

p<.005

50.6 (23.4) 48.3 (26.8) 95.9 (5.1) 97.4 (3.5)

10 20 30 40 50 60 70 80 90 100

Internal Jugular Subclavian

% Correct Pre Post MPS

12

T4: ¡CVC ¡Skill: ¡Long-­‑term ¡Reten5on ¡

Barsuk JH, et al. Acad Med 2010 Barsuk et al. Crit Care Med 2009

T2/T3: Patient Outcomes

Barsuk et al. Arch Intern Med. 2009 (p=.001)

85% REDUCTION

T3: CLABSI Rates

13

• A common concern:

– We cannot afford the supplies, and the salary support to do this in our institution.

Cohen et al. Simulation in Healthcare 2010

T3 ¡OUTCOMES: ¡BENEFITS ¡TO ¡SOCIETY ¡

• The total annual estimated savings

were approximately $820,000, 139 patient hospital days, and 120 MICU days

• When compared with the cost of
• ur intervention ($112,000), the net

savings was approximately $708,000

T4 Outcomes: Cost Effectiveness

Cohen et al. Simulation in Healthcare 2010

T3 ¡OUTCOMES: ¡BENEFITS ¡TO ¡SOCIETY ¡

• The total annual estimated savings

were approximately $820,000, 139 patient hospital days, and 120 MICU days

• When compared with the cost of
• ur intervention ($112,000), the net

savings was approximately $708,000

T4 Outcomes: Cost Effectiveness

14

Can it work at your hospital? T4: Small Community Hospital CLABSI Rates

2 4 6 8 10 12 Oct-08 Dec-08 Feb-09 Apr-09 Jun-09 Aug-09 Oct-09 Dec-09 Feb-10 Apr-10 Jun-10 Aug-10 Oct-10 Dec-10 Feb-11 Apr-11 Jun-11 Aug-11 Oct-11 Dec-11 Feb-12 Apr-12 Infection Rate per 1000 Catheter Days, No. MSICU

Simulator-trained residents enter the unit

Small Community Hospital CLABSI Rates

2 4 6 8 10 12 Oct-08 Dec-08 Feb-09 Apr-09 Jun-09 Aug-09 Oct-09 Dec-09 Feb-10 Apr-10 Jun-10 Aug-10 Oct-10 Dec-10 Feb-11 Apr-11 Jun-11 Aug-11 Oct-11 Dec-11 Feb-12 Apr-12 Infection Rate per 1000 Catheter Days, No. MSICU

Simulator-trained residents enter the unit

74% REDUCTION!

15

2 ¡cohorts ¡of ¡chief ¡residents ¡ T4: ¡SBML ¡Training ¡Collateral ¡Effects ¡

7% ¡ 16% ¡ 38% ¡ 11% ¡ 19% ¡ 38% ¡ 0% ¡ 5% ¡ 10% ¡ 15% ¡ 20% ¡ 25% ¡ 30% ¡ 35% ¡ 40% ¡ 2007 ¡ 2008 ¡ 2009 ¡

Pretest ¡ ¡Pass ¡Rate ¡

Internal ¡Jugular ¡ Subclavian ¡

p=.004 p=.028

Barsuk et al. Acad Med 2011

Paracentesis ¡

16

Paracentesis Skills

¡ Simulation-based Mastery Learning

Methods

• First year internal medicine residents
• N=58
• Pretest
• 3 hours education (Deliberate practice)
• Posttest (Minimum passing score)

Barsuk et al. J Grad Med Educ. 2012

17

T1 Paracentesis Simulated Skills Performance T2/T3 SBML Paracentesis Outcomes

Barsuk et al. Am J Med. 2013

T4: Cost of IR and SBML procedures

Attribute Interventional Radiology Procedures Interventional Radiology Cost Bedside Procedures Bedside Cost Blood products 100 units $100,125.00 40 units $34,698.00 Number of cases 331 cases $119.467.83 433 cases

• Simulation

costs

• $0
• $23,434.25

Total

• $219,592.83
• $58,132.25

Cost/case

• $663.42
• $134.01*

*Cost of simulation training per case based on average of 124 bedside procedures per year

(Other cost variables such as differences in physician billing and reimbursement or equipment for IR and bedside procedures were equal between interventional radiology and bedside procedures at our institution. Therefore, they are excluded from the table.) Barsuk et al. Sim in Healthc. Submitted

18 Advanced Cardiac Life Support

Methods

Training used a human patient simulator

• 6 core ACLS algorithms
• Pre-testing, 3 x 2-hours of practice,

and post testing

• ACLS expert teacher
• Deliberate practice
• Minimum Passing Score

Wayne et al. CHEST. 2008 Didwania et al. JGME 2011

T2/T3 Outcomes: American Heart Association Guidelines

Four key measures:

• Provision of basic life support as a first response
• Selection and dosage of the first drug administered
• Sequence and dosage of subsequent drugs administered
• Appropriate use of defibrillation, cardioversion, or pacing.

19

Wayne DB, et al. Chest 2008.

T2/T3 Outcomes

68% ¡ 86% ¡ 0% ¡ 20% ¡ 40% ¡ 60% ¡ 80% ¡ 100% ¡

2004 PGY-2 Non-mastery-trained Residents 2008 PGY-2 SBML-trained Residents Adherence to AHA Guidelines

Error ¡Bars ¡+/-­‑ ¡1 ¡SD ¡

T4: Culture? SBML-traied to non-SBML trained residents’ adherence to AHA Guidelines

Conclusions

• Simulation-based educational interventions

are effective quality improvement tools

• Deliberate practice and simulation-based

mastery learning should be used to ensure consistent learner and clinical outcomes

• More study is needed to demonstrate similar
• utcomes in other populations and skills

20

Mastery ¡Learning: ¡a ¡new ¡paradigm ¡

Improved outcomes in: bedside procedures, ACLS events, physical examination skills, ICU clinical skills, end of life discussions. GOAL: Educate Superb Clinicians