A case study of implementing simulation results in emergency stroke - - PowerPoint PPT Presentation

A case study of implementing simulation results in emergency stroke care

Dr Thomas Monks University of Exeter Medical School

Talk Overview

• Implementation of simulation results – any evidence?
• Background to the simulation study
• Overview of the model
• Timeline of implementation
• Empirical evaluation of system changes
• Evaluation conclusions

Implementation of simulation results

This talk describes the implementation and evaluation of changes to a stroke emergency pathway following a simulation study. What do I mean by implementation?

• Concrete: direct changes to the real system
• Abstract: learning or moving a debate forward

Implementation – the evidence

• There are lots of published case studies of simulation models
• Not many consider if results were implemented
• I don’t believe this is limited to one domain, but it has been

particularly well documented in healthcare by five systematic reviews between 1999 and 2011. “we were unable to reach any conclusions on the value of modelling in health care because the evidence of implementation was so scant” Fone et al. 2003

Why is the evidence missing?

Brailsford and Vissers (2011)

• Tension between what is seen as consultancy and research
• Different timelines for implementation and academic publication

Tako, Kotiadis and Vasilakis (2010)

• Lack of stakeholder involvement in key modeling stages
• Particularly conceptual modeling

Background to the simulation study

Context: acute stroke

The rapid loss of brain function due to a disruption in the blood supply to the brain

• Ischemic stroke (80%): lack of blood flow due to a blockage
• Hemorrhagic stroke: a bleed within the brain

Consequences of stroke

• There are around 110,000 strokes in England per year
• One quarter of patients with stroke are dead within one month, one third

by six months and one half by a year (Churlov and Donnan, 2012)

• Stroke accounts 9% of all deaths worldwide (12% in western countries)
• Many surviving patients are severely disabled

Treatment for acute ischemic stroke

• The only treatment for ischemic stroke is thrombolysis
• A clot busting drug called alteplase (recombinant tissue

plasminogen activator)

• Treatment is critically time dependent (time is brain)
• Risk of symptomatic intracerebral haemorrhage (4%–7%)
• It must be administered a short period from onset or the risks

begin to outweigh the benefits

Time dependent effectiveness

Treatment time Treat to get one attributable mRS 0-1 0-90 mins 91-180 mins 181-270 mins

• Research has largely focussed on extension of eligibility criteria;
• Our focus: analysis of the impact of reducing in-hospital delays;

Thrombolysis: high level pathway

Onset Pre-hospital care CT Scan Eligibility? Treatment 999 call Arrival to treatment time (ATT)

Annual strokes: 800 ATT: 60 minutes Thrombolysis: 4.5% Annual strokes: 625 ATT: 90 minutes Thrombolysis: 3.5% Annual strokes: 300 ATT: 70 minutes Thrombolysis: 2% Annual stroke: 650 ATT: 110 minutes Thrombolysis: 4%

The simulation project

1. What is the expected impact on the thrombolysis rate by extending the alteplase window from 3 to 4.5 hours from onset? 2. What is the clinical benefit of reducing in-hospital delays to treatment compared to extending the alteplase time window? 3. What in-hospital process changes are most effective in improving thrombolysis rates and reducing post-stroke disability? 4. Are the modelled benefits realised once implemented in the hospital? 5. Did the simulation project help implementation as expected?

Our assumptions about implementation

• 1. Involving the acute stroke team and emergency department in

conceptual modelling will aid the uptake of recommendations

• 2. The use of VIS within DES engages problem stakeholders and

increases the transparency of a model

• 3. Modelling provides structure in a debate between stakeholders

with competing interests

Quick overview of the model

Monks T, Pitt M, Stein K and James M.A. Maximizing the Population Benefit from Thrombolysis in Acute Ischemic Stroke: A Modeling Study of In- Hospital Delays. Stroke 2012; 43(10).

Methods: Discrete-event simulation

Key model outputs

Percentage of patients thrombolysed Patients with minimal disability due to treatment Urgent radiology workload (queue jumpers)

Key model inputs

Paramedic phone ahead (pre-alert) rate ED triage referral rate Thrombolysis contra-indication rate (other than time and age)

Key model simplifications

ED queuing modelled as time delays Process times independent from time remaining

Pitt, M., Monks, T., Agarwal, P., Worthington, D., Ford, G. A., Lees, K. R., Stein, K., & James, M. A. (2012). Will Delays in Treatment Jeopardize the Population Benefit From Extending the Time Window for Stroke Thrombolysis? Stroke, 43, 2992-2997.

Results presented as scenarios

2 4 6 8 10 12 14 16 10 20 30 40 50 60 70

Current Situation License Extension Triage Referral Paramedics Pre- alert Pre-alerts and Extension Pre-alerts, Ext., > 80 yrs Additional mRS 0-1 Patients Thrombolysed OTT 0-90 OTT 91-180 OTT 181-270 Additional mRS 0-1

Model uncertainty (4.5 hr license)

Input Low High Paramedic Pre-alerts 15% 85% Triage referrals 15% 85% Wake-up contra-indications Base Base + 40% Output Lower bound Upper bound Thrombolysis rate 8% 14% ATT 65 110 mRS 0-1 4 11 Significant interactions between pre-alert and triage referral rates.

Project timeline and narrative

Timeline of project and implementation

2011 2012 2013

Nov 2010: Preliminary Investigation Jan-Feb 2011: Problem structuring Jul 2011: Results reported Dec 2011: Triage referral & evaluation start May 2012: IST-3 Reports Aug 2012: Stroke phone Evaluation ends

Preliminary Investigation (Nov 2010)

• Meeting: medical school academics, head of the emergency

department (ED) and head of acute stroke team (AST)

• Quick analysis: patients potentially eligible for thrombolysis take

an average of 60 minutes to be scanned.

• Head of ED led process mapping
• A possible solution was recognised as referring FAST positive

patients directly to the AST as they are triaged.

• Agreement: AST would lead investigation with modeling support

Problem structuring (Jan-Feb 2011)

• Process mapping meetings with the AST (nurses and physicians)
• It became obvious that ambulance paramedics could help
• Paramedics could send a pre-alert of imminent FAST positive

arrivals so resources could be in place asap

• Implementation: who should be pre-alerted ED or AST?
• Persuasion: included patient disability as a model output
• Persuasion: included urgent scanning workload for radiology

Use of VIS (Mar-Jun 2011)

• We spent a lot of time developing the model so that it was very

clear what was happening in it

• VIS was mainly used as a face validation tool with AST
• We did use it to demonstrate (VIE) the impact of early alerting
• n individual patients; although the AST already bought into it!
• VIS proved a powerful tool for talking to other trusts (later)

Results: reaction of ambulance trust (Sept 2011)

• Final results were disseminated to the amb trust in Sept 2011.
• The ambulance trust response was very positive!
• In particular, they commented that it was rare to get feedback
• n what they as paramedics could do to aid patient outcomes
• They asked us to conduct a similar project with them on pre-

hospital delays

• They were keen to implement a pre-alert system in Exeter and

elsewhere.

Reaction of ED (Nov 2011)

• It took five months to organise a meeting with ED;
• Presentation given by the AST lead with modeller support
• A group of ED consultant’s were not interested in the
• perational logic of our model -> more the clinical assumptions
• This group did not believe the effectiveness data for

thrombolysis and believed the risks outweighed the benefits

• We did not model the risks because overall death rates are the

same in treated and untreated patients.

Reaction of ED (Nov 2011)

• The consultants were much more casual about process changes
• They suggested pre-alerts should go to the AST
• They were not concerned about overloading radiology
• Grateful for being consulted in such a manner.
• “The usual approach is to receive an e-mail demand”
• The decision was left with the ED consultants to debate.

Implementation events

• FAST positive patients referred at triage (Dec 2011)
• Stroke phone protocol (Aug 2012)
• We ran four similar projects with different trusts during this time

Did the changes work?

Did the changes work?

Before the intervention After During

Are patients treated quicker?

.005 .01 .015 .005 .01 .015 50 100 150 200

Before After

ATT

Graphs by Group

Before After N 93 58 Mean (SD) 90 (35) 71 (25) Median (IQR) 85 (46) 70 (35) 10th Percentile 52 51 90th Percentile 145 105

Is there strong evidence of more thrombolysis?

Overall difference: 2.2% (95% CI 0.3-4.1%) Difference after implementation phase two: 4%

Compliance to protocol (pre-alert rates)

• Pre-alerts didn’t really kick in until Sept 2012.

Compliance After Sept 2012 N (%) 95% CI Paramedic 52 (30%) 22-38% ED Triage 15 (11%) 4-16%

• The figures are corrected for the average FAST sensitivity

Discussion: Our assumptions about the intervention

Stakeholder involvement

• AST and ED involved in conceptual modeling.
• ED not involved enough? Involved the wrong consultant?
• We struggled to have credibility with one group
• Decision was difficult for ED
• (My experience elsewhere suggests it is difficult even with lots
• f involvement throughout)

Discussion: Our assumptions about the intervention

Visual Interactive Simulation

• I have no evidence that it increased engagement
• Stakeholders were much more excited about charts of results
• Very useful for checking the process logic with nurses
• It proved very useful for engaging other trusts who were keen to

improve their own thrombolysis rates.

• Less engaging due to relatively low volume of entities?

Discussion: Our assumptions about the intervention

Models help structure debates about change

• Most relevant to the discussion with ED.
• Focus on the validity of assumptions
• Encouraged expression of concerns about thrombolysis
• Uncovered the source of these concerns
• Model logic was discussed in view of implementation.

Conclusions

• Study provides some quantitative evidence of the (modest)

impact of simulation in healthcare

• Unpacks our assumptions about how a modeling intervention

would lead to system change

• Involvement: selection of participants not straightforward.
• VIS: Not as effective as expected, but useful for spin-offs
• Debate: Evidence that it helped, but not in the manner we

expected!

Thanks for listening Any questions?

Dr Thomas Monks t.monks@exeter.ac.uk