Usability of reports generated by a computerised dose prediction - - PowerPoint PPT Presentation

Melissa Baysari, Joanne Chan, Jane Carland, Sophie Stocker, Maria Moran, Richard Day

Macquarie University; UNSW Australia; St Vincent’s Hospital, Sydney

Usability of reports generated by a computerised dose prediction software

Therapeutic drug monitoring (TDM)

TDM involves monitoring drug concentrations in the blood and adjusting doses accordingly, in order to achieve the optimal effect of that drug TDM is most beneficial in patients with altered pharmacokinetic parameters, such as

• those with impaired renal function
• those that are critically ill
• those who are obese, etc.

BUT, maintaining optimal drug concentrations is a highly complex task Computerised dose prediction software can assist clinicians in this process

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

2

Computerised dose prediction

3

Patient data is inputted (by a pharmacologist, pharmacist, TDM team?)

• r data is automatically extracted from EMR

Algorithm (integrating relevant patient factors) Report, including dose recommendation Delivered to prescriber (e.g. as a link in EMR) Guide prescriber decision making

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Computerised dose prediction

4

Research has shown that computerised decision support can improve dosing: Dose calculators, order sets, alerts But limited evidence on dose prediction software in practice

• Research showing that recommendations from software are likely to lead to drug

concentrations within the therapeutic rangea These systems are not yet widely used in clinical practice - Why?

• Expensive? Resource intensive (who enters the data?)?
• Hard to use or interpret?

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

aClin Pharmacokinet 2013, 52:59-68; 2016, 55:1295-1335

Report for prescribers

5

Dose prediction software can be highly sophisticated and accurate in predicting individualised dosing regimens, but if the information in reports is not understood by doctors, the benefits of the software are not achieved Previous research? Some research on what features of guidelines are liked and used Very limited evaluations of reports from dose prediction software

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Previous evaluation of reports

6

• In one study, 12 dosing systems were assessed on a range of criteria by a

pharmacist and 2 clinical pharmacologists

• 1 criterion related to the report generated for prescribers
• Raters assigned a score to each report based on
• readability
• the inclusion of a relevant graph
• the inclusion of a free-text field
• whether it displayed a user’s identity
• whether it could be customised
• whether it could be converted to other formats

Reports scored 1 - 4.3 out of 5

Fuchs et al. Clinical Pharmacokinetics 52 (2013) 9-22

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Study rationale and aim

7

We know very little about how the reports are viewed by end-users (i.e. prescribers) So we set out to assess the perceived usability of a report generated from a dose prediction system findings communicated to vendor lead to improvements in report content and design

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Study site and dosing decision support

8

320-bed teaching hospital in Sydney The hospital had electronic medication management, ordering and reporting of lab and imaging tests. Patient notes were not electronic Hospital planned to implement DoseMe, after some additional testing and piloting DoseMe uses Bayesian forecasting to provide individualised dosing recommendations

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Method

9

Research team reviewed the report from DoseMe

(clinical pharmacologists, ID physician, pharmacists, medical student, human factors researcher)

Made some changes to the report to improve readability Tested this new report with prescribers

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

10

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Original report

• Lots of information, including historical

information

• Limited white space, with some text
• verlapping
• Inconsistent alignment of text

11

Patient information Main recommendation More detailed information Drug exposure graph

Participants and procedure

12

15 prescribers from a variety of wards (1 intern, 7 residents, 7 registrars) Prescribers were presented with a mock DoseMe report and asked: 1. What is your first impression of this report? 2. What information do you find useful and not useful? 3. Is there anything you find confusing? 4. What other information should be provided? 5. What sections of the report (box, text, graph) would you like provided to you? 6. What improvements could be made to the report, in terms of both content and layout?

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Results

Positive perception: Summary box

14

Prescribers liked that the top box provided a clear, straightforward recommendation and drew prescribers’ attention to the most important piece of information I like that, getting a big red box with what to do next

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Positive perception: Drug exposure graph

15

Seen to be a good way to visualise dosing, especially for patients on long-term courses of a drug I’m probably a more visual person so I like seeing this laid

• ut more, in a more visual way

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Positive perception: Drug exposure graph

16

Also seen to be more accessible than text-based information when time pressured I think people could easily misinterpret it if they're rushing through so I think the graph makes it a lot easier to see "oh,

• kay, we must increase to get to the

target outcome”

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Negative perception: Unfamiliar and ambiguous terminology

17

Prescribers were unfamiliar with the term AUC12 - a key concept in the report Some questioned the value of Area Under the Curve (AUC) to non specialist doctors But I don’t know how many clinicians would be familiar with using it (AUC)

• ther than infectious disease (ID) physicians or specific microbiology/ID

doctors…Because we don’t really use the area-under-the-curves in sort of a practical, clinical sense on a regular basis

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Negative perception: Unnecessary information

18

Although most prescribers saw value in including the drug exposure graph, it was also noted that the graph shows the same information as the text above it, which is confusing I think it would be useful just to see that, actually see that the graph has represented what’s written down. Because some people may think that they’re getting two amounts of information at first glance

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Overall perceptions

19

Despite displaying complex information, most prescribers viewed the report as useful and appreciated being shown the data to support the recommendation in the red box It's always good if the clinician can look at something and say that it sounds reasonable or not…that you're completely relying on a computer- generated system to give you information in case there's an error within it... I think it is nice to know on what basis that was done and that it gives you at least a feeling of justification

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Overall perceptions

20

But…a number of prescribers anticipated that in the context of a fast-paced time-poor environment, the upper box would likely be the only part of the report that is utilised I'm in a hurry, I'm super busy, I’m already doing 14-hour days. I've got 25 patients and they're all really sick. I literally am racing through the bloods…I literally just want to know what to do next and how to do it as quickly as possible

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Discussion

Main findings

22

• The mock report was effective in communicating the recommended dose of a

drug, but this recommendation was presented alongside information that was not understood or unlikely to be used by prescribers

• The aspects of the report viewed negatively by end-users related to a lack of

familiarity with TDM terminology

• If specialised terms (like AUC) are to be used in outputs for prescribers, they need

to be accompanied by training and/or the report should include clear definitions (e.g. when you hover over them with your mouse)

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Main lessons learnt

23

• This study has highlighted the value of seeking user input in the design of

computerised dose prediction software and their reports

• While these systems are designed with input from those with expertise in TDM

(i.e. clinical pharmacologists and pharmacists), involving prescribers early on in the process is likely to result in systems and outputs that are more useful, usable and accessible to users

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Where to now?

24

The study findings were communicated to the vendor ― Will hopefully result in changes to their report More work needed before dose prediction software can be used routinely ― Prospective studies evaluating impact in practice, both on prescribing and on patient outcomes ― Integration into workflow – who does what?

AUSTRALIAN INSTITUTE OF HEALTH INNOVATION FACULTY OF MEDICINE AND HEALTH SCIENCES

Thank you Contact: melissa.baysari@mq.edu.au

This research was supported by NHMRC Program Grant #1054146