Toward a standard model of feedback report and dashboard content - - PowerPoint PPT Presentation

▶
toward a standard model of feedback report and dashboard
SMART_READER_LITE
LIVE PREVIEW

Toward a standard model of feedback report and dashboard content - - PowerPoint PPT Presentation

Nov 01, 2023 143 likes •795 views

Toward a standard model of feedback report and dashboard content May 24, 2019 Zach Landis-Lewis Learning Health Sciences University of Michigan Symposium on Advancing the Science of Audit and Feedback Disclosure I have no competing

slide-1
SLIDE 1

Toward a standard model of feedback report and dashboard content

May 24, 2019 Zach Landis-Lewis

Learning Health Sciences University of Michigan Symposium on Advancing the Science of Audit and Feedback

slide-2
SLIDE 2

Disclosure

I have no competing interests to declare

2

slide-3
SLIDE 3

Takeaways

  • “Performance summary content” is an

important term to define for our community

  • Key types of performance summary content

○ Performance gaps and trends ○ Measures (i.e. indicators) ○ Time intervals

3

slide-4
SLIDE 4

Outline

  • 1. Introduction
  • 2. Objective
  • 3. A proposed model of feedback content
  • 4. Discussion

4

slide-5
SLIDE 5

Research focus

  • Can software tailor feedback messages for

situations that matter?

  • We encountered confusion when describing

the content of a display

slide-6
SLIDE 6

the problem

A&F terms are not well-defined ○ feedback ○ performance summary ○ comparator

slide-7
SLIDE 7

Why defining content matters

  • To understand mechanisms, we must

differentiate content and form

  • Good visualizations leverage relationships

between content and form elements

7

slide-8
SLIDE 8

Using taxonomy

  • Taxonomy: a hierarchical

classification scheme

  • “is a kind of” relationships
  • E.g. Linnaean

taxonomy Animal Chordate Vertebrate Mammal Bird

slide-9
SLIDE 9

Toward an ontology

  • taxonomy with

additional types of relationships

  • e.g. “part of”

Animal Chordate Vertebrate Mammal Bird

Spine

part of

9

slide-10
SLIDE 10

Value of ontologies

  • Describing our data
  • Scientific communication and learning

10

slide-11
SLIDE 11

Ontology development goals

  • Use our existing language and theory-based

terms

  • Write definitions with necessary and sufficient

characteristics

  • Use a standard (Basic Formal Ontology)

11

slide-12
SLIDE 12

Assumptions about ontologies

  • A work-in-progress that evolves
  • Preferred terms, not correct/incorrect terms
  • Challenging and time-consuming to develop
  • Systematic, open science approach is optimal
slide-13
SLIDE 13

Benefits of taxonomy and ontology

  • Better classification of research findings
  • Better consensus on knowledge, language
  • Better learning for new researchers
  • Better development of software for A&F

○ Dashboards ○ Reporting tools

slide-14
SLIDE 14

Scope: Performance summary content

Beck CA, Richard H, Tu JV, Pilote L. Administrative Data Feedback for Effective Cardiac Treatment: AFFECT, A Cluster Randomized Trial. JAMA. 2005 Jul 20;294(3):309–17.

14

slide-15
SLIDE 15

Scope

  • Feedback reports and dashboards have many

types of content ○ e.g. Patient lists, recommended actions

  • Scope for this talk:

Key information in a performance summary

15

slide-16
SLIDE 16

Feedback content vs form

  • Content

○ What we say ○ e.g. Feedback information, signal

  • Form

○ How we say it ○ e.g. Feedback delivery, visual display

16

slide-17
SLIDE 17

Feedback content vs form

Beck CA, Richard H, Tu JV, Pilote L. Administrative Data Feedback for Effective Cardiac Treatment: AFFECT, A Cluster Randomized Trial. JAMA. 2005 Jul 20;294(3):309–17.

17

slide-18
SLIDE 18

What is feedback content? (1 of 4)

  • ICEBeRG 2006

○ Comparative or not, anonymous or not?

  • Hysong et al 2009 and 2016 (FIT)

○ Sign (positive/negative) ○ Correct / incorrect ○ Correct solution ○ Attainment level ○ Velocity ○ Goal-setting type ○ Normative information ○ Norms ○ Discouraging ○ Praise

18

slide-19
SLIDE 19

What is feedback content? (2 of 4)

  • Ivers et al 2012

○ Summary of performance, recommended actions

  • Colquhoun et al 2016

○ Processes of care ○ Patient outcomes ○ Individual/group performance ○ Individual/aggregate patient cases ○ Identification of behavior ○ Graph presented ○ Type of comparison ■ Others’ performance ■ Guideline ■ Own/Others’ previous performance

19

slide-20
SLIDE 20

What is feedback content? (3 of 4)

Brown et al 2016: Interface components ○ Performance summaries ○ Patient lists ○ Patient data ○ Recommended actions

20

slide-21
SLIDE 21

What is feedback content? (4 of 4)

Brown et al 2019: CP-FIT Feedback display variables

○ Performance level ○ Patient lists ○ Specificity ○ Timeliness ○ Trend ○ Benchmarking ○ Prioritisation ○ Usability

Gude et al 2019: Comparators

○ Benchmarks ○ Explicit targets ○ Trends

21

slide-22
SLIDE 22

Outline

  • 1. Introduction
  • 2. Objective
  • 3. A proposed model of feedback content
  • 4. Discussion

22

slide-23
SLIDE 23

Objective

To propose a standard model of performance summary content for the purposes of:

  • Description: Organizing data and information

about A&F interventions

  • Learning: A&F research communication

23

slide-24
SLIDE 24

Outline

  • 1. Introduction
  • 2. Objective
  • 3. A proposed model of feedback content
  • 4. Discussion

24

slide-25
SLIDE 25

Performance summary

Beck CA, Richard H, Tu JV, Pilote L. Administrative Data Feedback for Effective Cardiac Treatment: AFFECT, A Cluster Randomized Trial. JAMA. 2005 Jul 20;294(3):309–17.

25

slide-26
SLIDE 26

Performance summary

Beck CA, Richard H, Tu JV, Pilote L. Administrative Data Feedback for Effective Cardiac Treatment: AFFECT, A Cluster Randomized Trial. JAMA. 2005 Jul 20;294(3):309–17.

26

slide-27
SLIDE 27

Performance summary

Beck CA, Richard H, Tu JV, Pilote L. Administrative Data Feedback for Effective Cardiac Treatment: AFFECT, A Cluster Randomized Trial. JAMA. 2005 Jul 20;294(3):309–17.

27

slide-28
SLIDE 28

Example

28

slide-29
SLIDE 29

Performance summary content

29

slide-30
SLIDE 30

Performance summary content

30

Information

slide-31
SLIDE 31

Performance summary content Measure

31

Information

slide-32
SLIDE 32

Performance measure

32

slide-33
SLIDE 33

Performance measure

  • Information about a method of measuring

clinical practice referring to the structures, processes, or outcomes of care (modified from Campbell et al 2003)

  • i.e. indicators, metrics

33

slide-34
SLIDE 34

Performance summary content Measure

34

Information

slide-35
SLIDE 35

Performance summary content Time interval Measure

35

Information

slide-36
SLIDE 36

Time interval

Beck CA, Richard H, Tu JV, Pilote L. Administrative Data Feedback for Effective Cardiac Treatment: AFFECT, A Cluster Randomized Trial. JAMA. 2005 Jul 20;294(3):309–17.

36

slide-37
SLIDE 37

Performance summary content Time interval Measure

37

Information

slide-38
SLIDE 38

Performance summary content Time interval Measure Feedback recipient

38

Information

slide-39
SLIDE 39

Performance summary content Time interval Measure Feedback recipient Comparator

39

Information

slide-40
SLIDE 40

40

slide-41
SLIDE 41

41

People,

  • rganizations,

benchmarks, goals

slide-42
SLIDE 42

42

People,

  • rganizations,

benchmarks, goals

slide-43
SLIDE 43

Performance summary content Time interval Measure Feedback recipient Comparator

43

Information

slide-44
SLIDE 44

Performance summary content Time interval Ascribee Measure Feedback recipient Comparator

44

Information

slide-45
SLIDE 45

Ascribee

  • Information about an entity that has an

attributed performance

  • i.e. feedback recipient, comparator

45

slide-46
SLIDE 46

Performance summary content Time interval Ascribee Measure Feedback recipient Comparator

46

Information

slide-47
SLIDE 47

Performance summary content Performance information Time interval Ascribee Measure Feedback recipient Comparator

47

Information

slide-48
SLIDE 48

Performance summary content Performance information Time interval Ascribee Measure Performance level Feedback recipient Comparator

48

Information

slide-49
SLIDE 49

Performance levels

Data about events, scores, percentages

49

slide-50
SLIDE 50

Performance level

  • Information about clinical practice that was

accomplished

  • i.e. performance score, data, or information
  • e.g. 81%, High, 23/42

50

slide-51
SLIDE 51

Performance summary content Performance information Time interval Ascribee Measure Performance level Feedback recipient Comparator

51

Information

slide-52
SLIDE 52

Performance summary content Performance information Time interval Ascribee Measure Performance level Performance gap Feedback recipient Comparator

52

Information

slide-53
SLIDE 53

Performance gaps

Distances between performance levels

53

slide-54
SLIDE 54

Performance gap

  • Information about a distance between

performance levels of a feedback recipient and a comparator

  • i.e. performance discrepancy
  • e.g. below average, top performer

54

slide-55
SLIDE 55

Performance summary content Performance information Time interval Ascribee Measure Performance level Performance gap Feedback recipient Comparator

55

Information

slide-56
SLIDE 56

Performance summary content Performance information Time interval Ascribee Measure Performance level Performance gap Performance trend Feedback recipient Comparator

56

Information

slide-57
SLIDE 57

No trend

57

slide-58
SLIDE 58

Performance trend

  • Information about movement that emerges

from performance levels displayed over time

  • i.e. velocity feedback
  • e.g. performance is increasing/decreasing

58

slide-59
SLIDE 59

Performance summary content Performance information Time interval Ascribee Measure Performance level Performance gap Performance trend Feedback recipient Comparator

59

Information

slide-60
SLIDE 60

Performance summary content Performance information Time interval Ascribee Measure Performance level Performance gap Performance trend Feedback recipient Comparator

M A P T

60

Information

slide-61
SLIDE 61

Outline

  • 1. Introduction
  • 2. Objective
  • 3. A proposed model of feedback content
  • 4. Discussion

61

slide-62
SLIDE 62

Limitations

  • Incomplete

○ Many other important types of content are not yet included

  • Slow-going, this represents ~3 years of work
  • Limited input from A&F community to date

62

slide-63
SLIDE 63

Toward a feedback intervention ontology

  • We are developing a computer-interpretable

form of MAPT

  • Purposes of the computable model:

○ Organizing data and information about feedback interventions ○ Learning about feedback mechanisms

63

slide-64
SLIDE 64

Implications for A&F research

  • A standard model of feedback content could

be useful for large-scale studies

  • Support organized efforts to address A&F

hypotheses at large scale

64

slide-65
SLIDE 65

Thank you

NIH National Library of Medicine K01 #5K01LM012528-02

DISPLAY Lab: https://github.com/Display-Lab Jessica Zhang Emily Dibble John Rincon-Hekking Veena Panicker Colin Gross Cooper Stansbury Dahee Lee Astrid Fishstrom Mentoring team: Anne Sales, Charles Friedman, Brian Zikmund-Fisher 65

zachLL@umich.edu Twitter: @zachll