Toward a standard model of feedback report and dashboard content May 24, 2019 Zach Landis-Lewis Learning Health Sciences University of Michigan Symposium on Advancing the Science of Audit and Feedback
Disclosure I have no competing interests to declare 2
Takeaways ● “Performance summary content” is an important term to define for our community ● Key types of performance summary content ○ Performance gaps and trends ○ Measures (i.e. indicators) ○ Time intervals 3
Outline 1. Introduction 2. Objective 3. A proposed model of feedback content 4. Discussion 4
Research focus ● Can software tailor feedback messages for situations that matter? ● We encountered confusion when describing the content of a display
the problem A&F terms are not well-defined ○ feedback ○ performance summary ○ comparator
Why defining content matters ● To understand mechanisms, we must differentiate content and form ● Good visualizations leverage relationships between content and form elements 7
Using taxonomy Animal ● Taxonomy: a hierarchical Chordate classification scheme ● “is a kind of” relationships ● E.g. Linnaean Vertebrate taxonomy Mammal Bird
Toward an ontology Animal ● taxonomy with Chordate additional types of relationships part of Spine Vertebrate ● e.g. “part of” Mammal Bird 9
Value of ontologies ● Describing our data ● Scientific communication and learning 10
Ontology development goals ● Use our existing language and theory-based terms ● Write definitions with necessary and sufficient characteristics ● Use a standard (Basic Formal Ontology) 11
Assumptions about ontologies ● A work-in-progress that evolves ● Preferred terms, not correct/incorrect terms ● Challenging and time-consuming to develop ● Systematic, open science approach is optimal
Benefits of taxonomy and ontology ● Better classification of research findings ● Better consensus on knowledge, language ● Better learning for new researchers ● Better development of software for A&F ○ Dashboards ○ Reporting tools
Scope: Performance summary content Beck CA, Richard H, Tu JV, Pilote L. Administrative Data Feedback for Effective Cardiac Treatment: AFFECT, A Cluster 14 Randomized Trial. JAMA. 2005 Jul 20;294(3):309–17.
Scope ● Feedback reports and dashboards have many types of content ○ e.g. Patient lists, recommended actions ● Scope for this talk: Key information in a performance summary 15
Feedback content vs form ● Content ○ What we say ○ e.g. Feedback information, signal ● Form ○ How we say it ○ e.g. Feedback delivery, visual display 16
Feedback content vs form Beck CA, Richard H, Tu JV, Pilote L. Administrative Data Feedback for Effective Cardiac Treatment: AFFECT, A Cluster 17 Randomized Trial. JAMA. 2005 Jul 20;294(3):309–17.
What is feedback content? (1 of 4) ● ICEBeRG 2006 ○ Comparative or not, anonymous or not? ● Hysong et al 2009 and 2016 (FIT) ○ Goal-setting type ○ Sign (positive/negative) ○ Normative information ○ Correct / incorrect ○ Norms ○ Correct solution ○ Discouraging ○ Attainment level ○ Praise ○ Velocity 18
What is feedback content? (2 of 4) ● Ivers et al 2012 ○ Summary of performance, recommended actions ● Colquhoun et al 2016 ○ Type of comparison ○ Processes of care ■ Others’ performance ○ Patient outcomes ■ Guideline ○ Individual/group performance ■ Own/Others’ previous ○ Individual/aggregate patient cases performance ○ Identification of behavior ○ Graph presented 19
What is feedback content? (3 of 4) Brown et al 2016: Interface components ○ Performance summaries ○ Patient lists ○ Patient data ○ Recommended actions 20
What is feedback content? (4 of 4) Brown et al 2019: CP-FIT Gude et al 2019: Feedback display variables Comparators ○ Performance level ○ Benchmarks ○ Patient lists ○ Explicit targets ○ Specificity ○ Trends ○ Timeliness ○ Trend ○ Benchmarking ○ Prioritisation ○ Usability 21
Outline 1. Introduction 2. Objective 3. A proposed model of feedback content 4. Discussion 22
Objective To propose a standard model of performance summary content for the purposes of: ● Description: Organizing data and information about A&F interventions ● Learning: A&F research communication 23
Outline 1. Introduction 2. Objective 3. A proposed model of feedback content 4. Discussion 24
Performance summary Beck CA, Richard H, Tu JV, Pilote L. Administrative Data Feedback for Effective Cardiac Treatment: AFFECT, A Cluster 25 Randomized Trial. JAMA. 2005 Jul 20;294(3):309–17.
Performance summary Beck CA, Richard H, Tu JV, Pilote L. Administrative Data Feedback for Effective Cardiac Treatment: AFFECT, A Cluster 26 Randomized Trial. JAMA. 2005 Jul 20;294(3):309–17.
Performance summary Beck CA, Richard H, Tu JV, Pilote L. Administrative Data Feedback for Effective Cardiac Treatment: AFFECT, A Cluster 27 Randomized Trial. JAMA. 2005 Jul 20;294(3):309–17.
Example 28
Performance summary content 29
Information Performance summary content 30
Information Measure Performance summary content 31
Performance measure 32
Performance measure ● Information about a method of measuring clinical practice referring to the structures, processes, or outcomes of care (modified from Campbell et al 2003) ● i.e. indicators, metrics 33
Information Measure Performance summary content 34
Information Time interval Measure Performance summary content 35
Time interval Beck CA, Richard H, Tu JV, Pilote L. Administrative Data Feedback for Effective Cardiac Treatment: AFFECT, A Cluster 36 Randomized Trial. JAMA. 2005 Jul 20;294(3):309–17.
Information Time interval Measure Performance summary content 37
Information Time interval Measure Performance summary content Feedback recipient 38
Information Time interval Measure Performance summary content Comparator Feedback recipient 39
40
People, organizations, benchmarks, goals 41
People, organizations, benchmarks, goals 42
Information Time interval Measure Performance summary content Comparator Feedback recipient 43
Information Time interval Measure Performance summary content Ascribee Comparator Feedback recipient 44
Ascribee ● Information about an entity that has an attributed performance ● i.e. feedback recipient, comparator 45
Information Time interval Measure Performance summary content Ascribee Comparator Feedback recipient 46
Information Time interval Measure Performance summary content Performance Ascribee information Comparator Feedback recipient 47
Information Time interval Measure Performance summary content Performance Ascribee information Comparator Performance level Feedback recipient 48
Performance levels Data about events, scores, percentages 49
Performance level ● Information about clinical practice that was accomplished ● i.e. performance score, data, or information ● e.g. 81%, High, 23/42 50
Information Time interval Measure Performance summary content Performance Ascribee information Comparator Performance level Feedback recipient 51
Information Time interval Measure Performance summary content Performance Ascribee information Comparator Performance level Feedback recipient Performance gap 52
Performance gaps Distances between performance levels 53
Performance gap ● Information about a distance between performance levels of a feedback recipient and a comparator ● i.e. performance discrepancy ● e.g. below average, top performer 54
Information Time interval Measure Performance summary content Performance Ascribee information Comparator Performance level Feedback recipient Performance gap 55
Information Time interval Measure Performance summary content Performance Ascribee information Comparator Performance Performance trend level Feedback recipient Performance gap 56
No trend 57
Performance trend ● Information about movement that emerges from performance levels displayed over time ● i.e. velocity feedback ● e.g. performance is increasing/decreasing 58
Information Time interval Measure Performance summary content Performance Ascribee information Comparator Performance Performance trend level Feedback recipient Performance gap 59
Information T M Time interval Measure Performance summary content P A Performance Ascribee information Comparator Performance Performance trend level Feedback recipient Performance gap 60
Outline 1. Introduction 2. Objective 3. A proposed model of feedback content 4. Discussion 61
Recommend
More recommend
