Creating a One-Page Evaluation Case Summary: Context, Narrative, - - PowerPoint PPT Presentation

Creating a One-Page Evaluation Case Summary: Context, Narrative, and Style Strategies

AcademyHealth Annual Research Meeting June 25, 2018, Seattle, WA Lynne Page Snyder, Ph.D., MPH Principal Research Scientist NORC at the University of Chicago

2

Overview

• Audience: who are they and what are their concerns?
• Context: framing the story
• Narrative: for CMMI evaluations, writing about numbers
• Style: guidance for drafting
• Example: one-page visual case summaries for first-round Health Care

Innovation Award (HCIA) evaluation

• For more information…

Acknowledgments: Health Care team at NORC; Mitali Dayal (CMMI).

Challenge = reporting that speaks to its audience

3

As LBJ Once Said, “A Range is for Cattle,…”

Source: Kat Jayne from Pexels.

4

“Give Me A Number.”

Source: Ryan McGuire, https://gratisography.com/.

5

One Page Case Summary for HCIA Evaluation

• Evaluation of first-round Health Care Innovation Award (HCIA),

High-Risk/Patient Targeting Portfolio

• Range of models
• Shared focus on highest-risk beneficiaries (Medicare, Medicaid)
• Mixed methods
• Client = CMMI
• The ask = one page visual case summary for each of 23 cases

(awardees), akin to a briefing document

• Constraints: time, budget, production resources (software), statement of

work (deliverable specifications)

• Flexible design to enable consistency across diverse cases

6

More About Goals for the One-Pager…

• Intended to convey part of a report, rather than be

summative or comprehensive

• Free-standing rather than a preview. Audience likely will

not read the report and may use the one pager to make a decision

• Informed by substantial literature on how to transform

evaluation reporting from chore (and bore) to engaging and effective communication

7

What Our Evaluation One-Pager is not…

• Data visualization or infographic (greater reliance on text)
• Dashboard (one-time, not intended for update)
• Peer-reviewed article (less standardized)
• Social media post (audience = client)

8

Audiences

• What does our audience most

want to know?

Contracting Representative, Client

Media & social media Policy influentials & decision makers Congress, the courts, executive branch agencies State & local government, civic groups Patients, families, & caregivers Health care: clinicians, practices, hospitals, skilled nursing facilities Health plans & payers

9

Context That the Audience Sees

• Define the situation
• What do we know? What don’t we know? What do we assume?
• Analogies: What is similar? What is different?
• Identify decision makers’ concerns
• What’s the objective?
• What’s the story? Timeline? The facts?
• What are the options for action? What is feasible for now?
• What expectations about causes and effects make certain options preferable

to others? What new evidence might change or challenge a presumption?

• Add context: people and organizations as actors moving through time

Source: Neustadt & May, Thinking in Time

10

More About Narrative: Telling a Story Using Numbers

• Making comparisons and explaining basis for same
• Testing claim(s)
• Respecting chance (null hypothesis)
• What makes an argument persuasive?
• Effect size
• Narrative detail and style
• Generalizability
• Being sufficiently important and compelling to change minds
• Credible

Sources: Abelson, Statistics as Principled Argument

“Good statistics involves principled argument that conveys an interesting and credible point.”

11

Writing About Numbers

For multivariate analysis:

• Set context
• Choose examples and analogies
• Tailor vocabulary to audience
• Decide whether and how to use text, tables, and/or figures
• Interpret number(s) in the text
• Specify both direction and size of association between variables
• Describe overall patterns, illustrate with examples, and note

exceptions Sources: Miller, Chicago Guide to Writing About Multivariate Analysis

12

Style: Best Practices for Reporting and Visualizations

• Graphics
• Keep visuals simple (images, graphs)
• Reduce clutter
• Refer to visualization checklist(s)
• Type
• Communicate through font style and sizes
• Develop a style sheet
• Color
• Keep the scheme simple
• Plan for 508 compliance
• Arrangement
• Follow design principles for layout, grouping, use of white space,

alignment

• Present no more than 5-7 points per section/page
• Use consistency (and breaks from consistency) to convey priorities

Source: Stephanie D.H. Evergreen & Ann Emery, Reporting & Evaluation Report Layout Checklist, in Evergreen, Presenting Data Effectively

13

Big Decisions about One Pager

• Frame/Context = COR, CMS front office
• Narrative =
• modest and limited positive findings on promising models
• findings based on data from interviews/site visits, program documents,

and claims

• analyses include multivariate regression models, difference-in-

differences, sample surveys, themes in qualitative data

• Style = free-standing, selectively comprehensive
• Importance of client review (akin to pretest)

14

What’s on the One-Pager?

Basic information

• About awardee: awardee name, summary description, classification of model

type

• About cooperative agreement: award $$, performance period, reach,

populations served, data sources for evaluation Evaluation Domains

Key findings Basic information Secondary findings

Footnote

Key findings

21

For more information,

• Robert P. Abelson. Statistics as Principled
• Argument. Hillsdale, NJ: Lawrence Erlbaum

Associates, 1995.

• Robert E. Stake. The Art of Case Study Research.

SAGE Publications, 1995.

• Ann Emery. Website on Information Design, at

http://annkemery.com/ .

• Stephanie Evergreen. Presenting Data Effectively.

SAGE Publications, 2014; and website, Evergreen Data: Intentional Reporting and Data Visualization, at http://stephanieevergreen.com/ .

• Jane E. Miller. Chicago Guide to Writing about Multivariate Analysis, 2nd edition. Chicago: Univ.

Chicago Press, 2013.

• Richard E. Neustadt and Ernest R. May. Thinking in Time. The Uses of History for Decision Makers.

Free Press, 1988.

• NORC at the University of Chicago. HCIA Complex/High-Risk Patient Targeting: Third Annual Report.
• 2017. At https://downloads.cms.gov/files/cmmi/hcia-chspt-thirdannualrpt.pdf .

Source: Kat Jayne from Pexels.

Thank You!

Lynne Page Snyder, Ph.D., MPH snyder-lynne@norc.org Office: 301.634.9569 Cell: 301.520.1809