Partnering and Pragmatic Trials in a Learning Health Care System - - PowerPoint PPT Presentation

Partnering and Pragmatic Trials in a Learning Health Care System

AcademyHealth Annual Research Meeting By: Anna Spier June 25, 2017

J-PAL’s U.S. Health Care Delivery Initiative (HCDI)

J-PAL | ACADEMYHEALTH 2

HCDI develops rigorous evidence of strategies to improve the quality and value of health care delivery by:

• Funding randomized

evaluations

• Connecting policymakers,

practitioners, and researchers to spur policy-relevant research on key health care issues

• Building evaluation capacity
• Disseminating evidence to

policymakers

I. Why RCTs?

• II. Building Research

Partnerships

• III. Examples

What is the effect of Medicaid?

4

• “Medicaid is

worthless or worse than no insurance”

J-PAL | ACADEMYHEALTH

Not true: Increases in utilization, perceived access and quality, reductions in financial strain, and improvement in self-reported health Not true: Medicaid increases use

• f ER (overall and for a broad

range of visit types)

Not true in short run: increases in health care use In long run, remains to be seen: increases in preventive care and improvements in self-reported health

• “Covering the

uninsured will get them out of the Emergency Room”

• “Health

insurance expansion saves money”

Randomized evaluations can provide clear answers

• Not always obvious what the effects of a given policy are
• Ex: Those with insurance are in worse health than those without

insurance

• Conclude that insurance makes people sicker? Or that individuals in

poor health are more likely to seek out insurance?

• Randomized evaluations randomly assign individuals to

treatment (program) or control (status quo)

• By construction, the treatment group and the control group will

have the same characteristics, on average

• Observable: age, income, measured health, etc.
• Unobservable: motivation, social networks, unmeasured health, etc.
• Clear attribution of subsequent differences to treatment

(program)

J-PAL | ACADEMYHEALTH

Limited use of RCTs in U.S. health care delivery

• Finkelstein and Taubman Science 2015
• Review of empirical papers in top medical, economics,

and health services journals: 18% of U.S. health care delivery interventions randomized

• Greater use of RCTs for U.S. medical studies
• 80% of U.S.-based medical treatment studies randomized
• True of both drug (86%) and non-drug (66%) interventions
• Greater use of RCTs for other social policy
• 36% of U.S. education studies
• 46% percent of international development studies

J-PAL | ACADEMYHEALTH

I. Why RCTs?

• II. Building Research

Partnerships

• III. Examples

Typical process

Defined intervention

• Partner has interesting

research question

• Interested in RCT

Initial scoping

• Is randomization

feasible?

• Is sample size sufficient?
• Is partner committed?

Identify interested researcher

• Based on research

interests/bandwidth

Design pilot

• Work out

implementation hurdles

• Proof of concept

Full RCT

• Incorporate what was

learned from pilot

J-PAL | ACADEMYHEALTH

8

Ideal research questions

• Intervention that is:

– Well-defined (protocol-driven, often well- established) – Policy-relevant/academically-interesting

• Serves a large enough sample to detect

anticipated effects

• Ability to randomize access to the intervention

(typically requires a capacity constraint or phased roll-out)

• For private sector: aligned with business interests

9

J-PAL | ACADEMYHEALTH

Ideal partnerships

10

• Willingness to experiment
• Large institution (statistical power)
• Access to administrative data
• Realistic expectations
• Executive-level support/sponsorship
• Engaged researcher

– Respects partner’s priorities – Works with partner to assess feasibility of evaluation – Thinks creatively about designing evaluation to address practical concerns – Helps navigate institutional or legal obstacles to data

J-PAL | ACADEMYHEALTH

Benefits of administrative data

Compared to surveys, administrative data may:

J-PAL | ACADEMYHEALTH 11

Image Credit: The Noun Project: Vaibhav Radhakrishnan; Chameleon Design; Aaron K. Kim; Alexander Bogomolov

Reduce research costs Lessen logistical burden Enable long-term follow-up Improve accuracy of findings

Common challenges to randomization

• PROGRAM DESIGN

– Resources exist to extend the program to everyone in the study area – Program has strict eligibility criteria – Program is an entitlement – Sample size is small

• IMPLEMENTATION

– Difficult for service providers to adhere to random assignment due to logistical or political reasons – Control group finds out about the treatment, benefits from the treatment, or is harmed by the treatment

J-PAL | ACADEMYHEALTH

12

When does a randomized evaluation not make sense?

• Too small: sample is to small to pick up a

reasonable impact

• Too early: still ironing out logistics
• Too late: already serving everyone who is

eligible, and no randomization was built in

• When a positive impact has been proven,

and we have the resources to serve everyone

13

J-PAL | ACADEMYHEALTH

I. Why RCTs?

• II. Building Research

Partnerships

• III. Examples

Nonprofit, community organization intervention: Care coordination

15

• Partner: Camden

Coalition of Healthcare Providers

• Intervention: Support to

high-need patients (“super-utilizers”) for

• ngoing outpatient care

and assistance in accessing social programs

Goal: Analysis of primary

• utcome (hospital re-

admissions) using Health Information Exchange

J-PAL | ACADEMYHEALTH

Large health system intervention: Clinical Decision Support (CDS)

16

J-PAL | ACADEMYHEALTH

• Partner: Aurora Health Care
• Intervention: CDS notifies physicians in real time when they

have ordered a diagnostic scan that is inconsistent with current professional guidelines

• Goal: Determine how CDS impacts ordering behavior of

physicians

Private company intervention: Workplace wellness

17

• Partner: BJ’s Wholesale

Club

• Intervention: Diet,

exercise, and mental health programming for employees over 1-year period

• Goal: Gauge impact on

five categories of

• utcomes gathered

from both primary and administrative data sources

J-PAL | ACADEMYHEALTH

Nurse Family Partnership

J-PAL | ACADEMYHEALTH

18

Feasibility: funding a novel expansion

PFS Project

SCDHHS Implementing Agencies J-PAL Philanthropists Social Finance SC DHEC The Children’s Trust NSO

• Medicaid Waiver
• Pay for Success (PFS)

contract

– contingent on randomized evaluation

J-PAL | ACADEMYHEALTH

19

Nurse home visits today

Study Population: 6,000 over four years Outcomes of interest: Short- and long-run impact on a wide range of health, education, employment, criminal justice and other

• utcomes.

Innovative expansion of NFP in South Carolina

J-PAL | ACADEMYHEALTH

Discussion & Questions

Anna Spier Senior Policy Associate J-PAL North America aspier@mit.edu

Thank you!

What Does HCDI Provide?

• Partnership development between policymakers,

practitioners, and researchers to spur policy-relevant research

• Technical assistance in identifying and scoping
• pportunities for rigorous evaluation of innovative

programs

• Targeted research funding
• Trainings and resources for researchers and

policymakers regarding rigorous evaluation

• Syntheses of existing evidence on key policy topics

22

J-PAL | ACADEMYHEALTH

J-PAL’s mission is to reduce poverty by ensuring that policy is informed by scientific evidence

J-PAL | ACADEMYHEALTH

J-PAL’s network of 145 professors use randomized evaluations to inform policy

J-PAL | ACADEMYHEALTH

We have 842 ongoing and completed projects across 8 sectors in 80 countries

J-PAL | ACADEMYHEALTH

26

What is a Randomized Evaluation?

Before the program starts, eligible individuals are randomly assigned to two groups so that they are statistically identical before the program.

Two groups continue to be identical, except for treatment Any differences in

• utcomes between

the groups can be attributed to the program

Treatment Control J-PAL | ACADEMYHEALTH

Why Randomize?

• Results from experiments can surprise us
• Decision makers are more likely to trust

the results

• Higher confidence to scale up

approaches or make changes in how money is spent

Photo Credit: Shutterstock.com

J-PAL | ACADEMYHEALTH

Why Randomize?

2013

Intervention

2014 2015

Hospi pital readmis missions

• ns

Yea ear

J-PAL | ACADEMYHEALTH

Why Randomize?

2013

Intervention

2014 2015

Hospi pital readmis missions

• ns

Yea ear

Counter-factual Impact

J-PAL | ACADEMYHEALTH

Why Randomize?

2013

Intervention

2014 2015

Hospi pital readmis missions

• ns

Yea ear

Impact Counter-factual

J-PAL | ACADEMYHEALTH

J-PAL | ACADEMYHEALTH

31

Future opportunities: what questions do you want answered?

Examples include:

• Choice of appropriate care (i.e. preventive screening

uptake)

• Who provides care? Where and how is care provided?

(i.e. post-partum length of stay, telemedicine)

• Efficient use of existing resources (i.e. staffing,

scheduling)

• Insurance contract and reimbursement design (i.e.

reference pricing, limited network plans)

• System-wide innovations (i.e. payment reform)

J-PAL | ACADEMYHEALTH

32

Research question examples

• How can providing information through varied methods

(email, letters, text) reduce gaps in care such as vaccinations, screenings, and appointment absenteeism?

• How do care management services at the time of

discharge impact 30 day re-admission rate, ED utilization and health outcomes at 6-12 months for patients?

• In what ways can providers be nudged to adhere to

clinical practice guidelines (e.g. EMR alert, app, incentives?)