Trans ansfor orming ming Tec echnic hnical al As Assis sistan - - PowerPoint PPT Presentation

Trans ansfor

• rming

ming Tec echnic hnical al As Assis sistan tance: ce: Using Using Evidence Evidence to Enric to Enrich h the Experience the Experience

Ann Person • Michelle Derr • Val Uccellani Len Finocchio • LaDonna Pavetti

Center for I Imp mproving ing Research Evide idence (CI (CIRE RE) ) Forum Washing ington, , DC DC January 22, 2015

2

Welcome

Ann Person, CIRE director

3

About CIRE

• Designs and uses an array of scientific research and evaluation

approaches in diverse settings

• Has more than 40 years of experience conducting a wide range
• f rigorous applied research using cutting-edge qualitative and

quantitative methods

• Strives to bridge the gap between policy research and practice

4

CIRE Mission

• Build capacity to understand and use evidence
• Plan rigorous and relevant evaluations
• Improve research methods and standards

5

Today’s Presenters

Michelle Derr, Mathematica Len Finocchio, Mathematica LaDonna Pavetti, Center on Budget and Policy Priorities Val Uccellani, Global Learning Partners

Evidence-Based Technical Assistance Michelle Derr

6

7

Con Conte text xt

• Federal, state, and local governments invest millions in

technical assistance (TA) to improve program quality

• Fiscal constraints place even greater importance on

investments that increase program efficiency and effectiveness

• Quality standards for TA are not well-defined
• TA content often lacks analytic approach and research evidence

8

Thr hree Pill ee Pillar ars s of

• f EBT

EBTA

• Drawing upon research evidence for TA content

– Analytic assessments to identify policy and program needs – Policy and program recommendations based on research findings

• Using a TA process that includes proven methods to help

adults learn and apply TA content – Methods include careful design of the learning “architecture,” achievement-based objectives, and intentional learner engagement

• Using evaluation techniques to measure the effectiveness of

policy and program change and the overall success of the TA

Reflection eflection Questions Questions

Scan over the three pillars of EBTA as presented in your resource packet, page 1. In what ways does the TA you deliver, support, or receive align with this description of EBTA? What are you curious about?

• Webinar audience: please use group chat to

exchange thoughts and questions

• In-person audience: please share in full group

9

Adult Learning and the Design of This Forum

10

Learning Design Process

Anticipated Change People Timing Place Situation Achieve- ments Content Learning Tasks

11

 Evidence-Based Technical Assistance: Framing and Reflection  EBTA in Action: Two Perspectives  EBTA Take-Aways  Adult Learning Principles

A Snapshot of Our Time Together

12

Forum Objectives

By the end of today, every participant will accomplish the following:

• Identify ways in which the TA you support or

implement is already evidence-based.

• Explore two diverse examples of evidence-

based TA (EBTA).

• Anticipate EBTA’s potential and ways to apply it

more fully in your own work.

• Examine proven principles of adult learning.

13

Evidence-Based Adult Learning Principles

Relevance Transparency Safety Engagement Productivity Support

14

6 Evidence-Based Adult Learning Principles

As you read the resource packet pages 2-3, consider: Which principles have been most operative in your TA experience?

15

Evidence-Based Technical Assistance in Action | Perspective #1 Len Finocchio

16

Transf ansfor

• rming T

ming Tec echnical As hnical Assis sistance: tance: Usi Using ng Evidence to Enric Evidence to Enrich the Experience h the Experience

Impr Improving ving Acc Acces ess s to to Den Dental tal Ser Service vices s for

• r Childr

Children en in in Med Medicaid icaid and and CHIP CHIP

Center for Improving Research Evidence

Len Finocchio, Dr.P.H., Senior Researcher Project Director, CMS Oral Health Initiative

January 22, 2015

17

18

Evidence of the Problem

• Early Childhood Caries (ECC) is one of the most common preventable

chronic diseases for children (Dyer et al, 2007)

– ECC is an infectious and progressive disease starting primary in teeth; unaddressed ECC will continue into permanent teeth

• Among children 2-5 untreated ECC is more prevalent among:

– Children in poverty (26%) than children above 200% FPL (12%) – Hispanic children of Mexican origin (29%) and non-Hispanic black children (24%) than among non-Hispanic white children (15%) (NCHS 2010)

• Among children 1 to 20 in Medicaid & CHIP, 48% received a preventive

dental service in FFY 2013 and 23% received a dental treatment service (CMS 2015)

• Early preventive dental services for children contribute to lower treatment

costs later in childhood (Lee et al, 2006)

19

CMS Oral Health Initiative

• The Centers for Medicare & Medicaid Services (CMS) is

committed to improving access to dental and oral health services for children in Medicaid and the Children’s Health Insurance Program (CHIP)

• In 2010, CMS launched the Children’s Oral Health Initiative (OHI)

and set goals for improvement by FFY 2015 – Goal #1: Increase by 10 percentage points the proportion of Medicaid and CHIP children ages 1 to 20 (enrolled for at least 90 days) who receive a preventive dental service – Goal #2: Increase by 10 percentage points the proportion of Medicaid and CHIP children ages 6 to 9 (enrolled for at least 90 days) who receive a sealant on a permanent molar

20

Technical Assistance to States

• Five-state OHI Learning Collaborative in which states receive TA

from clinical and policy experts, CMS, and other states – States supported to develop, implement, and measure action plans for reaching OHI goals

• Key strategy is the use of driver diagrams for planning objectives,

drivers of change, and interventions – Includes using data to assess interventions

• Mathematica developing curriculum to help states improve

collection and reporting of Early and Periodic, Screening, Diagnostic and Treatment (EPSDT) benefit dental data

• Center for Health Care Strategies developing Performance

Improvement Plan tools for Medicaid managed care plans – Focused on achieving measurable objectives

21

Aims, Drivers, Interventions

Aim Primary drivers Secondary drivers Interventions Increase by 10 percentage points the proportion of Medicaid and CHIP children ages 6 to 9 (enrolled for at least 90 days) who receive a sealant on a permanent molar.

22

Assessing Interventions (1)

Intervention Intervention assessment Specs / sources of assessment data Send practice-specific “report cards”— comparing a dental practice’s sealant application rate with that of peers—to Medicaid-contracted dental providers every six months. Include lists of patients in the target population at that practice who have not had this service.  Percentage of Medicaid- contracted dental practices in the state receiving an email, report card, and non-utilization list every six months.  Provider outreach department. Pay a bonus to practices with an increase (measured every six months) in the percentage of Medicaid-enrolled children ages 6–9 who receive a sealant on a permanent molar.  Change in percentage of Medicaid-contracted dental practices receiving a bonus every six months.  Medicaid department/staff responsible for distributing payments. Give practices a list of eligible patients who have not had a dental sealant.  Lists completed for and given to each practice in the network.  Provider network reps to confirm.

23

Assessing Interventions (2)

Secondary driver Goal/baseline Data specifications Data sources Increased dental provider motivation to apply sealants for the target population. 50% of providers receive bonus payment at each six- month mark. Percentage/number of Medicaid-contracted providers receiving a bonus every six months. Medicaid program data. Improved provider ability to identify targeted children eligible for the service. 50% of providers receive bonus payment at each six- month mark. Percentage/number of Medicaid-contracted providers receiving a bonus every six months. Medicaid program data. 95% of providers got a list of eligible patients who had not had a dental sealant. Percentage/number of Medicaid-contracted providers receiving a list. Practice outreach records maintained by contracted dental plans. Primary Driver Increased number of sealants applied in dental practices. Overall increase of 10% every six months. Rate of dental sealant application for children ages 6–9 in Medicaid. Provider claims data.

24

Evidence-Based Interventions

• American Dental

Association Center for Evidence-Based Dentistry

• CMS report: Innovative

State Practices for Improving The Provision of Medicaid Dental Services

• Increasing reimbursement

rates attract dentists to Medicaid (NASHP 2006)

• “Crowd-Sourcing” - relying
• n what was financially and

politically feasible in other states

25

Data-Driven Planning & Action (CMS-416 Data)

Total children (enrolled in Medicaid/CHIP for at least 90 days) receiving:

• Line 12a: any dental service (by or under supervision of dentist)
• Line 12b: a preventive dental service (by or under supervision
• f dentist)
• Line 12c: a dental treatment service (by or under supervision of

dentist)

• Line 12d: a sealant on a permanent molar tooth (by any type of

dental provider)

• Line 12e: a dental diagnostic service (by or under dentist

supervision)

• Line 12f: an oral health service provided by a non-dentist (not under

supervision of dentist)

• Line 12g: any dental or oral health service (12a+12f)

26

Challenges with Form CMS-416 Data

• Poor data quality = poor evidence, unreliable for planning
• Confusion about proper coding
• Problematic encounter data from some Medicaid providers (e.g.

Federally Qualified Health Centers, managed care organizations)

• Not understanding numerators vs. denominators
• Time lag with data reporting
• Oral Health Initiative solution to improve data:

– Web-based curriculum with step-by-step modules for states to improve the completeness and quality of CMS-416 dental data

27

Lessons

• Many use “evidence” in a fast and loose way
• “Evidence” is more than findings related to effectiveness

(e.g., “what worked in other states?”)

• Need to expand the evidence base of state policy and
• rganizational interventions
• “Natural experiment” opportunities from policy changes:

– Adding an optional adult dental benefit – Expanding Medicaid to childless adults

• If evidence shows that policy changes result in higher costs,

that may trump all other evidence

28

For More Information

Len Finocchio, Dr.P .H. Senior Researcher Project Director, Oral Health Initiative

lfinocchio@mathematica-mpr.com

• Webinar audience: please use group chat to exchange

thoughts and questions

• In-person audience: please share in full group

29

How does this example expand your perspective

• n TA?

What questions does it raise for you?

Reflection Questions

Evidence-Based Technical Assistance in Action | Perspective #2 LaDonna Pavetti

30

LaDonna Pavetti Vice President for Family Income Support

Transforming Technical Assistance: Using Evidence to Enrich the Experience Center for Improving Research Evidence Forum and Webinar Mathematica Policy Research January 22, 2015

The Technical Assistance Challenge

• Can we use executive

function concepts and principles to improve the effectiveness of employment programs that serve disadvantaged adults?

Temporary Assistance for Needy Families (TANF) as an Example: The Context

• Modest success, even in the most effective employment

programs

• A tough labor market
• Declining employment among single mothers with high school

education or less

• Cash assistance provided to very few families—employment is

the only route out of poverty

34

Even the Most Successful TANF Employment Programs Do Little to Increase Employment Stability

Outside Forces Created the Impetus for an Exploration of New Ideas

• Recognition of the limits of early childhood interventions—

and the long-term cost of not doing more to improve the caregiving environment

• An interest in two-generation approaches: Can we improve the
• utcomes of children and their parents simultaneously?
• Innovators on the ground with promising outcomes:

– Mobility Mentoring (Crittenton Women’s Union) – MOMS Partnership (Collaboration led by Yale Child Study Center)

Theory of Change Supporting a Focus on Adults

From the Frontiers of Innovation, Harvard Center on the Developing Child:

• Protecting children from the impacts of toxic stress requires selective skill

building—not simply the provision of information and support—for the adults who care for them;

• Interventions that improve the caregiving environment by strengthening the

executive function and self-regulation skills of parents will also enhance their employability, thereby providing an opportunity to augment child outcomes by strengthening the economic and social stability of the family; and

• Community-based initiatives and broad-based systems approaches are likely

to be more effective in promoting healthy development and reducing intergenerational disparities if they focus explicitly on strengthening neighborhood-level resources and capacities that buffer young children from the adverse impacts of toxic stress.

Moving from Theory to Practice with a Focus on Evidence

• What are executive function skills, and what is their

relevance to employment programs?

• What evidence do we have that executive function skills

can be improved or that otherwise paying attention to them will produce better outcomes for adults?

• Are there evidence-based interventions from other

fields that can be adapted to employment service programs?

Evidence from the TANF World That Moving in New Directions Could Yield Positive Results

Impact of Building Nebraska’s Families (Individualized Life Skills Education Home Visiting Program) on TANF Recipients with Substantial Barriers

Looking for Answers Outside

• f Familiar Worlds

Neuroscience

Cognitive Behavioral Therapy

Trauma-Informed Care

Mindfulness

Goal-Setting

ADHD Learning Disabilities

Self-Regulation

Grit

Coaching

Child Welfare Behavioral Economics Life Skills Training

Lessons Learned

• Academic research is relevant to the program world, but it

requires lots of translation

• Practice knowledge from other worlds that is rooted in theory

is extremely valuable

• Looking outside of familiar worlds can provide a fresh

perspective through which to try and improve program effectiveness

• Research and evidence can change buzzwords into new ideas

worth implementing and testing

What’s Next?

• Develop a new intervention that draws on evidence from

multiple fields; key component is evidence-based approach to goal-setting and achievement

• Identify a few states to be R&D partners
• Develop tools practitioners can use to implement the model
• Provide guidance on using the tools to ensure the model is

implemented with fidelity

• Rigorously evaluate the model to generate new evidence and

contribute to the field

• Webinar audience: please use chat to exchange thoughts

and questions

• In-person audience: please share in full group

How does this example expand your perspective

• n TA?

What questions does it raise for you?

Reflection Questions

42

Why EBTA?

• Raises expectations and standards for the TA content, process,

and evaluation

• Enhances relationships between research and practice

communities, improving TA resources and quality

• Improves the efficiency and effectiveness of

government programs

• Furthers a research agenda that is relevant and useful to

practitioners

43

Three Pillars of EBTA

• Drawing upon research evidence for TA content
• Using a TA process that includes proven methods to help

adults learn and apply TA content

• Using evaluation techniques to measure the effectiveness of

policy and program change and the overall success of the TA

44

EBT EBTA A in in You

• ur

r Wor

• rld

ld

• We hope this brief session has inspired and

motivated you.

• What final questions or thoughts do you have for us about

EBTA?

• What do you want to share, do, or discover next?

45

Adult Learning Principles Operating in This Forum

Relevance Transparency Safety Engagement Productivity Support

46

A Learning Sequence: The Four-A Model

Source: Global Learning Partners, 2015 http://www.globallearningpartners.com/index.php

47

Anchor Add Apply Away

 Welcome and The Design of this Forum  Evidence-based Technical Assistance (E-TA): Framing and Reflection  E-TA in Action: Three Perspectives  E-TA in Your World  E-TA Take-Aways  Honoring Adult Learning Principles

A Snapshot of Our Time Together

48

 Evidence-Based Technical Assistance (EBTA): Framing and Reflection  EBTA in Action: Two Perspectives  EBTA Take-Aways  Adult Learning Principles and Sequence

A Learning Sequence: The 4A Model

As you read page 4 in the resource packet, consider: How might the principles and sequence of this forum inspire your own work?

49

50

For More Information

• Mathematica’s Center for Improving Research Evidence

CIRE@mathematica-mpr.com

• Ann Person

aperson@mathematica-mpr.com