Following 12 Large-Scale Implementation Initiatives Amy D. - - PowerPoint PPT Presentation

Amy D. Herschell, Ph.D., Ashley Tempel Scudder, Ph.D., Sarah Taber-Thomas, Ph.D., & Kristen M. Schaffner, Ph.D.

A Mixed-Methods Study of System-Level Sustainability of an Evidence-Based Practice Following 12 Large-Scale Implementation Initiatives

Current Team

PCIT Across PA Alumni

A Statewide Trial to Compare Three Training Models for Implementing an EBT

Pittsburgh- based Team Pennsylvania Stakeholders and Steering Committee Science Experts PCIT Experts

Funder: NIMH R01 MH095750 Start Date: 9/18/12 Project Length: 5 years Project PI: Amy Herschell, Ph.D. Project Coordinator: Shelley Hiegel, M.Ed. Project Trainers: Ashley T. Scudder, Ph.D. Sarah Taber-Thomas, Ph.D. Kristen F. Schaffner, Ph.D. NCSP

2011 Pennsylvania Agencies Providing Parent-Child Interaction Therapy

Present Pennsylvania Agencies Providing Parent-Child Interaction Therapy

Last Updated – November, 2015

Statewide Steering Committee Membership Considerations

 Child Serving Systems

 Child Welfare  Education  Juvenile Justice  Mental Health

 Regions of the State  Urban/Rural  Ultimate Goal - Representative from any

group affected by PCIT Implementation – including Consumers

Steering Committee Members

Communications Consultant State OMHSAS President & CEO Behavior Health MCO Parent of children with BH concerns Parent who has completed PCIT Program Specialist Supervisor State Child Welfare Director of Pupil Services & Special Programs School District Director Penn State EPIS Center Senior Medical Director Behavioral Health MCO Statewide Child Psychiatric Consultant OMHSAS Service Systems Specialist Behavioral Health Alliance

• f Rural Pennsylvania

Executive Director Community-based MH Agency Clinician Community-based MH Agency Children’s Policy Specialist Pennsylvania Community Providers Association Director, Evidence-based Practice and Innovation Center (EPIC) ECMH Project Manager Pennsylvania Key Professor of Psychiatry, Psychology & Pediatrics Practice Improvement Specialist, The Pennsylvania Child Welfare Resource Center

Sustainability Defined

 Sustainability begins 2+ years after implementation

(Stirman et al., 2012)  Removal of formal training supports  Focus on fit into the environment

 Generally thought to be the process of

maintaining or improving a system’s ability to preserve a program’s function and utility

Sustainability Outcomes

 Sustainability is tricky

 75% have partial or low sustainability (Stirman et al., 2012)

 REALLY TRICKY at the Client Level

 Proportion of sites or providers sustaining (89%)  Proportion of eligible patients receiving an intervention

after the period of training and implementation (11%)

 EVEN TRICKIER at the State Level

 Lower level of sustainment at the state level

compared to the community level (Luke, 2014)

Dynamic Sustainability Framework

Current Study Aims

• 1. identify and evaluate specific methods

used to sustain PCIT in service systems across the United States

• 2. Examine factors promoting or hindering

long-term sustainability of PCIT following large-scale implementation initiatives

Inclusion Criteria

 Large scale training initiative:

 Training efforts across several counties or systems

 Sustaining Phase of Implementation

 Implementation Period: Prior to and during the

formal 12 month training period (e.g., training, consultation)

 Sustainability Period: Following the removal of

initial formal implementation supports (e.g., training, formal consultation, or funding)

Identification of Initiatives

 How?

 Database Searches: PsycINFO, Academic Search

Premier, Google Scholar

 Search Terms: evidence-based practice, evidence-based

treatment, dissemination, implementation, Parent-Child Interaction Therapy, sustainability, sustainment

 PCIT Trainer and the treatment developer

Identification of Initiatives

 How Many?

 21 identified

 6 did not meet “large scale” definition  2 were not yet in the sustaining phase  2 were really part of one initiative

 12 initiatives included from 13 states

Large-scale PCIT Initiatives

Enrolled

• 1. California
• 2. Delaware
• 3. Iowa
• 4. Michigan
• 5. Minnesota
• 6. Nebraska
• 7. North & South

Carolina

• 8. Oklahoma
• 9. Oregon
• 10. Pennsylvania
• 11. Tennessee

12.Washington

Measures & Procedures

 Implementation and training survey (25 items)

 Training resources, Implementation Timeline, Subsequent Training

Efforts, Methods & Consultation Approach

 Sustainability interview

 Open-ended  Active strategies, Barriers & Facilitators

 Barriers, Strategies and Sustainment Survey (19 items)

 Extent to which barriers were present and the degree to which

specific strategies were used; overall level of sustainment

 Program Sustainability Assessment Tool (40 items)

 Environmental Support, funding stability, partnerships

• rganizational capacity, program evaluation, program

adaption communications, and strategic planning

Participants

 12 initiatives

 37 individuals participated in the sustainability

interviews (M= 3.33, range = 1 to 8)

 10 PCIT trainers  9 state officials  5 behavioral health providers  4 individuals from private foundations  2 directors of oversight centers  1 judge  1 managed care representative  5 individuals working in academic settings

Codebook Development

System, Agency, and Therapist Factors Training and Implementation Factors Intervention Characteristics Funding Strategies to Sustain

Openness/ Resistance to EBPs Approach/ Philosophy Appeal of PCIT Federal Funds Infrastructure Policy Trained Clinician Characteristics Cost of PCIT State Funds Marketing PCIT Champion Training and Implementation Factors Cost-Benefit of PCIT Local Funds Integration into Existing Practices Beyond the Agency Support (+/-) Approach/ Philosophy MCO Funds New Settings/ Populations Agency Support (+/-) Trained Clinician Characteristics Private Insurance Funds Balancing Supply and Demand Therapist Support (+/-) Training and Implementation Factors Other Funds Continuing Education PCIT Service Reimbursement Within Agency Training Building Partnerships Fidelity Monitoring Tracking Clinical Competency Monitoring Clinical Outcomes

Descriptives

Min Max M SD % Clinicians Continuing to Provide 41 93 76.97 16.006 % Agencies Continuing to Provide 55 100 86.54 14.412 Total Clinicians Trained 27 >400 167.67 123.483 Self-Report of Overall Sustainability 2 7 5 1.537 PSAT Average 2.78 5.80 4.5229 .91853 Environmental Support 2.20 6.40 5.1667 1.18424 Funding Stability 2.00 5.60 4.2833 1.00348 Partnerships 2.40 7.00 4.6500 1.38334 Organizational Capacity 2.20 6.20 4.1333 1.30547 Program Evaluation 1.00 6.60 4.3167 1.75076 Program Adaptation 2.40 7.00 5.2333 1.15312 Communications 2 7 4.53 1.394 Strategic Planning 2.00 6.20 3.8667 1.22796

Model Summaries

DV IV R R2 R2

adj

changeR

2

Fchg p df1 df2 % clinicians

1. Integrationa

.716 .512 .442 .512 7.344 .030 1 7 % agencies

1. Integrationa

.646 .417 .352 .417 6.444 .032 1 9 Total clinicians

1. Integrationa

.693 .480 .428 .480 9.224 .013 1 10 PSAT

1. Integrationb

.681 .464 .411 .464 8.663 .015 1 10

1. Barrier of financial supportb

.821 .674 .602 .210 5.805 .039 1 9 Overall Sustainability

1. Integrationb

.904 .817 .798 .817 44.516 .000 1 10

1. Monitoring qualityb

.969 .939 .925 .122 17.917 .002 1 9

• Note. *Indicates significance at p<.001. % clinicians =% clinicians continuing to provide, % agencies =% agencies

continuing to provide, Total clinicians= Total clinicians trained, Overall Sustainability = Initiative Rating of Overall Sustainability. IV: a = Interviewer rating; b = Initiative Rating.

What seems to be the most important?

 Integration (quant & qual)  Financing (quant & qual)  Quality Monitoring (quant & qual)  Intervention Champions (qual)

Integrating into practice

 Training & education of other professionals (e.g.,

psychiatry residents, professionals in healthcare settings)

 training in the basics of PCIT  educating other professionals about the intervention (e.g.,

Troutman, 2011).

 Integrate PCIT into other services or programs,

 Allowing PCIT to “bleed into a lot of other programs.”

 Expand PCIT to new settings and populations

 “special time” at a residential facility  Teacher-Child Interaction Therapy

Money Matters

2 subcategories

 Implementation funding and financial support  Service reimbursement and billing practices  Incentives

 Increase funding dedicated to EBPs  Reimburse EBPs at higher session rate  Recognize practices as EBPs  Service accommodations

 Disincentives

 Existing/Competing services

 “You can have the best

trained therapists in the world but if nobody’s going to pay for it then there’s not much incentive to keep going…”

“So, for us it is very costly to do this stuff…. But then these children’s trajectories and their lives have

• changed. So for me that’s

worth whatever you’re putting into it.”

Monitoring of quality

 Monitoring quality at a system level is complex  Few initiatives monitored clinical outcomes at a

system level

 4 types of quality indicators

 Treatment fidelity  Clinician competencies in PCIT  Clinical service outcomes  Workforce outcomes

“You need longitudinal measurement in order to know the value of [PCIT]. And that’s the hard thing to sell in an economy that’s

• perating on a year-to-year budget.

At the end of the day, we have to prove [to payers] that evidence- based is actually something that makes money.”

Champions are important

 Champions were identified as one of the

most impactful factors

 One initiative formalized champions

PCIT

“going to make PCIT happen no matter what.” “go to bat for PCIT and talk to anyone who will listen.” “culture carriers” “flag bearers” people who “pound the PCIT drum” “creating a positive contagion” “carrying it forward”

Limitations

 One evidence-based treatment  Small sample size  Although every effort was made to include as

many individuals from each initiative as possible, not all individuals could be reached.

 Given the dynamic nature of sustainability, it is

possible that data collected at another time or from other informants may differ.

 Measurement limitations

 PSAT has psychometric data, but is novel and has

limited data with which to compare our findings.

Practical Implications

 Plan for integration

 Webinars for affiliated staff and system partners  Information sessions early on

 Allow (maybe even encourage)adaptations  Encourage local expertise – embedded at the

agency level (Within agency trainers)

 Partner with Behavioral Health Managed Care

Organizations

 Add continuous quality improvement processes

into implementation

 Develop and elevate “Champions”

Thank you for your time and interest!

Amy D. Herschell, Ph.D. Associate Professor of Psychology & Family Medicine 1234 Life Sciences Building 53 Campus Drive PO Box 6040 West Virginia University Morgantown, WV 26506-6040 Phone: 304-293-1719 Email: Amy.Herschell@mail.wvu.edu