A Mixed-Methods Study of System-Level Sustainability of an Evidence-Based Practice Following 12 Large-Scale Implementation Initiatives Amy D. Herschell, Ph.D., Ashley Tempel Scudder, Ph.D., Sarah Taber-Thomas, Ph.D., & Kristen M. Schaffner, Ph.D.
Current Team
PCIT Across PA Alumni
A Statewide Trial to Compare Three Training Models for Implementing an EBT Funder: NIMH R01 MH095750 Start Date: 9/18/12 Pennsylvania Pittsburgh- Stakeholders Project Length: 5 years based Team and Steering Committee Project PI: Amy Herschell, Ph.D. Project Coordinator: Shelley Hiegel, M.Ed. Science PCIT Experts Experts Project Trainers: Ashley T. Scudder, Ph.D. Sarah Taber-Thomas, Ph.D. Kristen F. Schaffner, Ph.D. NCSP
2011 Pennsylvania Agencies Providing Parent-Child Interaction Therapy
Present Pennsylvania Agencies Providing Parent-Child Interaction Therapy Last Updated – November, 2015
Statewide Steering Committee Membership Considerations Child Serving Systems Child Welfare Education Juvenile Justice Mental Health Regions of the State Urban/Rural Ultimate Goal - Representative from any group affected by PCIT Implementation – including Consumers
Steering Committee Members Parent of children Program Specialist Director of Pupil Services Parent who has with BH concerns Supervisor & Special Programs completed PCIT State Child Welfare School District Communications Consultant State OMHSAS Director Penn State EPIS Center President & CEO Behavior Health MCO Executive Director Community-based MH Agency Statewide Child Psychiatric Consultant OMHSAS Children’s Policy Specialist Pennsylvania Community Clinician Providers Association Community-based MH Service Systems Specialist Agency Behavioral Health Alliance ECMH Project of Rural Pennsylvania Manager Senior Medical Director Pennsylvania Key Behavioral Health MCO Professor of Practice Improvement Specialist, Psychiatry, Director, Evidence-based The Pennsylvania Child Welfare Psychology & Practice and Resource Center Pediatrics Innovation Center (EPIC)
Sustainability Defined Sustainability begins 2+ years after implementation (Stirman et al., 2012) Removal of formal training supports Focus on fit into the environment Generally thought to be the process of maintaining or improving a system’s ability to preserve a program’s function and utility
Sustainability Outcomes Sustainability is tricky 75% have partial or low sustainability (Stirman et al., 2012) REALLY TRICKY at the Client Level Proportion of sites or providers sustaining (89%) Proportion of eligible patients receiving an intervention after the period of training and implementation (11%) EVEN TRICKIER at the State Level Lower level of sustainment at the state level compared to the community level (Luke, 2014)
Dynamic Sustainability Framework
Current Study Aims 1. identify and evaluate specific methods used to sustain PCIT in service systems across the United States 2. Examine factors promoting or hindering long-term sustainability of PCIT following large-scale implementation initiatives
Inclusion Criteria Large scale training initiative : Training efforts across several counties or systems Sustaining Phase of Implementation Implementation Period: Prior to and during the formal 12 month training period (e.g., training, consultation) Sustainability Period: Following the removal of initial formal implementation supports (e.g., training, formal consultation, or funding)
Identification of Initiatives How? Database Searches: PsycINFO, Academic Search Premier, Google Scholar Search Terms: evidence-based practice, evidence-based treatment, dissemination, implementation, Parent-Child Interaction Therapy, sustainability, sustainment PCIT Trainer and the treatment developer
Identification of Initiatives How Many? 21 identified 6 did not meet “large scale” definition 2 were not yet in the sustaining phase 2 were really part of one initiative 12 initiatives included from 13 states
Large-scale PCIT Initiatives Enrolled 1. California 2. Delaware 3. Iowa 4. Michigan 5. Minnesota 6. Nebraska 7. North & South Carolina 8. Oklahoma 9. Oregon 10. Pennsylvania 11. Tennessee 12.Washington
Measures & Procedures Implementation and training survey (25 items) Training resources, Implementation Timeline, Subsequent Training Efforts, Methods & Consultation Approach Sustainability interview Open-ended Active strategies, Barriers & Facilitators Barriers, Strategies and Sustainment Survey (19 items) Extent to which barriers were present and the degree to which specific strategies were used; overall level of sustainment Program Sustainability Assessment Tool (40 items) Environmental Support, funding stability, partnerships organizational capacity, program evaluation, program adaption communications, and strategic planning
Participants 12 initiatives 37 individuals participated in the sustainability interviews (M= 3.33, range = 1 to 8) 10 PCIT trainers 9 state officials 5 behavioral health providers 4 individuals from private foundations 2 directors of oversight centers 1 judge 1 managed care representative 5 individuals working in academic settings
Codebook Development Training and Intervention Strategies to Sustain System, Agency, and Funding Implementation Characteristics Therapist Factors Factors Openness/ Resistance to Approach/ Appeal of PCIT Federal Funds Infrastructure EBPs Philosophy Policy Trained Clinician Cost of PCIT State Funds Marketing Characteristics PCIT Champion Training and Cost-Benefit of Local Funds Integration into Implementation PCIT Existing Practices Factors Beyond the Agency Approach/ MCO Funds New Settings/ Support (+/-) Philosophy Populations Agency Support (+/-) Trained Clinician Private Insurance Funds Balancing Supply Characteristics and Demand Therapist Support (+/-) Training and Other Funds Continuing Implementation Education Factors PCIT Service Reimbursement Within Agency Training Building Partnerships Fidelity Monitoring Tracking Clinical Competency Monitoring Clinical Outcomes
Descriptives Min Max M SD % Clinicians Continuing to Provide 41 93 76.97 16.006 % Agencies Continuing to Provide 55 100 86.54 14.412 Total Clinicians Trained 27 >400 167.67 123.483 Self-Report of Overall Sustainability 2 7 5 1.537 PSAT Average 2.78 5.80 4.5229 .91853 Environmental Support 2.20 6.40 5.1667 1.18424 Funding Stability 2.00 5.60 4.2833 1.00348 Partnerships 2.40 7.00 4.6500 1.38334 Organizational Capacity 2.20 6.20 4.1333 1.30547 Program Evaluation 1.00 6.60 4.3167 1.75076 Program Adaptation 2.40 7.00 5.2333 1.15312 Communications 2 7 4.53 1.394 Strategic Planning 2.00 6.20 3.8667 1.22796
Model Summaries R 2 R 2 DV IV R changeR F chg p df 1 df 2 adj 2 Integration a % clinicians 1. .716 .512 .442 .512 7.344 .030 1 7 Integration a % agencies 1. .646 .417 .352 .417 6.444 .032 1 9 Integration a 1. Total clinicians .693 .480 .428 .480 9.224 .013 1 10 Integration b PSAT 1. .681 .464 .411 .464 8.663 .015 1 10 1. Barrier of financial .821 .674 .602 .210 5.805 .039 1 9 support b Integration b Overall 1. .904 .817 .798 .817 44.516 .000 1 10 Sustainability Monitoring quality b 1. .969 .939 .925 .122 17.917 .002 1 9 Note. * Indicates significance at p <.001. % clinicians =% clinicians continuing to provide, % agencies =% agencies continuing to provide, Total clinicians= Total clinicians trained, Overall Sustainability = Initiative Rating of Overall Sustainability. IV: a = Interviewer rating; b = Initiative Rating.
What seems to be the most important? Integration (quant & qual) Financing (quant & qual) Quality Monitoring (quant & qual) Intervention Champions (qual)
Integrating into practice Training & education of other professionals (e.g., psychiatry residents, professionals in healthcare settings) training in the basics of PCIT educating other professionals about the intervention (e.g., Troutman, 2011). Integrate PCIT into other services or programs, Allowing PCIT to “bleed into a lot of other programs.” Expand PCIT to new settings and populations “special time” at a residential facility Teacher-Child Interaction Therapy
Money Matters 2 subcategories Implementation funding and financial support Service reimbursement and billing practices “So, for us it is very costly to “You can have the best do this stuff…. But then trained therapists in the these children’s trajectories Incentives world but if nobody’s and their lives have Increase funding dedicated to EBPs changed. So for me that’s going to pay for it then worth whatever you’re Reimburse EBPs at higher session rate there’s not much putting into it.” Recognize practices as EBPs incentive to keep Service accommodations going…” Disincentives Existing/Competing services
Recommend
More recommend
