Workgroup Update Evaluation Team Bethany Lee, PhD, Evaluation - - PowerPoint PPT Presentation

Families Blossom Evaluation Workgroup Update

Evaluation Team

Bethany Lee, PhD, Evaluation Director

October 21, 2016

Meet the Title IV-E Waiver Evaluation Team

Bethany Lee, PhD Evaluation Director Pam Freeman, PhD Lead Evaluator for CANS-F/ Trauma Elizabeth Greeno, PhD Lead Evaluator for EBPs

John Cosgrove Senior Data Analyst Rochon Steward Research Specialist

Maria Jose Horen Research Supervisor

Sara Betsinger, PhD Lead Cost Evaluator Eric Slade, PhD Cost Consultant

Rachel Margolis Research Asst David Chen Senior Data Analyst

Evaluation Components

Increasing Evidence-Based and Promising Practices

• Process Evaluation
• Outcomes Evaluation

Creating a Trauma-Informed System

• Process Evaluation
• Outcomes Evaluation

Cost Analysis

• Cost shifts
• Change in total expenses

Monitoring Trends in State-Wide Outcomes

EBP Models

• DSS Service Models

– Baltimore City: Solution Based Casework (SBC)

• Parenting Models

– Allegany County: Incredible Years (IY) – Anne Arundel County: Parent-Child Interaction Therapy (PCIT) – Harford County: Nurturing Parenting Program (NPP)

• Child Mental Health/Behavioral Health Models

– Anne Arundel County: Functional Family Therapy (FFT) – Baltimore County: Cognitive Behavior Therapy+ (CBT+)

SafeCare Comparative Effectiveness Trial: Howard and Prince George’s Counties

Start Dates

County EBP Start Dates Allegany Incredible Years June 2016 Anne Arundel Functional Family Therapy September 2016 Anne Arundel Parent-Child Interaction Therapy September 2016 Baltimore City Solution-Based Casework September 2016 Baltimore County CBT+ June 2016 Harford County Nurturing Parenting Program April 2016

Process Evaluation

• National Implementation Research Network

(NIRN) guidelines

• What planning, training, and other practices

happened in efforts to implement the evidence based practice? [what did we do?]

• Tracking of individuals served

and trained

• Fidelity to the models

Type of Data Data Element Specific Elements to Collect: Program - Level Staff Training Describe the number and type of staff involved in implementation (including demographics, education, and workplace characteristics) and the training they received. Practice- Level Eligibility Describe client eligibility to receive the new intervention, process for referring subjects for services, alternates to the new intervention (where applicable). Client- Level Participation Describe the number of children/families served and the type and duration of services provided. Practice- Level Fidelity Describe instruments to assess fidelity to the intended service model. Practice- Level Implementation Report any barriers during implementation, steps taken to address the barriers, and lessons learned.

Process data elements for each EBP

Domain Data Element Specific Elements to Collect: Safety, permanency MD CHESSIE Outcomes Entry, re-entry, length of stay, placement type, stability; Well-being, functioning Clinical

• utcomes

Standardized measures specific to the intervention Satisfaction Consumer Feedback Satisfaction with services, perceptions of impact Service use Auxiliary Public System Data (if relevant) Public mental health claims or juvenile justice involvement

Outcome Data elements for each EBP

Baltimore City: SBC

• Sample: All 100+ Baltimore City In-Home Family

Preservation workers will be trained in SBC. Estimated 2,000 families will be served with SBC per year.

• Implementation Status: Train-the-training

trainings occurred April-August (4 separate trainings). Training for in-home workers is slated for September 12th to October 16th. Trainings conducted by a certified SBC trainer.

Baltimore City: SBC

• Design: One-group longitudinal design.
• Measures: UMB SSW has collaborated with SBC

providers to track implementation and fidelity. Online surveys have been given to Baltimore City staff to assess job-related quality of life (Professional Quality of Life), job burnout (Maslach Burnout Inventory), and Supervision Satisfaction (subscale of organizational climate measure). Baseline measures; posttest will be given between 12-18 months post training.

• Secondary data from MD CHESSIE to monitor

investigations, substantiations, or removals will occur semi-annually and during the 12-months following SBC participation.

Allegany County: IY

• Sample: Any family with children ages birth to

12 that have involvement with Allegany Co

• DSS. Following IY guidelines for group size,

first year approximately 30 families and 50 children will be served.

• Implementation Status: IY currently

implemented in collaboration with Family

• Junction. First group was completed end of

September 2016.

Allegany County: IY

• Design: 3-Group Quasi-Experimental Design to compare

families and children receiving IY services to those receiving PCIT, and those receiving TAU from a comparable Maryland

• jurisdiction. Primary data will be collected at 2 time points:

baseline and posttest.

• Measures: Primary outcome measures will include child

behavioral change (Eyberg Child Behavior Inventory), parenting stress (Parenting Stress Index). Additional measures will include therapeutic relationship alliance (Working Alliance Inventory-Client).

• Secondary data from MD CHESSIE to monitor

investigations, substantiations, or removals during the 12- months following IY participation.

• Sample: Children ages 2-7 with severe

behavioral health problems and their parents, who are receiving in-home services in Anne Arundel County. In year 1, approximately 25 children will be served.

• Implementation Status: A PCIT provider has

been identified and it is estimated PCIT will begin in September 2016.

Anne Arundel County: PCIT

Anne Arundel County: PCIT

• Design: 3-Group Quasi-Experimental Design to compare families

and children receiving PCIT services to those receiving IY, and those receiving TAU from a comparable Maryland jurisdiction. Primary data will be collected at 2 time points: baseline and posttest.

• Measures: Primary outcome measures will include child behavioral

change (Eyberg Child Behavior Inventory) and parenting stress and attitudes (Parenting Stress Index – Short Form). Additional measures will include therapeutic relationship alliance (Working Alliance Index – Client Version) and quality of family life (asked via

• pen-ended questions).
• Secondary data from MD CHESSIE to monitor investigations,

substantiations, or removals during the 12-months following PCIT participation.

Harford County: NPP

• Sample: Any family in Harford County who has an
• pen child welfare case and has a child between

the ages of 5 to 11. In year 1, 2 NPP groups will be completed with estimated 14 families and 20 children served the first year.

• Implementation Status: NPP provider training of

facilitators occurred in January/February 2016. First NPP group occurred April-June (8 participants total). Second cohort began in August 2016.

Harford County: NPP

• Design: One-Group baseline-posttest design
• Measures: Primary outcome data collected at 2 time

points: baseline and posttest. As per the NPP model, baseline-posttest measures include the Adult Adolescent Parenting Inventory-2 (AAPI; measures parenting and child rearing attitudes) and the Nurturing Skills Quiz (measures quality of family life). Additional measures will include therapeutic relationship alliance (Working Alliance Inventory- Client).

• Secondary data from MD CHESSIE to monitor

investigations, substantiations, or removals during the 12-months following NPP participation.

Anne Arundel Co: FFT

• Sample: Families with youth ages 11-18 years

receiving in-home services, in a voluntary placement, or older adolescents returning home, presenting with behavioral issues. In year 1, approximately 15 families will be served.

• Implementation: FFT provided by Center for

Children, beginning around September 2016.

Anne Arundel Co: FFT

• Design: Clinical outcomes of FFT measured by a Single Group

Baseline-Posttest Design.

– Quasi experimental design used to measure child welfare and juvenile justice outcomes. FFT youth compared to similar youth receiving TAU

• Measures: Primary outcome measures will include clinical

measures that will assess the youth’s moods and behaviors (Youth Outcome Questionnaire; Youth Outcome Questionnaire – Self- Report) and change in family functioning (Client Outcome Measure – Parent; Client Outcome Measure – Adolescent) from perspective

• f the parent and youth; therapist outcome measure.
• Secondary data from MD CHESSIE to monitor investigations,

substantiations, or removals during the 12-months following FFT participation.

Baltimore County: CBT+

• Sample: Two cohorts, each of 40 community MH providers

and 40 CW staff will be trained; 80 families from each cohort expected to be served.

– For CW staff: In-Home Service workers will be the primary group

• trained. For children/youth: any child over the age of 5 w/o

cognitive limitations.

• Implementation Status: Training is provided by the

National Center for Evidence Based Practice in Child Welfare (NCEBPCW) at the UMB.

– Training for the 1st cohort occurred in June 2016; 36 mental health providers and 41 child welfare staff in attendance from THRIVE and Catholic Charities – Second training will occur in the fall

Baltimore County: CBT+

Implementation Status (Cont.)

Knowledge pre and post-training was collected for this training; score above 70% indicates participant passed knowledge assessment.

Participated in training Completed posttest Scored above 70% on posttest Mental Health Staff 36 35 (97%) 35 (100%) Child Welfare Staff 41 34 (83%) 32 (94%)

Baltimore County: CBT+

• Design: CBT+ evaluated at two levels

– Training participants evaluated using a single-group design pre-posttest design (knowledge) – Child-level data from youth served measured by the EBP Toolkit (part of the NCEBPCW). MH providers enter data into EBP toolkit; data includes standardized clinical measures

• Secondary data from MD CHESSIE to monitor

investigations, substantiations, or removals during the 12-months following CBT+ participation

SafeCare

• Georgia State University’s National SafeCare

Training and Research Center (NSTRC) will be implementing and evaluating SafeCare.

• State of Maryland has a separate contract

with NSTRC to conduct the Safe Care evaluation.

• UMB SSW will be tracking selected process

measures.

Next Steps for EBP

• Continue with evaluation plans
• Assess use of other EBPs in Maryland

– Using flex funds – Reach of Waiver Demonstration