The EPISCenter represents a collaborative partnership between the Pennsylvania Commission on Crime and Delinquency (PCCD), and the Prevention Research Center, College of Health and Human Development, Penn State University. The EPISCenter is funded by PCCD and the Department of Public Welfare. This resource was developed by the EPISCenter through PCCD grant VP-ST-24368.
Welcome New Pennsylvania Commission on Crime and Delinquency - - PowerPoint PPT Presentation
Welcome New Pennsylvania Commission on Crime and Delinquency - - PowerPoint PPT Presentation
Welcome New Pennsylvania Commission on Crime and Delinquency Grantee Orientation The EPISCenter represents a collaborative partnership between the Pennsylvania Commission on Crime and Delinquency (PCCD), and the Prevention Research Center,
Meet & Greet Activity
- Help yourself to refreshments
- Please join the table for your program
- Introduce yourselves
- Describe your organization
- Describe why your organization/community selected the
program including:
- Target Population
- Targeted Risk and Protective Factors
- Hopes for the future (after PCCD funding ends)
The EPISCenter represents a collaborative partnership between the Pennsylvania Commission on Crime and Delinquency (PCCD), and the Prevention Research Center, College of Health and Human Development, Penn State University. The EPISCenter is funded by PCCD and the Department of Public Welfare. This resource was developed by the EPISCenter through PCCD grant VP-ST-24368.
PCCD’s Investment in Evidence- based Programs
EPISCenter’s Three Key Functions
- Build general prevention
capacity
- Build program-specific
capacity
- Facilitate interaction/
communication between systems
Transla'onal ¡ Research
EPISCenter ¡
(Preven'on ¡ Support ¡System)
Build ¡general ¡preven'on ¡capacity ¡ among ¡prac''oners ¡and ¡policy ¡makers ¡
Technical ¡ Assistance
Con'nuous ¡Quality ¡ Improvement
EBP Grantees & Community Coalitions (Prevention Delivery System) Penn State’s Prevention Research Center (Prevention Synthesis & Translation System) Resource Center Steering Committee (Policy Makers & Funders) Rhoades, Bumbarger & Moore (in press). The Role of a State-Level Prevention Support System in Promoting High-Quality Implementation and Sustainability of Evidence-based Programs. American Journal of Community Psychology.
Pennsylvania’s EBP Dissemination in 1999…
Pennsylvania’s EBP Dissemination in 2014…
Resource ¡Center ¡ ¡ for ¡Evidence-‑based ¡Preven'on ¡and ¡ ¡ Interven'on ¡Programs ¡and ¡Prac'ces ¡
Support to Community Prevention Coalitions Improve ¡Quality ¡of ¡ ¡ Juvenile ¡Jus'ce ¡Programs ¡ and ¡Prac'ces ¡ Support to Evidence- based Programs
Multi-Agency Steering Committee (Justice, Welfare, Education, Health)
A unique partnership between policymakers, researchers, and communities to bring science to bear on issues of public health and public safety.
The EPISCenter is a project of the Prevention Research Center, College of Health and Human Development, Penn State University, and is funded by the Pennsylvania Commission on Crime and Delinquency and the Pennsylvania Department of Public Welfare as a component of the Resource Center for Evidence-Based Prevention and Intervention Programs and Practices.
The EPISCenter represents a collaborative partnership between the Pennsylvania Commission on Crime and Delinquency (PCCD), and the Prevention Research Center, College of Health and Human Development, Penn State University. The EPISCenter is funded by PCCD and the Department of Public Welfare. This resource was developed by the EPISCenter through PCCD grant VP-ST-24368.
PCCD Grantee Requirements
PCCD Representative
Quality Reporting
- Grantees are responsible for reporting in Egrants
quarterly.
- Report can be initiated after previous quarter is approved.
- Reports are due to be submitted by the 20th of the month after the
quarter is closed.
- Data collection tools and other documents are submitted as
attachments
Quality Assurance Process
- Purpose: To assess site functioning, review your data
collection process, and ability to demonstrate and communicate impact.
Start Date QA Process Date July 1, 2014 July-September 2015
*Funds should have been budgeted in your grant for this process.
Quality Assurance Process
- The (insert name of County/Agency) site is implementing with sufficient
- quality. Given the current implementation, it is reasonable to expect future
positive outcomes from the program. At this time, the areas identified as needing improvement are limited and as to be expected for the length of implementation time.
- The (insert name of County/Agency) site needs to make significant changes
to their implementation and is currently not implementing with the level of quality that is expected for the amount of time it has been implementing. For programs in this category, it is PCCD's intention to work with the developer and grantee to articulate a plan of corrective action and timetable to bring the program into compliance with the developer's requirements.
- The (insert name of County/Agency) site is not implementing the program as
designed and the developer does not believe that the site’s implementation quality is correctable within a reasonable amount of time.
Outcomes Report
- Purpose-
- Summarize Data Collected to Measure
- Program Reach
- Implementation Quality
- Impact
Start Date OR Due Date July 1, 2014 April 2016
Outcomes Report
- Quantitative Data
- Performance measures and survey data
- Qualitative Data
- Feedback from participants, staff, and program partners
- Changes to project plan, project modification requests
Outcomes Report
- Section 1: Description of Population Served
- Section 2: Indicators of Program Impact
- Section 3: Indicators of Implementation Quality
- Section 4: Lessons Learned
- Section 5: Program Sustainability
Top Three Reasons to Call a PCCD Analyst
- Project Modification Requests - Program and Fiscal
changes to your application
- Quarterly Program/Fiscal Report questions
- Any fiscal questions/concerns should be directed to the
fiscal contact listed in Egrants
The EPISCenter represents a collaborative partnership between the Pennsylvania Commission on Crime and Delinquency (PCCD), and the Prevention Research Center, College of Health and Human Development, Penn State University. The EPISCenter is funded by PCCD and the Department of Public Welfare. This resource was developed by the EPISCenter through PCCD grant VP-ST-24368.
EPISCenter Technical Assistance
Roger Spaw EPISCenter Prevention Coordinator
Evolution of Technical Assistance
- 1998: PCCD invests in prevention programming.
- Difficult to monitor implementations of a wide variety of programs.
- Grantees experienced mixed outcomes; some programs worked
well, some failed.
- It was hard to know about problems until “after the fact”.
- No formal technical assistance was available.
Evolution of Technical Assistance
- 2001: Penn State’s Prevention Research Center begins
providing Technical Assistance to PCCD grantees.
- Grantees get support from the beginning, but programs are varied
and not all of them have high levels of evidence of effectiveness.
- Technical Assistance was mostly reactive.
- No standard approach to quality assurance
- Performance Measures varied greatly from grantee to grantee
- Difficult to know if programs were having desired impact
Evolution of Technical Assistance
- 2008: EPISCenter was established to support programs
with a higher level of evidence of effectiveness.
- “Blueprints” programs www.blueprintsprograms.com
- EPISCenter now PROACTIVELY targets Technical
Assistance for 11 programs.
- Goal: To advance high quality implementation, impact assessment,
and sustainability of programs in order to maximize the positive impact of prevention science on outcomes for Pennsylvania youth.
Evolution of Technical Assistance
How?: Explaining the Magic
“I think you should be a bit more explicit here in step 2.”
Why Technical Assistance
- Goal: To advance high quality implementation, impact
assessment, and sustainability of programs in order to maximize the positive impact of prevention science on
- utcomes for Pennsylvania youth.
- Objectives:
- Assess and support program implementation
- Promote program quality and fidelity
- Support the data collection process
- Maximize the ability to demonstrate and communicate impact
- Increase local investment & sustainability
How? Group Training and Networking
- Networking Meetings
- PCs organize regular networking opportunities to allow grantees
who are doing the same program to share lessons learned, tools, and resources.
- Trainings
- Often PCs will identify a common training need for several
- grantees. They then identify experts who can meet this need and
facilitate in-person or webinar training events. These trainings can be geared towards all of the EBPs or just one program specifically.
How? Personal Consultations
- On-Site Consultations
- PC visits you in person in your community to review the Web-based
resources and discuss any successes or barriers
- Typically twice a year, but varies depending on program and need
- Initial consultation always includes completion of an
Implementation Plan with action steps for PC and Grantee
- Web-Based Consultations
- Skype, Adobe Connect, and Go-To Meeting are frequently used to
connect with grantees
- Email & Phone Calls
- Both great ways to contact your PC with a question
How? Web-based Resources
- Web Based Resources: www.EPISCenter.psu.edu
- Logic Models
- Frequently Asked Questions
- Readiness Guides
- Implementation Manuals
- Impact Measurement Tools & Guidance
- Performance Measure Reporting Tools and Guidance
- Links to research articles, Developer Websites, etc.
- Templates for Outcomes Reporting to PCCD and community
stakeholders
- Training videos
Implementation Plan
Demonstrating Program Impact
§ Solid data collection systems § Implementing with a commitment to quality and fidelity § Consistently analyzing process and outcomes data
Who Provides Technical Assistance?
- EPISCenter Prevention Coordinators or “PCs”
- Prevention Coordinators:
- Lee Ann Cook – IYS (MST, FFT, MTFC)
- Kris Glunt – LST, TND, OBPP
- Roger Spaw - ART, BBBS
- Mary Ann Demi- SFP 10-14, PATHS
- Elizabeth Campbell – MST, FFT, MTFC
The EPISCenter represents a collaborative partnership between the Pennsylvania Commission on Crime and Delinquency (PCCD), and the Prevention Research Center, College of Health and Human Development, Penn State University. The EPISCenter is funded by PCCD and the Department of Public Welfare. This resource was developed by the EPISCenter through PCCD grant VP-ST-24368.
Planning & Start-up
Lee Ann Cook, MSS EPISCenter Prevention Coordinator
Don’t Reinvent The Wheel!
- Use EPIS Readiness Tools
Did you follow recommendations for:
- Training Information & Coordination?
- Local training is often available
- Materials Purchasing Guidelines?
- Do you really need that video or those stickers?
- Real World Time Requirements
- Staff Requirements?
- How many are required?
- What qualifications are required?
Break Out
- EPISCenter PC will review specific “start up” resources
- Discuss: How is start up going for you?
- Successes?
- Barriers?
- How can EPIS help?
Lunch
- On your own
- Be back by 1:30 p.m. Please!
The EPISCenter represents a collaborative partnership between the Pennsylvania Commission on Crime and Delinquency (PCCD), and the Prevention Research Center, College of Health and Human Development, Penn State University. The EPISCenter is funded by PCCD and the Department of Public Welfare. This resource was developed by the EPISCenter through PCCD grant VP-ST-24368.
Implementation Quality (IQ) – Ensuring the effectiveness of your program
Mary Ann Demi, M.S. Research Coordinator, EPISCenter
Effective Program
- What does it mean when a program has been determined
effective/evidence based?
- The intervention is based on a theory of change that is documented
in a clear logic or conceptual model
- Documentation exists that it has been effectively implemented in
the past with:
- In a scientific manner
- With high and rigorous standards
- With credible and positive effects
- The intervention is reviewed by a panel of informed prevention
researchers
Basis for “Evidence”
- Reduces risk due to a strong research design
- Desired risk reduction (outcomes) is sustained over time
- Desired outcomes are replicated in multiple sites
Fidelity Leads to Effectiveness
- What is fidelity?
- Fidelity is:
- Delivering a prevention or intervention program and adhering to the to
the original protocol or program model.
- Implementing a program without removing parts of the program that are
essential to the program’s effectiveness—its core components.
- Implementing a program without adding parts to the program that
are not part of the original protocol
FOLL FOLLOW THE RECIPE W THE RECIPE
Logic Model
- The “Logic Model’ is the recipe and is
- Developed based on tested theory of change
- Has key components (ingredients) and without those key
components recipe may fail
- Often follow a logical sequence of cause and effect:
If this, than that ….. Program Activity Decreased Risk Behavioral Outcomes
Logic Model Example:
Strengthening Families Program
Model created by Penn State EPISCenter
Deviating from program design…
- “When communities ‘tweak’ a program to suit their own
preferences or circumstances, they wind up with a different program whose effectiveness is unknown.”
Blueprints for Violence Prevention www.colorado.edu/cspv/blueprints
Elements of Implementation Quality
- Dosage
- Adherence
- Quality of Delivery
- Program Reach
- Participant Responsiveness
- Dosage
- All lessons are taught
- Lessons are taught the entire length of time prescribed
- All materials correspond with lessons
Elements of Implementation Quality
- Fidelity/Model Adherence
- Understand the theory behind the program
- Comply with recommended staff qualifications
- Contract with developers to train staff in the use of the model
- Enroll intended participants (universal versus targeted)
- Monitor fidelity using developed tools that identify key components
Elements of Implementation Quality
- Quality of Delivery
- Program staff are trained by developer
- Staff have characteristics that target population can relate to,
- Staff share positive and enthusiastic attitudes
- Facilities are appropriate for group size, safe, and welcoming
- Materials and supplies are prepared in advance, sufficient for
number of participants, and are in good shape and of good quality
- Efforts to keep participants engaged in the program (importance of
quality, trained and staff experience working with the target population)
Elements of Implementation Quality
- Program Reach (target audience)
- Delivered to intended audience
- Age range
- Youth, Parent, or Family
- Individual versus Group
- Sufficient number of participants to run groups effectively
Elements of Implementation Quality
Ways to Measure Implementation Quality
- Fidelity Observation
- Implementer Self-Report
- Administrative Documents
- Participant Self-Reports
Why IQ or Fidelity Monitoring?
- Measure IQ à Unlock black box
- Ex: What is happening in the program?
- Measure IQ à Better IQ
- “What gets assessed gets addressed!”
- Better IQ à Better Outcomes
- Show funding agencies what you’ve done
- We’re here to support the best possible implementation in
your communities.
- Contact your Prevention Coordinator for expert assistance
- Consult our website for additional resources
- www.episcenter.psu.edu
Need implementation Help?
The EPISCenter represents a collaborative partnership between the Pennsylvania Commission on Crime and Delinquency (PCCD), and the Prevention Research Center, College of Health and Human Development, Penn State University. The EPISCenter is funded by PCCD and the Department of Public Welfare. This resource was developed by the EPISCenter through PCCD grant VP-ST-24368.
Data Collection and Reporting
Lee Ann Cook
Process Data
- How many?
- How often?
- How well?
Outcomes Data
- Did you achieve your intended impact?
- Baseline Pre Test or Survey
- Follow-up Post Test or Survey
- Success Stories
Data Analysis & Reporting
- PCCD Excel Spreadsheet Data Analysis & Reporting
Tools
- Generally each is designed to hold one year of data
- Update and submit to PCCD via egrants on a quarterly basis
- Provides summary data on each performance measure so that
providers can make adjustments to implementation as necessary
Instructions
Process Tab
Outcomes Tab
Raw Data Entry-Pre/Post
Raw Data Entry-Fidelity
Why SOOO much Data Collection?
Strengthening Families 10-14 2012-2013 Reach
- Active Grants
- Facilitators Trained
- Programs Completed
- 478 Adults Enrolled
- 437 Youth Enrolled
- 74% of Participants Completed the Program
- 77% of Participants Completed Pre/Post Measures
Life Skills Training Fidelity 2012-2013
PA LST Model Fidelity Model Fidelity
The goal is to observe 20 percent of the lessons taught to assess model
- fidelity. For LST “Minimum
Fidelity” is defined as having taught at least 75 percent of the lesson as designed by the developer.
84% ¡ 16% ¡
Lessons ¡Observed: ¡103 ¡
Mee'ng ¡Min. ¡ fidelity ¡ Needing ¡ improvement ¡
Strengthening Families 10-14 2012-2013 Impacts
54% 54% 63% 45% 56%
0% 10% 20% 30% 40% 50% 60% 70%
Parents - Increased Substance Use Rules/ Expectations Parents - Improved Standard Setting Parents - Improved Family Problem Solving Youth - Improved Parent-Child Relationship Quality Youth - Improved Peer Pressure Refusal Skills
NUMBER OF PARTICIPANTS REPORTING POSITIVE CHANGE 349 parents/caregivers and 353 youth with completed pre/post tests 189 188 219 158 198
Big Brothers Big Sisters Impacts 2010-2013
35% 43% 11% 26% 42%
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%
Family Relationship Peer Relationships Avoid ATOD Avoid Delinquency Academic Performance
Littles' Improvements
722 Annual Outcome Evaluations (POE and YOS)
Data Reporting
- Its never to early to start sharing data
- Internally
- Collaborative Board Reporting
- School Board
- How?
- Community Reporting Templates
- Outcome Report Templates
- Comprehensive Outcome Report Required at the
end of Year Two.
Project Towards No Drug Abuse Community Report Template
Future Improvements: Logic Model Tabs
Future Improvements: More Descriptive Analysis
The EPISCenter represents a collaborative partnership between the Pennsylvania Commission on Crime and Delinquency (PCCD), and the Prevention Research Center, College of Health and Human Development, Penn State University. The EPISCenter is funded by PCCD and the Department of Public Welfare. This resource was developed by the EPISCenter through PCCD grant VP-ST-24368.
Kris Glunt
The 5 W’s of Sustainability
Defining Sustainability
- “…the continuation of community health or quality of life
benefits over time.”
- A Blueprint for Sustainabiliy From Surviving to Thriving adapted from
Sanislaus County Children and Families
WHEN should you begin planning?
- From the beginning:
- Consider efficiency from the start
- Have data collection procedures in place
- Utilize the resources provided through EPIS
- Schedule regular meetings to discuss
YESTERDAY!!
WHO should be involved in this process?
- Administrators
- Staff
- Community Support
- Identify Key Stakeholders
- Strategic partnerships with similar interests/goals
- Identify Key Champions
- Serve as advocates for your program/agency
- People within and outside of your organization
WHAT tools can be used for this process?
- Plan for Sustainability (A Blueprint for Sustainability From
Surviving to Thriving)
- Sustainability: The Result of Building Relationships
- Sustainability Planning Worksheet
- Community/Outcomes Reports
- Quality Assurance Letter Verification
- http://episcenter.psu.edu/PSIBGF
Sustainability Tools/Resources:
Tools/Resources Continued:
WHERE to turn for resources/funding?
- In-kind Resources
- Foundations
- Government
- Corporations
- Businesses
- United Way
- Community-Service Organizations
- Faith-based Organizations
- Individuals
- Schools/PTOs
WHERE to turn…be strategic:
- List selling points:
- State the benefits to the participants
- State the benefits to the community
- Tie your product/program to the funder’s objectives and
terminology
- Highlight how your outcomes match the potential funders’ goals
WHY start planning so soon?
- Planning for sustainability is a challenging process
- 2 years will go fast
- It’s important to start preparing from the beginning
- Continue to make a positive impact in your community
- Continue to allow good people to do good work
HOW should we begin this process?
- Consider efficiency from the start
- Use the “Plan for Sustainability” as a guide to get started
- Schedule regular meetings to develop a plan and make
changes as necessary
- Have data collection procedures in place from the start
- Discuss with your EPIS PC
- Maintain strong ties with your Collaborative Board
- Visit http://episcenter.psu.edu/PSIBGF for more resources
For more information:
- Planning for Sustainability: Impact Beyond Grant Funding
- http://episcenter.psu.edu/PSIBGF