Practical Program Evaluation: Using CDC’s Evaluation Framework By: Thomas J. Chapel, MA, MBA Chief Evaluation Officer CDC/OADPG Tchapel@cdc.gov 404-639-2116
“... If you blindfold someone, put them in a large field, and tell them to walk forward in a straight line. Within minutes and without fail… 2
“... they will walk in a circle. Robert Krulwich, NPR, All Things Considered, 11/23/10 3
Objectives State CDC Evaluation Framework steps and standards Describe the added value of strong program description Describe how to make strong decisions on evaluation focus 4
Defining Evaluation Evaluation is the systematic investigation of the merit, worth, or significance of any “object” Michael Scriven Program is any organized public health action/activity implemented to achieve some result 5
Integrating Processes to Achieve Continuous Quality Improvement What do Continuous Quality we do? Improvement (CQI) cycle Planning — What actions Why are we will best reach our goals How do we doing and objectives. do it? well or Performance poorly? measurement — How are we doing? How are we Evaluation — W hy are we doing? doing well or poorly? 6
“ Research seeks to prove , evaluation seeks to improve …” M.Q. Patton
Enter the CDC “Evaluation” Framework 8 8
Enter the CDC Good M&E = use of findings Evaluation Framework 9
Enter the CDC Good M&E= use of findings Evaluation Framework Focus is situation -specific 10
Enter the CDC Good M&E = use of findings Evaluation Framework Early steps key to best focus Focus is situation -specific 11
Step 2: Describing the Program: Complete Logic Model Short- Intermediate Long-term term Effects/ Effects/ Inputs Activities Outputs Effects/ Outcomes Outcomes Outcomes Context Assumptions 12
What the program and its staff actually do Short- Intermediate Long-term term Effects/ Effects/ Inputs Activities Outputs Effects/ Outcomes Outcomes Outcomes Context Assumptions 13
Results of activities: Who/what will change? Short- Intermediate Long-term term Effects/ Effects/ Inputs Activities Outputs Effects/ Outcomes Outcomes Outcomes Context Assumptions 14
Finding Activities and Outcomes Plan Actions/ Objectives Goals Tactics LT Eval ST or MT Outcomes Activities Outcomes or Impacts Outcome Measures Process Measures PM Impact Measures Progress Measures Key Performance Indicators Impl. Measures Success Factors 15
Core Funding — Activities and Outcomes Activities Outcomes • • Hire staff and establish org locus People with disabilities are • Develop advisory committee integrated into promotion and • Analyze existing data prevention activities • Evaluate and monitor activities • Plans [of others?] address • Develop strategic planning process needs of people with • Develop strategy disabilities • Develop access strategies • • [Others?] implement access Active collaboration and cooperation of partners and other strategies • [Who?] implements effective health promo • Increased access to promotion and wellness programs and prevention activities • Strong policy [function?] • Prevent secondary conditions • Strong planning function and strategic plan • Improve health and quality of • Strong disability report and dissemination of life of people with disabilities report 16 16
Core Funding — Logic Model Table Activities Outcomes Hire staff and establish Active collaboration People with Prevent secondary org locus and cooperation of disabilities are conditions partners integrated into Develop advisory promotion and Improve health and committee [Who?] implements prevention activities quality of life of people effective health promo with disabilities Analyze existing data and wellness Plans [of others?] programs address needs of Evaluate and monitor people with disabilities activities Strong policy [function?] [Others?] implement Develop strategic access and other planning process Strong planning strategies function and strategic Develop strategy plan Increased access to promotion and Develop access Strong disability report prevention activities strategies and dissemination of report 17 17
Logic Model Table: Module A: Evidence- Based Interventions Activities Outcomes Consult with Facilitate policy Increased social and Improve health and committee [change?] physical accessibility quality of life of PWDs of environments Facilitate physical (home, school, work) access 18 18
Logic Model Table: Module E: Training Activities Outcomes Use/modify existing Offer training Change KAB of curricula (para)professionals in care and health Collaborate with promotion with people with disabilities partners Consult with faculty and staff of PH training programs 19 19
Practical Program Evaluation Setting an Evaluation Focus
Evaluation Focus — Evaluation Can Be About Anything Evaluation can focus on any/all parts of the logic model Evaluation questions can pertain to Boxes---did this component occur as expected Arrows---what was the relationship between components
Phases and Types of Evaluation Before (More) Program Mature Program New Program Established Stage Program Begins Program Phase FORMATIVE SUMMATIVE Evaluation Needs Process Outcome Impact Type Assessment Evaluation Evaluation Evaluation Is the program Is the program To what extent is (Some) achieving it’s Is the program achieving its the need being met? Questions operating as short-term long-term What can be done to planned? outcomes/ outcomes and Asked address this need? objectives? impacts?? Source: Based on slides from Jennifer Nichols, Porter Novelli
Enter the CDC “Evaluation” Framework 23 23
Setting Focus: Some Rules Based on “utility” standard: Purpose: Toward what end is the evaluation being conducted? User: Who wants the info and what are they interested in? Use: How will they use the info? 24
(Some) Potential Purposes Show accountability Test program implementation “Continuous” program improvement Increase the knowledge base Other… Other… 25
(Some) Potential Purposes Show accountability Test program implementation “Continuous” program improvement Increase the knowledge base Other… Other… 26
Practical Program Evaluation Elaborating Our Program “Theory”— Mediators
Filling in the Blanks…. 28 28
Where “mediators” live in our logic model… Short- Intermediate Long-term term Effects/ Effects/ Inputs Activities Outputs Effects/ Outcomes Outcomes Outcomes Context Assumptions 29
Prevention Program — Simple Logic Model Capacity Building Change Physical Environments Surveillance Communication Prevent and Control Problem Research and Partnership Development Change Social Environments Leadership 30
Prevention Program — Elaborated Logic Model ACTIVITIES OUTCOMES Identify key SURVEILLANCE factors and populations Adopt changes in Propose policy policies, laws and changes Identify modifiable regulations RESEARCH & risk and protective factors and DEVELOPMENT consequences. Change physical Develop/test environment interventions. Create/identify best method and models. Support/develop CAPACITY Change frontline Diffuse supply of Prevent and Adopt practices BUILDING established/ infrastructure. tools, practices and control and programs takes root. Identify skills and programs problem needs. Identify channels, COMMUNICATION Change knowledge, audiences, and key Generate attitudes and beliefs. demand for tools. behavior. Change social environment Identify strategic PARTNERSHIP partners. Access to leaders. Strong Access to partnerships LEADERSHIP Forum for convening. key groups. at all levels. Develop research and other agendas. Increased resources 8/2/2011
“Reality Checking” the Focus Based on “feasibility” standard: Stage of Development: How long has the program been in existence? Program Intensity: How intense is the program? How much impact is reasonable to expect? Resources: How much time, money, expertise are available? 32
Core Funding — Logic Model Table Activities Outcomes Hire staff and establish Active collaboration People with Prevent secondary org locus and cooperation of disabilities are conditions partners integrated into Develop advisory promotion and Improve health and committee [Who?] implements prevention activities quality of life of people effective health promo with disabilities Analyze existing data and wellness Plans [of others?] Mediators Mediators programs address needs of Evaluate and monitor people with disabilities activities Strong policy [function?] [Others?] implement Develop strategic access and other planning process Strong planning strategies function and strategic Develop strategy plan Increased access to promotion and Develop access Strong disability report prevention activities strategies and dissemination of report 33 33
Recommend
More recommend
