Logic Models: A framework for program planning and evaluation - - PowerPoint PPT Presentation

▶
logic models a framework for program planning and
SMART_READER_LITE
LIVE PREVIEW

Logic Models: A framework for program planning and evaluation - - PowerPoint PPT Presentation

Sep 28, 2023 151 likes •498 views

Logic Models: A framework for program planning and evaluation Ellen Taylor-Powell, Ph.D. Evaluation Specialist University of Wisconsin-Extension-Cooperative Extension Nutrition, Food Safety and Health Conference Baltimore, Maryland March 31,

slide-1
SLIDE 1

Logic Models: A framework for program planning and evaluation

Ellen Taylor-Powell, Ph.D. Evaluation Specialist University of Wisconsin-Extension-Cooperative Extension Nutrition, Food Safety and Health Conference Baltimore, Maryland March 31, 2005

1 University of Wisconsin-Extension, Program Development and Evaluation

slide-2
SLIDE 2

What we’ll cover

  • Logic model role in accountability
  • Value to you
  • Examples
  • Things to watch out for
  • Q and A

2 University of Wisconsin-Extension, Program Development and Evaluation

slide-3
SLIDE 3

A bit of history Dates to late 1960’s Dates to late 1960’s Current accountability demands Current accountability demands

  • Public Sector -

Public Sector - GPRA PRA

  • Non-Profit Sector

Non-Profit Sector

  • Private Sector

Private Sector

  • International Agencies

International Agencies

  • Evaluation

Evaluation

3 University of Wisconsin-Extension, Program Development and Evaluation

slide-4
SLIDE 4
  • What gets measured gets done
  • If you don’t measure results, you can’t tell success

from failure

  • If you can’t see success, you can’t reward it
  • If you can’t reward success, you’re probably

rewarding failure

  • If you can’t see success, you can’t learn from it
  • If you can’t recognize failure, you can’t correct it.
  • If you can demonstrate results, you can win public

support.

Osborne and Gaebler, 1992

Accountability era

4 University of Wisconsin-Extension, Program Development and Evaluation

slide-5
SLIDE 5

Results is the name of the game! “The important question any constituent needs to ask is whether or not the program achieves results.”

President Bush, NPR news, 2-7, 05

5 University of Wisconsin-Extension, Program Development and Evaluation

slide-6
SLIDE 6

Generic logic model

Results Strategy

Inputs Outputs Outcomes Outcomes Outcomes Activities Participation

» » » » »

Inputs Outputs Outcomes Outcomes Outcomes Activities

» » » »

C O N T E X T C O N T E X T

A diagram of the theory of how a program is supposed to work A graphic depiction of relationships between activities and results

6 University of Wisconsin-Extension, Program Development and Evaluation

slide-7
SLIDE 7

“I think you should be more explicit here in Step Two.”

7 University of Wisconsin-Extension, Program Development and Evaluation

slide-8
SLIDE 8

8 University of Wisconsin-Extension, Program Development and Evaluation

slide-9
SLIDE 9

9 University of Wisconsin-Extension, Program Development and Evaluation

Staff Money Partners Develop parent ed curriculum Deliver series of interactive sessions

Parents increase knowledge of child dev Parents better understanding their own parenting style Parents use effective parenting practices Improved child- parent relations

Research

INPUTS OUTPUTS OUTCOMES

Facilitate support groups

Parents gain skills in effective parenting practices

Simple logic model

Parents identify appropriate actions to take Strong families

Targeted parents attend

SITUATION: During a county needs assessment, majority of parents reported that they were having difficulty parenting and felt stressed as a result

slide-10
SLIDE 10

Logic model of a training workshop

Situation: Funder requires grantees to include a logic model in their funding request; grantees have limited understanding of logic models and are unable to fulfill the funding requirement

Trainer Funds Equipment Research base Training curriculum Increase knowledge of logic models Increase ability to create a useful logic model of program Increase confidence in using logic models Improved planning Improved evaluation 3 hour training

  • Interactive

activities

  • Group work
  • Practice
  • Q and A

Create meaningful logic models Use logic models in own work

OUTCOMES INPUTS OUTPUTS

10 University of Wisconsin-Extension, Program Development and Evaluation

Grantees Accountable here Fulfill requirement

  • f funder
slide-11
SLIDE 11

Programs are not linear!

INPUTS OUTPUTS OUTCOMES

Program investments Activities Participation Short Medium What we invest What we do Who we reach What results Long- term

11 University of Wisconsin-Extension, Program Development and Evaluation

slide-12
SLIDE 12

Language: What do you mean by…

  • Goal = Impact
  • Impact = Long-term outcome
  • Objectives (participant focused) =

Outcomes

  • Activities = Outputs

–Outputs may signify “tangible” accomplishments as a result of activities

12 University of Wisconsin-Extension, Program Development and Evaluation

slide-13
SLIDE 13

What does a logic model look like?

  • Graphic display of boxes and

arrows; vertical or horizontal

–Relationships, linkages

  • Any shape possible

–Circular, dynamic –Cultural adaptations;

storyboards

  • Level of detail

–Simple –Complex

  • Multiple models

13 University of Wisconsin-Extension, Program Development and Evaluation

slide-14
SLIDE 14

So, why bother? What’s in this for you? “This seems like a lot of work.” “Where in the world would I get all the information to put in a logic model? “I’m a right brain type of person – this isn’t for me.” “Even if we created one, what would we do with it?”

14 University of Wisconsin-Extension, Program Development and Evaluation

slide-15
SLIDE 15

What we are finding:

  • Provides a common language
  • Helps us differentiate between “what we do”

and “results” --- outcomes

  • utcomes
  • Increases understanding about program
  • Guides and helps focus work
  • Leads to improved planning and management
  • Increases intentionality and purpose
  • Provides coherence across complex tasks,

diverse environments

15 University of Wisconsin-Extension, Program Development and Evaluation

slide-16
SLIDE 16
  • Enhances team work
  • Guides prioritization and allocation of

resources

  • Motivates staff
  • Helps to identify important variables to

measure; use evaluation resources wisely

  • Increases resources, opportunities,

recognition

  • Supports replication
  • Often is required!

16 University of Wisconsin-Extension, Program Development and Evaluation

slide-17
SLIDE 17

Testimonials

“Wow – so that is what my program is all about” “I’ve never seen our program on one page before” “I’m now able to say no to things; if it doesn’t fit within our logic model, I can say no. “ “I can do this” “This took time and effort but it was worth it; our team never would have gotten here otherwise.” “It helped us to think as a team – to build a team program vs. an individual program.”

17 University of Wisconsin-Extension, Program Development and Evaluation

slide-18
SLIDE 18

18 University of Wisconsin-Extension, Program Development and Evaluation

slide-19
SLIDE 19

19 University of Wisconsin-Extension, Program Development and Evaluation

Research inputs Res’rchers Accurate research available and shared Extension inputs Producer inputs Policy inputs Policy is followed Reductions in ammonia emissions Producers

Multi agency partnership: Abating ammonia emissions from dairy farms

Adopt BMPs Conduct research Disseminate & educate Develop & set standards Test & feedback INPUTS OUTPUTS OUTCOMES

Powell et al, 2005

slide-20
SLIDE 20

Multi agency partnership: Research sub-logic model

20 University of Wisconsin-Extension, Program Development and Evaluation

Scientists

Accurate research available

Staff Funding Equipm’t

Research widely communi- cated

  • Publication
  • Popular

press

  • Pres’tations
  • Reports

Increased ability to determine ammonia emissions at different scales Increased understanding of relationship between measurement and actual emissions Increased skills in non-traditional science Increased knowledge of sources, processes

  • f ammonia

emissions Conduct process & operational level experim’ts Interpret, validate results Scale up/out results Incorporate farmer feedback Generate funding Educate re. complexities, components,

  • pportunities

Reductions in ammonia emissions Res’rchers Partners

Existing knwl’dge

Powell et al, 2005

slide-21
SLIDE 21

Benefits of logic modeling to this partnership

  • Provided framework to discuss and articulate joint

work

  • Helped facilitate conversation with focus on agreed

upon goal that might not happened otherwise

  • Shows contribution of each partner and how result

depends upon all

  • Keeps end outcome upfront and center
  • Provides way to communicate about the partnership

that has been presented at national conference

  • Separates indicators of achievement - # papers

published, # and type of experiments completed – from theory of change. Indicators of achievement are part of evaluation plan for the partnership.

21 University of Wisconsin-Extension, Program Development and Evaluation

slide-22
SLIDE 22

Tobacco Control: Global View

Community programs Chronic disease pr’grms School programs

22 University of Wisconsin-Extension, Program Development and Evaluation

Research: evidence- base Policy Advocates Practitioners

Policy change

Policy makers Current and potential users Disparate populations Key stakeh’ders Publics

System change

Funders

Change in access Change in support Change in knowledge, attitudes, skills, motivation

Enforcement Evaluation and Surveillance Counter- marketing Administration & management

Individual change

Partners Decreased

smoking

Statewide programs

Reduced exposure to ETS

Cessation programs Reduce mortality, morbidity

slide-23
SLIDE 23

Tobacco Control: Statewide View - Community Program

Research: evidence- base Policy Advocates

Practitioner 23 University of Wisconsin-Extension, Program Development and Evaluation

Demon- strations of support

Reduce mortality, morbidity

  • Policy

makers

  • Current

and potential users

  • Disparate

populations

  • Key

stakeh’ders

Policies

impl’mented

enforced Change in K,A,S,M

Funders

Change in access Change in support Promote smoke-free policy change Prevent youth initiation, reduce use Treat tobacco addiction Individual change

Partners

Coalition development

  • Coalition

members

  • Key

stakeh’ders

Effective coalition functioning Change in

  • KAS,
  • Self-efficacy,
  • Intent

Success- ful TC imple- mentation

Decreased

smoking Reduced exposure to ETS

slide-24
SLIDE 24

Tobacco Control: Local view - smoke-free environments

OUTCOMES

24 University of Wisconsin-Extension, Program Development and Evaluation Mgrs of public areas/events Organize and implement SF campaign Public Demonstrations of support SF:

  • Municipal

buildings, grounds, & vehicles

  • Public

areas & events

  • Worksites
  • Residence

Change in intent to make services/support available Increased knowledge and skills to participate in SF public policy change SF public policies implemented SF public policies adhered to and enforced SF policies drafted, improved Elected officials Worksite contacts Increased commitment, support, demand for SF environments Increased availability of cessation support and services Increased awareness of importance of SF public policies Residential

  • wners, mgrs

Increased knowledge of SF benefits and options Form committee Develop grassroots support Educate community Organize earned media Identify and work with supportive policy makers Community activists Media Coalition Time Money Partners including youth Research and best practices

slide-25
SLIDE 25

Programs as “systems”

Org 1

Multi-Org partnership Single

  • rganization

Community

Org 2 Org 3 Org 4

OUTPUTS SHORT-TERM OUTCOMES MED-TERM OUTCOMES LONG-TERM OUTCOMES

Source: Adapted from CDC: http://www.phppo.cdc.gov/phtn/DLSummit2004/1

INPUTS

25 University of Wisconsin-Extension, Program Development and Evaluation

slide-26
SLIDE 26

Logic Model and Planning Logic Model and Planning

  • Applies at any level: national plan, statewide

plan, individual plan of work, specific project/activity plan

  • Model vs. more detailed program

plan/management plan

  • Focus on outcomes: “start with end in mind”

26 University of Wisconsin-Extension, Program Development and Evaluation

slide-27
SLIDE 27

Logic model and evaluation

27 University of Wisconsin-Extension, Program Development and Evaluation

Outcome evaluation: To what extent are desired changes occurring? For whom? Is the program making a difference? What seems to work? Not work? What are unintended

  • utcomes?

Process evaluation: How is program implemented? Fidelity of implementation? Are activities delivered as intended? Are participants being reached as intended? What are participant reactions? Needs/asset assessment: What are the characteristics, needs, priorities of target population? What are potential barriers/facilitators? What is most appropriate?

slide-28
SLIDE 28

University of Wisconsin-Extension, Program Development and Evaluation

Staff Money Partners Research Parents gain skills in effective parenting practices Develop parent ed curriculum Deliver series of interactives essions Parents increase knowledge of child dev Parents better understand their own parenting style Parents use effective parenting practices Improved child- parent relations Facilitate support groups Parents identify appropriate actions to take Strong families Targeted parents attend

EVALUATION: What do you (and others) want to know about this program?

To what extent are relations improved? Does this result in stronger families? To what extent did behaviors change? For whom? Why? What else happened? To what extent did knowledge and skills increase? For whom? Why? What else happened? Who/how many attended/did not attend? Did they attend all sessions? Supports groups? Were they satisfied – will they come again? How many sessions were actually delivered? How effectively? # and quality of support groups? What amount

  • f $ and time

were invested?

slide-29
SLIDE 29

Data collection plan

  • 1. Focus:
  • 1. Focus:
  • 5. D
  • 5. Data collection

ta collection

  • 2. Questi

Question

  • ns
  • 3. I
  • 3. Indic

dicators

  • 4. Timin
  • 4. Timing

Sources Sources Met Methods

  • ds

Sampl Sample Ins Instrumen ruments

29 University of Wisconsin-Extension, Program Development and Evaluation

slide-30
SLIDE 30

30 University of Wisconsin-Extension, Program Development and Evaluation

Components of a Success Story

Logic model and reporting

slide-31
SLIDE 31

What logic model is not…

  • A theory
  • Reality
  • An evaluation model or method

It is a framework for describing the relationships between investments, activities and results. It provides a common approach for integrating planning, implementation, evaluation and reporting.

31 University of Wisconsin-Extension, Program Development and Evaluation

slide-32
SLIDE 32

Cautions:

  • Time consuming - paperwork
  • Too much focus on outcomes
  • Too little focus on testing the theory
  • Perfecting the key to the wrong lock
  • Attending to context only at front end
  • Universal vs. context specific mechanisms
  • Viewing logic model as reality

– “Pyrennes not the Alps”

32 University of Wisconsin-Extension, Program Development and Evaluation

slide-33
SLIDE 33

Summing up

  • Demonstrates accountability with focus on
  • utcomes
  • Links activities to results: Prevents

mismatches

  • Integrates planning, implementation,

evaluation and reporting

  • Creates understanding
  • Promotes learning

A way of thinking – not just a pretty graphic

33 University of Wisconsin-Extension, Program Development and Evaluation

slide-34
SLIDE 34

References

“We build the road and the road builds us.”

  • Sri Lankan saying
  • www.uwex.edu/ces/lmcourse
  • www.uwex.edu/ces/pdande
  • http://www.cdc.gov/eval/index.htm
  • http://ctb.ku.edu/
  • http://www.innonet.org/
  • http://www.eval.org/

34 University of Wisconsin-Extension, Program Development and Evaluation