ROBERT E. HOKE INDEPENDENT EVALUATION CONSULTANT GOALS FOR THIS - - PowerPoint PPT Presentation

EVALUATION MODELS AND METRICS PART II: UTILIZING YOUR GRANTEES’ REPORTS TO MEASURE IMPACT

ROBERT E. HOKE INDEPENDENT EVALUATION CONSULTANT

GOALS FOR THIS SESSION

 Build on the lessons from part one  Apply tools of the evaluator in analyzing grantee reports  Introduce to a framework of population and performance metrics  Share successes and challenges of measuring and reporting impact

2

My question is: are we making an impact?

3

EXERCISE

 Think of a previous report with aggregated grantee performance/impact information that you thought was particularly effective.  What were the characteristics of that report that made it effective?  Share with your colleagues at your table the background of the report and what made it effective.  Any common characteristics? 4

SO YOU HAVE A STACK OF EVALUATIONS FROM YOUR GRANTEES: NOW WHAT?

5

UTILIZATION FOCUSED EVALUATION (UTE)

INTENDED USES FOR INTENDED USERS

6

POTENTIAL USES OF EVALUATION REPORTS FROM GRANTEES

 Accountability: Oversight and Compliance  Monitoring: Progress on Process Benchmarks  Effectiveness: Desired Outputs and Results  Program Improvement: Implementation Changes  Knowledge Generation: General Understanding of Problem  Impact: Individual and Collective Impact of Grantees

7

POTENTIAL USERS OF EVALUATION REPORTS FROM GRANTEES

 Program Officer or Other Foundation Staff  Foundation Volunteer Leadership  Donors  Community Partners  Grantees

8

WHAT EVERYONE IS TALKING ABOUT

Collective Impact

• a very specific

model of

• rganizing

community-wide efforts based on Cincinnati's Strive Initiative (e.g., backbone

• rganization)

collective impact

• a more

generalized concept of joint decision making about community goals, outcomes, and priorities 9

KEY EVALUATION QUESTIONS

RESULTS BASED EVALUATION FRAMEWORK

• What did you do?
• How well did you

do it?

• Is anyone better
• ff?

MICHAEL QUINN PATTON

• What?
• So what?
• Now what?

10

COMBINED PROGRAM LOGIC MODEL

11

EVALUATING PORTFOLIOS: ITS NOT JUST COMPARING APPLES AND ORANGES

12

WHAT DO WE CHOOSE TO REPORT?

13

What we tend to report. What we aspire to report.

RESULTS BASED ACCOUNTABILITY (RBA) FRAMEWORK

Mark Friedman: Fiscal Policy Study Institute Results-Based Accountability™ (RBA), also known as Outcomes- Based Accountability™ (OBA), is a disciplined way of thinking and taking action that communities can use to improve the lives of children, youth, families, adults and the community as a whole.  Performance Accountability  Population Accountability

14

PERFORMANCE ACCOUNTABILITY

Performance level accountability for the performance of programs, agencies service systems We identify measures / indicators that tell us the service / program is performing well In RBA, there are three different types of performance measures: (1) How much service was provided? (2) What was the quality of service? (3) Are the clients better off because of the service?

15

POPULATION ACCOUNTABILITY

Population level accountability for the well-being of an entire population in a geographic area. Population level accountability concerns conditions we hope to

• attain. Example: “Children in our community are healthy.”

We identify measures / indicators that tell us the outcome is achieved for the population. Measures related to population accountability are solely focused

• n whether the populations are better off because of the services

provided.

16

RBA: FROM THE ENDS TO THE MEANS

 What are the quality of life conditions we want for the children, adults and families who live in our community?  What would these conditions look like if we could experience them?  How can we measure these conditions?  How are we doing on the most important measures?  Who are the partners that have a role to play in doing better?  What works, including low-cost and no-cost ideas?  What do we propose to do?

17

APPLY RBA TO IMPROVING HIGH SCHOOL GRADUATION RATES

PERFORMANCE

Examining our portfolio of programs

• How much service

did we provide?

• How good was

that service?

• Are the clients

better off?

POPULATION

Examining the population in the community

• Was there a

positive change in the high school drop-out rate as a result of our programs? 18

WHAT IS THE OVERLAP BETWEEN POPULATION AT RISK AND THE CLIENTS OF YOUR PROGRAMS?

Clients of Programs Population at Risk

19

WHAT IS THE OVERLAP BETWEEN POPULATION AT RISK AND THE CLIENTS OF YOUR PROGRAMS?

Clients of Programs Population at Risk

The larger the overlap the greater the connection between performance and population measures

20

PERFORMANCE ACCOUNTABILITY: BENEFITS AND CHALLENGES

BENEFITS

• Timely Data
• Causal link fairly

strong

• Shows

accountability to donors

• Helpful for

program improvement

CHALLENGES

• No information on

impact on target population

• Blind adherence to

program plans

• Program

participants may not be in target population 21

POPULATION ACCOUNTABILITY: BENEFITS AND CHALLENGES

BENEFITS

• More flexibility
• Stronger focus on

at-risk populations

• Community-wide

collaborations and buy-in

• Speaks to donors’

interest in results

CHALLENGES

• Data may not be

timely

• More difficult to

show causal link

• Increased

dependence on

• utside partners

22

PERFORMANCE MEASUREMENT IS ABOUT PROVING WE DID THINGS IN THE RIGHT WAY: “WE FUND GOOD PROGRAMS” POPULATION MEASUREMENT IS ABOUT PROVING WE DID THE RIGHT THINGS: “OUR FUNDING HAD POSITIVE IMPACT”

23

TRIANGULATION

Triangulation in evaluation:

• multiple methods
• quantitative (data and statistics)
• qualitative methods (interviews, focus groups)
• various stakeholders

24

Using both performance and population measures: “WE FUND GOOD PROGRAMS THAT HAVE POSITIVE IMPACT ON OUR COMMUNITY”

LOGIC MODELS AND RBA

25

26

EXERCISE: APPLY RBA TO HIGH SCHOOL GRADUATION RATES

RESOURCES AND GOOD EXAMPLES

27

BACK TO THE EVALUATORS’ TOOLKIT

Apply the lessons from the first session on evaluating grantee measurement and methods to aggregation of their results:

• Are your methods rigorous and appropriate?
• Are you applying a one-size fits all approach or

focusing only on reporting results from programs that had rigorous evaluations?

• Are the evaluation findings consistent with your

theory of change or program logic? 28

EVALUATION TERMS YOU SHOULD KNOW

• Meta-Evaluation
• Evaluating Strategy

29

CONVERSATIONS WITH YOUR GRANTEES

• Are you sharing the information you learned from your

analysis of the grant reports?

• Are you giving the grantee feedback on how to improve

their performance reports?

• Are you engaging the grantees in post-grant review of how

to improve the process?

• Have you considered having a focus group of grantees to

discuss how to change the grant program to have a greater impact on the community change desired?

• Have you considered allowing the grantees to work

together to recommend a standard set of reporting measures?

30

UNANSWERED QUESTIONS?

31

CONTACT INFORMATION

Robert E. Hoke Independent Consultant 5301 E. 9th Street Indianapolis, Indiana 46219 317-356-6367 robert@roberthoke.com

32