Presentation Summary Participatory Project Review (PPR) 1. PPR - - PDF document

2/1/2010 1

Participatory Project Review (PPR)

An Overview

January 2010 Scott Chaplow e Senior M& E Officer scott.chaplow e@ ifrc.org

1

Presentation Summary

• 1. PPR overview
• 2. PPR steps
• 3. PPR limitations
• 4. PPR Benefits
• 5. Group discussion.

2 Scott Chaplowe, IFRC Performance & Accountability Department

What is a PPR?

• A Participatory Project Review (PPR) is a participatory

approach to evaluation that can be adapted to different contexts and interventions according to need.

• Draws upon a variety of methodologies that can be tailored to

specific project needs, interests, timeframes, resources, and capacity.

• The PPR is more than an evaluation/review exercise – it is a

project activity itself, involving participants in a meaningful way towards project success!

3 Scott Chaplowe, IFRC Performance & Accountability Department

Participatory Appeal

“Learning should always be the main focus of any type of

• evaluation. Where the traditional kind of external evaluation may

produce good quality evaluations results and recommendations, quite often these remain with the senior management of the C l h l i i di i d l l

• program. Consequently, the learning is not disseminated let alone

incorporated among important stakeholders, such as project staff, representatives of the communities and relevant government authorities or other NGOs in the region. The participatory type of evaluation involving multiple stakeholders is therefore emerging rapidly.”

(Susanne van Lieshout and Natasha Cassiere, 2007. “Maximize Learning from Participatory Evaluation with Peer Review,” MDF South Asia.)

4 Scott Chaplowe, IFRC Performance & Accountability Department

2/1/2010 2

PPR = Combination of Methods

Empowerment Evaluation (David Fetterman).

“Program participants – including clients – conduct their own evaluations; an

• utside evaluator often serves as a coach or additional facilitator depending on

internal program capabilities…The group thus can serve as a check on its own members, moderating the various biases and agendas of individual participants.”

The Most Significant Change technique (Rick Davies and Jessica Dart).

“(T)he collection of significant change (SC) stories emanating from the field level, and the systematic selection of the most significant of these stories by panels of designated stakeholders or staff. It is a form of monitoring because it

• ccurs throughout the program cycle and provides information to help people

manage the program. It contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole.

5 Scott Chaplowe, IFRC Performance & Accountability Department

Other Participatory Approaches

• Vulnerability & Capacity Assessment (VCA)

“VCA is a participatory investigative process designed to assess the risks that people face in their locality, their vulnerability to those risks, and the capacities they possess to cope with a hazard and recover from it.” (IFRC)

• Participatory Rapid (Rural)Appraisal (PPR)

“Participatory Rural Appraisal (PRA) comprises a set of techniques aimed at shared learning between local people and outsiders” (World Bank)

• Participatory Impact Assessment (PIA)

“(Participatory Impact Assessment (PIA) is an extension of Participatory Rural Appraisal (PRA) and involves the adaptation of participatory tools combined with more conventional statistical approaches specifically to measure the impact of humanitarian assistance and development projects on people’s lives. (Feinstein International Center, Tufts University).

6 Scott Chaplowe, IFRC Performance & Accountability Department

Prior PPR Experience

Sri Lanka: Canadian Red Cross community health program, 2007. Sri Lanka: American Red Cross ceramic water filter project in water/sanitation program, 2008. Maldives: American Red Cross PSP program, 2008. Peru: International Federation Red Cross/Crescent (IFRC) response to the Peru Earthquake, 2009.

7 Scott Chaplowe, IFRC Performance & Accountability Department

How is a PPR flexible?

• Not a blueprint for assessment, but should be tailored to

specific interests, timeframes, resources, and capacity.

• Can be used for multiple purposes: initial assessment, review,

evaluation, etc. ,

• Can be conducted at different times: beginning, middle, or end
• f an intervention (project/program).
• Can be used alone, or in combination with other methods.
• Can be internal, or facilitated with external assistance and

participation.

• Can be done once (one-off), or multiple times.

8 Scott Chaplowe, IFRC Performance & Accountability Department

2/1/2010 3

How does a PPR evaluation w ork?

1 PPR: 1 day (6 hours + 2 hours for lunch, tea break, etc.) 10 – 20 people M l i l PPR d i h ddi i l h h Multiple PPRs repeated with additional groups through purposeful sampling.

• Program HQ – project team and senior management
• Branch Level – field team key managers and field officers.
• Community Level - How to representatively sample communities

within program capacity (time & resources).

• Consensus PPR to organize action steps into Action Plan.

9 Scott Chaplowe, IFRC Performance & Accountability Department

Key PPR Features

Safe and anonymous identification of opinions Participatory discussion of opinions

Participatory discussion of opinions

Democratic prioritization of opinions

10 Scott Chaplowe, IFRC Performance & Accountability Department

Participatory Top-Down

Beneficiaries are a consulted data source Beneficiaries decide if/what/how to l t Beneficiaries decide questions to

Participatory Continuum

Beneficiaries participate in data ll ti & Beneficiaries are an

• bserved

data source data source (interviews & focus groups) evaluate. answer.

FWRS Federation-wide Reporting System

In PPR, stakeholders are involved in the: Data Collection Data Analysis Identification of Lessons and Recommendations

collection & analysis data source Beneficiaries are a secondary data source

11 Scott Chaplowe, IFRC Performance & Accountability Department

Key PPR Steps

Step #3

What have we d ?

Step #2 Start-up Step #1 Prep

done?

Step #6 Follow-up

• Action Plan
• Write-Up

Step #4

What has changed?

Step #5

What should be done in the future?

12 Scott Chaplowe, IFRC Performance & Accountability Department

2/1/2010 4

Key PPR Steps

Step #3

What have we d ?

Step #2 Start-up Step #1 Prep

done?

Step #6 Follow-up

• Action Plan
• Write-Up

Step #4

What has changed?

Step #5

What should be done in the future?

13 Scott Chaplowe, IFRC Performance & Accountability Department

PPR Preparation

Timing: start 2 months prior, allowing proper time to inform and involve key stakeholders. Refer to IFRC Evaluation Policy. Stakeholder Meeting

Determine evaluation need and if PPR is appropriate. Identify PPR manager (someone in program/country). Identify available time, money, personnel, resources. Clarify sample frame and purposeful selection criteria. Identify key decision makers and deal breakers.

14 Scott Chaplowe, IFRC Performance & Accountability Department

PPR Preparation

TOR Development Identify facilitators and arrange for TOT Gatekeeper Visits Gatekeeper Visits

• Identify, inform, and obtain initial consent of

participants, as well as any local permission, etc.

Arrange PPR Logistics

• PPR scope and duration: 1 week or 1 month?
• Place/venues, transportation, refreshments.
• Prepare materials!

15 Scott Chaplowe, IFRC Performance & Accountability Department

Key PPR Steps

Step #3

What have we d ?

Step #2 Start-up Step #1 Prep

done?

Step #6 Follow-up

• Action Plan
• Write-Up

Step #4

What has changed?

Step #5

What should be done in the future?

16 Scott Chaplowe, IFRC Performance & Accountability Department

2/1/2010 5

PPR Start Up

Timing: ~ 1 hour. Introductions Inform & Consent

Manage Expectations (M& E)!!!!!! Participant understanding is critical. Stress that it w ill be their exercise.

Guidelines & Logistics

17 Scott Chaplowe, IFRC Performance & Accountability Department

Inform & Confidentiality

Should clearly answ er WHAT & WHY Should reassure CONFIDENTIALIY! Should reassure CONFIDENTIALIY! “What is shared in the room stays in the room.” Address any QUESTIONS or CONCERNS!

18 Scott Chaplowe, IFRC Performance & Accountability Department

Key PPR Steps

Step #3

What have we d ?

Step #2 Start-up Step #1 Prep

done?

Step #6 Follow-up

• Action Plan
• Write-Up

Step #4

What has changed?

Step #5

What should be done in the future?

19 Scott Chaplowe, IFRC Performance & Accountability Department

WHAT HAVE WE DONE?

Objective: to warm up participants mind to the what has been done - factual. Timing: ~1.5 hours. g Method: this can be facilitated in multiple ways, based on consultative planning process with team. Mission-ing Timeline

20 Scott Chaplowe, IFRC Performance & Accountability Department

2/1/2010 6

Key PPR Steps

Step #3

What have we d ?

Step #2 Start-up Step #1 Prep

done?

Step #6 Follow-up

• Action Plan
• Write-Up

Step #4

What has changed?

Step #5

What should be done in the future?

21 Scott Chaplowe, IFRC Performance & Accountability Department

WHAT HAS CHANGED?

Objective: Elicit participant’s perspective of impact. Timing: ~1.5 – 2 hours M th d b f ilit t d i lti l Method: can be facilitated in multiple ways:

• 1. Elicit ideas (ensure anonymity): four walls, idea toss,

floor/wall organizing, etc.

• 2. Discuss
• 3. Vote: award voting stars (dots) according to # of ideas.

22 Scott Chaplowe, IFRC Performance & Accountability Department

Most Significant Change (MSC) “Domains”

Domains: broad categories of possible significant changes:

Open ended: “Looking back over the X months/years, what do you think was the most significant change in in your life?” M “L ki b k th X th / h t d More narrow: “Looking back over the X months/years, what do you think was the most significant change in in your community?” More narrow : “Looking back over the X months/years, what do you think was the most significant change in in your health?” More narrow : “Looking back over the X months/years, what do you think was the most significant change in in your diet?”

23 Scott Chaplowe, IFRC Performance & Accountability Department

Domains do not have to be changes!

Domains can be phrased as questions:

• What do you think the disaster response effort?
• What do you think about the delivery of the relief

supplies?

• How equitable was the delivery of the relief supplies?

24 Scott Chaplowe, IFRC Performance & Accountability Department

2/1/2010 7

Key PPR Steps

Step #3

What have we d ?

Step #2 Start-up Step #1 Prep

done?

Step #6 Follow-up

• Action Plan
• Write-Up

Step #4

What has changed?

Step #5

What should be done in the future?

25 Scott Chaplowe, IFRC Performance & Accountability Department

What should be done in the future?

Objective: Identify how participants think:

• 1. The project should continue – action items.
• 2. Similar projects should (or should not) be done in the

future. Timing: ~1.5 – 2 hours Method: can also be facilitated in multiple ways:

• 1. Elicit ideas (ensure anonymity): four walls, idea toss,

floor/wall organizing.

• 2. Discuss
• 3. Vote: award voting stars (dots) according to # of ideas.

26 Scott Chaplowe, IFRC Performance & Accountability Department

Key PPR Steps

Step #3

What have we d ?

Step #2 Start-up Step #1 Prep

done?

Step #6 Follow-up

• Action Plan
• Write-Up

Step #4

What has changed?

Step #5

What should be done in the future?

27 Scott Chaplowe, IFRC Performance & Accountability Department

PPR Follow -Up

Objectives:

• Develop and action plan with specific items to follow-up.
• Frame expectations with PPR participants.
• Feedback outcome of PPR to stakeholders
• Feedback outcome of PPR to stakeholders.

Timing: depends on the specific intervention being evaluated, and the method chosen at this stage.

28 Scott Chaplowe, IFRC Performance & Accountability Department

2/1/2010 8

Follow -Up Method

Follow-up can be facilitated in multiple ways, depending the desired project outcomes form the PPR. Are recommendations for the continuation of the project, for future projects, or both?

29 Scott Chaplowe, IFRC Performance & Accountability Department

Example of Consensus PPR

Representatives are elected at the end of each PPR. Program HQ – project team and senior management Branch Level – field team key managers and field officers. Community Level - How to representatively sample communities within program capacity (time & resources) communities within program capacity (time & resources). Consensus PPR to organize action steps into Action Plan. Consensus Planning Meeting: elected representatives from multiple PPRs come together to reach consensus. Deliverable: a concrete Action Plan of how to move forward.

30 Scott Chaplowe, IFRC Performance & Accountability Department

Report Writing

Critical for stakeholders to agree on who is to prepare final report and when. Facilitation team should meet after PPRs to review to notes and key findings. Have facilitators write up each PPR. Use tables to prepare data for report. Provide support on basic report structure. Share back report or summarized version with stakeholders at all levels.

31 Scott Chaplowe, IFRC Performance & Accountability Department

PPR Limitations

Not an independent, “objective,” external evaluation, but based on stakeholder perception. If there are sensitive issues or technical hardware questions. M id May want to consider:

• a. Complimenting the PPR with some targeted key

informant interviews (likely conducted by the PPR lead facilitator, who can be external to the project team),

• b. Different evaluation methodology than PPR.

32 Scott Chaplowe, IFRC Performance & Accountability Department

2/1/2010 9

PPR can be time intensive.

• Needs time commitment of project team.
• However, considerable savings in not hiring an external

consultant or team

PPR Limitations

consultant or team

• Also, added value as investment in capacity building,

team building, and becomes a part of program component.

33 Scott Chaplowe, IFRC Performance & Accountability Department

Participant differences can inhibit participation Differences due to hierarchy-power status because of gender, privilege, ethnicity, religion etc. Problem: discussion can dominated by the most

PPR Limitations

y powerful voices in the group – due to . Solution: Can help to neutralize power difference through arrangement of discussion/work groups.

34 Scott Chaplowe, IFRC Performance & Accountability Department

Careful not to create false expectations.

• PPR facilitators can address this with clear

communication.

• Also in the group discussion of feasibility, capacity, and

PPR Limitations

Also in the group discussion of feasibility, capacity, and resources of action steps during the discussion and voting process.

35 Scott Chaplowe, IFRC Performance & Accountability Department

PPRs require a skilled, lead facilitator!

• Facilitators need careful training and practice

because more difficult than just asking questions

PPR Limitations

and writing answers.

36 Scott Chaplowe, IFRC Performance & Accountability Department

2/1/2010 10

Potential PPR Benefits

1. Empower project participants to provide input into a project that affects THEIR lives! 2. Learn how stakeholders feel about the program and how to move forward in their own words how to move forward in their own words. 3. Reinforces project participation, understanding, and

• wnership.

4. Identify recommendations that are more likely to succeed due to overall participation in process.

37 Scott Chaplowe, IFRC Performance & Accountability Department

5. Identify culturally appropriate lessons and recommendations that come from the experiences and realities of the stakeholders. 6 Improve listening articulating and acceptance of different

Potential PPR Outcomes

6. Improve listening, articulating, and acceptance of different viewpoints. 7. Builds the local capacity of project staff and participants to monitor and evaluation initiatives, and communicate and reach consensus (democratic process). 8. Less costly than an external evaluation.

38 Scott Chaplowe, IFRC Performance & Accountability Department

Who can name the key PPR steps?

Step #3

What have we d ?

Step #2 Start-up Step #1 Prep

done?

Step #6 Follow-up

• Action Plan
• Write-Up

Step #4

What has changed?

Step #5

What should be done in the future?

39 Scott Chaplowe, IFRC Performance & Accountability Department