M&E FUNDAMENTALS What is this presentation about? 1 M, E, - - PowerPoint PPT Presentation

M&E FUNDAMENTALS

Time to reflect and discuss some key concepts

Participatory Monitoring, Evaluation and Learning

A brief Introduction

What is this presentation about?

• 1 M, E, M&E
• 2 PM&E
• 3 PM&EL

Information for Who, for What?

• Important to ask such questions to understand what

is going on.

Discuss

Roles in M&E Use of Information – how? Donors Policy-Makers Planners Managers Field-Workers “Beneficiaries”

Discuss the elements of this matrix in your groups. Think about current reality.

Monitoring Evaluation

Convergence

Consider the meaning of terms like; “Impact Monitoring” and “Formative (ongoing) Evaluation”

• 1. Understanding the

situation/ identification

• 8. Impact

Assessment

• 6. Reviews
• 2. Design
• 3. Appraisal and Approval

Where are lessons Learned & applied?

• 5. Mid-Term

Evaluation

• 7. Terminal

Evaluation

Live Project Summative Evaluation

THE RESULTS CHAIN

The results chain

Inputs Activities Outputs Outcomes Impact

Results Implementation

Inputs

The resources used in an intervention: People Money, materials, infrastructure, equipment Intangibles - know-how, ideas, reputation

Activities/processes

What the intervention does to turn inputs into outputs

Inputs

Activities/ processes

Outputs

Outputs

The products and services delivered by the intervention to its direct targeted people or institutions

The “edge of your intervention”

Outcomes

Shorter-term changes - to which the intervention has contributed to a substantial extent. The ‘use’ of outputs by primary stakeholders.

Outcomes

Often at two or three levels Long-term Outcome Intermediate outcome Immediate/short-term

• utcome

Impact

Longer-term, wider, more profound changes to which the intervention has contributed to some extent. In a ‘Development Project/programme’, the impact must contribute significantly to long-term improvement in the lives of disadvantaged people. Often equated with Ultimate Outcome, Goal, Development Outcome

Construct a results chain for this training programme, writing each individual result on a separate strip of paper

Impact

?

Intermediate outcome

?

Short-term outcome

?

Outputs

?

Activities

?

Inputs People, materials, infrastructure, intellectual property, reputation, money

A results chain for a training programme

A brief history of PM&E

• WHOSE REALITY COUNTS? 30 years of participatory research traditions,

including participatory action research (PAR), participatory learning and action (including Participatory Rural Appraisal or PRA), and farming systems research (FSR) or farming participatory research (FPR)

• WHO COUNTS REALITY? By the 1980s, concepts of participatory monitoring and

evaluation had already entered the policy making domain of larger donor agencies and development organisations, (FAO, USAID, DANIDA, DFID, SIDA, NORAD,World Bank).

• Communities and community-based organisations have long been monitoring

and evaluating their work (without labelling it as such), developing their own procedures for recording and analysing information, and using that information for making decisions.

Adapted from Estrella and Gaventa 2003

Why of PM&E?

The engagement of all stakeholders

• Ownership, effectiveness and

sustainability

• Saving the planet from people
• Saving the planet for people
• Saving the planet with people

Behaviour Attitude Sharing Methods

Participatory Methodologies: Principles and values

Ownership and control

Respecting local knowledge and wisdom; individuals have control over data collection, analysis and use.

Empowerment

Creating a space where individuals are able to articulate and reflect on their own perspectives around livelihoods and development

Inclusion and representation

Making sure that the most vulnerable groups and marginalised are included in knowledge generation

Robert Chambers

Coupal 2001 and Narayan 2002

• Conventional

M&E Participatory M&E Who

External experts Community members, project staff, facilitator

What

Predetermined indicators of success principally cost and production

• utputs
• People identify their own indicators of success,

which may principally cost and production outputs include production outputs

How

Focus on ’scientific objectivity’; distancing evaluators from other participants; complex procedures; delayed, limited access to results

• Self-evaluation; simple methods adapted to local

culture; open, immediate sharing of results through local involvement in evaluation processes

• Methods Survey, questionnaire, semi-

structured interviewing, focus group discussions Range of methods such as Participatory Learning and Action, Appreciative Inquiry, and testimonials

When

Usually upon completion of project/programme; sometimes also mid-term

• More frequent, small-scale evaluations
• Why

Donor Accountability, usually summative, to determine if funding continues

• To empower local people to initiate, control and

take corrective action Capacity building, increasing ownership over results, and multi-stakeholder accountability

Source: Civicus: Turning PrinciplEs Into Action

Balance Accuracy vs Legitimacy Photo

What all this means in practice?

• Accepting that circumstances and factors surrounding

complex social interventions and projects, such as the BBNPP, tend to be dynamic and emergent, require a major focus on learning. We understand PMEL as the reflection-action Learning processes happening during and after any social intervention aimed to maximize contribution to transformative changes including implementing organisations’ changes.

• From a learning perspective to MEL, participation of

different stakeholders is not just considered a “good practice” but constitutes a core, pragmatic principle to inform processes of organisational learning and decision-making. Since knowledge is to be created by actors on the ground, programmes need to be informed by stakeholders’ interpretation and participation in makings sense of their experiences.

• Very expert based and extractive evaluative processes and

can hinder long term and locally led transformative processes (Whitty, 2013; Eyben, 2013; Shutt, 2009). On the

• ther hand, the excessive emphasis on upward accountability

(or accountability to donors) makes programmes focus on attribution to change rather than contribution to change. Thus, this emphasis provides simplistic and idealistic pictures

• f the reality that programmes aim to influence.

What was our level of listening?

A B C D E F G H I Count Rank A A A D A A A H A A 6 2= B C B E B G H B B 3 5 C C E C G H C C 4 4 D E F G H D D 2 6 E E E H E E 6 2= F G H I F 1 7= G H G G 5 3 H H H 8 1 I I 1 7=

Issues raised

Issue Wins (Score) Rank Health sector not addressed by the project 8 1st Pressure to include candidates of UP. 7 2nd Limited selection targets create tension among excluded poor* 6 3rd Sudden directives from PMU with inadequate preparation time 4 4th = Villager’s unfair criticism of CDOs to visiting management 4 4th = Split allocation of assets limits investment 4 4th = Elderly beneficiaries are difficult to engage 2 5th Religious opposition to working with women 1 6th Personal health risks working in the Chars 7th

Issues Scoring

• In one group,

address the following question;

– “ What are the main challenges of M&E in practice”?

• Allow anyone to voice an issue

– Record agreed captions for each issue on flipchart – Letter … A, B, C … etc sequentially

• When discussion cools and main points are made,

call a halt.

• Use the grid to compare issues systematically, and

vote on the most important in each pair.

• Record ‘wins’ on the grid.

TIMELINE AS A TOOL

Most Significant Change

Most Significant Change

An empowering method for participatory monitoring and evaluation A learning and change oriented approach that allows us to know about our contribution to change Based mainly on qualitative and emergent (not pre-determined) change indicators

The Most Significant Change (MSC) technique is a form of participatory monitoring and evaluation.

It is participatory because many project stakeholders are involved both in deciding the sorts of change to be recorded and in analysing the data. It is a form of monitoring because it occurs throughout the program cycle and provides information to help people manage the program. It contributes to evaluation because it provides data on impact and

• utcomes that can be used to help assess the performance of the

program as a whole.

Essentially, the process involves the collection of significant change (SC) stories emanating from the field level, and the systematic selection of the most significant of these stories by panels of designated stakeholders or staff. The designated staff and stakeholders are initially involved by ‘searching’ for project impact. Once changes have been captured, various people sit down together, read the stories aloud and have regular and often in- depth discussions about the value of these reported changes. When the technique is implemented successfully, whole teams of people begin to focus their attention on program impact.

Why use MSC?

1. It is a good means of identifying unexpected changes. 2. It is a good way to clearly identify the values that prevail in an

• rganisation

3. It is a form of monitoring that requires no special professional skills. 4. There is no need to explain what an indicator is. Everyone can tell stories about events they think were important. 5. It encourages analysis as well as data collection because people have to explain why they believe one change is more important than another. 6. It can build staff capacity in analysing data and conceptualising contribution. 7. It can deliver a rich picture of what is happening. 8. It can be used to monitor and evaluate bottom-up initiatives that do not have predefined outcomes against which to evaluate (emergent sense-making).

The Most Significant Change (MSC) approach is based on analysing personal accounts of change and deciding which of these accounts is the most significant – and why.

The are three basic steps in setting up this option:

• 1. Deciding the types of stories that should be collected (stories about what - for

example, about practice change or health outcomes or empowerment)

• 2. Collecting the stories and determining which stories are the most significant
• 3. Sharing the decisions with stakeholders and contributors so that learning about

what is valued happens.

10 Steps towards learning and change

• 1. How to start and raise interest
• 2. Defining the domains of change
• 3. Defining the reporting period
• 4. Collecting SC stories
• 5. Selecting the most significant of the stories
• 6. Feeding back the results of the selection process
• 7. Verification of stories
• 8. Quantification
• 9. Secondary analysis and meta-monitoring

10.Revising the system.

What was our level of listening?

Post field work reflection

• How was the experience?
• What are your insight about the method?
• How was your level of listening?

Theory of Change approach and model

Condition 1 Condition 5 Condition 4 Condition 3 Condition 2

CC CC CC CC CC SPHERE OF CONTROL STRATEGY 1

ASSUMPTION ASSUMPTION ASSUMPTION ASSUMPTION ASSUMPTION

STRATEGY 2 STRATEGY 3 STRATEGY 4 STRATEGY 5 OD STRATEGY

TFNC, UNICEF PM Office, Tanzania, Multisectoral Nutrition Action Plan : Desired Change

Desired Change A change in state and stakeholders envisioned to happen in a determined timeframe

Conditions for Change

These are events and phenomena that need to be in place in order for the Desired Change to happen

Assumptions

Assumptions are the rationale of our interventions and they sit between Desired Change and Conditions answering the question: “Why do you think this condition is necessary in order for the Desired Change to happen?

Contributions to Condition

These are the contribution in terms of individuals and organisations that is provided in order to influence the Condition for Change

There is a standard for optimal nutrition behaviours for children which leads to the reduction of malnutrition

The presence and the enforcement of laws and policies helps the change

• f consumption

patterns

CONDITION Civil society organisation's are promoting and influencing sustainable management of NR

ASSUMPTION

Civil society are key actors in influencing planning, decision making and good governance of natural resources

УСЛОВИЕ Организации гражданского общества влияют на устойчивый менеджмент природных ресурсов и продвигают его

ПРЕДПОЛОЖЕНИЕ

Гражданское общество – ключевой агент, влияющий на планирование, принятие решений и управление природными ресурсами

• 2. EBM is understood by

civil society and applied in MSP processes in dialogue with other stakeholders

CONDITION

CONDITION