Evaluation Talk From Uganda, Ukraine and beyond Evaluation lessons - - PowerPoint PPT Presentation

Evaluation Talk

From Uganda, Ukraine and beyond – Evaluation lessons from parliamentary strengthening?

A practical example of the design and execution of a multi-country end-evaluation

By: Karin Weber Kampala, April 9th 2014

• 1. Introduction
• 2. Scope of evaluation
• 3. TWC evaluation in numbers
• 4. Purpose of evaluation
• 5. Evaluation criteria
• 6. Evaluation methodology
• 7. Evaluation framework
• 8. Limitations & Challenges
• 9. Top tips

2

Presentation Outline

Introducing…

Karin Weber

• Senior Consultant
• M&E specialist
• Worked as independent consultant, NGO director, United

Nations M&E specialist, MoFA policy advisor in Europe, Africa and Asia. Delta Partnership

• International development consultancy firm
• Offices in UK, Kenya and Uganda
• Recently became part of WYG
• Multi-sector and multi-disciplinary

TWC end evaluation in numbers…

TWC end evaluation 7 Consortium members 1 lead 7 Countries in 5 different regions 50 Implementing partners 9 Evaluation Criteria 130+ respondents 5 languages 4 thematic areas 5 outputs

Evaluation Scope (1/3)

• The programme: ‘’Strengthening Human Resource

Development in Southern Parliaments’’

• Implemented by The Westminster Consortium (TWC):
• led by the Westminster Foundation for Democracy (WFD)
• Commonwealth Parliamentary Association UK Branch,
• The House of Commons Overseas Office,
• The International Bar Association’s Human Rights Institute
• The National Audit Office (NAO)
• The Thomson Reuters Foundation (TRF)
• The University of Essex Institute for Democracy and

Conflict Resolution

• Funded by DFID’s Governance and Transparency Fund

(GTF)

Evaluation Scope (2/3)

• TWC programme aimed to establish self-

sustaining Parliamentary Studies Centres (PSC) and building the capacity of local partners to deliver effective parliamentary development

• Twinning model between UK partners and key

partners in each country.

• Four thematic focus areas:
• 1. Parliamentary Process and Management;
• 2. Human Rights and the Rule of Law;
• 3. Financial Oversight; and
• 4. Access to Information.

Evaluation Scope (3/3)

• Period under evaluation: 2008 – 2013
• Geographical scope: UK, Georgia, Ukraine,

Uganda, Mozambique, Lebanon and Morocco

Purpose of Evaluation

In line with GTF guidelines the purpose of the Final Evaluation was to:

• Identify the impact of the programme and ways that this

may be sustained

• Record and share lessons
• Account to local stakeholders for the programme’s

achievements

• Improve future programme design and management
• Verify funds were used effectively and efficiently to deliver

results

• Enable DFID to evaluate the performance of the GTF as a

whole, making sure the overall portfolio has increased accountability and responsiveness.

Evaluation Criteria

• Relevance
• Impact
• Economy
• Efficiency
• Effectiveness
• Equity
• Value for money
• Sustainability
• Replicability

Methodology

• The international evaluation team: Karin Weber and Aileen Lyon
• The end evaluation was conducted between September 2012 and

August 2013

• Step 1: Planning meetings with the client
• Step 2: Stakeholder mapping (to identify potential respondents)
• Step 3: Development of an evaluation matrix
• Step 4: Development of an evaluation plan
• Step 5: Development of interview topic list/ questionnaires
• Step 6: Conducting the desk study
• Step 7: Carrying out 6 country visits (up to 5 days in each country)

– Mid-term update report to client including 3 visit reports and emerging findings – Country reports validated with in-country teams

• Step 8: Analysis and draft reporting
• Step 9: Validation meeting with TWC team
• Step 10: Finalisation of evaluation report

Example of Evaluation Framework

Criteria Key evaluation questions Methodology Data sources/key stakeholder Effectiveness

• To what extend have the planned

results been achieved to date?

• What has been the most

significance change of the programme?

• Which factors contributed to the

success of the programme?

• What were the key challenges

affecting programme performance?

• Were there any unintended results

in the programme? Were they positive or negative and in which ways did they affect the different stakeholders? Desk study Semi- structured interviews Focus Group Discussions Project documents National level studies and surveys Project staff, Ministerial policy advisors

Cont.

Criteria Key evaluation questions Methodology Data sources/key stakeholder Efficiency

• How economically were

resources/inputs (funds, expertise, time, etc.) converted to programme results?

• To what extend did the budget and

actual expenditure at the country level reflect identified programme priorities? Value for Money analysis Cost-benefit analysis Desk study Project documents Finance

• fficers

Audit reports Relevance

• Was the programme formulated

and implemented in accordance with national and local strategies?

• To what extend did the programme

answer the needs of the intended beneficiaries? Interviews Desk study Satisfaction survey Project document National policies Beneficiaries

From ToR to Framework, Tools and Analysis

Analysis and Reporting

Labelling, grouping, summarising Report outline/ chapters

Tools for data collection

Interview topic lists & Questionnaires Desk review checklists & Achievement rating scale

Evaluation framework

Evaluation Criteria & Evaluation Questions Methodology & Data sources & Key stakeholders

Evaluation objectives

In Terms of reference Link to Evaluation criteria

Desk Study

TWC Programme documents

• Log frame
• Budget
• Quarterly reports
• Annual reports
• Financial reports
• Risk management

report

• Case studies

Other documents

• Final Evaluation Guidelines for

Governance and Transparency Fund Grant Holders

• National level data (incl. bills,

laws, policies, statistics etc.)

• Academic literature related to

topic

• Articles by trained journalists
• Handbooks and guides

developed by the programme

• Partner documents
• Websites

Evaluation documents

• Mid Term Review

Report

• Management

response to MTR

• Internal

evaluations

Methodology (cont.)

• Country visits:

– Observation, Semi-structured interviews and Focus Group Discussions with:

• TWC Programme Managers and Administration and Finance Officers
• local partners
• MPs and Parliamentary staff
• Local trainers and Trainees
• Journalists and representatives from Civil Society Organisations (CSO).
• DFID and other donors working with the parliament in-country

– Use of a translator in Georgia, Ukraine, Morocco, Lebanon and Mozambique – Use of voice recorder where permission was granted by respondent – Additional in-country document review where relevant

• Performance analysis - to assess performance against the

logframe, for outputs, outcome and impact

Methodology (cont.)

• Data analysis and synthesis

– Notes from document review – Transcriptions from interviews and FGDs – Labelling and synthesis in line with evaluation matrix – Further analysis of financial data to assess Value for Money

• Reporting

– Draft report in line with good practice, the ToR and GTF guidance – Discussion and validation with the TWC team – Incorporation of feedback – Finalisation of the evaluation report.

Challenges & Limitations (1/2)

• Limited evidence

– Especially at the higher purpose (outcome) and goal (impact) levels. – Focus on monitoring activities and expenditure during implementation instead of intermediate and long-term results – Limited use of the logframe for monitoring (seen in-country as a HQ document) – Poor follow-up after trainings, study tours and other capacity building activities – No systematic monitoring of bills passed and implementation of laws – Lack of clarity on the responsibility for data collection during monitoring

• Data collection during country missions took place at

different times in the last programme year (between October 2012 and June 2013)

Challenges & Limitations (2/2)

• In some cases WFD staff members were present

during interviews; risk of bias

• Security (e.g. Lebanon mission postponed due to

car bombing next to Parliament)

• Political sensitivity (e.g. parliamentary

proceedings in Lebanon are not public)

• The end evaluation steering committee consisted
• f TWC members only; no independent members
• Gender unbalanced evaluation team
• Different quality of translators

Top Tips (1/2)

• Check specific donor guidelines for end evaluations
• Frequently refer back to ToR and evaluation matrix
• Ensure a good preparation and initial analysis before the

country missions:

– Go straight to the monitoring data and reports early on to get a grip on the data they have available – Ensure that you have a ToC up front – not just the logframe. Jointly develop one if ToC doesn’t exist. You can then test the assumptions and how it all worked out as you go into in-country conversations

• Check country reports, data and facts with respondents

and/or managers before the final analysis of the data

• Make time for a joint analysis with all the team who worked
• n it – much more efficient

Top Tips (2/2)

• Ensure highest adherence to ethical standards,

including:

– Independence (emphasise this with client where necessary, don’t be shy!) – Honesty, integrity and transparency (report on limitations)

– Respect for dignity and diversity – Confidentiality- protecting sources of information

• Apply triangulation in methodology and data sources
• Use participatory M&E methods and integrate gender

and human rights

• Keep the client informed, also during the process

Thank you! Questions?

For more information: karin@deltapartnership.com www.deltapartnership.com For the full evaluation report: http://www.wfd.org/upload/docs/GTF%20394%20Annex%20A6%20- %20Final%20Evaluation.pdf

21