Overview of Overview of Evaluation in Evaluation in the UN - - PowerPoint PPT Presentation

Overview of Overview of Evaluation in Evaluation in the UN Secretariat the UN Secretariat

Prepared by Prepared by the Inspection and Evaluation Division of the Inspection and Evaluation Division of the Office of Internal Oversight Services the Office of Internal Oversight Services for the Committee for for the Committee for Programme Programme and Coordination and Coordination

June 6 June 6th

th 2011

2011

2 2

The IED Management Team The IED Management Team

Eddie Yee Woo Eddie Yee Woo Guo Guo Demetra Demetra Arapakos Arapakos Arild Arild Hauge Hauge Beth Beth Daponte Daponte Chandi Chandi Kadirgamar Kadirgamar

3 3

Overview Overview

1. 1.

Introduction Introduction

2. 2.

The The Programme Programme Planning Cycle Planning Cycle

3. 3.

The Role of Evaluation The Role of Evaluation

4. 4.

Evaluation in the Secretariat Evaluation in the Secretariat

5. 5.

The Work of OIOS The Work of OIOS‐ ‐IED IED

6. 6.

Issues Issues

4 4

LEGISLATIVE MANDATES

STRATEGIC FRAMEWORK FOR A TWO- YEAR PERIOD, WITH PRIORITIES

• Objectives
• Expected accomplishments
• Indicators of achievement

BUDGET OUTLINE

PROGRAMME BUDGET (BIENNIAL)

• Initial proposals
• Revised estimates
• Programme budget implications

statements

• Unforeseen and extraordinary expenses
• First budget performance report
• Second budget performance report
• Closing of accounts

MONITORING OF IMPLEMENTATION Programme performance report (biennial)

• Status of outputs delivered
• Extent to which expected accomplishments

(results) were achieved

EVALUATION: Assesses relevance, efficiency and effectiveness (including impact)

• Self-evaluations conducted by managers

to determine what worked and what did not work

• Programme or thematic evaluations

conducted by OIOS

• Other external evaluations

PROGRAMME PLANNING CYCLE

5 5

The Role of Evaluation The Role of Evaluation

• Provides objective assessments of the

Provides objective assessments of the relevance, efficiency and effectiveness relevance, efficiency and effectiveness (including impact) of (including impact) of programmes programmes to to support decision support decision‐ ‐making and reflection by making and reflection by the General Assembly, the Secretary the General Assembly, the Secretary General, General, Programme Programme managers and staff managers and staff

• Supports accountability and learning

Supports accountability and learning

6 6

Conditions for a Conditions for a credible evaluation function: credible evaluation function:

1. 1.

Operational independence Operational independence

2. 2.

Adequate resources Adequate resources

3. 3.

Professional competence Professional competence

4. 4.

Methodological Rigor Methodological Rigor

5. 5.

Transparency Transparency

6. 6.

Utility Utility

7 7

Evaluation in the UN Secretariat Evaluation in the UN Secretariat

8 8

Secretariat Secretariat Programmes Programmes, by type , by type

Peace & Security Development Humanitarian Mgt & Support

DESA ECA ECE ECLAC ESCAP ESCWA OHRLLS OSAA UNCTAD ITC UNEP UN-Habitat EOSG DGACM DM DPI DSS OLA

DPKO DFS DPA ODA OHCHR; UN OHCHR; UN-

• Women

Women

OCHA UNHCR UNRWA OOSA UNODC

9 9

$- $2,000,000.00 $4,000,000.00 $6,000,000.00 $8,000,000.00 $10,000,000.00 $12,000,000.00 $14,000,000.00 $16,000,000.00

DM DPA UNRWA UNHCR All (22 other) DPKO

Programme/s

The 5 largest UN programmes compared to rest (by budget)

DM DPA UNRWA UNHCR All (22 other) DPKO

10 10

Independent Evaluation Independent Evaluation & Self Evaluation in the Secretariat & Self Evaluation in the Secretariat

Programme Programme Managers; Division Managers; Division Managers; Section Chiefs; Staff Managers; Section Chiefs; Staff General Assembly; General Assembly; Secretary General; Secretary General; Programme Programme Managers Managers Primary Primary users users Mixed methods: May or may not Mixed methods: May or may not be as per Norms and Standards be as per Norms and Standards for Evaluation for Evaluation Mixed Methods: Qualitative Mixed Methods: Qualitative and Quantitative, as per and Quantitative, as per Norms and Standards for Norms and Standards for Evaluation Evaluation How? How? Internal Reports, PPR and the Internal Reports, PPR and the OIOS Biennial report on OIOS Biennial report on Evaluation Evaluation OIOS Reports to GA and to OIOS Reports to GA and to Programme Programme Managers Managers Where? Where? Depends on the Depends on the programme programme Once every 12 years Once every 12 years When? When? Project level; Operational issues; Project level; Operational issues; Uneven coverage; Uneven coverage; Programme Programme specific thematic evaluations, specific thematic evaluations, Subprogrammes Subprogrammes and and Programme Programme as whole; as whole; Secretariat wide thematic Secretariat wide thematic What? What? Programme Programme Evaluation Units Evaluation Units OIOS IED OIOS IED Who? Who? Self Evaluation Self Evaluation Independent Evaluation Independent Evaluation

11 11

(Self) Evaluation Units (Self) Evaluation Units

• f the Secretariat
• f the Secretariat Programmes

Programmes

1. 1.

29 29 Programmes Programmes (not including OIOS) (not including OIOS)

2. 2.

Lack operational independence Lack operational independence -

• 9 have stand

9 have stand-

• alone

alone Evaluation Units, 20 do not Evaluation Units, 20 do not

3. 3.

Inadequate resources: Budget has declined; large Inadequate resources: Budget has declined; large programmes programmes (e.g. (e.g. DPKO, UNRWA, DPA, DM, DPKO, UNRWA, DPA, DM, DESA DESA) have ) have little or no little or no dedicated evaluation dedicated evaluation capacity capacity

4. 4.

Uneven competence; no training Uneven competence; no training

5. 5.

Some transparency Some transparency

6. 6.

Some utility Some utility

Secretariat programmes, by size, risk and evaluation budget

DGACM DM DPA DPI DPKO DSS ECA ECE ECLAC EOSG ESCAP ESCWA ITC OCHA OHCHR OLA DESA UNCTAD UNEP UN‐HABITAT UNHCR UNODC UNRWA OHRLLS OOSA OSAA ODA 1.4 2 2.6 0.0 500.0 1,000.0 1,500.0 2,000.0 2,500.0 3,000.0 3,500.0 4,000.0

2009 Evaluation budget (in thousands)

Risk (IED 2008)

13 13

OIOS OIOS Inspection and Evaluation Division (IED) Inspection and Evaluation Division (IED)

14 14

What we do What we do – – our outputs

• ur outputs

1. 1.

Programme Programme Evaluations Evaluations – – “ “whole of whole of programme programme” ” evaluations evaluations focused on assessing results achieved focused on assessing results achieved

2. 2.

Thematic Evaluations Thematic Evaluations – – covering cross covering cross‐ ‐cutting Secretariat cutting Secretariat‐ ‐ wide issues wide issues

3. 3.

Inspections Inspections – – review of an unit or issue at risk and extent it review of an unit or issue at risk and extent it adheres to normative standards or good practice adheres to normative standards or good practice

4. 4.

Biennial report Biennial report – – summary of Secretariat evaluations and summary of Secretariat evaluations and assessment of their quality, as well as M&E capacity assessment of their quality, as well as M&E capacity

5. 5.

Triennial reviews Triennial reviews – – follow follow‐ ‐up on implementation of CPC up on implementation of CPC endorsed recommendations endorsed recommendations

6. 6.

Quarterly Compliance Reports on Performance Reporting Quarterly Compliance Reports on Performance Reporting

7. 7.

Ad Hoc Evaluations Ad Hoc Evaluations

15 15

How we do it How we do it – – our approach

• ur approach

1. 1.

Programme Programme Logic Model to determine Logic Model to determine Programme Programme Theory Theory Inputs > Activities > Outputs > Outcomes > Impact Inputs > Activities > Outputs > Outcomes > Impact

2. 2.

Mixed Methods Mixed Methods – – Qualitative and Quantitative Qualitative and Quantitative

3. 3.

Methods include Methods include – – Desk Reviews, Content Analysis, Desk Reviews, Content Analysis, Structured Interviews, Surveys (Client and Local Structured Interviews, Surveys (Client and Local Population), Direct Observation, Focus Group Discussions, Population), Direct Observation, Focus Group Discussions, Case Studies, Comparative Analysis, Benchmarking. Case Studies, Comparative Analysis, Benchmarking.

4. 4.

Recommend Actions and Monitor Implementation Recommend Actions and Monitor Implementation

5. 5.

Adhere to UNEG Norms and Standards Adhere to UNEG Norms and Standards

16 16

Programmes Programmes Evaluated Evaluated

2008 2008 – – DPA DPA 2009 2009 – – OHCHR; OHRLLS; OSAA OHCHR; OHRLLS; OSAA 2010 2010 – – UNRWA; DM (including UNRWA; DM (including UNOs UNOs) ) 2011 2011 ‐ ‐ DESA DESA 2012 2012 – – UNHCR; OCHA; UNEP UNHCR; OCHA; UNEP 2013 2013 – – ECA; UNODC ECA; UNODC To be evaluated To be evaluated – – ESCAP, EOSG, ESCWA, ECLAC, UN ESCAP, EOSG, ESCWA, ECLAC, UN Habitat, DPI, DGACM, UNCTAD, ITC, DSS, ODA, Habitat, DPI, DGACM, UNCTAD, ITC, DSS, ODA, ECE, OLA, OOSA ECE, OLA, OOSA

17 17

Peacekeeping Missions Evaluated Peacekeeping Missions Evaluated

2009 2009 – – UNOCI UNOCI 2010 2010 – – UNMIL; MINUSTAH UNMIL; MINUSTAH 2011 2011 – – UNMIS UNMIS 2012 2012 – – MINUSTAH; MONUSCO MINUSTAH; MONUSCO

18 18

OIOS Thematic Evaluations OIOS Thematic Evaluations

2005 2005 ‐ ‐ HQ and Field Linkages in Poverty Eradication HQ and Field Linkages in Poverty Eradication 2006 2006 – – Knowledge Management Knowledge Management 2008 2008 – – Lessons Learned: Protocols and Practices Lessons Learned: Protocols and Practices 2009 2009 – – UN Coordinating Bodies UN Coordinating Bodies 2009 2009 – – Disarmament, Demobilization and Reintegration Disarmament, Demobilization and Reintegration 2009 2009 ‐ ‐ Secretary General Secretary General’ ’s Climate s Climate‐ ‐Neutrality Commitment Neutrality Commitment 2010 2010 – – UN UN‐ ‐Business Partnerships for Climate Change Business Partnerships for Climate Change 2010 2010 ‐ ‐ Gender Mainstreaming Gender Mainstreaming 2011 2011‐ ‐ Cooperation of DPKO/DFS with Regional Organizations Cooperation of DPKO/DFS with Regional Organizations 2013 2013‐ ‐ Internal Justice Internal Justice

19 19

Ad Hoc Assignments Ad Hoc Assignments

2006 2006 – – Children and Armed Conflict Children and Armed Conflict 2008 2008 – – The The Peacebuilding Peacebuilding Fund Fund 2009 2009 – – Integrated Global Management of Conference Services Integrated Global Management of Conference Services 2009 2009 – – Human Resources Management of DESA Human Resources Management of DESA 2011 2011 – – Public Information Resources Public Information Resources

20 20

Issues Issues

‐ ‐

Weak Evaluation Culture; limited use of Evaluation Weak Evaluation Culture; limited use of Evaluation

‐ ‐

Current 12 year cycle for Current 12 year cycle for programme programme evaluation of evaluation of Secretariat Secretariat programmes programmes is too long is too long

‐ ‐

Inadequate evaluation of peacekeeping activities Inadequate evaluation of peacekeeping activities

‐ ‐

Methodological challenges to assess attribution and Methodological challenges to assess attribution and impact impact

‐ ‐

Lack of clarity on issue of OIOS evaluation oversight for Lack of clarity on issue of OIOS evaluation oversight for extra extra‐ ‐budgetary activities budgetary activities

‐ ‐

Lack of support for Lack of support for programme programme level (self) evaluation; level (self) evaluation; delineation of DM versus OIOS responsibility for support delineation of DM versus OIOS responsibility for support to self to self‐ ‐evaluation evaluation

21 21

Questions and Discussion Questions and Discussion