Overview of Overview of Evaluation in Evaluation in the UN - PowerPoint PPT Presentation

Overview of Overview of Evaluation in Evaluation in the UN Secretariat the UN Secretariat Prepared by Prepared by the Inspection and Evaluation Division of the Inspection and Evaluation Division of the Office of Internal Oversight Services the Office of Internal Oversight Services for the Committee for Programme Programme and Coordination and Coordination for the Committee for th 2011 June 6 th 2011 June 6

The IED Management Team The IED Management Team Eddie Yee Woo Guo Guo Eddie Yee Woo Demetra Arapakos Arapakos Demetra Arild Hauge Hauge Arild Beth Daponte Daponte Beth Chandi Kadirgamar Kadirgamar Chandi 2 2

Overview Overview Introduction Introduction 1. 1. The Programme Programme Planning Cycle Planning Cycle The 2. 2. The Role of Evaluation The Role of Evaluation 3. 3. Evaluation in the Secretariat Evaluation in the Secretariat 4. 4. The Work of OIOS‐ ‐IED IED The Work of OIOS 5. 5. Issues Issues 6. 6. 3 3

PROGRAMME PLANNING CYCLE LEGISLATIVE MANDATES STRATEGIC FRAMEWORK FOR A TWO- EVALUATION: Assesses relevance, YEAR PERIOD, WITH PRIORITIES efficiency and effectiveness (including • Objectives impact) • Expected accomplishments • Self-evaluations conducted by managers • Indicators of achievement to determine what worked and what did not work BUDGET OUTLINE • Programme or thematic evaluations conducted by OIOS • Other external evaluations PROGRAMME BUDGET (BIENNIAL) • Initial proposals MONITORING OF • Revised estimates • Programme budget implications IMPLEMENTATION statements Programme performance report • Unforeseen and extraordinary expenses (biennial) • First budget performance report • Status of outputs delivered • Second budget performance report • Extent to which expected accomplishments • Closing of accounts (results) were achieved 4 4

The Role of Evaluation The Role of Evaluation Provides objective assessments of the Provides objective assessments of the � � relevance, efficiency and effectiveness relevance, efficiency and effectiveness (including impact) of programmes programmes to to (including impact) of support decision‐ ‐making and reflection by making and reflection by support decision the General Assembly, the Secretary the General Assembly, the Secretary General, Programme Programme managers and staff managers and staff General, Supports accountability and learning Supports accountability and learning � � 5 5

Conditions for a Conditions for a credible evaluation function: credible evaluation function: Operational independence Operational independence 1. 1. Adequate resources Adequate resources 2. 2. Professional competence Professional competence 3. 3. Methodological Rigor Methodological Rigor 4. 4. Transparency Transparency 5. 5. Utility Utility 6. 6. 6 6

Evaluation in the UN Secretariat Evaluation in the UN Secretariat 7 7

Secretariat Programmes Programmes, by type , by type Secretariat Peace & Security DPKO DFS DPA ODA DESA ECA OOSA UNODC ECE ECLAC EOSG ESCAP DGACM Mgt & ESCWA OHCHR; UN- -Women Women DM OHCHR; UN Development Support OHRLLS DPI OSAA DSS UNCTAD OLA ITC UNEP UN-Habitat OCHA UNHCR UNRWA Humanitarian 8 8

The 5 largest UN programmes compared to rest (by budget) $16,000,000.00 $14,000,000.00 $12,000,000.00 $10,000,000.00 DM DPA $8,000,000.00 UNRWA UNHCR $6,000,000.00 All (22 other) DPKO $4,000,000.00 $2,000,000.00 $- DM DPA UNRWA UNHCR All (22 other) DPKO Programme/s 9 9

Independent Evaluation Independent Evaluation & Self Evaluation in the Secretariat & Self Evaluation in the Secretariat Independent Evaluation Self Evaluation Independent Evaluation Self Evaluation Who? OIOS IED Programme Evaluation Units Evaluation Units Who? OIOS IED Programme What? Subprogrammes and and Project level; Operational issues; What? Subprogrammes Project level; Operational issues; Programme as whole; as whole; Uneven coverage; Programme Programme Programme Uneven coverage; specific thematic evaluations, specific thematic evaluations, Secretariat wide thematic Secretariat wide thematic When? When? Once every 12 years Once every 12 years Depends on the programme Depends on the programme Where? OIOS Reports to GA and to Internal Reports, PPR and the Where? OIOS Reports to GA and to Internal Reports, PPR and the Programme Managers Managers OIOS Biennial report on Programme OIOS Biennial report on Evaluation Evaluation How? Mixed Methods: Qualitative Mixed methods: May or may not How? Mixed Methods: Qualitative Mixed methods: May or may not and Quantitative, as per be as per Norms and Standards and Quantitative, as per be as per Norms and Standards Norms and Standards for for Evaluation Norms and Standards for for Evaluation Evaluation Evaluation Primary General Assembly; Programme Managers; Division Managers; Division Primary General Assembly; Programme users Secretary General; Managers; Section Chiefs; Staff users Secretary General; Managers; Section Chiefs; Staff Programme Managers Programme Managers 10 10

(Self) Evaluation Units (Self) Evaluation Units of the Secretariat Programmes Programmes of the Secretariat 29 Programmes Programmes (not including OIOS) (not including OIOS) 29 1. 1. Lack operational independence - - 9 have stand 9 have stand- -alone alone Lack operational independence 2. 2. Evaluation Units, 20 do not Evaluation Units, 20 do not Inadequate resources: Budget has declined; large Inadequate resources: Budget has declined; large 3. 3. programmes (e.g. (e.g. DPKO, UNRWA, DPA, DM, DPKO, UNRWA, DPA, DM, programmes DESA) have ) have little or no little or no dedicated evaluation dedicated evaluation DESA capacity capacity Uneven competence; no training Uneven competence; no training 4. 4. Some transparency Some transparency 5. 5. Some utility Some utility 6. 6. 11 11

Secretariat programmes, by size, risk and evaluation budget OHCHR DPKO UNEP 2.6 UNHCR DM OCHA ECA UNODC DSS UN ‐ HABITAT Risk (IED 2008) DPA UNRWA DESA 2 ODA DPI UNCTAD EOSG DGACM ITC ESCAP ECLAC OSAA OLA ECE ESCWA OHRLLS OOSA 1.4 0.0 500.0 1,000.0 1,500.0 2,000.0 2,500.0 3,000.0 3,500.0 4,000.0 2009 Evaluation budget (in thousands)

OIOS OIOS Inspection and Evaluation Division (IED) Inspection and Evaluation Division (IED) 13 13

What we do – – our outputs our outputs What we do Programme Evaluations Evaluations – – “ “whole of whole of programme programme” ” evaluations evaluations Programme 1. 1. focused on assessing results achieved focused on assessing results achieved Thematic Evaluations – – covering cross covering cross‐ ‐cutting Secretariat cutting Secretariat‐ ‐ Thematic Evaluations 2. 2. wide issues wide issues Inspections – – review of an unit or issue at risk and extent it review of an unit or issue at risk and extent it Inspections 3. 3. adheres to normative standards or good practice adheres to normative standards or good practice Biennial report – – summary of Secretariat evaluations and summary of Secretariat evaluations and Biennial report 4. 4. assessment of their quality, as well as M&E capacity assessment of their quality, as well as M&E capacity Triennial reviews – – follow follow‐ ‐up on implementation of CPC up on implementation of CPC Triennial reviews 5. 5. endorsed recommendations endorsed recommendations Quarterly Compliance Reports on Performance Reporting Quarterly Compliance Reports on Performance Reporting 6. 6. Ad Hoc Evaluations Ad Hoc Evaluations 7. 7. 14 14

How we do it – – our approach our approach How we do it Programme Logic Model to determine Logic Model to determine Programme Programme Theory Theory Programme 1. 1. Inputs > Activities > Outputs > Outcomes > Impact Inputs > Activities > Outputs > Outcomes > Impact Mixed Methods – – Qualitative and Quantitative Qualitative and Quantitative Mixed Methods 2. 2. Methods include – – Desk Reviews, Content Analysis, Desk Reviews, Content Analysis, Methods include 3. 3. Structured Interviews, Surveys (Client and Local Structured Interviews, Surveys (Client and Local Population), Direct Observation, Focus Group Discussions, Population), Direct Observation, Focus Group Discussions, Case Studies, Comparative Analysis, Benchmarking. Case Studies, Comparative Analysis, Benchmarking. Recommend Actions and Monitor Implementation Recommend Actions and Monitor Implementation 4. 4. Adhere to UNEG Norms and Standards Adhere to UNEG Norms and Standards 5. 5. 15 15

Programmes Evaluated Evaluated Programmes 2008 – – DPA DPA 2008 2009 – – OHCHR; OHRLLS; OSAA OHCHR; OHRLLS; OSAA 2009 2010 – – UNRWA; DM (including UNRWA; DM (including UNOs UNOs) ) 2010 2011 ‐ ‐ DESA DESA 2011 2012 – – UNHCR; OCHA; UNEP UNHCR; OCHA; UNEP 2012 2013 – – ECA; UNODC ECA; UNODC 2013 To be evaluated – – ESCAP, EOSG, ESCWA, ECLAC, UN ESCAP, EOSG, ESCWA, ECLAC, UN To be evaluated Habitat, DPI, DGACM, UNCTAD, ITC, DSS, ODA, Habitat, DPI, DGACM, UNCTAD, ITC, DSS, ODA, ECE, OLA, OOSA ECE, OLA, OOSA 16 16

Peacekeeping Missions Evaluated Peacekeeping Missions Evaluated 2009 – – UNOCI UNOCI 2009 2010 – – UNMIL; MINUSTAH UNMIL; MINUSTAH 2010 2011 – – UNMIS UNMIS 2011 2012 – – MINUSTAH; MONUSCO MINUSTAH; MONUSCO 2012 17 17