National Evaluation System Dr Ian Goldman, Head: Evaluation and - - PowerPoint PPT Presentation

▶
national evaluation system
SMART_READER_LITE
LIVE PREVIEW

National Evaluation System Dr Ian Goldman, Head: Evaluation and - - PowerPoint PPT Presentation

Nov 26, 2022 253 likes •473 views

The Presidency Department of Performance Monitoring and Evaluation National Evaluation System Dr Ian Goldman, Head: Evaluation and Research PSPPD PSC 26 August 2014 Overall situation with evaluations 39 evaluations completed, or underway,

slide-1
SLIDE 1

Dr Ian Goldman, Head: Evaluation and Research PSPPD PSC 26 August 2014

The Presidency Department of Performance Monitoring and Evaluation

National Evaluation System

slide-2
SLIDE 2

The Presidency: Department of Performance Monitoring and Evaluation

Overall situation with evaluations

  • 39 evaluations completed, or underway, of

which 37 are NEP evaluations.

2 Improvement plans being implemented Served at Cabinet Approved reports Research underway TORs approved Preparatio n stage Stuck

8 2 11 23 2 2

  • Cost benefit - R50 billion of programmes evaluated

(over MTEF). 10% improvement = R5 billion, annual cost of evaluation system R50 million, so very high return on investment.

slide-3
SLIDE 3

The Presidency: Department of Performance Monitoring and Evaluation

Nos of departments involved

  • 15/46 national

departments now involved in evaluations

  • Nos of national +

provincial departments involved in evaluations risen from 13% to 23%

3

slide-4
SLIDE 4

The Presidency: Department of Performance Monitoring and Evaluation

NEP 2012/13

Dept Title of evaluation Status as 5 May dti Implementation/ design evaluation of the Business Process Services Programme Final report approved. Report

  • n way to Cabinet

DBE Impact Evaluation of Grade R Final report approved. Report been to Cabinet Health Implementation Evaluation of Nutrition Programmes addressing Children Under 5 Final report approved and been to Cab Committee DRDLR Implementation Evaluation of the Land Reform Recapitalisation and Development Programme Final report approved and been to Cab Committee DRDLR Implementation Evaluation of CRDP Final report approved and been to Cab Committee DHS Implementation Evaluation of the Integrated Residential Dev. Programme Evaluation underway. DHS procurement has delayed DHS Implementation Evaluation of the Urban Settlements Development Grant (USDG) Evaluation underway. DHS procurement has delayed DBE Impact Evaluation of the National School Nutrition Programme (NSNP) DBE requested to drop. Reallocated to 2014/15. 4 Completing Underway About to start

Delays!

slide-5
SLIDE 5

The Presidency: Department of Performance Monitoring and Evaluation

NEP 2013/14

Dept Title of evaluation

Presidency Implementation Evaluation of Government’s Coordination Systems dti Evaluation of Export Marketing Investment Assistance Incentive programme (EMIAI) dti Evaluation of Support Programme for Industrial Innovation (SPII) dti Impact Evaluation of Technology and Human Resources for Industry Programme (THRIP) Military Veterans Evaluation of Military Veterans Economic Empowerment and Skills Transferability and Recognition Programme. DST Evaluation of National Advanced Manufacturing Technology Strategy SARS Impact Evaluation on Tax Compliance Cost of small businesses COGTA Impact evaluation of the Community Works Programme (CWP) DRDLR Evaluation of the Land Restitution Programme DAFF Impact Evaluation of CASP DAFF Implementation Evaluation of MAFISA DHS Baseline for informal settlements targeted for upgrading DHS Evaluating interventions by DHS to facilitate access to the city. DHS Diagnostic of whether the provision of state-subsidised housing has addressed asset poverty for households and local municipalities DPME Impact Evaluation of the Outcomes Approach

5 Completing by Sept Underway About to start

Delays!

slide-6
SLIDE 6

The Presidency: Department of Performance Monitoring and Evaluation

2014/15

6

Dept Evaluation DEA Evaluation of the Effectiveness of Environmental Governance in the Mining Sector (EEGM) DHET Design Evaluation of the Policy on Community Education and Training Colleges (PCETC) DHS Impact Evaluation of the Social Housing Programme (SHP) DST Evaluation of the Indigenous Knowledge Systems Policy (IKSP) DSD Diagnostic Evaluation/Programme Audit for Violence Against Women and Children (AVAWC) DSD Diagnostic Review of Coordination of the Social Sector Expanded Public Works Programme SAPS Economic Evaluation of the incremental investment into the SAPS Forensic Services DAFF/DRDLR Impact Evaluation of the Ilima Letsema Programme and Irrigation Schemes DAFF Impact evaluation of MAFISA (quantitative) – through 3ie DAFF/DRDLR Policy Evaluation of Small Farmer Support DBE Evaluation of the Funza-Lushaka Bursary Scheme DBE Impact evaluation of National School Nutrition Programme DRDLR Impact evaluation of Land Restitution Programme – through 3ie DPME Impact/implementation evaluation of the MPAT system DPME Implementation evaluation of the dept strategic planning and APP system

Underway TORs developed No TORs yet

Delays!

slide-7
SLIDE 7

The Presidency: Department of Performance Monitoring and Evaluation

Evaluations selected for 2015/16

  • Impact/implementation evaluation of the evaluation system (DPME).
  • Implementation evaluation of citizen-based monitoring programme

(DPME).

  • Agricultural Extension Recovery Plan (DAFF).
  • Evaluation of the New School Curriculum (DBE).
  • Evaluation of the Asset Forfeiture Unit Sub-programme (NPA).
  • Diagnostic evaluation of the Non-Profit Organisations Regulatory

Framework and Legislation (DSD).

  • Implementation Evaluation of the National Drug Master Plan in

addressing all forms of Substance abuse (DSD).

  • Evaluation of the effectiveness of the South African Post-School

Education and Training (PSET) quality assurance regime (DHET).

  • Evaluation of Early Grade Reading in SA (DBE).
  • Evaluation of the Mining Charter
  • Evaluation of the service delivery improvement planning system

7

slide-8
SLIDE 8

The Presidency: Department of Performance Monitoring and Evaluation

8

MTSF Outcome

All evaluations since 2009 Nos of evaluations in NEPs

1. Improved quality of basic education. 15 5 2. A long and healthy life for all South Africans. 3 1 3. All people in South Africa are and feel safe. 4 3 4. Decent employment through inclusive economic growth. 14 9 5. A skilled and capable workforce to support an inclusive growth path. 9 3 6. An efficient, competitive and responsive economic infrastructure network. 1 7. Vibrant, equitable and sustainable rural communities with food security for all. 12 8 8. Sustainable human settlements and improved quality of household life. 7 6 9. A responsive, accountable, effective and efficient local government system. 3 2

  • 10. Environmental assets and natural resources that

are well protected and continually enhanced. 5 1-2

  • 11. Create a better South Africa and contribute to a

better and safer Africa and World.

  • 12. An efficient, effective and development oriented

public service and an empowered, fair and inclusive citizenship 10 5

  • 13. Social Protection

8 6

  • 14. Social Cohesion

3 1

slide-9
SLIDE 9

The Presidency: Department of Performance Monitoring and Evaluation

9

Programme being evaluated

Outcome needed from evaluation results Close Major changes needed Minor changes needed No changes needed

Early Childhood Development (ECD) X Business Process Services Programme X Grade R (reception year of schooling) X Nutrition Programmes addressing under 5s X Land Recapit. and Dev Prog (RECAP) X Comprehensive Rural Dev Prog (CRDP) X Export Marketing Investment Assistance X Support Prog for Industrial Innovation (SPII) X Land Restitution Programme X Government Coordination System (clusters/MinMECs/Implementation Forums) X Micro Agric Financial Institution (MAFISA) X Presidential Intervention in KSD X

Many programmes need major changes

slide-10
SLIDE 10

The Presidency: Department of Performance Monitoring and Evaluation

Progress with National Evaluation System

  • 3 National and Provincial Evaluation plans to date, last week selected

evaluations for 4th

  • 39 evaluations completed, underway (11 completed, 2 public)
  • 18 guidelines, evaluation standards, competences, trained >600

government staff

  • Building demand – emphasising learning, reports to Cabinet, working

with Parliament, publicising reports, training for DGs/DDGs in use of evidence

  • Evaluation Repository – audit of evaluations since 2006, plus new - 103

evaluations on the website

  • 2 provincial evaluation plans (WC and GP), working with 4 more

provinces, and 5 departments have departmental evaluation plans

  • First Annual Report on the National Evaluation System (for 2013/14)

now available

10

slide-11
SLIDE 11

The Presidency: Department of Performance Monitoring and Evaluation

11

slide-12
SLIDE 12

The Presidency: Department of Performance Monitoring and Evaluation

12

slide-13
SLIDE 13

The Presidency: Department of Performance Monitoring and Evaluation

13

slide-14
SLIDE 14

The Presidency: Department of Performance Monitoring and Evaluation

Early indications of policy impact

  • Role of improvement plans - produced for ECD, Land Restitution,

Grade R, CRDP, RECAP and Business Process Services.

  • First 6 month report for ECD, 2nd due (late).
  • Apart from the ECD evaluation, the next evaluation was only

commissioned in October 2012 – ie 18 months prior to this report. It is therefore early days to see concrete impacts of the evaluations.

  • Early days, some examples of policy influence can already be seen:
  • New ECD policy drafted responding to the ECD Diagnostic Review, including

the need to target children from conception;

  • Renewed focus on quality of Grade R rather than just rollout as a result of

the Grade R evaluation – included in MTSF.

  • Renewed focus on nutrition in children resulting from the evaluation of

nutrition interventions for children under 5, and a stunting target in the Medium-Term Strategic Framework. The Improvement Plan will take this further.

  • Revised Business Process Outsourcing Scheme launched by Minister Davies

in UK

14

slide-15
SLIDE 15

The Presidency: Department of Performance Monitoring and Evaluation

Grade R – short-term targets

15

slide-16
SLIDE 16

The Presidency: Department of Performance Monitoring and Evaluation

Key challenges

  • Inadequate supply of skilled evaluators - range of skills

needed from proposal writing, evaluation methodology, writing reports

  • DPME working with universities and World Bank,

promote training of SPs

  • Insufficient PDI participants
  • Changing panel criteria to promote access, but keep quality, have

30% PDI requirement in CFPs

  • Some evaluations are taking a long time to be brought

by departments to cluster (often 6+ months) and thence Cabinet.

  • DPME will bring if takes longer than 4 months.
  • Departments taking long (6 months+ delay) to finalise

improvement plans arising from evaluations, and to report on progress.

  • Include in MPAT standards and discuss with AG.

16

slide-17
SLIDE 17

The Presidency: Department of Performance Monitoring and Evaluation

Challenges (2)

  • Weak programme planning
  • Cabinet agreed need for support programme using

government/donor resources. Also develop MPAT standard to

drive change.

  • Insufficient funding for complex evaluations
  • Programmes should reserve between 0.1% and 5% of

programme budgets for evaluation.

  • Some departments who are funding NEP evaluations

themselves are taking a very long time to procure (notably DHS).

  • These should be in departmental evaluation plans and

DPME procures for all evaluations selected for NEP. Need to meet Minister DHS to discuss delays on 5 evaluations.

  • Some sectors/outcomes not proposing evaluations –

notably Health, Economic Infrastructure, Environment, Public Service, Local Government.

  • FOSAD suggested propose and 6 evaluations being proposed

to Cabinet on Crime, Health, Public Service

17

slide-18
SLIDE 18

The Presidency: Department of Performance Monitoring and Evaluation

Taking the evaluation system forward

  • Undertaking audit of implementation

programmes in gov. – may well be 1000! If each were to be evaluated every 5 years = 200/year – big challenge

  • To do that extending national evaluation

system to provinces in 2014-15/2015-16 and to departments from 2015-16. Is an issue of capacity to support 46 national depts and 9 provinces.

  • Focus on programme planning (again needs

resources)

18

slide-19
SLIDE 19

The Presidency: Department of Performance Monitoring and Evaluation

Research

  • Work starting on a diagnostic which will inform a

research strategy – balance between work we do on actual research, vs support to influence the research system so it provides research relevant to the

  • utcomes
  • Some work being undertaken internally in DPME
  • PSPPD supporting a research exercise in a number of

provinces and departments to see what is happening and what role we should play

  • Working to develop immediately a repository of key

research around each outcome

  • Assignment on DG workload – starting

19

slide-20
SLIDE 20

The Presidency: Department of Performance Monitoring and Evaluation

  • Thank you
  • Dr Ian Goldman
  • Head: Evaluation and Research
  • ian@po-dpme.gov.za
  • Www.thepresidency-dpme.gov.za

20