DPME Evaluations NQF Impact Workshop, Midrand Antonio Hercules - - PowerPoint PPT Presentation

NQF Impact Workshop, Midrand Antonio Hercules (DPME) 09 February 2016

DPME Evaluations

Outline

• 1. Evaluation: what is it? [2 minutes!]
• 2. NES and Types of Evaluation. what these are [6

mins!]

• 3. NQF Impact Study vs. NES Impact Evaluation.

the study and impact evaluation [5 minutes!]

• 4. 2017/18 NES Implementation Evaluation

current NES evaluation [5 minutes!]

• 5. Presentation Key messages

2

Evaluation – easy word, specific meaning

As a field, its value and its shortcomings

1.

Field:

• about 30-40 years,
• imprecise but improving,
• specifically deals with policy/programme/project as the object,
• specialised,
• mixed methods,
• sensitive to bias

2.

Value:

• offers “objective” external and “expert” internal approach options.
• great window for renewal/improvement,
• highly flexible,
• decades of practice
• strongly evidence-based

3

Evaluation – easy word, specific meaning

3.

Shortcomings

• Complex
• Duration
• Financial investment
• Must have buy-in
• Needs good and committed managers/political principals to

take forward

• Evaluation is different from “normal”

evaluative reflection in policy analysis/research

4

National Evaluation System and Evaluation

Six (6) guidelines (for policy/prog./proj.)

• 1. Diagnostic*: … “identification”

“research” – info. foundations

• 2. Design*: “formulation”
• 3. Implementation: performance

(results), everything short of impact

• 4. Impact: using quantitative RCT

approach conventionally

• 5. Economic*: …economic benefits
• 6. Synthesis*: …synthesis of pre-existing

evaluations

5

NQF Impact Study vs. NES Impact Evaluation

1.

NQF Impact Study has value in its own right 

2.

Key question: would Study pass in the NES? 

3.

Main reasons:

• Largely impact evaluation TOR, but methodology does not

deliver accepted evaluation design

• More of a “synthesis” evaluation pulling together existing

bodies of knowledge -> produces overview, trends, insights, but doesn’t subject underlying evaluation design of supporting reports to data verification/validation testing

• Does not appear to comply with accepted large system

impact evaluation design (correlations, regressions, etc.)

• Still very useful, and well-timed for NQF Act policy

implementation evaluation (NES – NEP 2016/2017)

6

Typical Impact Evaluation Approach

7

• Policy
• Goals
• Results

1 Results Frame- work

• Theory of

Change

• Logframe
• Operational

Strategy

• “Nesting”

2 Logic

• Literature
• Interviews
• Focus

Groups

• Survey
• Databases

3 Evidence

• Policy

Narrative

• Evidence
• Analysis
• Results
• Conclusions

4 Assess- ment

• Sound
• Specific
• Based on

evidence

5 Recom- mend- ations

2017/18 NES Implementation Evaluation

Purpose: to assess the implementation of the NQF Act relative to its goal(s) and objectives in the period of review, including its associated policies and regulations. Successes and challenges in its implementation will be identified, and recommendations offered regarding improvements to implementation of the NQF Act in the future.

8

1.

(Relevance and Appropriateness) To what extent is the theory of change (intervention logic) of the NQF Act adequately robust, including its main underlying assumptions?

2.

(Effectiveness) To what extent has the implementation

• f the NQF Act been effective in achieving its policy

goal(s), objectives and intended outcomes?

3.

(Efficiency) To what extent has the implementation of the NQF Act been efficient?

4.

(Impact) What is the emerging impact of the NQF, if any?

5.

What needs to be done to improve the implementation

• f the NQF Act, and which aspects of the NQF Act need

to be reviewed?

9

2017/18 NES Implementation Evaluation

Sufficient teaching time

Nutritious meals delivered to schools on time Nutritious meals eaten Improved health and nutritional status of South African primary school children Better educational performance Improved levels of primary school attendance Improved concentration in class

Activities

Local business prospers Department of Education pays for food delivered

Outputs

Children have adequate nutrition

Outcomes Impact Assumptions

Local producers provide the food/ingredients at fair price

Inputs

Funds Local SMMEs appointed to render services Decentralised purchasing system Fair procurement Payments to producers/suppliers made on time Food prepared locally and safely

Immediate Intermediate

Schools have adequate food storage facilities

National S chool Nutrition Programme TOC

10

TOC example: Aspen Institute (2003)

12

Thanks!

Antonio Hercules: Director: Evaluations & Research. antonio@presidency-dpme.gov.za Head: Evaluation and Research, DPME. Ian Goldman. Ian@presidency-dpme.gov.za www.thepresidency-dpme.gov.za