IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS ARIEL FISZBEIN - - PowerPoint PPT Presentation

USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS

Overseas Development Institute May 4, 2010

ARIEL FISZBEIN CHIEF ECONOMIST, HUMAN DEVELOPMENT WORLD BANK

Motivation

‘Since it is difficult to distinguish the good from the bad prophet, we must be suspicious of all prophets: it is better to avoid revealed truths, even if we feel exalted by their simplicity and splendor , even if we find them comfortable because they come at no cost. It is better to be content with more modest and less inspiring truths that are laboriously conquered, step by step, with no shortcuts, by studying, discussion and reasoning, and that can be verified and demonstrated'’ Primo Levi

2

The evaluation problem

 Impact: difference in outcome with and without

• program. Cannot observe simultaneously.

 with and without comparisons deceptive if participation

depends on individual attributes that influence outcomes

 before/after comparisons can be misleading if other things

happened during the period

 Need to estimate counterfactual (control or comparison)

 Treated & control have same characteristics on average,  Only reason for difference in outcomes is the intervention

3

Multiple design options

 Experimental (often using phase in)  Quasi-experiments/non-experimental

Regression Discontinuity (RD) Difference in difference – panel data Other (Instrumental Variables, matching, etc)

 In all cases, these will involve knowing the rule

for assigning treatment

 Advantages of prospective design

4

IE at the World Bank

5

175 98

25 50 75 100 125 150 175 200

...Ongoing ...Under discussion 273 Impact Evaluations…

As of April 28, 2010

Strong focus on HD areas

6

Social Protection 20% Health 17% Education 13% ECD & Nutr. 7% Other 9% Gov . 4% Urban Upgr. 6% Agric. 6%

• Fin. &
• Priv. Sec.

8% Local Dev. 10%

The World Bank Impact Evaluation Program: Thematic Areas HD thematic areas: 58%

• f total World

Bank IEs

Strategic themes

7

6 9 9 13 14 16 19 20 5 10 15 20 25 Education, Other Pay For Performance in Health Malaria HIV Early Childhood Development Active Labor Market Programs School Accountability Conditional Cash Transfers

Number of Ongoing Impact Evaluations Thematic areas

The World Bank Impact Evaluation program on Human Development

Building a global evidence base on the impact of HD programs

8

1 3 10 17 21 54

10 20 30 40 50 60 Europe & Central Asia Middle East & North Africa East Asia & the Pacific Latin America & Caribbean South Asia Africa

Number of Ongoing Impact Evaluations

Regions

The World Bank Impact Evaluation program on Human Development

Priority thematic areas

1.

Conditional Cash Transfers

2.

Paying for Performance in Health (P4P)

3.

School Accountability

4.

Malaria Control

5.

Active Labor Market Programs / Youth Employment

6.

HIV/AIDS Prevention

7.

Early Childhood Development

9

CCTs have become very popular….

10

CCTs have become very popular….

11

First generation evaluation questions

12

 Impact on

consumption and poverty

 Side effects?

 Impacts on service

utilization

 Impacts on human

development

• utcomes

Impacts on consumption and poverty

Mexico (1999) Nicaragua (2002) Colombia (2006) Honduras (2002) Average transfer (%

• f per capita

consumption) 20% 30% 13% 11% Impact on per capita consumption (%) 8.3%** 20.6%** 10%** 7%* Impact: headcount index (% points) 1.3** 5.3** 2.9*

• Impact: poverty gap

(% points) 3.0** 9.0** 7.0** 2.0* Impact: sqd. poverty gap (% points) 3.4** 8.6** 2.2** 2.0*

13

Impacts on school enrollment

Age range Baseline enrollment Impact (% points) Size of transfer Colombia

8-13 14-17 91.7% 63.2% 2.1** 5.6*** 17%

Chile

6-15 60.7% 7.5*** 3-7%

Ecuador

6-17 75.2% 10.3*** 10%

Mexico

Grade 0-5 Grade 6 Grade 7-9 94.0% 45.0% 42.5% 1.9 8.7*** 0.6 20%

Nicaragua

7-13 72.0% 12.8*** 30%

Cambodia

Grade 7-9 65.0% 31.3*** 2%

Bangladesh

11-18 44.1% 12.0** 1%

Pakistan

10-14 29.0% 11.1*** 3%

14

Impacts on health service utilization

Age range Baseline level Impact (% points) Size of transfer Colombia

<24 months 24-48 months n.a. n.a. 22.8** 33.2*** 17%

Chile

0-6 years 17.6% 2.4 7%

Ecuador

3-7 years n.a. 2.7 10%

Honduras

0-3 years 44.0% 20.2*** 9%

Mexico

0-3 years 69.8% 8.4 20%

Nicaragua

0-3 55.4% 13.1* 27%

15

Impacts on education and health

• utcomes

 Only mixed success in terms of improving

final outcomes in education and health: Education: Increases in school enrollment and

years of completed schooling have not come hand- in-hand with improved learning outcomes

Health: Some programs, but my no means all,

have improved child nutrition (as measured by height-for-age, hemoglobin status)

16

Second generation evaluation questions

17

 Role of conditions: Testing CCTs vs. UCTs

(Burkina Faso, Malawi, Morocco and Yemen)

 Does it matter who receives the cash? Mothers

• vs. fathers (Burkina Faso, Morocco and Yemen);

Girls vs. parents in Malawi

 Design of payments: Cambodia; Colombia

Priority thematic areas

1.

Conditional Cash Transfers

2.

Paying For Performance in Health

3.

School Accountability

4.

Malaria Control

5.

Active Labor Market Programs / Youth Employment

6.

HIV/AIDS Prevention

7.

Early Childhood Development

18

Results-based financing (RBF) ≈ Pay-for-performance (P4P)

Provision of payment for the attainment of well- defined results Transfer of money or material goods conditional

• n taking a measureable action or achieving a

predetermined performance target (CGD, 2009) Donor Central government Local government Private insurer

$

Service providers Facilities / NGOs Central government Local governments

RBF takes many forms…

Payers Payees

19

Impact evaluation of P4P in Health

 Effects on non-contracted

services provided?

 Impact on equity of

services?

 Impact on out-of-pocket

expenditures?

Impact Evaluations:

Will the introduction of pay for performance for patient services impact the quality and performance of hospitals/health centers?

20

P4P in Rwanda

21

National P4P scheme to supplement input-based budgets with bonus payments based on the quantity and quality of maternal and child healthcare (14 output indicators)

“Paying Primary Health Care Centers for Performance in Rwanda” (Basinga et al., 2010)

22

Impact on proportion of institutional deliveries

36.3 49.7 34.9 55.6

30.0 40.0 50.0 60.0 Baseline (2006) Follow up (2008) Proportion of of institutional deliveries

Control facilities Treatment (PBF facilities)

7.3 % increase due to PBF

“Paying Primary Health Care Centers for Performance in Rwanda” (Basinga et al., 2010)

23

Impact on quality of prenatal care

• 0.10
• 0.13

0.15

• 0.15
• 0.10
• 0.05

0.00 0.05 0.10 0.15 0.20 Baseline (2006) Follow up (2008) Standardized Prenatal effort score

Control facilities Treatment (PBF facilities)

15 % Standard deviation increase due to PBF

P4P in Rwanda: Lessons

24

 No effect on the number of prenatal care visits or on

immunization rates. Need demand side incentives?

 Greatest effect on services that had the highest

payment rates and are under the provider control.

 Financial performance incentives can improve both use

• f and quality of health services.

 An equal amount of financial resources without the

incentives would not have achieved the same gain in

• utcomes.

25

P4P experiment in Uganda

Lundberg, Marek and Pariyo (2007) Contracting for Primary Health Care in Uganda

 Three arms: (1) performance bonus, (2) freedom to

allocate base grant, (3) control.

 Performance bonus based on meeting targets (pre-

natal care, attended births, immunizations, etc.)

 Results after 18 months:

bonus facilities perform the same as or worse

than others;

freedom-to-allocate facilities perform

significantly better than others.

26 P4P experiment in Uganda

 This performance bonus didn’t work.

Amounts not large enough? Scheme too complicated? Not enough time? (some evidence of learning

curve)

 Facilities may allocate budgets more effectively than

the Ministry of Health

Priority thematic areas

1.

Conditional Cash Transfers

2.

Paying For Performance in Health

3.

School Accountability

4.

Malaria Control

5.

Active Labor Market Programs / Youth Employment

6.

HIV/AIDS Prevention

7.

Early Childhood Development

27

Strategies for improving learning outcomes

 Supply side policies (classrooms, textbooks,

extra tutors, teacher training)

 Demand side policies (CCTs?)  Child endowments/readiness to learn -- ECD,

de-worming

 Accountability reforms: Change the

environment in which decisions are made

28

WDR 2004 framework of accountability

29

Rationale for Education Accountability Schemes

30

How do we transform this teacher…. …into this teacher?

School Accountability

31

3 Modalities on School Accountability

32

• 1. School-based Management (SBM)
• 2. Information for accountability projects
• 3. Teacher contracting and pay for performance

projects

How can SBM improve outcomes?

33

Those at the local level have more or better

information and different objectives

Key decisions about school personnel Key decisions about spending Changes in the educational process Resource mobilization Changes in the school climate

School Based Management in Mexico

(Apoyo a la Gestión Escolar, AGE)

34

 Grants + training to Parent Associations (PA).  AGEs increase school autonomy through improved

mechanisms for participation of directors, teachers and PA in the management of the schools.

 Effects on: School failure, repetition, drop out  Use phasing in of program for evaluation

School Base Management in Mexico

35 Gertler P. et al, 2007. “Empowering Parents to Improve Education: Evidence from Rural Mexico”

School Base Management in Mexico

36 Gertler P. et al, 2007. “Empowering Parents to Improve Education: Evidence from Rural Mexico”

School Base Management in Mexico

37 Gertler P. et al, 2007. “Empowering Parents to Improve Education: Evidence from Rural Mexico”

3 Modalities on School Accountability

38

• 1. School-based Management
• 2. Information for accountability (IFA)projects
• 3. Teacher contracting and pay for performance

projects

How could IFA improve outcomes?

39

 Empower parents and communities vis a vis local &

school authorities

– Increased importance of education in the political sphere – Increased attention to results valued by parents  Enable better technocratic decision-making

• Help guide allocation decisions

 Under “strong accountability” systems: – Threat of sanction will spur increased teaching effort  When there is school choice: – “Market-pressure” for performance

Newspaper Campaign in Uganda

– In 1995 only 13 % of non-wage recurrent spending reached schools. – From 1996-2001, amounts and dates of grant to local governments published in local and national newspapers.

40

Impact of information campaign

Source: Reinikka and Svensson (2001), Reinikka and Svensson (2003a)

IMPACT ON LEAKAGE: A school close to a newspaper

• utlet experienced 25

percentage points less leakage compared to a school one standard-deviation (30 km) further away from a newspaper

• utlet.

IMPACT ON LEARNING: A one standard deviation reduction (30 km) in distance lead to a 0.09 standard deviation increase in average test scores.

41

Information Campaigns in India

42

 Village-level education committees (VECs), representing parents

and village leaders, created by state education policy

 Presumption that information campaigns are needed to

empower parents/communities

 Two sets of experiments  PRATHAM --Banerjee et al. (2008) “Pitfalls of participatory

programs: Evidence from a Randomized Evaluation in Education in India”

 Three states in India –WB + Nike Foundation --Pandey et al. (2008)

Community Participation in Public Schools: Impact of Information Campaigns in three Indian States.

Impact of Information Campaign

(Banerjee et al.)

43

 Program:

 Advocacy and information about VECs =(1);  (1)+teaching how “learning reports cards” are done=(2);  (2)+ Teaching: Capacity Building to Directly Improve Learning (reading

classes)=( 3)

 Information campaigns ((1) and (2)) had no effect on

VEC activism or on learning outcomes

 Trained teaching volunteers (3) did have a significant

effect on “reading” learning outcomes

Impact of Information Campaign

(Pandey et al.)

REGION Teacher Attendance Teacher Activity % Received Uniform Participation rate of parents in school inspections Scholarships Students received Improvement in Reading UP

 11 %

Unchanged

 18 %

(girls)

 .11  33 %

(general caste)

 27 %

(Grade 3)

MP

Unchanged

 30%  14 %

(general caste)

 .02  26 %

(SC/ SC)

 14 %

(Grade 3)

Karnataka

Unchanged Unchanged N/A Unchanged N/A

 15 %

(Grade 4)

Pandey, Priyanka et al. Community Participation in Public Schools: Impact of Information Campaigns in three Indian States. South Asia Human

• Development. Presentation. World Bank. 2008.

44

44

3 Modalities on School Accountability

45

• 1. School-based Management
• 2. Information for accountability projects
• 3. Teacher contracting and pay for performance

projects

How could TI improve outcomes?

46

 Accreditation and merit-based incentives lead to

better teaching professionals joining the system.

 Decentralized hiring and monitoring of teacher

performance leads to higher attendance rates.

 Career advancement incentives stimulate sustained

effort, improving student performance.

 Pay-for-performance incentivizes improved quality

and quantity of teaching improving student test scores.

Teacher Pay Incentives in Andra Pradesh (Muralidharan and Sundararaman, 2008)

47

 Three policies

 Group and individual pay incentives based on average

improvement in test scores of all students

 Block grants to schools (extra teachers, more inputs)  Assessments conducted by an independent NGO

 Results (two rounds)

 Incentives increase student performance by 0.22 SD  1st year both incentives work well. 2nd year individual

incentives work better

 Additional inputs improve scores only by 0.08 SD  Teachers liked the program

Priority thematic areas

1.

Conditional Cash Transfers

2.

Paying For Performance in Health

3.

School Accountability

4.

Malaria Control

5.

Active Labor Market Programs / Youth Employment

6.

HIV/AIDS

7.

Early Childhood Development

48

Look for more information….

49

www.worldbank.org/hdchiefeconomist

..and remember Primo Levi

‘It is better to be content with more modest and

less inspiring truths that are laboriously conquered, step by step, with no shortcuts, by studying, discussion and reasoning, and that can be verified and demonstrated'’

50