IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS ARIEL FISZBEIN - PowerPoint PPT Presentation

USING EVIDENCE TO INFORM POLICY: IMPACT EVALUATION OF HUMAN DEVELOPMENT PROGRAMS ARIEL FISZBEIN CHIEF ECONOMIST, HUMAN DEVELOPMENT WORLD BANK Overseas Development Institute May 4, 2010

Motivation 2 ‘Since it is difficult to distinguish the good from the bad prophet, we must be suspicious of all prophets: it is better to avoid revealed truths, even if we feel exalted by their simplicity and splendor , even if we find them comfortable because they come at no cost. It is better to be content with more modest and less inspiring truths that are laboriously conquered, step by step, with no shortcuts, by studying, discussion and reasoning, and that can be verified and demonstrated' ’ Primo Levi

The evaluation problem 3  Impact: difference in outcome with and without program. Cannot observe simultaneously.  with and without comparisons deceptive if participation depends on individual attributes that influence outcomes  before/after comparisons can be misleading if other things happened during the period  Need to estimate counterfactual (control or comparison)  Treated & control have same characteristics on average,  Only reason for difference in outcomes is the intervention

Multiple design options 4  Experimental (often using phase in)  Quasi-experiments/non-experimental  Regression Discontinuity (RD)  Difference in difference – panel data  Other (Instrumental Variables, matching, etc)  In all cases, these will involve knowing the rule for assigning treatment  Advantages of prospective design

IE at the World Bank 5 273 Impact Evaluations… As of April 28, 2010 200 175 175 150 125 98 100 75 50 25 0 ...Ongoing ...Under discussion

Strong focus on HD areas 6 The World Bank Impact Evaluation Program: Thematic Areas Local Social Fin. & Dev. Protection Priv. Sec. 10% 20% 8% Agric. 6% Urban Upgr. 6% Health 17% Gov . 4% HD thematic Other areas: 58% 9% ECD of total World Education & Nutr. 13% Bank IEs 7%

Strategic themes 7 The World Bank Impact Evaluation program on Human Development Thematic areas Conditional Cash Transfers 20 School Accountability 19 Active Labor Market Programs 16 Early Childhood Development 14 HIV 13 Malaria 9 Pay For Performance in Health 9 Education, Other 6 Number of Ongoing Impact Evaluations 0 5 10 15 20 25

Building a global evidence base on the impact of HD programs 8 The World Bank Impact Evaluation program on Human Development Regions 54 Africa 21 South Asia 17 Latin America & Caribbean 10 East Asia & the Pacific 3 Middle East & North Africa 1 Europe & Central Asia 0 10 20 30 40 50 60 Number of Ongoing Impact Evaluations

Priority thematic areas 9 Conditional Cash Transfers 1. Paying for Performance in Health (P4P) 2. School Accountability 3. Malaria Control 4. Active Labor Market Programs / Youth Employment 5. HIV/AIDS Prevention 6. Early Childhood Development 7.

CCTs have become very popular…. 10

CCTs have become very popular…. 11

First generation evaluation questions 12  Impact on consumption and poverty  Side effects?  Impacts on service utilization  Impacts on human development outcomes

Impacts on consumption and poverty 13 Mexico Nicaragua Colombia Honduras (1999) (2002) (2006) (2002) Average transfer (% 20% 30% 13% 11% of per capita consumption) Impact on per capita 8.3%** 20.6%** 10%** 7%* consumption (%) Impact: headcount 1.3** 5.3** 2.9* -- index (% points) Impact: poverty gap 3.0** 9.0** 7.0** 2.0* (% points) Impact: sqd. poverty 3.4** 8.6** 2.2** 2.0* gap (% points)

Impacts on school enrollment 14 Age range Baseline Impact (% Size of enrollment points) transfer 8-13 91.7% 2.1** 17% Colombia 14-17 63.2% 5.6*** 6-15 60.7% 7.5*** 3-7% Chile Ecuador 6-17 75.2% 10.3*** 10% Grade 0-5 94.0% 1.9 Mexico Grade 6 45.0% 8.7*** 20% Grade 7-9 42.5% 0.6 7-13 72.0% 12.8*** 30% Nicaragua Cambodia Grade 7-9 65.0% 31.3*** 2% 11-18 44.1% 12.0** 1% Bangladesh 10-14 29.0% 11.1*** 3% Pakistan

Impacts on health service utilization 15 Age Baseline Impact Size of range level transfer (% points) <24 n.a. 22.8** 17% Colombia months 24-48 n.a. 33.2*** months Chile 0-6 years 17.6% 2.4 7% 3-7 years n.a. 2.7 10% Ecuador 0-3 years 44.0% 20.2*** 9% Honduras 0-3 years 69.8% 8.4 20% Mexico 0-3 55.4% 13.1* 27% Nicaragua

Impacts on education and health outcomes 16  Only mixed success in terms of improving final outcomes in education and health: Education: Increases in school enrollment and years of completed schooling have not come hand- in-hand with improved learning outcomes Health: Some programs, but my no means all, have improved child nutrition (as measured by height-for-age, hemoglobin status)

Second generation evaluation questions 17  Role of conditions: Testing CCTs vs. UCTs (Burkina Faso, Malawi, Morocco and Yemen)  Does it matter who receives the cash? Mothers vs. fathers (Burkina Faso, Morocco and Yemen); Girls vs. parents in Malawi  Design of payments: Cambodia; Colombia

Priority thematic areas 18 Conditional Cash Transfers 1. Paying For Performance in Health 2. School Accountability 3. Malaria Control 4. Active Labor Market Programs / Youth Employment 5. HIV/AIDS Prevention 6. Early Childhood Development 7.

Results- based financing (RBF) ≈ Pay -for-performance (P4P) 19 Provision of payment for Transfer of money or material goods conditional the attainment of well- on taking a measureable action or achieving a defined results predetermined performance target (CGD, 2009) RBF takes many forms… Payers Payees Donor Service providers Central government Facilities / NGOs $ Local government Central government Private insurer Local governments

Impact evaluation of P4P in Health 20 Will the introduction of pay for performance for patient services impact the quality and performance of hospitals/health centers? Impact Evaluations :  Effects on non-contracted services provided?  Impact on equity of services?  Impact on out-of-pocket expenditures?

P4P in Rwanda 21 National P4P scheme to supplement input-based budgets with bonus payments based on the quantity and quality of maternal and child healthcare (14 output indicators)

“ Paying Primary Health Care Centers for Performance in Rwanda ” (Basinga et al., 2010 ) 22 Impact on proportion of institutional deliveries Proportion of of institutional deliveries 60.0 55.6 7.3 % increase due to PBF 49.7 50.0 40.0 36.3 34.9 30.0 Baseline (2006) Follow up (2008) Control facilities Treatment (PBF facilities)

“ Paying Primary Health Care Centers for Performance in Rwanda ” (Basinga et al., 2010) 23 Impact on quality of prenatal care 0.20 Standardized Prenatal effort score 0.15 0.15 0.10 15 % Standard deviation increase due to PBF 0.05 0 0.00 -0.05 -0.10 -0.10 -0.13 -0.15 Baseline (2006) Follow up (2008) Control facilities Treatment (PBF facilities)

P4P in Rwanda: Lessons 24  No effect on the number of prenatal care visits or on immunization rates. Need demand side incentives?  Greatest effect on services that had the highest payment rates and are under the provider control.  Financial performance incentives can improve both use of and quality of health services.  An equal amount of financial resources without the incentives would not have achieved the same gain in outcomes.

P4P experiment in Uganda Lundberg, Marek and Pariyo (2007) Contracting for Primary Health Care in Uganda 25  Three arms: (1) performance bonus, (2) freedom to allocate base grant, (3) control.  Performance bonus based on meeting targets (pre- natal care, attended births, immunizations, etc.)  Results after 18 months:  bonus facilities perform the same as or worse than others;  freedom-to-allocate facilities perform significantly better than others.

26 P4P experiment in Uganda  This performance bonus didn’t work.  Amounts not large enough?  Scheme too complicated?  Not enough time? (some evidence of learning curve)  Facilities may allocate budgets more effectively than the Ministry of Health

Priority thematic areas 27 Conditional Cash Transfers 1. Paying For Performance in Health 2. School Accountability 3. Malaria Control 4. Active Labor Market Programs / Youth Employment 5. HIV/AIDS Prevention 6. Early Childhood Development 7.

Strategies for improving learning outcomes 28  Supply side policies (classrooms, textbooks, extra tutors, teacher training)  Demand side policies (CCTs?)  Child endowments/readiness to learn -- ECD, de-worming  Accountability reforms: Change the environment in which decisions are made

WDR 2004 framework of accountability 29

Rationale for Education Accountability Schemes 30 How do we transform this teacher…. …into this teacher?

School Accountability 31

3 Modalities on School Accountability 32 1. School-based Management (SBM) 2. Information for accountability projects 3. Teacher contracting and pay for performance projects