LESSONS FROM AFRICA Annette N. Brown www.3ieimpact.org Outline of - - PowerPoint PPT Presentation

www.3ieimpact.org Annette N. Brown

IMPACT EVALUATION: LESSONS FROM AFRICA

2012 AFDB Evaluation Week: Evaluation for Development

www.3ieimpact.org Annette N. Brown

Outline of talk

• What is impact evaluation
• Colin Powell’s rules redux: rules for good

impact evaluation studies

• Evidence I: Institutions
• Evidence II: Roads
• Evidence III: Sustainability

www.3ieimpact.org Annette N. Brown

Impact Evaluation

An evaluation that examines the counterfactual in order to attribute changes in outcomes to an intervention.

www.3ieimpact.org Annette N. Brown

Good Impact Evaluation

A good impact evaluation explores all the links in the causal chain. This involves using mixed methods. Not quantitative and qualitative, but counterfactual and factual.

www.3ieimpact.org Annette N. Brown

Mixed methods

Howard White, “The use of mixed methods in randomized control trials”, 3ie 2012.

www.3ieimpact.org Annette N. Brown

Colin Powell’s rules

• Tell me what you know
• Tell me what you don’t know
• Then tell me what you think
• Always distinguish which is

which

www.3ieimpact.org Annette N. Brown

Good impact evaluation rules

Colin Powell’s rules redux

• Tell me the counterfactual
• Tell me the factual
• Tell me what you don’t know
• Then tell me what you think
• Always distinguish which is which

www.3ieimpact.org Annette N. Brown

Evidence I: Institutions

“Effective conflict resolution is essential to

• rder and development. Every local

business deal, land boundary, or loan offers a possible conflict. When local dispute institutions function well, they prevent and resolve conflict.” (p. 1)

“Building institutions at the micro-level: Results from a field experiment in property dispute and conflict resolution” Christopher Blattman, Alexandra Hartman, and Robert Blair, Working paper, October 2012.

www.3ieimpact.org Annette N. Brown

The intervention

http://reedsinthewind.blogspot.com/2012/10/week-four-not-cote-divoire-but- liberia.html

www.3ieimpact.org Annette N. Brown

The counterfactual

All residents Residents with a dispute

Any serious dispute Results in violence Any un- resolved dispute Results in violence Resolved Resolved informal mech- anism Satisfied with

• utcome

Treatment 0.005

• 0.010
• 0.020** -0.028

0.080*** 0.042* 0.065* Mean, control group 0.221 0.122 0.0698 0.554 0.683 0.193 0.578 ATE as %

• f control

2%

• 8%
• 28%
• 5%

12% 22% 11% # obs 5,411 5,411 5,411 1,210 1,210 1,210 1,210 R-squared 0.173 0.145 0.063 0.121 0.073 0.065 0.085 Table 2: Impacts on land disputes (adapted)

www.3ieimpact.org Annette N. Brown

The factual

http://globalvoicesonline.org/2011/11/20/liberia-giving-free-press-a-second- chance/

www.3ieimpact.org Annette N. Brown

The factual

“Those who participated in the workshop, now they are available and it is okay for a person to go to them, and these people use the same skills they learned in the workshop and talk to both people involved in the dispute to solve it.” (p. 33)

www.3ieimpact.org Annette N. Brown

What we don’t know

• Whether the intervention changes norms
• Whether the intervention changes

attitudes

• Longer term effects, both the progression
• f resolutions over time and whether the

impacts of the trainings are sustainable long after the intervention is completed

www.3ieimpact.org Annette N. Brown

What we think

“The intervention in Liberia did not reduce the level or severity of conflict in the space

• f one or eighteen months, but it did change

dramatically the manner and success of conflict resolution, especially with the most common and important form of conflict: land disputes.” (p. 36)

www.3ieimpact.org Annette N. Brown

Evidence II: Roads

“Better roads lower transaction costs associated with agricultural activities and in doing so have the potential to reduce the costs of acquiring inputs, increase output prices, reduce the impact of shocks, and permit entry into new, more profitable activities.”

• Stefan Dercon, Daniel O. Gilligan, John Hoddinott, and Tassew

Woldehanna, “The Impact of Agricultural Extension and Roads on Poverty and Consumption Growth in Fifteen Ethiopian Villages,” American Journal of Agricultural Economics 91(4) (November 2009)

• Stefan Dercon and John Hoddinott, “Livelihoods, Growth, and Links to

Market Towns in 15 Ethiopian Villages,” FCND Discussion Paper 194, July 2005.

www.3ieimpact.org Annette N. Brown

The intervention

http://web.worldbank.org/WBSITE/EXTERNAL/NEWS/0,,contentMDK:21275 250~menuPK:34471~pagePK:34370~piPK:34424~theSitePK:4607,00.html

www.3ieimpact.org Annette N. Brown

The counterfactual

Poor Consumption Growth Log consumption

• 0.131**
• 0.396**

Access to all- weather road

• 0.069**

0.163** Sample size 4,771 4,771 Table 3. Determinants of Consumption Growth and Poverty Status: Basic Results (adapted)

www.3ieimpact.org Annette N. Brown

The factual

http://www.lilsoak.com/?tag=fair-trade-jewelry

www.3ieimpact.org Annette N. Brown

What we don’t know

The evaluation does not explore the non- economic linkages between villages and local market towns (e.g. access to healthcare, government officials, etc.) that may account for some of the decrease in poverty and/or increase in consumption growth.

www.3ieimpact.org Annette N. Brown

What we think

“Better roads in these localities make it easier for households to access local market towns, which in turn are linked to larger urban centers.” (p. 1018) “Public investments have the potential to play important roles in facilitating increased growth and faster poverty reduction.” (p. 1019)

www.3ieimpact.org Annette N. Brown

Evidence III: Sustainability

“The difficulty of scaling up is particularly pressing when the programme is complex, where the intervention stretches over time, and where the

• rganisational setting and skills of the service

providers matter greatly for the quality of the services provided.” (p. 809)

Lars Ivar Oppedal Berge, Kjetil Bjorvatn, Kartika Sari Juniwaty, and Bertil Tungodden, “Business Training in Tanzania: From Research-driven Experiment to Local Implementation,” Journal

• f African Economies, Vol. 21, number 5 (2012).

www.3ieimpact.org Annette N. Brown

The intervention

Promotion of Rural Initiatives and Development Enterprises (PRIDE)

http://info.textileexchange.org/te-farm-blog/bid/119393/5-Ambitions-for- Organic-Cotton-in-2012

www.3ieimpact.org Annette N. Brown

The counterfactual

Figure 3. Business Knowledge

www.3ieimpact.org Annette N. Brown

The factual

Figure 1. Attendance per Session.

www.3ieimpact.org Annette N. Brown

The factual

Figure 2. Subjective Evaluation

www.3ieimpact.org Annette N. Brown

What we don’t know

• We do not have measures of the final

business impacts—the incomes of the entrepreneurs or the revenues and profits from their businesses.

• We do not know why the internal training

performed poorly relative to the external

• training. Characteristics of trainers?

Perceptions of quality going in?

www.3ieimpact.org Annette N. Brown

What we think

“We conclude that the estimated effect of research-led interventions should be interpreted as an upper bound of what can be achieved when scaling up such interventions locally.” (p. 809)

www.3ieimpact.org Annette N. Brown

Conclusion

Good impact evaluation studies can help us to understand what works, how, and why by examining both counterfactual and factual evidence.