Presentation on Impact Measurement Concern Worldwide - 22 nd July - - PowerPoint PPT Presentation

IIMDA - Initiative on Impact Measurement in Development Aid

Presentation on Impact Measurement

Concern Worldwide - 22nd July 2008 Michael King Trinity College Dublin

IIMDA - Initiative on Impact Measurement in Development Aid

Outline

• 1. Establishment of IIMDA
• 2. Overview of Impact Measurement

Methodologies

• 3. Case Study 1: Electoral Violence in

Nigeria

IIMDA - Initiative on Impact Measurement in Development Aid

• 1. IIMDA
• Trinity College/IIIS Initiative led by

Pedro Vicente and Michael King with research partners in UCD, Oxford University, Columbia University and Tufts University.

• IIMDA Mission

– Research: To develop understanding

• n the impact of innovative

development aid projects. – Dissemination: To communicate lessons learned from impact evaluations to policy makers and development practitioners. – Training: To develop the impact evaluation skills of development practitioners and researchers based in Ireland and partner countries.

Speakers at the IIMDA Conference on Impact Evaluation (April 17th 2009)

IIMDA - Initiative on Impact Measurement in Development Aid

IIMDA Developments 2009

• Ongoing project: Counteracting Illegal Behaviour

during Elections in Mozambique: The Role of Observers and Cell Phones (in partnership with Oxford and Tufts Universities).

• Forthcoming Projects: Agreed to develop joint

research projects with Trocáire, Christian Aid Ireland and Microfinance Opportunities in Washington DC.

• Wish to explore the demand for intensive

training courses in impact evaluation.

IIMDA - Initiative on Impact Measurement in Development Aid

Undertake more systematic and in-depth sectoral and country-wide assessments of aid

3

Support more innovative activities and ensure that lessons are learned

9

Support efforts to expand and extend NGO assessments of impact

8

Revisit the historical distinction between humanitarian and development aid

7

Extend, deepen and improve assessments of humanitarian aid.

6

Ensure aid assessment includes analysis of policy coherence

5

Take stock of the relationships between aid and the achievement of the MDGs

4

Judiciously expand the number, scope and range of assessments of discrete aid projects.

2

Ensure that comprehensive assessments of aid impact encompass both ex ante and systemic as well as ex post assessments.

1

Dissemination Training Research

ABIA Recommendations and IIMDA

Source: ABIA ((2007) Measuring Impact: The Global and Irish Aid Programme

IIMDA - Initiative on Impact Measurement in Development Aid

2. Impact Measurement (Evaluation)

• Impact evaluation studies estimate the

effect of an intervention on final welfare

• utcomes, rather than assessing the

project outputs or the project implementation process.

IIMDA - Initiative on Impact Measurement in Development Aid

Evaluation and Monitoring Framework

Outcomes

(Impact)

Outputs Inputs Activities

Welfare effects on target group directly or partly attributable to the project. Goods and services produced by the project Actions and tasks carried out to transform inputs into outputs Financial, human and material resources required

Impact Evaluation Implementation Monitoring Quantitative Quantitative Methodologies Qualitative Qualitative

Source: Adopted from Asian Development Bank (2006) Impact Evaluation: Methodological and Operational Issues.

IIMDA - Initiative on Impact Measurement in Development Aid

Log Frame

• The logical framework (“log frame”)

considers the higher-order project results (“outcomes” and “impacts”) but alone lacks the tools to estimate robustly project impact.

• In practice, the log frame is generally used

to assess the operational flow of project inputs and outputs and assessing the validity of the underlying assumptions.

IIMDA - Initiative on Impact Measurement in Development Aid

Log Frame and Impact Evaluation Methods

IIMDA - Initiative on Impact Measurement in Development Aid

Alternative Impact Evaluation Methodologies

IIMDA - Initiative on Impact Measurement in Development Aid

Random Assignment

IIMDA - Initiative on Impact Measurement in Development Aid

Types of Randomization Mechanisms

• 1. Lottery design: Applicants are simply

randomly assigned to the treatment group and the control group.

– Used when the program resources can cover only a fraction of eligible participants and there is no reason to discriminate among applicants. – Perceived as a fair and transparent means to decide who will receive the program benefits and who will not.

IIMDA - Initiative on Impact Measurement in Development Aid

Types of Randomization Mechanisms

• 2. Phase-in design can be used when the

program is designed to cover the entire eligible population but in a phased in manner.

– Everyone is told that they will end up receiving the program benefits but at different times. – The timing of actually receiving the program benefits can be randomized.

IIMDA - Initiative on Impact Measurement in Development Aid

Types of Randomization Mechanisms

• 3. Encouragement design is used when

everyone is immediately eligible to receive the program benefits and there is enough funding to cover the entire eligible population, but not everyone will necessarily take advantage of the program.

– Can randomly select a group of people and offer them specific incentives to encourage them to use the program. – The remaining population without the incentives is used as the control group.

IIMDA - Initiative on Impact Measurement in Development Aid

Issues in RCTs

• For RCTs to be successful treatment and control groups

must remain clean and unchanged as originally designed throughout the study period. The following three issues require specific attention.

– Attrition: This is the situation when some members of the treatment or control group or both, drop out from the sample. – Spillover: This occurs when the program impact is not confined to program participants. – Noncompliance: This is another complication in randomized

• evaluation. It occurs when some members of the treatment

group do not get treated or get treated improperly, or some members of the control group get treated.

IIMDA - Initiative on Impact Measurement in Development Aid

Double Difference

IIMDA - Initiative on Impact Measurement in Development Aid

Case Study

• Votes and Violence: Evidence from an

Informational Campaign in Nigeria (Pedro Vicente and Paul Collier)

• Evaluation of the impact of a randomised

anti-violence grassroots campaign.

• Conducted by ActionAid International

Nigeria

IIMDA - Initiative on Impact Measurement in Development Aid

Project Background

• Political violence in African elections

In April and May 2003, at least one hundred people were killed and many more injured during federal and state elections in Nigeria. In a number of locations, elections simply did not take place as groups of armed thugs linked to political parties and candidates intimidated and threatened voters in order to falsify results.

– June 2004, Human Rights Watch

IIMDA - Initiative on Impact Measurement in Development Aid

Research Questions

• 1. Can a NGO-conducted campaign against

electoral violence help in undermining this phenomenon?

• 2. Does violence diminish voter turnout?
• 3. Who are the candidates that are perceived as

‘violence-prone’?

• 4. Who are the people with whom the campaign

works particularly well?

• 5. Are there network effects of the campaign

against electoral violence?

IIMDA - Initiative on Impact Measurement in Development Aid

Campaign Details

• 1. Distribution of materials
• T-shirts (3,000, i.e. 1 for every 2 households on

average), Caps (3,000), Hijabs (1,000)

• Posters (3,000), Stickers (3,000), Leaflets

(5,000)

• 2. Road shows (using jingles in 3 languages, Yoruba,

Hausa, Pidgin English)

• 3. At least one Community Meeting at each of the 12

locations

• 4. At least one Popular Theatre representation at each
• f the 12 treated locations
• 5. Opportunity to report violence on election day.

IIMDA - Initiative on Impact Measurement in Development Aid

Figure 5: The Time Frame of the Experiment asking about violence asking about violence April: Elections Time Line Past Year

Pre-Election Survey Post-Election Survey Anti- Violence Campaign

Feb: May/Jun: Jan/Feb:

Campaign Timeline

Independent Journalists in all 24 EAs

1200 Respondents (96% Re-surveyed)

IIMDA - Initiative on Impact Measurement in Development Aid

Campaign Literature

IIMDA - Initiative on Impact Measurement in Development Aid

IIMDA - Initiative on Impact Measurement in Development Aid

Popular Theatres

IIMDA - Initiative on Impact Measurement in Development Aid

Anti-violence Roadshows

IIMDA - Initiative on Impact Measurement in Development Aid

• 1. Focused on AfroBarometer’s sampled EAs
• 2. Treatment areas chosen randomly
• 3. Non-random component: 2 states in each of the main 3 regions of

the country (Southwest, Southeast, North) chosen by looking at recent history of violence

• 4. Control EAs chosen by choosing the closest EAs from AB’s list, in

the same state, of the same type (Large Urban, Small Urban, Rural)

• 5. Households and corresponding respondents within a census area

chosen randomly using AB’s standard techniques

• 6. Conditions for sampling within a household: 18+ years, residence

in EA

Sampling Process

IIMDA - Initiative on Impact Measurement in Development Aid

IIMDA - Initiative on Impact Measurement in Development Aid

Table A2: Differences across Treatment and Control Areas - Demographics (Panel plus Oversample)

Control Treatment Difference Number of Observations

• 0.26

1.01 0.03 0.74 0.04 0.05

• 0.03

0.04 0.52 0.34

• 0.04

0.17

• 0.06

0.11 0.09 0.09 0.12 0.13

• 0.09

0.13 0.31 0.2 household size 6.43 6.46 1500 Basic Demographics female 0.5 0.5 1500 age 32.95 32.69 1497 Ethnicity and Religion schooling (0-9) 4.31 4.83 1500 single 0.38 0.42 1449 married 0.58 0.55 1449 muslim 0.34 0.25 1499 religious intensity (1-6) 4.76 5.07 1485 christian 0.62 0.74 1499 hausa 0.16 0.1 1500 yoruba 0.32 0.28 1500 igbo 0.07 0.16 1500

Comparison: Treatment & Control

IIMDA - Initiative on Impact Measurement in Development Aid

1) Effect of the Campaign on Violence

where:

• VC is violence and crime,
• i, l, t are subscripts for individuals, locations, and time (before / after)
• T is a binary variable with value 1 for treated locations,
• X is a vector of controls (demographic, attitudinal), potentially time-varying
• Y is a geographical fixed effect.

2) Effect of the Campaign on Voting

where:

• V is strict voting behavior (intended-before and actual reported-after)

ilt l l l it ilt

T ft eT dt cY bX a VC ε + + + + + + = *

ilt l l l it ilt

T nt mT kt jY hX g V ε + + + + + + = *

Model Specification

IIMDA - Initiative on Impact Measurement in Development Aid

Violence/Crime Outcome Variables:

• Perceptions of politically-motivated violent behavior:
• in general, respondent’s area, from the top and from the

bottom

• Actual political violence events: Local Journals
• compiled by independent local journalists at all surveyed

locations

• including descriptions of political violence events

(sources: town meetings, police)

IIMDA - Initiative on Impact Measurement in Development Aid

• Empowerment against political violence
• Perceptions
• ‘Postcard’ variable

IIMDA - Initiative on Impact Measurement in Development Aid

Sample Results

Electoral Strategies

• Violent intimidation was identified to

reduce voter turnout: A 0.78pp decrease in turnout per 1pp decrease in electoral security – strategy linked primarily to

• pposition groups.
• Vote-buying and ballot fraud tend to be

associated with the incumbent.

IIMDA - Initiative on Impact Measurement in Development Aid

Impact of Campaign on Violence

• The campaign improved responses on the level of political freedoms

(by 9-12%).

– Political Freedoms: You feel secure ‘joining a party’, on ‘voting freely’,

• n ‘being free from insecurity’.
• The campaign improved responses on free and fair elections (by

18%-21%).

• Campaign lead to a 12% decrease for ‘conflict within the local

community’.

• Postcard (empowerment to counteract violence) 27% of subjects

returned postcard to the mail. Treated - 14% more frequently send card.

• Campaign lead to a 13% reduction in the intensity of electoral

violence (significant at the 10% level)

• Not so clear effects on actual violent events.

IIMDA - Initiative on Impact Measurement in Development Aid

• 20%
• 10%

0% 10% 20% 30% 40% 50%

Conflict within Community Gang Activity Physical Intimidation Intensity (Journals) Security Empower Knowledge Postcard

% Subjective Scale (exc. Postcard)

Chart 2: Impact of the Anti-Violence Campaign Differenced After-Before the Campaign (exc. Postcard)

Treatment Control

Source: Own Data (Surveys, Journals - Nigeria 2007). % for Postcards refers to % postcards received back in the mail.

IIMDA - Initiative on Impact Measurement in Development Aid

Impact of Campaign on Voting

• No plan to vote: Number of people who changed their

mind and voted was 10% higher in treated areas.

• Turnout increases by 13% for each unit of ‘security from

violence originated by politicians at the local level’ on a 6 point scale.

• The campaign reduced the vote for the 3rd party in the

Presidential Elections by 4% because of the ‘vote against violent politicians’. Voting not altered for the largest 2 parties.

• In the regional elections campaign reduced the vote for
• pposition parties and in some cases strengthened the

vote for the incumbent.

IIMDA - Initiative on Impact Measurement in Development Aid

• 15%
• 10%
• 5%

0% 5% 10% 15% 20% 25% 30%

Change in Turnout Change in PDP Change in ANPP Change in AC Change in Incumbent Change in 2nd Party Change in 3rd Party

Chart 3: Impact of the Campaign on Voting Behavior Differenced After-Before the Campaign

Treatment Control

Source: Own Data (Surveys - Nigeria 2007).

Presidential Elections Gubernatorial Elections

IIMDA - Initiative on Impact Measurement in Development Aid

Other Key Results

• Campaign had the largest impact on

smaller households, working outside the home, not owning land, and members of the Yoruba people.

• Networks seem to be playing a relevant

role in reinforcing and spreading the direct message of the campaign.

IIMDA - Initiative on Impact Measurement in Development Aid

Thank you. Any Questions on…. IIMDA? Methodologies? Case Study?