Results in Complex Contributions 8 November 2012 09.15-12.00 - - PowerPoint PPT Presentation

Capturing and Communicating Results in Complex Contributions

8 November 2012 09.15-12.00

Development Talks

Susanne Wadstein

Director Department for Organisational Development Sida

Development Talks

Time Item 09.15 - 09.30 Welcome (Susanne Wadstein, Sida) 09.30 - 09.45 Definition of results (Lennart Peck, Sida) 09.45 - 10.30 Capturing results (Michael Woolcock, World Bank) 10.30 - 10.40 Coffee Break 10.40 - 10.55 “What about the results” (Charlotte Örnemark, Nordic Consulting Group) 10.55 - 11.10 An Example: “Vietnam, Laos & Sri Lanka: Evaluation of long- term development co-operation” (Annika Nordin Jayawardena, Sida) 11.10 - 12.00 Panel discussion & questions

RESULTS

OECD/DAC Definition of Results

“The output, outcome or impact … intended or unintended… positive and/or negative…

• f a development intervention”

Results of what?

For example:

• a single project or programme
• Sweden’s cooperation with a country
• the joint efforts of partner country and donors
• a strategy or a policy
• influencing factors in the environment

Results at what point?

For example:

• after X number of years
• “primary”, “secondery”, “tertiary” etc. effects
• “output”, “outcome” and “impact”

Results for whom?

For example:

• The individual citizen
• ”Women”, ”rural population”, ”the poor” etc.
• Citizens in a village, region or country

Results in terms of what?

For example:

• Products, goods and services
• Changes of individual or organizational behaviour,

attitudes, knowledge etc.

• Changes in the level of service provision,

protection of human rights etc.

• Changes in terms of individual wellbeing

An example: Results of electrification

.

Results of Results at what point Results in terms of Results for whom The results of the Swedish contribution… …after completion… … were new connections… … for 2 000 households The results of joint donor cooperation with government… … in an impact perspective… … was access to television, better security and improved business… … for Mrs. x and her family The result of access to donor funds… … after some time… … was a reduction of government’s own spending… … on citizens needing electricity

Degrees of causation

• Necessary and sufficient
• Necessary but not sufficient
• Sufficient but not necessary
• Neither sufficient nor necessary - but contributing

Different causal patterns

One cause – One result One cause – Multiple results Multiple causes – One result Multiple causes – Multiple results

Attribution – Contribution – Confusion

” to contribute to create conditions that will enable poor people to improve their lives”.

Four key questions:

• Did the intervention make a difference?
• How has the intervention made a difference?
• To what extent can a specific result be attributed

to the intervention?

• Will the intervention work elsewhere?

Some of our challenges

• Establishing attribution/contribution
• Capturing both the qualitative and the quantitative
• Aggregating results
• Generalising
• Communicating complex things in a simple way

Development Talks

Michael Woolcock

Lead Social Development Specialist The World Bank

“But How Generalizable is That?” A Framework for Assessing the Internal and External Validity of Complex Development Interventions

Michael Woolcock

World Bank and Harvard University

mwoolcock@worldbank.org

SIDA, Stockholm

8 November 2012

Theory is when you know everything but nothing works. Practice is when everything works but nobody knows

• why. We have put together theory

and practice: nothing is working… and nobody knows why!

Albert Einstein

Overview

• Background
• From assessing IV to EV in ‘complex’ projects

– Importance of trajectories, theories of change

• A framework for integrating

– Design elements – Causal density – Implementation dynamics – Context compatibility

• Implications for EV, and for case studies

Primary source material

• Bamberger, Michael, Vijayendra Rao and Michael Woolcock (2010)

“Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development”, in Abbas Tashakkori and Charles Teddlie (eds.) Handbook of Mixed Methods (2nd revised edition) Thousand Oaks, CA: Sage Publications, pp. 613-641

• Barron, Patrick, Rachael Diprose and Michael Woolcock (2011)

Contesting Development: Participatory Projects and Local Conflict Dynamics in Indonesia New Haven: Yale University Press

• Woolcock, Michael (2009) ‘Toward a Plurality of Methods in Project

Evaluation: A Contextualized Approach to Understanding Impact Trajectories and Efficacy’ Journal of Development Effectiveness 1(1): 1-14

• Woolcock, Michael and Arathi Rao (2012) ‘But How Generalizable is

That? A Framework for Assessing the External Validity of “Complex” Development Interventions’ Mimeo

Background

• Rising obsession with ‘causality’, RCTs

– Pushed by donors, foundations (e.g., Gates) – Yet also serious critique

• In medicine: Rothwell (2005)
• In philosophy: Cartwright (2011)
• In economics: Deaton (2010), Heckman, Ravallion…

– Especially as it pertains to EV

• Engber (2011) on ‘Black 6’ (biomedical research)
• Heinrich et al (2011) on ‘WEIRD’ people (social psychology)
• Across time, space, groups, scale
• How to assess ‘social’, ‘participatory’ projects?

– Barron, Diprose and Woolcock (2011) – Mansuri and Rao (2012) – How to compare roads, irrigation, ‘empowerment’?

22

A typology of interventions…

‘Simple’ Nets, pills, roads ‘Complicated’ Agriculture, microcredit ‘Complex’ Education, health ‘Chaotic’ Local justice reform, CDD Theory

 Predictive precision  Cumulative

knowledge

 Subject/object gap

High Mechanisms

 # Causal pathways  # of ‘people-based’

transactions

 # Feedback loops

Few Outcomes

 Plausible range  Measurement

precision

Low Many Wide Narrow

23

…or aspects of interventions (e.g., health)

‘Simple’ Build clinics ‘Complicated’ Inoculation (logistics) ‘Complex’ Ambulatory care ‘Chaotic’ Adolescent sexual behavior Theory

 Predictive precision  Cumulative

knowledge

 Subject/object gap

High Mechanisms

 # Causal pathways  # of ‘people-based’

transactions

 # Feedback loops

Few Outcomes

 Plausible range  Measurement

precision

Low Many Wide Narrow

Contesting Development

Participatory Projects and Local Conflict Dynamics in Indonesia

PATRICK BARRON RACHAEL DIPROSE MICHAEL WOOLCOCK

Yale University Press, 2011

Summary of findings

Type of Impact Context Capacity Low High Program Functionality Program Functionality Low High Low High Direct Forums (places)

• ++
• -*

Facilitators (people) Indirect Group Relations ++ + +++ Behavioral +++ + Normative + +++

* While we noted higher rates of KDP-triggered conflict in high capacity areas, such conflict is much less likely to escalate and/or turn violent. Hence negative impacts are greater in low capacity areas, where program functionality is poor.

Lessons for evaluating ‘chaotic’ projects

In Evaluation 101, we assume… Impact = f (Design) | Selection, Confounding Variables

Adequate for ‘simple’ interventions with a ‘good-enough’ counterfactual. But this is inadequate for assessing ‘complex’ interventions: * design is multi-faceted (i.e., has high ‘causal density’) * interaction with context is pervasive, desirable * implementation quality is vital (high discretion) * trajectories of change are probably non-linear (perhaps unknowable ex ante)

Lessons for evaluating ‘chaotic’ projects

Impact = f ([DQ, CD], IE, CC) | SE, CV, RE

DQ = Design quality (weak, strong) CD = Causal density (low, high), or ‘discretionary mechanisms’ (few, many; tight, loose; seen, unseen) IE = Implementation effectiveness (low, high) CC = Context compatibility (resistant, supportive) SE = Selection effects (non-random placement, participation) CV = Confounding variables RE = Reasoned expectations (where by when?) * In Social Development projects (cf. roads, immunizations): CD is high, loose, often unseen; IE and CC are variable; RE is often unknown (unknowable?)

Pervasive problem

• SD projects are inherently very complex, thus:

– Very hard to isolate ‘true’ impact – Very hard to make claims about likely impact elsewhere – Understanding how (not just whether) impact is achieved is also very important

• Process Evaluations, or ‘Realist Evaluations’, can be most

helpful (see work of Ray Pawson, Patricia Rogers et al)

Time t = 0 t = 1 Net Impact

From IV to EV in complex interventions: Understanding impact trajectories

Time t = 0 t = 1 Net Impact

Understanding impact trajectories

“Same” impact claim, but entirely a function of when the assessment was done

Time t = 0 t = 1 Net Impact

Understanding impact trajectories

A B C

Time t = 0 t = 1 Net Impact

Understanding impact trajectories

A B C

?

D t = 2

Initial thoughts on external validity

• Logic of (elite) research and most ‘development

effectiveness’ debates leads to a focus on Design

– The better to identify, replicate ‘best practices’, ‘tools’ – Preferably validated via an RCT (the ‘gold standard’)

Initial thoughts on external validity

• Logic of (elite) research and most ‘development

effectiveness’ debates leads to a focus on Design

– The better to identify, replicate ‘best practices’, ‘tools’ – Preferably validated via an RCT (the ‘gold standard’)

• To better address IV and EV, especially of complex

interventions, we need (a) an operational definition

• f “complexity”, (b) a pragmatic typology of Designs,

Implementation and Contexts, integrated with (c) an explicit theory of change

– The better to specific the conditions under which certain

• utcomes are likely to be observed

– Enhancing frequency and rigor of case studies is crucial

(1) Projects Four analytical questions

Is your activity… Does producing successful outcomes from your policy…. TI: Transaction Intensive? Require many agents to act or few, over extended time periods? LD: Locally Discretionary? Require that the implementing agents make finely based distinctions about the “state of the world”? Are these distinctions difficult for a third party to assess? KT: Based on Known Technology? Require that agents innovate to achieve desired outcomes ? HS: High Stakes? Require that the agents resist large temptations to do something besides implement the policy that would produce the desired

• utcome?

Classification of “activities” in health

LD: Locally Discretionary TI: Transaction Intensive HS: High Stakes KT: Known ‘Technology’ Iodization of salt No No No Yes Vaccinations No Yes No Yes Ambulatory curative care Yes Yes No(ish) Yes Regulation of private providers Yes Yes Yes Yes Encouraging preventive health Yes Yes No No Technocratic (implementation light; policy decree) Logistical (implementation intensive, but easy) Implementation Intensive ‘Downstream’ (of services) Complex (implementation intensive, motivation hard), need (continuous?) innovation Implementation Intensive ‘Upstream’ (of obligations)

(2) Implementation

How things stand…

Time

Design Implementation Evaluation

Effort Prestige Resources

(2) Implementation

…What we need (esp. for ‘complex’ projects)

Time

Design Implementation Evaluation

Effort Prestige Resources

(2) Implementation

• Importance vastly underappreciated, misunderstood

– Mosse (2005)… (Hard to name much else in development)

• Usually assume weak implementation is a function of

agents’ skills, behaviors…

– i.e., inadequate ‘training’, pervasive ‘corruption’, lack of ‘political will’, low ‘work ethic’

• Which can ‘fixed’ by engaging agents in extensive ‘capacity building’
• …but agents are employed by organizations, which are

part of systems

– Need to understand interactions between all three – ‘Escaping Capability Traps’ (Andrews, Pritchett, Woolcock 2012)

• ‘Implementation Quality’ – Strong, Weak

(3) Contexts

• Obviously enormously varied…
• Understanding requires time, patience, analysis

– political economy, anthropology, sociology etc

• Key questions, for present purposes:

– Do prevailing (most administratively salient) elites fundamentally support or resist the initiative? – Is the project design, and the implementation apparatus(domestic and/or international), broadly perceived as legitimate?

• ‘Context Compatibility’ – Positive, Negative

Putting it all together

Project Design Features Technocratic Logistical Implementation Intensive (‘Downstream’) Implementation Intensive (‘Upstream’) Complex Implementation Quality Strong Weak Strong Weak Strong Weak Strong Weak Strong Weak Context Compatibility +

• +
• +
• +
• +
• +
• +
• +
• +
• +
• External Validity

High Low

Even with low EV interventions, the ideas and processes behind them may still travel well

Putting it all together

Project Design Features Technocratic Logistical Implementation Intensive (‘Downstream’) Implementation Intensive (‘Upstream’) Complex Implementation Quality Strong Weak Strong Weak Strong Weak Strong Weak Strong Weak Context Compatibility +

• +
• +
• +
• +
• +
• +
• +
• +
• +
• External Validity

High Low

Even with low EV interventions, the ideas and processes behind them may still travel well

Putting it all together

Project Design Features Technocratic Logistical Implementation Intensive (‘Downstream’) Implementation Intensive (‘Upstream’) Complex Implementation Quality Strong Weak Strong Weak Strong Weak Strong Weak Strong Weak Context Compatibility +

• +
• +
• +
• +
• +
• +
• +
• +
• +
• External Validity

High Low High Low

Even with low EV interventions, the ideas and processes behind them may still travel well

Coffee break

10.30-10.40

Development Talks

Charlotte Örnemark

Institutional development, learning and participatory M&E specialist Nordic Consulting Group

What about the results?

Lessons from long-term process support to strengthen results-based management (RBM) for Swedish framework NGOs operating in Western Balkans Charlotte Örnemark, cor@ncg.dk

”It’s all about the project logic...”

Lesson 1: RBM in self-contained ’PCM bubbles’ tend to be fictive, responding more to donor or ’aid system needs’ than to social change processes that are embedded in contexts that by nature are chaotic and non-linear.

”It’s all about aggregation ...”

Look

• k!

! A total al of 529 phone ne call lls made to gover vernment ent

• fficial

ficials!! !!

Lesson 2: Increased centralised demand for results can acutally lead to a decrease in overall system performance, unless accompanied by adapted approaches and investments (more of the same won’t necessarily work).

”It’s all about behavioural change ...”

Lesson 3: We may not be able to make people change, but we can influence and track how external influence makes patterns of interactions evolve towards positive (or negative) alignment for social change. Given that...

• People influence people.
• Institutions organise people and ideas into

systems.

• Systems are often bounded, context-driven

and interact with other systems...

Social change processes Example: Change agents (boundary partners) Partner

• rgs.

Swedish FOs Sida /UD Inter- national commit- ments FO National Gov Inter- national commit- ments National reforms / poverty reduction plans and results frame-works UNDAF MDGs

Sphere of interest Sphere of influence Sphere of control Infl./interest

PO PO PO PO PO PO

• Core org.

values

• Key

indicators (reflecting core values) Country strategy / country programme

• Added

value to partners Rights- holders Duty- bearers Media, civil society Reduce discrim- ination Women’s move- ment

Select Partner Organisation (PO) carefully based on joint values + potential for influence Mutual /joint accountability commitments Mutual /joint accountability commitments Boundary partners Few but clearly defined / in line with FO mandate

What kind of indicators?

Lesson 4: It’s always good to be SMART in one’s results assessments. But for interventions in the area of complex social change processes it is even more important to be REAL.

The ’REAL’ results framework for complex social change

Lesson 5: Practical implications

• Invest in results frameworks where a

multitude of actors feed in meaningful information in a timely manner (used for system-wide learning, engagement)

• Use flexible/adaptive monitoring tools

that can capture emerging results patterns

• ”Give the power back”: Demystify RBM
• Shift from SMART to REAL
• Make RBM a day-to-day concern, not an

administrative or ’control’ function

Development Talks

Annika Nordin Jayawardena

Deputy Director Department for Programme Co-operation Sida

Lessons from 136 years of Swedish aid

Evaluation of Swedish aid to Vietnam, Laos and Sri Lanka

• Unique opportunity to analyse more than 130 years of experiences and impacts of long-

term programmes and projects

• Possibility to compare experiences from different countries with different preconditions

Select conclusions from the evaluations

• Swedish development co-operation had an impact on poor people’s lives in all three

countries, but the results varies, and in hindsight not all interventions were strategic.

• Swedish aid contributed to pulling millions of Vietnamese out of poverty.
• The contribution to poverty reduction is likely to have been less successful in Laos and
• nly marginal in Sri Lanka.

1 Background

The evaluators draw conclusions about long-term Swedish development co-operation

Period: 1967-2011

• Total aid: 3.45 billion USD1

General conclusions:

• Focus on poverty was

limited, but created preconditions for effective future aid

• Strategic support
• Sensitive issues

(“motvindsfrågor”)

• Development co-op

facilitated relations beyond development co-op

… … …

Period: 1958-2009

• Total aid: 1.2 billion USD1

General conclusions:

• Effective beginning to co-op
• Principled stance was

allowed because of long- term commitment

• Good intentions but

ineffective in relation to intended impacts on national level Period: 1974-2010

• Total aid: 700 million USD1

General conclusions:

• Slow start due to ineffective

aid (tied aid, limited capacity and poor co-ordination between donors)

• Focus on capacity

development

2 Main conclusions

1 Constant prices, 2008.

Overall:

• Only marginal impact on

poverty reduction Overall:

• Contributed to lifting large part
• f population out of poverty

Overall:

• Considerable impact on

poverty reduction

Macro economic analysis

• Development and aid trends
• Assessment of changes in quality of life, particularly

income poverty

Qualitative analysis

• Country specific needs & aid effectiveness principles
• Thematic & longitudinal case studies
• Interviews, focus groups, analysis of available docs

Longitudinal evaluation makes it possible to highlight long-term development trends

• Trends identified by quantitative analysis on

macro level to give overall picture of development in respective countries;

• Qualitative analysis of longitudinal case

studies highlight development on micro level;

• Subsequent analysis of whether observations on

macro level are supported by observations on micro level and vice versa.

Evaluation method

Phase out Micro Macro Phase in

Validation

3 Methodological overview

Development co-operation with Vietnam: From paper mill to local democracy

Bai Bang Doi Moi Chia Se Phase I: 1969-85 Phase II: 1986-99 Phase III: 2000-11

• Largest Swedish aid

project ever – Did not meet acute needs – Mixed evidence for effective aid – Today privately run paper mill

• Wider understanding for

sustainable use of natural resources

• Created pre-conditions for

continued development co-

• peration
• Sweden only donor with

good relations – Could support economic reform in initial phases

• Swedish economic model

used as tool – Several examples of economic models given – Policy dialogue, education & technical support

• Experimental & long-term

co-operation for local democracy – Last programme in succession of projects/ programmes in same area as Bai Bang – Has affected national development plans with rights-based approach,

• incl. women’s rights

4 Case study - Vietnam

Experiences and lessons learned from Swedish development co-operation

• Strike a balance between principles and pragmatism.
• Understand underlying causes of aid impacts and be flexible. Drivers of change

and actors.

5 Experiences & lessons learned

• Long-term approaches build trust and relations that allow for engagement in

sensitive issues.

• Sustainable results require development of competences, capacity and

institutions. Principles & pragmatism

1

Long-term approach & presence

2

Sustainability

4

Learning & flexibility

3

Development Talks

Panelists Organisation Michael Woolcock The World Bank Charlotte Örnemark Nordic Consulting Group Janet Vähämäki Pontus Modéer Rightshouse Angelica Broman Sida Martina Fors Sida Moderator Lennart Peck Sida

Thank you!