How do we interpret evidence? Aidan Coville, Economist, World Bank - - PowerPoint PPT Presentation

▶

Apr 24, 2023 231 likes •519 views

How do we interpret evidence? Aidan Coville, Economist, World Bank DIME Kenya Evidence Forum - June 14, 2016 Using Evidence to Improve Policy and Program Designs Impact of sewerage connec0ons? diarrhea 25% Sewerage connecPons 20% in an

SLIDE 1

Kenya Evidence Forum - June 14, 2016

Using Evidence to Improve Policy and Program Designs

How do we interpret “evidence”?

Aidan Coville, Economist, World Bank DIME

SLIDE 2

Impact of sewerage connec0ons?

20% 12%

0% 5% 10% 15% 20% 25% Before AMer

diarrhea Sewerage connecPons in an informal seQlement Impact?

SLIDE 3

Is this the impact of providing sewerage?

A. YES
B. NO

YES NO

65% 35%

SLIDE 4

Impact of DIME

260 509

100 200 300 400 500 600 2012 2013

Nigeria GDP (USD Billions) DIME started working with Nigerian Ministry of Health

SLIDE 5

Is this the impact of DIME?

A. YES
B. NO

YES NO

76% 24%

SLIDE 6

Monitoring

Collects data on treatment

groups to:

Track performance over Pme
Tell us whether we’re moving

in the right direcPon

Describes what is happening,

but not why or whether this is because of our intervenPon Impact EvaluaPon

Assigns intervenPon to treatment

and control groups to:

Measure counterfactual: what

would have happened?

Establish causal link between

intervenPon and outcome

So we can measure impact of

policy, compare instruments, make beQer decisions and improve policy over Pme

Monitoring vs. impact evalua0on

SLIDE 7

2 4 6 8 10 12 14 Before AMer Control Group Treatment Group

(+) Impact of the program??

(+) Impact of other (external) factors

The Value of a Control Group

SLIDE 8

2 4 6 8 10 12 14 Before AMer Control Group Treatment Group

(+) Impact of the program

(+) Impact of other (external) factors

The Value of a Control Group

SLIDE 9

Counterfactual criteria

Treated & comparison groups

Have idenPcal iniPal average characterisPcs (observed and

unobserved)

The only difference is the treatment
Therefore the only reason for the difference in outcomes is due to

the treatment

We generate these equal groups through randomized allocaCon: on

average control and treatment groups are the same

SLIDE 10

The produc0on of useful research

It needs to be rigorous and relevant It needs to be used

SLIDE 11

Tablet Ques0onnaire

How can we produce evidence to more effecPvely inform

policy?

What are your beliefs about the effecPveness of parPcular

intervenPons?

What aQributes of a study do think make the study most useful/

relevant?

SLIDE 12

Imagine a rouleCe table

SLIDE 13

Now to make it simpler let’s imagine it has 100 numbers… (apparently this really exists…)

SLIDE 14

Ques0on: what number do we think it will land on?

SLIDE 15

In this case all numbers are equally likely between 1 and 100

SLIDE 16

For each choice we make, we can es0mate the chances / probability that the actual number lands below or above our choice

SLIDE 17

If we choose “75”, what is the probability that a number higher than 75 will come up?

SLIDE 18

What If we choose “50”? There is a 50% probability that a number higher than 50 will come up.

SLIDE 19

Finally, let’s try 25… what is the probability of a larger number being drawn?

SLIDE 20

You’ve just had the stats course you always wanted to avoid in university!

The rouleQe example describes percenCles
The median or 50th percenCle is the number where the true result

has a 50% chance of being above or below that number

The 75th percenPle is the number where the true result has a 25%

chance of being above or a 75% chance of being below that number

And…
The 25th percenPle is the number where the true result has a 75%

chance of being above or a 75% chance of being below that number

And so on…

SLIDE 21

Es0ma0ng a program impact

In reality, we make esPmates (or best guesses) all the Pme…
While gambling is pure luck, our esPmates are based on informed

guesses using our available knowledge.

Let’s take a shot at esPmaPng the impact of a program:
Later you will learn about the impact of providing subsidies to rural

households to connect to the electricity grid in Kenya.

Let’s guess what the impact will be of reducing the cost of connecPon from

35,000 to KSh 25,000 ?

SLIDE 22

Median (50 Median (50th

th p

per ercen cen0le): le): What is your median es0mate of the program impact (the number where you think its equally likely to have been a larger or smaller impact)?

A. 10% connected
B. 20% connected
C. 30% connected
D. 40% connected
E. 50% connected
F. 60% connected
G. 70% connected

10% connected 20% connected 30% connected 40% connected 50% connected 60% connected 70% connected

26% 10% 10% 3% 3% 39% 10%

SLIDE 23

What is your es0mate of the 75 75th

th p

per ercen cen0le le of the program impact (the number where you think there’s a 25% chance that the TRUE impact is bigger and a 75% chance it is a smaller impact)?

A. 10% connected
B. 20% connected
C. 30% connected
D. 40% connected
E. 50% connected
F. 60% connected
G. 70% connected

10% connected 20% connected 30% connected 40% connected 50% connected 60% connected 70% connected

13% 23% 7% 17% 17% 10% 13%

SLIDE 24

What is your es0mate of the 25 25th

th p

per ercen cen0le le of the program impact (the number where you think there’s a 75% chance that the TRUE impact is bigger and a 25% chance it is a smaller impact)?

A. 10% connected
B. 20% connected
C. 30% connected
D. 40% connected
E. 50% connected
F. 60% connected
G. 70% connected

10% connected 20% connected 30% connected 40% connected 50% connected 60% connected 70% connected

27% 21% 15% 12% 9% 6% 9%

SLIDE 25

Now you’re ready to take this on your own…

ONLINE OPTION:

hQp://bit.ly/1UMOkME

Note the capital and small leQers

SLIDE 26

SLIDE 27