Ra ndo mize d Co ntro lle d T ria ls, De ve lo pme nt E c o no mic - - PowerPoint PPT Presentation

▶
ra ndo mize d co ntro lle d t ria ls de ve lo pme nt e c
SMART_READER_LITE
LIVE PREVIEW

Ra ndo mize d Co ntro lle d T ria ls, De ve lo pme nt E c o no mic - - PowerPoint PPT Presentation

Jun 22, 2023 390 likes •831 views

Ra ndo mize d Co ntro lle d T ria ls, De ve lo pme nt E c o no mic s a nd Po lic y Ma king in De ve lo ping Co untrie s Esthe r Duflo De p a rtm e nt o f Ec o no m ic s, MIT C o -Dire c to r J-PAL [Jo int w o rk w ith Ab hijit Ba ne rje e

slide-1
SLIDE 1

Ra ndo mize d Co ntro lle d T ria ls, De ve lo pme nt E c o no mic s a nd Po lic y Ma king in De ve lo ping Co untrie s

Esthe r Duflo De p a rtm e nt o f Ec o no m ic s, MIT C o -Dire c to r J-PAL [Jo int w o rk w ith Ab hijit Ba ne rje e a nd Mic ha e l Kre m e r]

slide-2
SLIDE 2

Ra ndo mize d c o ntro lle d tria ls ha ve g re a tly e xpa nde d in the la st two de c a de s

  • Ra ndo mize d c o ntro lle d T

ria ls we re pro g re ssive ly a c c e pte d a s a to o l fo r po lic y e va lua tio n in the US thro ug h ma ny b a ttle s fro m the 1970s to the 1990s.

  • I

n de ve lo pme nt, the ra pid g ro wth sta rts a fte r the mid 1990s

– K

re me r e t a l, studie s o n K e nya (1994)

– PROGRE

SA e xpe rime nt (1997)

  • Sinc e 2000, the g ro wth ha ve b e e n ve ry ra pid.

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

2

slide-3
SLIDE 3

Ca me ro n e t a l (2016): RCT in de ve lo pme nt

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

3

50 100 150 200 250 300 1975 1980 1985 1990 1995 2000 2005 2010 2015

Pub lic a tio n Ye a r

Figure 1: Number of Published RCTs

slide-4
SLIDE 4

BRE AD Affilia te s do ing RCT

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

4

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1980 or earlier 1981-1990 1991-2000 2001-2005 2006-today

PhD Year

Figure 4. Fraction of BREAD Affiliates & Fellows with 1 or more RCTs

* Total Number of Fellows and Affiliates is 166.

slide-5
SLIDE 5

T

  • p Jo urna ls

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

5

slide-6
SLIDE 6

Ma ny se c to rs, ma ny c o untrie s

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

6

slide-7
SLIDE 7

Why ha ve RCT ha d so muc h impa c t?

  • F
  • c us o n ide ntific a tio n o f c a usa l e ffe c ts (a c ro ss the

b o a rd)

  • Asse ssing E

xte rna l Va lidity

  • Ob se rving Uno b se rva b le s
  • Da ta c o lle c tio n
  • I

te ra tive E xpe rime nta tio n

  • Unpa c k impa c ts

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

7

slide-8
SLIDE 8

F

  • c us o n I

de ntific a tio n… a c ro ss the b o a rd!

  • T

he ke y a dva nta g e o f RCT wa s pe rc e ive d to b e a c le a r ide ntific a tio n a dva nta g e

  • With RCT

, sinc e tho se who re c e ive d a tre a tme nt a re ra ndo mly se le c te d in a re le va nt sa mple , a ny diffe re nc e b e twe e n tre a tme nt a nd c o ntro l must b e due to the tre a tme nt

  • Mo st c ritic isms o f e xpe rime nt a lso fo c us o n limits to

ide ntific a tio n (impe rfe c t ra ndo miza tio n, a ttritio n, e tc . ) o r thing s tha t a re no t ide ntifie d e ve n b y ra ndo mize d tria ls (distrib utio n o f tre a tme nt e ffe c ts, e ffe c ts e lse whe re ).

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

8

slide-9
SLIDE 9

F

  • c us o n I

de ntific a tio n… a c ro ss the b o a rd!

  • Be fo re the e xplo sio n o f RCT

in de ve lo pme nt, a lite ra ture

  • n RCT

in la b o r a nd pub lic fina nc e ha s tho ug ht o f o the r wa ys to ide ntify c a usa l e ffe c ts

  • I

n de ve lo pme nt e c o no mic s, the re wa s a jo int de ve lo pme nt o f the two lite ra ture s (na tura l e xpe rime nt a nd RCT ), whic h ha s ma de b o th lite ra ture s stro ng e r, a nd pe rha ps le ss diffe re nt tha n we initia lly tho ug ht the y wo uld b e :

– Na tura l e xpe rime nts think o f RCT

a s a na tura l b e nc hma rk (no t just a n hypo the tic a l g o ld sta nda rd).

– De ve lo pme nt o f me tho ds to g o b e yo nd simple c o mpa riso n o f

tre a tme nt a nd c o ntro l in e xpe rime nts, a nd ric he r de sig ns

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

9

slide-10
SLIDE 10

E nc o ura g e me nt de sig n

Diffe re nc e in ta ke up c a use d b y e nc o ura g e me nt Pe o ple who ta ke up pro g ra m

slide-11
SLIDE 11

F

  • c us o n I

de ntific a tio n… a c ro ss the b o a rd!

  • Be fo re the e xplo sio n o f RCT

in de ve lo pme nt, a lite ra ture o n RCT in la b o r a nd pub lic fina nc e ha s tho ug ht o f o the r wa ys to ide ntify c a usa l e ffe c ts

  • I

n de ve lo pme nt e c o no mic s, the re wa s a jo int de ve lo pme nt o f the two lite ra ture s (na tura l e xpe rime nt a nd RCT ), whic h ha s ma de b o th lite ra ture s stro ng e r, a nd pe rha ps le ss diffe re nt tha n we initia lly tho ug ht the y wo uld b e :

–

Na tura l e xpe rime nts think o f RCT a s a na tura l b e nc hma rk (no t just a n hypo the tic a l g o ld sta nda rd). E xtre me ly we ll ide ntifie d no n ra ndo mize d studie s.

–

De ve lo pme nt o f me tho ds to g o b e yo nd simple c o mpa riso n o f tre a tme nt a nd c o ntro l in e xpe rime nts, a nd ric he r de sig ns

  • Ultima te ly, the a dva nta g e o f RCT

in te rms o f ide ntific a tio n is a ma tte r o f de g re e , ra the r tha n a funda me nta l diffe re nc e .

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

11

slide-12
SLIDE 12

Why ha ve RCT ha d so muc h impa c t?

  • F
  • c us o n ide ntific a tio n o f c a usa l e ffe c ts (a c ro ss the

b o a rd)

  • Asse ssing E

xte rna l Va lidity

  • Ob se rving Uno b se rva b le s
  • Da ta c o lle c tio n
  • I

te ra tive E xpe rime nta tio n

  • Unpa c k impa c ts

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

12

slide-13
SLIDE 13

E xte rna l Va lidity

  • Will re sults o b ta ine d so me whe re g e ne ra lize e lse whe re ?
  • A fre q ue nt c ritic ism o f RCT

is tha t the y do n’ t g ua ra nte e e xte rna l va lidity

  • Whic h is q uite rig ht, b ut it is no t like the y a re le ss

e xte rna lly va lid…

  • And b e c a use the y a re inte rna lly va lid, a nd b e c a use yo u

c a n c o ntro l whe re the y will ta ke pla c e :

– c o mpa re d a c ro ss c o nte xts. –

the y c a n b e purpo se fully run in diffe re nt c o nte xts

– Pre dic tio n c a n b e ma de o f wha t the e ffe c ts o f re la te d pro g ra ms

c o uld b e .

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

13

slide-14
SLIDE 14

Ba ye sia n Hie ra rc hic a l Mo de lling o f a ll the MF re sults : Pro fits Me a g e r (2015)

slide-15
SLIDE 15

Ba ye sia n Hie ra rc hic a l Mo de ling -- Me ta a na lysis (c o nsumptio n)

15

slide-16
SLIDE 16

E xa mple 2: T a rg e ting the Ultra Po o r Pro g ra m: Co o rdina te d e va lua tio n in se ve ra l c o untrie s

Beneficiary

Productive asset transfer Health Consumption support Technical skills training Home visits Savings

Ba ne rje e e t a l, 2015

16

slide-17
SLIDE 17

Co untry b y c o untry re sults: Asse ts

Ba ne rje e e t a l, 2015

  • 0.1

0.2 0.5 0.8

Endline 1 Endline 2

Asse t c ha ng e (sta nda rd de via tio ns)

17

slide-18
SLIDE 18

Co untry b y c o untry re sults: Co nsumptio n

  • 5%

0% 5% 10% 15% 20%

Endline 1 Endline 2

% Cha ng e in pe r c a pita c o nsumptio n

18

slide-19
SLIDE 19

Struc ture d Spe c ula tio n

  • Ultima te ly, if the re sults a re simila r it is nic e , b ut if the y a re

diffe re nt the e x-po st a na lysis is spe c ula tive .

  • Ba ne rje e , Cha ssa ng , Sno wb e rg (2016) pro po se to b e

e xplic it a b o ut suc h spe c ula tio n, a nd tha t re se a rc he rs sho uld pre dic t wha t the e ffe c t ma y b e fo r o the r inte rve ntio ns, o r in o the r c o nte xts.

  • T

his c a n the n mo tiva te running suc h e xpe rime nts, a nd g ue sse s c a n b e fa lsifie d.

  • E

xa mple : Dupa s (2014)—E ffe c t o f sho rt run sub sidie s o n lo ng run a do ptio n de pe nd o n the timing o f c o sts a nd b e ne fits, a nd ho w q uic kly unc e rta inty a b o ut the m is re so lve d: this a llo ws he r to c la ssify the g o o ds.

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

19

slide-20
SLIDE 20

Why ha ve RCT ha d so muc h impa c t?

  • F
  • c us o n ide ntific a tio n o f c a usa l e ffe c ts (a c ro ss the

b o a rd)

  • Asse ssing E

xte rna l Va lidity

  • Ob se rving Uno b se rva b le s
  • Da ta c o lle c tio n
  • I

te ra tive E xpe rime nta tio n

  • Unpa c k impa c ts

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

20

slide-21
SLIDE 21

Ob se rving uno b se rva b le s

  • So me thing s simply c a nno t b e o b se rve d in the wild, with

na tura lly o c c urring va ria tio n

  • Ne g a tive inc o me ta x e xpe rime nt wa s de sig ne d a s a n

e xpe rime nt to se pa ra te inc o me a nd sub stitutio n e ffe c ts

  • Ma ny e xpe rime nts in de ve lo pme nt a re de sig ne d like wise

to c a pture suc h e ffe c ts:

– K

a rla n Zinma n Ob se rving Uno b se rva b le s

– Co he n Dupa s a nd Ashra f Dupa s Sha piro : se le c tio n a nd tre a tme nt

e ffe c t o f pric e s.

– Be rtra nd e t a l. Co rruptio n in driving lic e nc e s in De lhi.

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

21

slide-22
SLIDE 22

Why ha ve RCT ha d so muc h impa c t?

  • F
  • c us o n ide ntific a tio n o f c a usa l e ffe c ts (a c ro ss the

b o a rd)

  • Asse ssing E

xte rna l Va lidity

  • Ob se rving Uno b se rva b le s
  • Da ta c o lle c tio n
  • I

te ra tive E xpe rime nta tio n

  • Unpa c k impa c ts

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

22

slide-23
SLIDE 23

I nno va tive da ta c o lle c tio n

  • I

nno va tive da ta c o lle c tio n do e s no t re q uire a n e xpe rime nt.

  • But e xpe rime nts ha ve two fe a ture s whic h ha ve

mo tiva te d c re a tivity in me a sure me nt

– We kno w pre c ise ly wha t we a re trying to me a sure : pa yo ff to the

pe rso n who is de sig ning the q ue stio nna ire

– We kno w tha t the re will like ly b e e no ug h po we r to me a sure suc h

e ffe c ts

  • As a re sult, lo ts o f inno va tio n in me a sure me nt:

– Bo rro wing fro m o the r fie lds : psyc ho lo g y, po litic a l sc ie nc e ,

a g ric ulture , we b sc ra ping , we a ra b le te c ho lo g y,

– I

nve nting ne w me tho ds: e .g . Olke n 2007

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

23

slide-24
SLIDE 24

Why ha ve RCT ha d so muc h impa c t?

  • F
  • c us o n ide ntific a tio n o f c a usa l e ffe c ts (a c ro ss the

b o a rd)

  • Asse ssing E

xte rna l Va lidity

  • Ob se rving Uno b se rva b le s
  • Da ta c o lle c tio n
  • I

te ra tive E xpe rime nta tio n

  • Unpa c k impa c ts

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

24

slide-25
SLIDE 25

I te ra tive e xpe rime nta tio ns

  • So me g re a t na tura l e xpe rime nt le a ve us with so me

una nswe re d q ue stio ns:

– Why a re e lite sc ho o l no t wo rking fo r the ma rg ina l c hild? – Why a re (so me ) c ha rte r sc ho o l wo rking so we ll?

  • One o the r a dva nta g e o f e xpe rime nts is tha t o ne is ne ve r

stuc k with o ne pa rtic ula r surprising a nswe r: yo u c a n c o ntinue to e xpe rime nt in the sa me se tting till yo u ha ve so me c la rity.

  • E

xa mple : Duflo , K re me r, Ro b iso n multi-ye a r wo rk o n fe rtilize r.

– Pe o ple do n’ t use fe rtilize r, e ve n tho ug h it is pro fita b le – One se t o f e xpe rime nt o n fina nc ing – One se t o n le a rning a nd so c ia l le a rning .

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

25

slide-26
SLIDE 26

Why ha ve RCT ha d so muc h impa c t?

  • F
  • c us o n ide ntific a tio n o f c a usa l e ffe c ts (a c ro ss the

b o a rd)

  • Asse ssing E

xte rna l Va lidity

  • Ob se rving Uno b se rva b le s
  • Da ta c o lle c tio n
  • I

te ra tive E xpe rime nta tio n

  • Unpa c k impa c ts

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

26

slide-27
SLIDE 27

Unpa c k impa c ts

  • T

his is a re la te d po int, b ut mo re na rro wly fo c use d o n po lic y de sig n.

  • T

he re a re ma ny ma ny po ssib le wa ys to de sig n a pa rtic ula r pro g ra ms

  • Usua lly, o ne ve rsio n is trie d o ut
  • But if it wo rks wha t wa s e sse ntia l?

– E

ffo rt to unpa c k Co nditio na l Ca sh T ra nsfe r

– E

xa mple o f do ing e ve rything a t o nc e : Ra skin pro g ra m, I ndo ne sia

  • Ma ny pe o ple do no t re c e ive the ric e the y a re e lig ib le fo r, o r o ve r pa y
  • Wo uld tra nspa re nc y he lp?

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

27

slide-28
SLIDE 28

T he y distrib ute 4 ve rsio n o f a c a rds to e lig ib le villa g e rs in 378 villa g e s, ra ndo mly c ho se n o ut

  • f 572

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

28

slide-29
SLIDE 29

Othe r so urc e s o f va ria tio n a nd re sults

  • T

he y a lso va rie d:

–

Pub lic (c o mmo n) kno wle dg e o f the pro g ra m

–

F ra c tio n o f pe o ple who g e t the phyisic a l c a rd

  • Re sults:

–

Ma king the c a rd distrib utio n pub lic kno wle dg e ma ke s it mo re e ffe c tive

–

T he physic a l c a rd ma tte r: info rma tio n (in the fo rm o f list) a lo ne is no t suffic ie nt

–

(Pe rc e ptio n o f) a c c o unta b ility do e s no t se e m to ma ke muc h o f a diffe re nc e .

  • T

he g o ve rnme nt de c ide d to sc a le the ve rsio n o f the c a rd with mo st info a nd the list to 65 millio n b e ne fic ia rie s!

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

29

slide-30
SLIDE 30

Wha t ha s b e e n the po lic y impa c ts o f RCT s?

  • I

s the Ra skin Ca se uniq ue o r unusua l?

–

A study de sig ne d b y re se a rc he rs with se ve ra l tre a tme nts a nd a n unde rlying e c o no mic mo de l, de stine d to b e pub lishe d in a to a c a de mic jo urna l

–

b ut tha t still ha d la rg e po lic y impa c t

  • So me ha ve a rg ue d tha t the re se a rc h impa c t o f RCT

ha s po te ntia lly c o me a t the e xpe nse o f re a l-wo rld impa c t:

–

Re se a rc he rs’ a nd po lic y ma ke rs inte re sts ma y dive rg e

–

Re se a rc h slo w do wn the pro c e ss o f ite ra tio n

  • E

vide nc e

–

Out o f 700 pro je c ts o n g o ing o r c o mple te d o n the J-PAL site , the re a re

  • nly 9 sto ry o f sc a le up o r po lic y impa c t.

–

Ho we ve r, this is no t a c e nsus o f J-PAL study (o r o f RCT ). Sto ry se le c te d fo r hig h impa c ts : the sum o f pe o ple re a c he d is a b o ut 200 millio n.

30

slide-31
SLIDE 31

Ove r 200 millio n pe o ple re a c he d thro ug h sc a le -ups o f pro g ra ms e va lua te d b y J-PAL re se a rc he rs

Pr

  • gr

am Pe ople Re ac he d (mn)

Sc ho o l-b a se d De wo rming 95 Ra skin: Sub sidize d Ric e (I ndo ne sia ) 66 T e a c hing a t the Rig ht L e ve l (I ndia ) 34 Ge ne ra si: Co nditio na l Co mmunity Blo c k Gra nts (I ndo ne sia ) 6 Chlo rine Dispe nse rs fo r Sa fe Wa te r (E a st Afric a ) 0.5 F re e I nse c tic ida l Be dne ts Po lic y influe nc e Po lic e Skills T ra ining Po lic y influe nc e T OT AL 202 mn

31

slide-32
SLIDE 32

Ge tting a se nse o f o ve ra ll influe nc e is diffic ult

  • Re turns to R&D hig hly ske we d.
  • Mo st sc ho la rly a rtic le s a re ne ve r c ite d
  • Mo st sta rt-ups fa il
  • Ve nture c a pita lists g e t mo st o f the ir re ve nue fro m a sma ll numb e r o f

inve stme nts

  • Still hug e pa yo ffs to R&D
  • I

de a s ta ke a lo ng time to pe rc o la te thro ug h the syste m, a nd ma ny RCT a re fa irly re c e nt

  • Ma ny RCT

find tha t thing s DO NOT wo rk a s we ll a s ho pe d (mic ro c re dit, smo ke le ss sto ve s)

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

32

slide-33
SLIDE 33

RCT s a nd re a l wo rld impa c t: T he c a se study o f DI V

  • T
  • so lve the “c e nsus” pro b le m, we fo c us o n o ne c a se study: USAI

D DI V.

  • USAI

D’ s De ve lo pme nt I nno va tio n Ve nture s (DI V) o ffe rs a n o ppo rtunity to c o mpa re o utc o me s in se le c te d sa mple o f a wa rd winne rs:

  • DIV ha s ope n a pproa c h: no to p-do wn re stric tio n o n se c to r, stra te g y
  • Gra nte e s inc lude so c ia l e ntre pre ne urs, NGOs a nd de ve lo pme nt

re se a rc he rs

  • Sta g e d fina nc ing (Pilot ≤ $150,000; Testing ≤ $1,500,000; Transition to

Scale ≤ $15,000,000)

  • Ope nne ss on sc a ling stra te g y (Co mme rc ia l, pub lic -se c to r o r hyb rid

funding

  • E

mpha sis o n c o st e ffe c tive ne ss;

  • Atte ntio n to ma na g e me nt te a m, e xte rna l c o mmitme nts, b ut no

rig id litmus te st

33

slide-34
SLIDE 34

Me tho ds

  • Cove ra g e : 43 DI

V a wa rds ma de fro m 2010-2012; to ta l va lue $17.3m

  • He re just e xa mine re a c h, the e stima te d numb e r o f

pe o ple e xpo se d to the o rig ina l a nd a da pte d ve rsio ns o f the inno va tio n, a fte r the DI V funding .

  • Do no t c o mpa re me a sure s o f the size o f impa c t pe r

b e ne fic ia ry

  • Do no t e stima te the like liho o d tha t re a c h will b e susta ine d
  • r inc re a se d in the future
  • Do e s no t a ssume the c re dit to furthe r e xpa nsio n a ll g o e s to

DI V

  • One (o f se ve ra l) c o mpo ne nts o f so c ia l re turn c a lc ula tio n
  • Spe c ific a lly, we fo c us o n numb e r o f a wa rds re a c hing

mo re tha n 100k o r mo re tha n 1 M pe o ple .

34

slide-35
SLIDE 35

DI V-Suppo rte d se ve ra l I nno va tio ns Re a c hing > 100K Pe o ple

5 INNOVAT IONS RE ACHE D MORE T HAN 1 MIL L ION PE OPL E

  • Vo te r Re po rt Ca rds (2

a wa rds)

  • E

le c tio n Mo nito ring T e c hno lo g y

  • Co nsume r Ac tio n a nd

Ma ta tu Sa fe ty

  • Dig ita l Atte nda nc e

Mo nito ring

  • Dispe nse rs fo r Sa fe Wa te r

6 INNOVAT IONS RE ACHE D MORE T HAN 100K AND L E SS T HAN ONE MIL L ION PE OPL E

  • Sc a ling Co mmCa re fo r

Co mmunity He a lth Wo rke rs (2 a wa rds)

  • d.lig ht I

nno va tive F ina nc ing fo r So la r Syste ms

  • Susta ina b le Distrib utio n fo r

I mpro ve d Co o ksto ve s

  • Re c ruiting a nd Co mpe nsa ting

Co mmunity He a lth Wo rke rs

  • Visio nSpring Bo Ptic a l Ca re
  • Re ne wa b le Po we re d Mic ro g rids

fo r Rura l L ig hting

35

slide-36
SLIDE 36

Hig h re a c h o f inno va tio ns with RCT s/ invo lve me nt o f de ve lo pme nt e c o no mic s re se a rc he rs

  • 24 a wa rds ha d RCT

/ re se a rc he r invo lve me nt, o f whic h:

  • 42% (10 a wa rds) re a c he d mo re tha n 100,000 pe o ple
  • 25% (6 a wa rds) re a c he d mo re tha n o ne millio n pe o ple
  • 19 a wa rds did no t ha ve RCT

, o f whic h:

  • 16% (3 a wa rds) re a c he d mo re tha n 100,000 pe o ple
  • No a wa rds ha ve ye t re a c he d mo re tha n o ne millio n

pe o ple

  • Ove ra ll DI

V numb e rs fa vo ra b le re la tive to ma ny impa c t inve sto rs

  • Arb itra g e o ppo rtunity fro m o pe nne ss to multiple type s o f

inno va tio n?

  • Disc ipline o f e vide nc e use ful?

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

36

slide-37
SLIDE 37

While e a rly sta g e a wa rds ha ve lo w pro b a b ility o f a tta ining re a c h, the y ha ve hig h e xpe c te d re a c h pe r do lla r spe nt

Awar d Stage Numbe r

  • f

Awar ds Awar d Value F r ac tion Re ac hing Mor e than 100,000 pe ople F r ac tion Re ac hing Mor e than 1,000,000 pe ople Pe ople Re ac he d E xpe nditur e pe r Pe r son Re ac he d Stage 1 (< $100,000)

24 $2,353,136 17% (4/ 24) 8% (2/ 24) 6,723,733 $0.35

Stage 2 (<$1,000,000)

18 $9,557,926 39% (7/ 18) 17% (3/ 18) 16,931,044 $0.56

Stage 3 (<$15M)

1 $5,516,606 100% (1/ 1) 100% (1/ 1) 2,754,193 $2.00

37

slide-38
SLIDE 38

Pa thwa ys to re a c h

  • DI

V a wa rds fo r inno va tio ns with RCT s re a c he d > 100,000 pe o ple thro ug h a var

ie ty of par tne r ships: – Co untry g o ve rnme nts (e .g . Za mb ia CHW re c ruiting , I

ndia b io me tric mo nito ring )

– Do no rs (e .g . c o o ksto ve s in E

thio pia a nd Suda n)

– NGOs/ So c ia l E

nte rprise s (e .g . Dispe nse rs fo r Sa fe Wa te r)

– Priva te se c to r firms (e .g . ne wspa pe rs, b a nks, insura nc e

c o mpa nie s, Qua lc o mm, Sa fa ric o m)

  • T

hre e o f five inno va tio ns re a c hing mo re tha n o ne millio n pe o ple ha d e a rlie r RCT s de mo nstra ting impa c t, po te ntia l fo r c o st e ffe c tive ne ss: re se a rc he r/ pro je c t se le c tio n?

38

slide-39
SLIDE 39

Why mig ht pro je c ts invo lving RCT b e mo re like ly to ha ve future re a c h?

  • Co nvinc ing fo rc e o f e vide nc e [mo st pro je c ts tha t do no t

invo lve RCT try to sc a le thro ug h re ta ils sa le , whic h is ha rde r]

  • No thing to do with the RCT

pe r se :

– Clo se invo lve me nt o f re se a rc he rs he lp ide a s g ro unde d in b a sic

sc ie nc e pe rc o la te re se a rc h (like in b io te c h).

– I

n pa rtic ula r: I nflue nc e o f b e ha vio ra l e c o no mic s/ info rma tio n: fo c us o n lo w c o st inte rve ntio ns , whic h a re mo re like ly to sc a le

  • Se le c tio n o f g o o d pro je c ts [willing to do a n RCT

]

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

39

slide-40
SLIDE 40

Co nc lusio n

  • T

he pro je c ts e va lua te d b y RCT tha t the n ha ve re a c he d ma ny pe o ple te nd to b e lo w-c o st, we ll de fine d, simple .

  • So wha t ha s ma de RCT

use ful a s a re se a rc h to o l (a b ility to ite ra te , ze ro -do wn to c o mpo ne nt, te st a the o ry) is e xa c tly wha t ha s turne d o ut to ma ke the m po lic y re le va nt: de ta ils ma tte r tre me ndo usly, a nd RCT te nd to g e t the de ta ils rig ht.

  • An a lte rna tive pa thwa y: BRAC, PROGRE
  • SA. Co mple xe d

inte rve ntio ns re plic a te d in ma ny c o nte xts.

  • And a third o ne : inno va tio n within e xisting g o ve rnme nts a nd

institutio ns.

–

T a mil Na du inno va tio n F und, Nudg e Unit

–

Guja ra t Po llutio n Co ntro l Bo a rd

J-PAL | T HE RO LE O F RANDO MIZED EVALUAT IO NS IN INFO RMING PO LIC Y

40

slide-41
SLIDE 41

Co nc lusio n

  • F
  • r RCT

to mo ve fro m the la rg e re se a rc h impa c ts to la rg e po lic y impa c t, we ne e d a ra ng e o f c o mple me nta ry institutio ns:

–

Me ta -a na lysis

–

Re vie w a rtic le

–

Re vie w pa ne ls

–

Re g istry o f E xpe rime nts ha s sta rte d a nd is suc c e ssful (706 studie s a s o f June 8),

  • Appro pria te suppo rt a nd e xpe rime nt to suppo rt the le a rning

ne e de d to mo ve fro m suc c e ssful pilo t to po lic y a t sc a le

–

I te ra tio ns to de sig n sc a la b le (ro b ust) ve rsio ns a nd me a sure the ir e ffe c ts

–

E q uilib rium e ffe c ts

–

Po litic a l e c o no my / industria l o rg a niza tio n o f imple me nta tio n

J-PAL | T HE IMPAC T O F RC T

O N RESEARC H AND PO LIC Y 41