Replication, Reproduction and the Credibility of Micro-econometric Studies of the Impact of Microfinance and Informal Sector Borrowing in Bangladesh Maren Duvendack 1, 2 Richard Palmer-Jones 1 6 June 2012 1. UEA, 2. ODI 1
Replication & impact heterogeneity Many good reasons to conduct replications, one is to explore impact heterogeneity: Crucial for drawing appropriate policy conclusions, confounds causal effects because results only hold for the groups identified, and not others Neglecting impact heterogeneity misleading by inappropriately merging groups which respond quite differently to the treatment and indeed experience different treatments Sub-groups known feature of MF context Replication should seek to know the conditions under which the results hold, often by repeating the experiments Thus, heterogeneity and replication are closely linked. We want to know what works, for whom, under what circumstances. 2
Introduction: Microfinance evaluations Microfinance hype: MF has long been seen as silver bullet for alleviating poverty and empowering women through providing financial services to the poor Studies suggesting social and economic benefits: Hulme and Mosley (1996), Coleman (1999), Pitt and Khandker (1998), Khandker (1998 and 2005), Rutherford (2001) and Morduch and Haley (2002) Critical voices: Roodman and Morduch (2011), Duvendack et al (2011), Stewart et al (2011), Bateman (2010) and Dichter and Harper (2007), Roy (2010) First two RCTs in the sector (Banerjee et al, 2009; Karlan and Zinman, 2009) raising doubts about the causal link between MF and poverty alleviation. Most influential MF IE to date: Pitt and Khandker (1998) 3
Why is Pitt and Khandker so important? Methodologically innovative Large original World Bank survey in 1991-2 With follow up panel in 1998-9 Complex and sophisticated analysis (WESML-LIML) Most rigorous impact evaluation of microfinance Key work of main academic author(s) Widely cited by high level MF advocates such as M. Yunus 4
Introduction to Pitt and Khandker (1998) Iconic study finding positive impacts of MF especially when lending to women (male and female groups) Quasi-experimental design & eligibility condition used to identify impact Primary eligibility criterion: landownership (0.5 acres = 50 decimals) Overall sample:1,798; 1,538 households from treatment villages, 260 from controls “Treatment” Village “Control” Village Land owned/cultivated Would not Not-eligible non be eligible participants 0.5 acres Eligible participants Would be eligible Eligible but non participants 5
Sub-groups & Microfinance PnK simply ignore alternative sources of finance but they appear in their data: Status Treatment villages, Control villages, no. of individuals with no. of individuals with multiple sources (in %) multiple sources (in %) Eligible 4 7 Not eligible 2.5 8 Sub-groups should have been in the design to explore impact heterogeneity One should test against next best alternative, research design neglected alternatives Sub-groups little analysed and not vs MF 6
Sub-group comparisons Lack of sub-group heterogeneity undermines claim that MFIs make unique contribution to poverty reduction. 1: MF None Other MF None 2: Borr Other MF 3: Borr 7
Sub-group PSM results Y MF + Y MF vs Y MF vs Y Multiple +Y Borr Outcome variables Y None Y Borr vs Y None Comparison 1 2 3 Kernel matching, 0.05 Log per capita expenditure (Taka) -0.004 0.035*** -0.114*** Log women non-landed assets 0.846*** 0.501*** 1.183*** (Taka) Girl school enrolment, aged 5-17 0.070*** 0.073*** 0.029 years Boy school enrolment, aged 5-17 0.037 0.057*** 0.007 years • Summary: Sub- group analysis undermines PnK’s claims, no obvious advantage of MF vs other sources
Conclusions Too easy to believe MF is beneficial without considering evidence in balanced way, thus sub-group analysis crucial undermines PnK’s original claims, supported by other data sources Award prestige only if public deposit of original data and code allowing replication and reproduction "The more freely researchers circulate their data and code, the easier it is for others to subject that work to the scrutiny needed for science to proceed. The stakes are particularly high for research that influences policy“ (Roodman and Morduch, 2011:45). Had data and code disclosure policy been in place at the time PnK was published, we might have resolved the current debate over this study a while ago (Roodman and Morduch, 2011). Continuing need for high quality studies with ethical reporting and publication practices (enabling replication) Ensure opportunities for independent research by people from different methodological and epistemological backgrounds. 9
Q & A Session For further questions or comments please email: m.duvendack@odi.org.uk; or r.palmer-jones@uea.ac.uk 10
Appendix: PSM Results – By Gender Y MF vs Y MF + Multiple +Y Borr Y MF vs Y Borr Outcome variables Y None vs Y None Comparison 1 2 3 Kernel matching, 0.05 Women -0.002 0.002 -0.111*** Log per capita 1 expenditure (Taka) 2 Men -0.002 0.067*** -0.121*** Women 0.948*** 0.961*** 1.510*** Log women 3 non-landed assets 4 Men 0.441 0.041 0.834** Women 0.074** 0.081*** 0.046 Girl enrolment, 5 aged 5-17 years 6 Men 0.052 0.056** 0.029 Women 0.049* 0.053** 0.040 Boy enrolment, 7 aged 5-17 years 8 Men 0.007 0.050** -0.033 Source: Authors calculations. Notes: *statistically significant at 10%, **statistically significant at 5%, ***statistically significant at 1%. Results refer to the differences in the mean values between matched samples. t-tests before and after matching employed to investigate the differences in the mean values for each covariate X across matched samples; the test provided conclusive results.
Appendix: Sensitivity Analysis PSM estimate for log of women non-landed assets for Y MF : 0.846*** (comparison 1) - sensitive to selection on unobservables? Significance levels Hodges-Lehmann point 95% Confidence estimates intervals Gamma (Γ) Minimum Maximum Minimum Maximum Minimum Maximum 1 < 0.0001 < 0.0001 0.652 0.652 0.106 1.132 1.2 < 0.0001 < 0.4344 0.031 1.209 -0.300 1.590 1.3 < 0.0001 < 0.8037 -0.144 1.419 -0.437 1.759 1.4 < 0.0001 < 0.9641 -0.297 1.587 -0.539 1.899 Source: Authors calculations.
Recommend
More recommend
