Kore Lavi DFAP Final Qualitative Performance Evaluation Webinar - - PDF document

FOLLOW-UP TO QUESTIONS RAISED IN KORE LAVI EVALUATION WEBINAR

This document aims to respond to questions and comments raised during the Kore Lavi DFAP Final Qualitative Performance Evaluation Webinar. Some questions were answered during the webinar. The response to each question or comment indicates whether the response was live or post-event. This webinar was held by the Implementer-Led Evaluation and Learning (IMPEL) award on May 22, 2020. The evaluation was led by Tulane University, and IMPEL Chief of Party, Peter Horjus, presented findings in this webinar, with support from Dr. Nancy Mock and John Berry.

Questions and Comments from Webinar Q&A Box with Responses

SIMAST is a good tool for targeting beneficiaries. MAST must work for the appropriation of the tool by local authorities and communities.

• Live Response: Agreed. Efforts were made by KL to build capacity at MAST to run the

program, but they will continue to need support. What was the average length of time that people received vouchers?

• Live Response: Recipients were identified at the beginning of the activity and continued to

receive them until end of project. In essence from 2013/2014 to 2019. For this definition of graduation, I think that we must take into account socio-economic integration. This graduation does not amount to a program but an entire environment which influences the life

• f the communities.
• Post-event Response: Absolutely. Graduation is not necessarily a simple concept, and it may

not be as simple as a basic economic measure. That is why it's important from the start to define what graduation means, and how it will be used programmatically- and if it's an appropriate goal for the project to aim for in all cases. Regarding SIMAST, it will be important to determine how it will recover its costs in the future, especially the human resource costs required to ensure its proper administration, management, maintenance and updating. Also, there has been discussion regarding the potential to integrate ways to track household livelihoods so that it could track graduation. This would definitely require additional investment.

Kore Lavi DFAP Final Qualitative Performance Evaluation Webinar Q&A Reponses

IMPEL Webinar | May 22, 2020

IMPEL | Implementer-Led Evaluation and Learning 2 Kore Lavi DFAP Final Qualitative Performance Evaluation | Webinar Q&A Reponses 2

• Live Response: Agreed. The sustainability of the system will require ongoing investment in

developing and maintaining the system. Several donors have expressed interest in supporting the system. Is there clear documentation on the graduation now? If yes, please suggest or share them with us.

• Post-event Response: Not that I (Peter) am aware of. The Indicator Performance Tracking

Table (IPTT) for the indictor during the first 5 years states "This indicator has been removed based on the approval of the donor (PREP 2016)," and the IPTT for the extension period indicates that "This indicator has been removed from the indicator list through PREP 2019." What are the things you consider before determining the top up value per household (HH), and how do you coordinate with vendors to ensure prices of food items?

• Post-event Response: Kore Lavi (KL) used the MFIs and their own monitoring staff to ensure

that the prices set by participating vendors matched those of the market in general. This worked well, no reports of price gouging of the voucher recipients. If the asker is referring to inflation/food price increases when saying “top up value per household”, then this could be the local CPI for food (from IHSI), or it could be tied to the cost of the basic food basket in the local market, depending on what data is available. Did the evaluators consider that 50% of Safety net members joined VSLAs due to that monthly transfer? Many of them have undertaken some income-generating activities with the loan from the VSLA groups.

• Post-event Response: Absolutely- we present this in the report (from IPTT data), which will

be shared with all webinar registrants when published. The 50% of voucher recipients in VSLAs is an important achievement. The result of HHs undertaking improved/additional income-generating activities with loans from VSLAs is unfortunately not well measured. Can you explain why the voucher amount wasn't perceived or considered as transformational? In the context of Haiti, what can be considered as a minimum amount to make a nutrition voucher transformational? What would you consider as features of a transformational nutrition voucher aimed at vulnerable households?

• Post-event Response: This was stated by a few KIs and in some of the FGD. The word

'transformational' is of course somewhat subjective. The implication in the discussions was generally that the value of the vouchers was certainly helpful, but it wasn't always sufficient to allow the household to make large, lasting changes in their lives and livelihoods. (As said in the presentation, though, we are not saying here that no households achieved large lasting changes in their lives and livelihoods). The goal of a food voucher program may not necessarily need to be for such a significant transformation. In the context of safety nets, it may be simply to prevent the household from sliding further into poverty and food

• insecurity. It is also not just about the vouchers alone, but other opportunities that are

made available to households and how the vouchers may just allow households to take advantage of those other opportunities (livelihood, agriculture, etc.). Other households may not be readily transformed (elderly, infirmed, for example). The reason for SO3 not getting continued was mainly due to lack of resources - a large cut to the annual budget.

• Post-event Response: That was our understanding too. There may be an important small

shade of difference here (budget cut vs. activity cut). We will check and be sure we are reflecting that correctly.

IMPEL | Implementer-Led Evaluation and Learning 3 Kore Lavi DFAP Final Qualitative Performance Evaluation | Webinar Q&A Reponses 3

At the time, Kore Lavi followed the PM2A methodology that only provided rations for the mother and child pair as promoted by FFP at the time.

• Post-event Response: Thank you! This is very helpful information into the reason for the

choice of the individual rations vs. family rations. From the start, setting a threshold of inflation that should prompt adjustment to the value of vouchers could be very helpful regarding decision making

• Post-event Response: Agreed. And communicated with beneficiaries so they know what to
• expect. As mentioned above, inflation could be the CPI, local food CPI, the cost of a

minimum food basket at the local markets, etc. Additionally, there could be some plans put in place for deflation (say, against the dollar) - I would not think that decreasing the voucher value would be appropriate, since HHs come to rely on the consistency of the voucher each

• month. However, it could have implications for overall project budget.

Is there any plan to link VSLAs with existing financial institutions?

• Live Response: Yes, definitely linkages with MFIs. In addition, there was discussion of a

federation of VSLAs. Regarding fresh food vouchers' impact on local market, can you point to any specific impact? Quantitative or Qualitative?

• Live Response: Vendors described an increase in sales and in new clients. This may not be

sustainable after the end of the voucher system.

• Post-Event Response: This was qualitative data (very commonly reported). We mention that

having quantitative indicators in the M&E for the project to measure the impact on vendors would be a good idea- there are most certainly positive impacts on the vendors, but it's not well quantified. Why have the Evaluation Team listed VSLAs as part of SO4? They were included in SO2.

• Post-Event Response: This should have been clarified better during the presentation. A few

pieces fell on multiple strategic objectives, at least conceptually. We were thinking of the bigger concept of strengthening Haitian institutional capacity (SO4) and put VSLAs as well as the work with MFIs, etc. under SO4- but agreed, those were really part of SO2. We did not delineate that way in the report. What are the nutrition activities that was conducted and who was responsible in conducting those activities?

• Live Response: Nutrition activities were integrated in training on healthy foods, hygiene, etc.

Respondents indicated positive behavior change and increased nutrition knowledge. Was the Kore Lavi voucher system strong enough to avoid fraud? For instance, a participant may defeat the system by receiving more coupons than allowed, whether with the help of corrupt project staff or by himself. Or maybe project staff use coupon for his/her own benefit?

• Live Response: There were systems in place to avoid fraud and they seemed to be effective.

According to respondents there was very little “leakage.” Was there any issue with the vendors providing the local foods?

• Post-Event Response: There were one or two exceptional reports of vendors struggling to

meet the increased demand for local foods when vouchers were distributed, and having to reach further afield to source local products (neighboring communes/departments). This

IMPEL | Implementer-Led Evaluation and Learning 4 Kore Lavi DFAP Final Qualitative Performance Evaluation | Webinar Q&A Reponses 4

was not a commonly reported issue, and those rare cases were discussed in conjunction with the fact that the vendors were doing good business! Are you going to collect quantitative data?

• Live Response: Yes, we completed the first round of data collection in January of this year.

We postponed the second round of data collection (May/June) due to COVID 19. We plan to resume and complete the second and third (final) round of data collection in November/December of this year as planned. How do you handle cases of missing voucher and impersonation by participants? And how long does it take?

• Post-Event Response: Kore Lavi had a very carefully monitored system to ID participants in

the paper voucher distribution. There was also a hotline that anyone could call if they suspected abuse. The Evaluation Team is unsure how the project staff handled lost vouchers, however. Were mechanisms put in place to ensure that vendors sell local products effectively? Did beneficiaries report any issues on food quality and quantity?

• Post-Event Response: Yes, KL had very effective mechanisms to monitor the

markets/vendors. Vendors worked close with the project and with the MFIs, including hygiene training for the vendors. Vendors were also carefully assessed before being included in the program to be sure they provided quality and hygienic goods. Ten percent of the poorest households means that most are landless and ag/livelihood interventions would have been unsustainable. The overall objective was to establish a GOH safety net program that was long term.

• Post-Event Response: Yes. We generally referred to those sorts of interventions as

"ag/livelihood" so that landless households would be targeted with non-ag livelihood

• support. Where agriculture is a possibility for the household, though, there is the additional

benefit of the synergy with the increased demand for local foods created by the vouchers. What is your [Kora Lavi’s] exit and sustainability plan for participants, vendors, community volunteers etc.?

• Post-Event Response: Voucher recipients were given cash infusions in the last month of the

project, referred to in KL as the 'resilience package', which was substantially larger than the usual amount, and if I'm not mistaken, it was given in cash rather than voucher to allow households to purchase livelihood assets (tools, livestock, etc.). There was no issue with sharing - it was not a nutrition program.

• Live Response: The sharing issue was seen in the nutrition component.
• Post-Event Response: The sharing between HHs of the voucher benefits was generally

viewed as a positive thing- building social capital, etc. What are the challenges you encountered with the Red Rose and SIMAST?

• Post-Event Response: The ongoing challenges that SIMAST faces included:
• Ensuring sustainable access to the resources (financial and technical) to maintain the

database

• Regularly updating the system with quality data
• Fully transferring ownership of the system and the location of the servers to the

Government of Haiti

IMPEL | Implementer-Led Evaluation and Learning 5 Kore Lavi DFAP Final Qualitative Performance Evaluation | Webinar Q&A Reponses 5

• Avoiding the pull to be everything for everybody and drifting from the mission of

supporting social safety nets to add data from other government ministries (i.e. agriculture, water, etc.) VSLA is the big tool for graduation.

• Post-Event Response: To some degree. The ET felt that where graduation was possible, it

would most likely be a combination of vouchers, VSLA, and livelihood support that would potentially have the biggest impact. Can you please discuss the sustainability of the interventions?

• Post-Event Response: Sustainability is covered in more detail in the report, which will be

shared with all webinar registrants once it is published. In relation to sustainability, any chances that the social protection program contributing to COVID- 19 responses or lessons in this regard?

• Post-Event Response: Existing voucher systems can be very effectively used to respond to

shocks such as COVID-19. Although the voucher part of the project has ended, the SIMAST may be of use in targeting future social protection programming related to COVID-19. Any plans for an ex-post evaluation further down the road to compare long-term results of project participants in KL against participants in other types of approaches?

• Post-Event Response: As far as the ET knows, only the on-going quantitative SO2 impact
• survey. There is a question in there of what sorts of assistance HHs have gotten, so

depending on what the responses are, there could be some exploration of that. However, it's not part of the design to look at that specifically (comparing approaches). What are the cultural norms that affected implementation and how where you able to mitigate the challenge?

• Live Response: Sharing food rations was definitely a cultural norm that affected
• implementation. But this had both up and down sides in terms of building social cohesion.
• Post-Event Response: See above- the nutrition rations targeted to individuals may have had

its impacts diluted because of sharing. But the sharing between HHs, particularly of the voucher benefits, may have increased social capital and been a benefit for the household. Can you briefly talk about the other donors that have invested in the SIMAST and safety net system?

• Live Response: WFP took the lead in supporting development of SIMAST and expressed the

intent to continue to support it in the future.

• Post-Event Response: Additionally, the EU, WB, and others (see report when published) had

either expressed interest or started to support the SIMAST system at the time of the evaluation. It seems like the main recommendation for institutional capacity is more resource-heavy capacity building, which most of the time is affected by the quality training and learning opportunities provided, and constrained by poor retention of capacity built personnel. How do we retain capacity in a context of unpredictable mobility of civil servants across departments and agencies on the one side, and challenges related to localising capacity in a context of programme implementation led by different partners?

• Post-Event Response: Building institutional capacity only to lose it is a real issue, and one

that a single program like KL can't solve. However, KL did at least get a start at it where other programs have not made this a priority in the past. The way forward is to continue the

IMPEL | Implementer-Led Evaluation and Learning 6 Kore Lavi DFAP Final Qualitative Performance Evaluation | Webinar Q&A Reponses 6

capacity building work that KL started with other future programs and to focus on both the national and local level. Strengthening relations between local and national level government staff is also part of capacity building and should continue as well. Capacity development in a country like Haiti takes long term commitment. As long as the donor money comes in, the GOH will continue the project. The reality is that the US needs to be in it for the long haul if they want any results. Is there any commitment from WB to supporting this system and possibly to slowly scaling up?

• Live Response: The WB expressed interest in the system and will be implementing a new

activity that includes SIMAST. Talking about coverage, is there any lesson learned on how a project like this can be better scoped, zoned, and concentrated for more impact?

• Post-Event Response: Since this was a pilot project, the goal was not to provide national
• coverage. Depending on the objectives of future safety net projects, it may be a combination
• f geographic targeting to identify poorer areas for priority in being covered, and then using

a HDVI-like approach for identifying HHs in these areas for inclusion. The challenge is that in many cases in Haiti, the difference between poorest households and those just a rung up is very very small. Yet projects cannot target huge swaths of the population, it would be financially implausible. One interesting dynamic I've found in Haiti as compared to some other countries is a proud approach to valuing local food as compared to imported food (es diri peyi vs diri Miami). Were the elements of this project that encouraged that narrative and did it contribute to success/sustainability?

• Live Response: The local food component was definitely a success of the activity. However,

local foods are somewhat more expensive than imported, so increased consumption may not be sustainable.

• Post-Event Response: The knowledge and attitude towards a diverse diet were definitely

positively impacted. The sustainability of the practice is less certain- the SO2 impact survey may help answer this question. Has the KL voucher system been institutionalized? MAST has the capacity to lead it by itself now. If not, what critical areas require continued assistance in order to reach this goal?

• Post-Event Response: The SIMAST system is still 'in process' of being institutionalized. Key

areas include financial support and the eventual creation of a revenue stream for the SIMAST system, and ensuring the capacity to manage the database itself. Data collection for SIMAST was done by CNSA, who has strong capacity for this type of activity already. The voucher distributions ended with the end of the project. We should note that disaggregated data about poverty for each Commune was not available until now.

• Post-Event Response: There is a wealth of data available now on the communes covered by

the SIMAST. It did appear that that data may be underutilized, however. The ET did not see many examples of the use of disaggregated KL data from SIMAST data outside of the KL context (not referring here to the positive and encouraging use of the system by other actors/donors for data collection/targeting). WFP is not a donor though, and was one of the partners under KL. I’m thinking more of the EU, the Bank, etc.

IMPEL | Implementer-Led Evaluation and Learning 7 Kore Lavi DFAP Final Qualitative Performance Evaluation | Webinar Q&A Reponses 7

• Live Response: Both the EU and WB are aware of and supportive of SIMAST. They both have
• ngoing social protection programs.

Ten percent was the number due to the project budget constraints.

• Post-Event Response: That may be. We were also told that the justification for the 10% was

simply that it was national prevalence of extreme poverty in Haiti. Are there cases or perceived benefits that the vouchers may have led to the creation or recovery of a livelihood or created a path to a market of good or service? Can those can be considered among main changes that have been brought about by KL?

• Post-Event Response: Absolutely. There are many cases of this. However, it is not well
• quantified. We may get at this a bit more with the SO2 impact survey (though livelihood

changes/creation specifically is not the main focus of that study). We have been hearing that VSLA’s were well received and were widely seen to be positive. Can you please address what were the key quantifiable impacts of VSLAs on the beneficiary HHs?

• Post-Event Response: We will see more quantifiable data hopefully in the SO2 impact
• survey. There is also the study that KL did "Kore Lavi Safety Net Beneficiary Resilience

Assessment" that uses the SenseMaker approach to measure impacts of VSLAs on households. Have you been able to assess and depict any negative impact of KL on the targeted communities, such as inflation in some local food price?

• Post-Event Response: We looked specifically at inflation of food prices, and found no

evidence of this. It was also well monitored by KL during the project to ensure this did not happen as a result of the project.

This brief is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of the Implementer-Led Evaluation & Learning (IMPEL) award and do not necessarily reflect the views of USAID or the United States Government.