Building an Evaluation Toolkit for your FC/IE Initiatives IFIE Webinar 12/02/2014
Overview Context Monitoring & Evaluation – Objectives Evaluation criteria Analysis of proposals received Conclusions 2
Context Broad-Based Black Economic Empowerment (B-BBEE Act) – to achieve redress through facilitating economic participation to previously marginalised SA citizens Financial Sector Charter Code – define, measure, assess B-BBEE in financial services sector ASISA Foundation = Savings and Investment’s industry response – o to facilitate the effective achievement of the intentions of B-BBEE Act through the delivery of Consumer Financial Education (CFE) o pooled resources – collaboration and reach o governance and oversight 3
Monitoring and Evaluation (M&E) Independent M&E specialists Report to Foundation’s Board of Trustees Collaborate with project implementation team from project planning phase to ensure: o Common understanding of project goals and objectives o Practical measurement criteria/metrics determined upfront Gain insights into various M&E approaches and related costs Prospective M&E specialist service providers were asked to submit a proposal specifying their methodology and approach to performing M&E on the proposed pilot project 4
M&E Objectives To assess the effectiveness and impact (if possible) of two different implementation methodologies using the following key elements: o interactive workshops (9600 people) o industrial theatre (1800 people) o support provided by posters o radio drama with short message service (sms) competition attached o sms facility for further FE messages sent o incorporation specific life stages; population demographics and geographies o advocacy and stakeholder engagements which precede the campaign to garner support o training and use of local facilitators 5
M&E Objectives (cont’d) To review achievements and identify challenges and areas for improvement in both methodologies To generate key learnings to inform future strategic and programming decisions To determine if the programme met its specific training objectives by assessing the programme's impact on participant’s knowledge and behaviours To determine a basis for assessing longer-term behavioural impact. To facilitate potential value added opportunities eg longitudinal studies 6
Evaluation Criteria Independence Experience in M&E and other relevant CFE M&E and research Proposed project specialists and references Any relationships with ASISA members Understanding of Project Intent Detail primary objectives of the M&E to be done Specific methodology and proposed evaluation tools Appropriateness of proposed methodology Sample size, statistically determined or not Sample profiles – life stages/geographic/racial 7
Evaluation Criteria (cont’d) Coverage Sufficiency of evaluation methodology to draw necessary conclusions Fieldwork - who /how trained/quality checks Reporting process and timing Timelines - sufficient and aligned to project Ethical approach to respondents Intellectual Property - ownership Collaboration with project implementers BEE and Transformation credentials and practice 8
Evaluation Criteria (cont’d) Budget o Total costs o Analysis o Exclusions from budget o Flexibility Added value Uncertainties/Questions Other 9
M&E Proposals Analysis Experience o Received proposals from three (3) independent entities, very experienced and skilled M&E specialists, each would be able to add value to the programme through their insights and previous experience Focus o Although they all understood the project intent, there was a different focus in each approach Approach o Two (A&B) reflected similar approaches using predominantly surveys / questionnaires and focus group discussions ( FGDs ) o The third (C) focused heavily on the establishing impact of the programmes, through a detailed household survey only across the required demographic profiles (life stages / peri-urban /urban) 10
M&E Proposals Analysis ( cont’d) Methodology o A&B both proposed qualitative and quantitative methodologies to assess if the objectives of the project had been met o A focused on the components of the project and proposed questionnaires, spot checks immediately following the industrial theatre and FGD, with beneficiaries of programme, control group as well as facilitators of the programme o B focused on life stages in determining the selected respondents for questionnaires and FGD and did not include spot checks or surveys 11
M&E Proposals Analysis ( cont’d) Samples o A & C proposed statistically selected samples A – for 90% precision 270 respondents C- for 95% precision 960 respondents o B suggested smaller sample sizes (160) with a focus of representative characteristics of the population and being able to gather more in-depth feedback All 3 proposed gathering baseline (pre) and endline (post) information None of the respondents specifically included the evaluation of the poster, radio and sms campaign nor the efficacy of the stakeholder intervention A & B did not make a distinction in assessing rural/peri-urban vs urban respondents B identified opportunities for skills transfer into local communities through training locals to do face to face questionnaires. A & C proposed the use their 12 internal skilled resources
M&E Proposals Analysis (cont’d) Sufficiency of methodology to draw conclusions o A – assess knowledge transfer ; participant and facilitator’s experience ; make projections of likely long term impact on participant’s behaviours ( logframe) o B – assess knowledge transfer , participant’s experience ; methodology enables them to set up the framework for a future longer term (longitudinal) research study to assess impact on participant’s behaviours o C - methodology focused on performing this longitudinal impact study and knowledge transfer, but at a high cost 13
M&E Proposals Analysis ( cont’d) Timelines o A & B’s timelines co -incided with project period and a few weeks thereafter o C’s timelines included gathering baseline data during the project but most of the evaluation taking place 6 to 18 weeks post project intervention Other o All implied willingness to collaborate with project implementers o A had stronger BEE credentials o Only B provided insights into establishing an ethical approach to respondents as well as their views of ownership rights of intellectual property (admittedly this was not requested in the RFP) 14
M&E Proposals Analysis ( cont’d) Budget o A & B provided an analysis of their proposed budget. However A chose to reflect hours per person per project activity and total costs, while B reflected cost per project activity o C provided a lumpsum cost o A & B project costs were similar, approximating 20% of the pilot project costs o C’s lumpsum approximates 125% of project cost and excluded control group o B’s budget excluded longitudinal behaviour tracking research Telephonic interactions were held with both A&B to ensure we properly understood their approach . Evaluation Outcome o Comfortable that both A&B would collaborate well with the CFE project team, be flexible in their approach, as required, and add value to the project. o C not considered acceptable due to cost and methodology o CEO motivated for the appointment of A 15
Conclusions Ensure M&E goals and objectives are aligned to project goals and objectives as presented to various project stakeholders – M&E outcomes impacts potential for funding and sustainability When assessing M&E specialists for the assignment: o Define evaluation criteria upfront and be specific in your request for proposal o Determine which criteria should be weighted more heavily in the evaluation and how Ensure M&E programme is defined early to ensure their inclusion from the planning phase of the FE project Follow your instincts to facilitate a team approach to achieving project objectives 16
Thank You Ruth Benjamin-Swales
Recommend
More recommend
