Evaluation: Hot tips from The Ian Potter Foundation Dr S quirrel Main, Research and Evaluation Manager
Before we begin… Please note that IPF staff may be taking photos/ videos during this workshop for social media purposes. If you do not wish for your image to be included in the publicity, please let any staff member know.
Presentation Overview S tandards of evidence Level 1: Grantee tips and KPIs Level 2: Endgame and sustainability Level 3: Resources and ideas Dissemination: Infographics
Evaluation vs. chasing unicorns…
S tandards of Evidence
S tandards of Evidence
Level 1: Top tips 1. Train the volunteers conducting surveys 2. Ensure redundancy in the key knowledge holders (bikes- cancer-hearts) 3. Hold immediate staff-wide dissemination training as soon as possible 4. S tart early: consultations with stakeholders take a long time 5. Capture baseline data 6. Use pre-existing, validated surveys (North Carolina Family Assessment S cales)
Level 1: S MART KPIs Take a moment to review your goals in pairs. Listen to your partner’s goals and provide feedback. Are they SMART ? S pecific M easurable A chievable R ewarding/ Relevant T ime-bound How can your partner improve their measurements? Have time? Take a moment to review your long-term outcomes in pairs… How will your partner begin to measure/ collect data for their longer-term outcomes? Challenges? S olutions?
Level 2: Evaluation key steps Det ermine Involve Circulat e Finalise Review first Disseminat e endgame st akeholders t ender proposal draft
Level 2: Decide your endgame Endgame Example 1 Open source Firefox 2 Replication Job S upport 3 Government adoption OoHC pilot 4 Commercial adoption Pharmaceuticals 5 Mission achievement One Disease 6 S ustained service S TREAT S ource: Alice Guglev and Andrew S t ern. What ’s Y our End-game? Global Development Incubat or. 30 January 2014.
Endgame influences measurement: 26 not 14 weeks…
Level 2: Tailor your evaluation… S hort - Medium- Long- Input s Act ivit ies Out put s t erm t erm t erm out comes out comes out comes • How does the program work? • What outcomes resulted from the • What did the program program? accomplish? • When and to whom? • What has influence how the • What is the program’s program works? effectiveness? (Findings, not Rec’s) • How do changes compare to other programs/ sectors?
Level 2: Who would you invite?
From Level 2 to Level 3: Why bother with a control group? S ource: S tephen Taylor, Volker S choer and Thabo Mabogoane
Level 3: Evaluation resources • TIPFEP • Example RFQ • BetterEvaluation.org • IssueLab (ROIs!) • Each other!
Level 3: No silos
Level 3: Data rehearsal
Level 3: S tructured Pyramid Analysis Plan Goal/ outcome Dependent variables Data sources
Enter your SMART goal here Facilitate the rapid uptake of the new technology to advance WRH research proj ects. Dependent Variable 2 Dependent Variable 3 Dependent Variable 1 Data source 2a Data source 2b Data source 3a Data source 3b Data source 1a Data source 1b
Enter your SMART goal here Facilitate the rapid uptake of the new technology to advance WRH research proj ects. Dependent Variable 2 Dependent Variable 3 Dependent Variable 1 Data source 2a Data source 2b Data source 3a Data source 3b Data source 1a Data source 1b
Enter your SMART goal here Train 50 users at workshops, with 20 becoming fully independent users and machine usage rates >= 40 hours per week. Dependent Variable 2 Dependent Variable 3 Dependent Variable 1 number of fully independent number of hours of usage per at t endance at hands-on users of t he inst rument week workshops Data source 2a Data source 2b Data source 3a Data source 3b Data source 1a Data source 1b
Enter your SMART goal here Train 50 users at workshops, with 20 becoming fully independent users and machine usage rates >= 40 hours per week. Dependent Variable 2 Dependent Variable 3 Dependent Variable 1 number of fully independent number of hours of usage per at t endance at hands-on users of t he inst rument week workshops Data source 2a Data source 2b Data source 3a Data source 3b Data source 1a Data source 1b Int ernal Count s of Int ernal N/ A Tot al unique S ign-off list of inst rument Excel column inst rument count of ‘ qualified usage H “ workshop usage workshop independent dat abase, part icipant dat abase, at t endees users’ by t ot al hours (1 t it le” count of (from chief Jan 2017 (st udent , post - unique user workshop sign- t echnician, t hrough 31 Dec doc, research IDs in) t ot al count 2018) ÷ 52 assist ant , et c)
Enter your SMART goal here 70% of children participating in DPIL report an increased affinity for reading Dependent Variable 2 Dependent Variable 3 Dependent Variable 1 Data source 2b Data source 3a Data source 1b Data source 2a Data source 3b Data source 1a
Enter your SMART goal here 70% of children participating in DPIL report an increased affinity for reading Dependent Variable 2 Dependent Variable 3 Dependent Variable 1 Data source 2b Data source 3a Data source 1b Data source 2a Data source 3b Data source 1a
Enter your SMART goal here 70% of children participating in DPIL program report an increase of at least 1-point on their reading motivation from baseline to 12-months into the program. Dependent Variable 2 Dependent Variable 3 Dependent Variable 1 “ affinit y for reading” – “ affinit y for reading” – point baseline average difference aft er 12 mont hs Data source 2b Data source 3a Data source 1b Data source 2a Data source 3b Data source 1a Difference between Average score on Child unique Child unique score on 5-question 5-quest ion ident ifier. ident ifier. Habitual Reading Habit ual Reading Column Column Motivation Mot ivat ion “ Child ID “ Child ID Questionnaire Quest ionnaire number” (Möller and number” Bonerad, 2007), (Möller and excel excel baseline and 12- Bonerad, 2007). spreadsheet , spreadsheet , month. S tored on Baseline. S t ored column B. column B. excel spreadsheet, on excel columns S . spreadsheet , columns I-M.
Enter your SMART goal here 70% of children participating in DPIL program report an increase of at least 1-point on their reading motivation from baseline to 12-months into the program. Dependent Variable 2 Dependent Variable 3 Dependent Variable 1 “ affinit y for reading” – “ affinit y for reading” – point [Ext ernal dat a examples] baseline average difference aft er 12 mont hs Data source 2b Data source 3a Data source 1b Data source 2a Data source 3b Data source 1a Difference between Average score on Child unique Child unique score on 5-question AIHW-S HS 5-quest ion ident ifier. ident ifier. ABS (cite Habitual Reading “ Other Habit ual Reading Column Column Motivation specific Mot ivat ion “ Child ID “ Child ID outcomes Questionnaire table Quest ionnaire number” (Möller and number” national” / cube!) Bonerad, 2007), (Möller and excel excel (cite specific baseline and 12- Bonerad, 2007). spreadsheet , spreadsheet , table month. S tored on Baseline. S t ored column B. column B. / cube!) excel spreadsheet, on excel columns S . spreadsheet , columns I-M.
Level 3: Data sources
Hot tip: S implicity
Table 2: Statistical Analysis of Pre-Post Workshop Questionnaire - Paired t-test Table 2 outlines the statistical analysis of the data presented in Tables 3 to 7. Pre & Post Test n= Pre-test Mean Post test-Mean Difference Significance Level P< 0.001 Question 1 59 2.64 3.98 1.33 P< 0.001 Question 2 59 3.64 4.17 0.52 P< 0.001 Question 3 59 2.76 4,20 1.44 P< 0.001 Question 4 59 3.37 4.17 0.79 Question 5 59 3.19 4.20 1.01 P< 0.001 Table 2 shows the mean differences between pre workshop and post workshop responses for each question from 59 participants. Data was subject to a paired t-test to determine the p value for each question. This indicated the outcome to be highly significant at p<0.01 as the difference in mean is significant and not due to chance, which indicates attitudinal change and knowledge transfer.
Or this? Growth in financial knowledge: Responses from 59 workshop participants 5 4.2 4.2 4.0 4 3.4 3.2 3 2.6 2 1 It's a wast e of time to think In order t o avoid debt s and The only way t o pay bills about managing money financial st ress it ’ s and saving is t o spend less import ant t o underst and t han you earn how loans, credit cards, int erest and mobile phone plans work Pre-t est Mean Post t est -Mean
Conclusion S tandards of evidence: Move up a level Level 1: S MART KPIs AND learn from others Level 2: Know your endgame AND involve stakeholders Level 3: Plan data collection AND avoid silos Disseminat ion: Infographics
Recommend
More recommend
