Evaluation: Hot tips from The Ian Potter Foundation Dr S quirrel - - PowerPoint PPT Presentation

▶
evaluation hot tips from the ian potter foundation
SMART_READER_LITE
LIVE PREVIEW

Evaluation: Hot tips from The Ian Potter Foundation Dr S quirrel - - PowerPoint PPT Presentation

Apr 05, 2023 494 likes •842 views

Evaluation: Hot tips from The Ian Potter Foundation Dr S quirrel Main, Research and Evaluation Manager Before we begin Please note that IPF staff may be taking photos/ videos during this workshop for social media purposes. If you do

slide-1
SLIDE 1

Evaluation: Hot tips from The Ian Potter Foundation

Dr S quirrel Main, Research and Evaluation Manager

slide-2
SLIDE 2

Before we begin…

Please note that IPF staff may be taking photos/ videos during this workshop for social media purposes. If you do not wish for your image to be included in the publicity, please let any staff member know.

slide-3
SLIDE 3

Presentation Overview

S tandards of evidence

Level 1: Grantee tips and KPIs

Level 2: Endgame and sustainability

Level 3: Resources and ideas

Dissemination: Infographics

slide-4
SLIDE 4

Evaluation vs. chasing unicorns…

slide-5
SLIDE 5

S tandards of Evidence

slide-6
SLIDE 6

S tandards of Evidence

slide-7
SLIDE 7

Level 1: Top tips

  • 1. Train the volunteers conducting surveys
  • 2. Ensure redundancy in the key knowledge holders (bikes-

cancer-hearts)

  • 3. Hold immediate staff-wide dissemination training as

soon as possible

  • 4. S

tart early: consultations with stakeholders take a long time

  • 5. Capture baseline data
  • 6. Use pre-existing, validated surveys (North Carolina

Family Assessment S cales)

slide-8
SLIDE 8

Level 1: S MART KPIs

Take a moment to review your goals in pairs. Listen to your partner’s goals and provide feedback.

 Are they SMART?  Specific  Measurable  Achievable  Rewarding/ Relevant  Time-bound

How can your partner improve their measurements? Have time? Take a moment to review your long-term outcomes in pairs…

 How will your partner begin to measure/ collect data

for their longer-term outcomes?

 Challenges?

S

  • lutions?
slide-9
SLIDE 9

Level 2: Evaluation key steps

Det ermine endgame Involve st akeholders Circulat e t ender Finalise proposal Review first draft Disseminat e

slide-10
SLIDE 10

Level 2: Decide your endgame

S

  • urce: Alice Guglev and Andrew S

t ern. What ’s Y

  • ur End-game?

Global Development Incubat or. 30 January 2014.

Endgame Example 1 Open source Firefox 2 Replication Job S upport 3 Government adoption OoHC pilot 4 Commercial adoption Pharmaceuticals 5 Mission achievement One Disease 6 S ustained service S TREAT

slide-11
SLIDE 11
slide-12
SLIDE 12

Endgame influences measurement: 26 not 14 weeks…

slide-13
SLIDE 13

Level 2: Tailor your evaluation…

Input s Act ivit ies Out put s S hort - t erm

  • ut comes

Medium- t erm

  • ut comes

Long- t erm

  • ut comes
  • How does the program work?
  • What did the program

accomplish?

  • What has influence how the

program works?

  • What outcomes resulted from the

program?

  • When and to whom?
  • What is the program’s

effectiveness? (Findings, not Rec’s)

  • How do changes compare to other

programs/ sectors?

slide-14
SLIDE 14

Level 2: Who would you invite?

slide-15
SLIDE 15

From Level 2 to Level 3: Why bother with a control group?

S

  • urce: S

tephen Taylor, Volker S choer and Thabo Mabogoane

slide-16
SLIDE 16
  • TIPFEP
  • Example RFQ
  • BetterEvaluation.org
  • IssueLab (ROIs!)
  • Each other!

Level 3: Evaluation resources

slide-17
SLIDE 17

Level 3: No silos

slide-18
SLIDE 18

Level 3: Data rehearsal

slide-19
SLIDE 19

Level 3: S tructured Pyramid Analysis Plan

Dependent variables

Data sources

Goal/

  • utcome
slide-20
SLIDE 20

Enter your SMART goal here

Dependent Variable 1 Data source 1a Data source 1b Dependent Variable 2 Data source 2a Data source 2b Dependent Variable 3 Data source 3a Data source 3b

Facilitate the rapid uptake of the new technology to advance WRH research proj ects.

slide-21
SLIDE 21

Enter your SMART goal here

Dependent Variable 1 Data source 1a Data source 1b Dependent Variable 2 Data source 2a Data source 2b Dependent Variable 3 Data source 3a Data source 3b

Facilitate the rapid uptake of the new technology to advance WRH research proj ects.

slide-22
SLIDE 22

Enter your SMART goal here

Dependent Variable 1 Data source 1a Data source 1b Dependent Variable 2 Data source 2a Data source 2b Dependent Variable 3 Data source 3a Data source 3b

Train 50 users at workshops, with 20 becoming fully independent users and machine usage rates >= 40 hours per week.

number of fully independent users of t he inst rument number of hours of usage per week at t endance at hands-on workshops

slide-23
SLIDE 23

Enter your SMART goal here

Dependent Variable 1 Data source 1a Data source 1b Dependent Variable 2 Data source 2a Data source 2b Dependent Variable 3 Data source 3a Data source 3b

Train 50 users at workshops, with 20 becoming fully independent users and machine usage rates >= 40 hours per week.

Tot al unique count of workshop at t endees (from workshop sign- in)

number of fully independent users of t he inst rument number of hours of usage per week at t endance at hands-on workshops

Count s of Excel column H “ workshop part icipant t it le” (st udent , post - doc, research assist ant , et c) Int ernal inst rument usage dat abase, t ot al hours (1 Jan 2017 t hrough 31 Dec 2018) ÷ 52

N/ A

Int ernal inst rument usage dat abase, count of unique user IDs S ign-off list of ‘ qualified independent users’ by chief t echnician, t ot al count

slide-24
SLIDE 24

Enter your SMART goal here

Dependent Variable 1

Data source 1a Data source 1b

Dependent Variable 2

Data source 2a Data source 2b

Dependent Variable 3

Data source 3a Data source 3b

70%

  • f children participating in DPIL

report an increased affinity for reading

slide-25
SLIDE 25

Enter your SMART goal here

Dependent Variable 1

Data source 1a Data source 1b

Dependent Variable 2

Data source 2a Data source 2b

Dependent Variable 3

Data source 3a Data source 3b

70%

  • f children participating in

DPIL report an increased affinity for reading

slide-26
SLIDE 26

Enter your SMART goal here

Dependent Variable 1

Data source 1a Data source 1b

Dependent Variable 2

Data source 2a Data source 2b

Dependent Variable 3

Data source 3a Data source 3b

70%

  • f children participating in DPIL program report an

increase of at least 1-point on their reading motivation from baseline to 12-months into the program.

Average score on 5-quest ion Habit ual Reading Mot ivat ion Quest ionnaire (Möller and Bonerad, 2007).

  • Baseline. S

t ored

  • n excel

spreadsheet , columns I-M.

“ affinit y for reading” – baseline average “ affinit y for reading” – point difference aft er 12 mont hs

Difference between score on 5-question Habitual Reading Motivation Questionnaire (Möller and Bonerad, 2007), baseline and 12-

  • month. S

tored on excel spreadsheet, columns S.

Child unique ident ifier. Column “ Child ID number” excel spreadsheet , column B. Child unique ident ifier. Column “ Child ID number” excel spreadsheet , column B.

slide-27
SLIDE 27

Enter your SMART goal here

Dependent Variable 1

Data source 1a Data source 1b

Dependent Variable 2

Data source 2a Data source 2b

Dependent Variable 3

Data source 3a Data source 3b

70%

  • f children participating in DPIL program report an

increase of at least 1-point on their reading motivation from baseline to 12-months into the program.

Average score on 5-quest ion Habit ual Reading Mot ivat ion Quest ionnaire (Möller and Bonerad, 2007).

  • Baseline. S

t ored

  • n excel

spreadsheet , columns I-M.

“ affinit y for reading” – baseline average “ affinit y for reading” – point difference aft er 12 mont hs

Difference between score on 5-question Habitual Reading Motivation Questionnaire (Möller and Bonerad, 2007), baseline and 12-

  • month. S

tored on excel spreadsheet, columns S.

[Ext ernal dat a examples]

Child unique ident ifier. Column “ Child ID number” excel spreadsheet , column B. Child unique ident ifier. Column “ Child ID number” excel spreadsheet , column B.

ABS (cite specific table / cube!) AIHW-S HS “ Other

  • utcomes

national” (cite specific table / cube!)

slide-28
SLIDE 28

Level 3: Data sources

slide-29
SLIDE 29

Hot tip: S implicity

slide-30
SLIDE 30
slide-31
SLIDE 31

Pre & Post Test n= Pre-test Mean Post test-Mean Difference Significance Level Question 1 59 2.64 3.98 1.33

P< 0.001

Question 2 59 3.64 4.17 0.52

P< 0.001

Question 3 59 2.76 4,20 1.44

P< 0.001

Question 4 59 3.37 4.17 0.79

P< 0.001

Question 5 59 3.19 4.20 1.01

P< 0.001

Table 2 shows the mean differences between pre workshop and post workshop responses for each question from 59 participants. Data was subject to a paired t-test to determine the p value for each question. This indicated the

  • utcome to be highly significant at p<0.01 as

the difference in mean is significant and not due to chance, which indicates attitudinal change and knowledge transfer. Table 2: Statistical Analysis of Pre-Post Workshop Questionnaire - Paired t-test Table 2 outlines the statistical analysis of the data presented in Tables 3 to 7.

slide-32
SLIDE 32

Or this?

2.6 3.4 3.2 4.0 4.2 4.2 1 2 3 4 5 It's a wast e of time to think about managing money In order t o avoid debt s and financial st ress it ’ s import ant t o underst and how loans, credit cards, int erest and mobile phone plans work The only way t o pay bills and saving is t o spend less t han you earn

Growth in financial knowledge: Responses from 59 workshop participants

Pre-t est Mean Post t est -Mean

slide-33
SLIDE 33

Conclusion

 S

tandards of evidence: Move up a level

 Level 1: S

MART KPIs AND learn from others

 Level 2: Know your endgame AND involve

stakeholders

 Level 3: Plan data collection AND avoid silos  Disseminat ion: Infographics

slide-34
SLIDE 34