USAIDs Early Grade Reading and Early Grade Math Impact Evaluations: - - PowerPoint PPT Presentation

usaid s early grade reading and early grade math impact
SMART_READER_LITE
LIVE PREVIEW

USAIDs Early Grade Reading and Early Grade Math Impact Evaluations: - - PowerPoint PPT Presentation

USAIDs Early Grade Reading and Early Grade Math Impact Evaluations: Lessons Learned National Education Week Subtitle placeholder Reforming the Education Sector for Effective Service Delivery: Embracing Innovations Monica Gadkari Chief of


slide-1
SLIDE 1

USAID’s Early Grade Reading and Early Grade Math Impact Evaluations: Lessons Learned

Subtitle placeholder National Education Week Reforming the Education Sector for Effective Service Delivery: Embracing Innovations

Monica Gadkari Chief of Party, USAID Partnership for Education, Evaluating Systems mgadkari@socialimpact..com

slide-2
SLIDE 2

Program Designs Understanding the Impact Evaluation Impact and outcome results Fidelity of implementation Recommendations

Presentation Outline

FOOTER GOES HERE

9/5/19 2 This presentation looks at lessons learned with regards to teacher performance from USAID’s Early Grade Reading and Math Programs:

slide-3
SLIDE 3

PROGRAM DESIGNS

9/5/19 3

Numeracy Pilot Early Grade Reading Program Revised syllabus with greater emphasis on conceptual understanding and mathematical reasoning IMPROVED PEDAGOGY Use of Ghanaian language and focus on phonics versus “whole language” approach to literacy acquisition Teacher resource guide with weekly work schemes, games and activities, performance standards, and assessment tasks MATERIALS Ghanaian language materials including scripted lesson plans, pupil workbooks, supplementary readers, and assessment tools Training of Math Coaches who facilitate INSET and weekly learning circles to help teachers put the resources into practice TRAINING Teachers trained on use of new materials through training events and weekly school-based INSET meetings Math Coaches and Head Teachers to provide regular classroom observation and coaching/feedback COACHING & MONITORING Head Teachers, Curriculum Leads, and Circuit Supervisors regularly observe and coach teachers on instructional practices

slide-4
SLIDE 4

EVALUATION QUESTIONS

9/5/19 4 Impact / Goal Questions

Math After 1 year, to what extent does the Numeracy Pilot improve early grade mathematics performance for P1 and P2 learners, as measured by the Early Grade Mathematics Assessment (EGMA) and select subtests of the Ghana Early Numeracy Assessment (GENA)? Reading After 2 years, to what extent does the Early Grade Reading Program improve P1 and P2 reading skills, as measured by the Early Grade Reading Assessment (EGRA)?

slide-5
SLIDE 5

EVALUATION RESEARCH DESIGN

9/5/19 5

Sampling frame: all schools in Shai Osudoku and New Juanbeng (121 total) 60 schools randomly assigned to numeracy pilot 61 schools randomly assigned to control

Math

Randomized Controlled Trial (RCT)

slide-6
SLIDE 6

EVALUATION RESEARCH DESIGN

FOOTER GOES HERE

9/5/19 6

Intervention Comparison

Quasi-Experiment Using Statistical Matching

Reading

slide-7
SLIDE 7

PROGRAM IMPACT MATH

FOOTER GOES HERE

9/5/19 7 Math

Pupil Mathematics Performance

EGRA: Procedural understanding of math 3.8% increase in P1 5.3% increase in P2 Small reduction in zero scores GENA: Conceptual Understanding of math 17.3% increase in P1 18.9% increase in P2 Small reduction in zero scores

slide-8
SLIDE 8

PROGRAM IMPACT READING

FOOTER GOES HERE

9/5/19

Letter Sounds

s y m o Ɛ y m N u k Ɔ a

Non-Word Reading

kɛw rɔm mɛn nuw lim lɔm gɔm lam mun

Abasɛm tiawa bi ni. Mepɛ sɛ wokenkan no dennennen, ne ntɛmntɛm ma me. Wokenkan wie a, mebisabisa nsɛm bi afa nea woakenkan no ho. Meka sɛ “Fi Ase” a kenkan abasɛm no sɛnea wubetumi biara. Wudu asɛmfua bi so na wunnim a, kɔ asɛmfua foforo so.

Oral Reading Fluency Reading Comprehension

­ 18.54

letter sounds per minute

­ 4.04

words per minute

­ 5.17

words per minute

Pre-Reading Initial Reading Fluency/Comprehension

Listening Comprehension

s y m o Ɛ y m N u k Ɔ a

­ 6.17

Percent

­ 4.72

Percent

slide-9
SLIDE 9

PROGRAM IMPACT READING

0.9 1.54 8.64 6.9 22.72 33.76 5 10 15 20 25 30 35 40 Average Top 25% Top 10% Control Treatment

The average gains are not representative of all students. The most significant gains exist amongst the top 10% and top 25% of students. The driver of the low averages are zero scorers

slide-10
SLIDE 10

The following observed changes to teacher performance show greater use of teaching approaches that promote a conceptual understanding of math

TEACHER PERFORMANCE MATH

9/5/19 10 Math

29% more likely to ask students to find and share problem solving strategies 15% more likely to engage in mathematical communication 14% more likely to provide students

  • pportunities to use mathematical reasoning

10-15% more likely to use active learning techniques in classroom Reduced frequency of assessment but greater response to assessment data

slide-11
SLIDE 11

TEACHER PERFORMANCE READING

9/5/19 11 Reading

7% increase in time spent on learning activities 28% increase in coaching and mentoring 20% increase in coaching based on

  • bservation

Reduced frequency of assessment Improvement in implementation of 4 out

  • f 5 teaching best practices

Most outcomes surrounding changes in pupil and teacher behaviour were positive

slide-12
SLIDE 12

FIDELITY OF IMPLEMENTATION MATH

9/5/19 12

Fully Implemented

  • Training of head teachers
  • Training of math coaches
  • Provision of Materials

Partially implemented

  • Math coaches provided only

20% of planned INSET training

  • Math coaches, head teachers,

and circuit supervisors provided support at 30% dosage

  • Supplementary materials not

developed consistently

In terms of use, lesson plans and teacher guides were widely used, while classroom materials, even when present, were used inconsistently

slide-13
SLIDE 13

FIDELITY OF IMPLEMENTATION READING

About 80% of treatment schools in a state of “full” or “near full” implementation fidelity on multiple components.

On-Track Component Lesson Performance Component Pupil Materials Component Teacher Materials Component Supplemental Materials Component Teacher In- Training Component HT/CL Coaching Component SBI Coaching Component CS Monitoring Component

slide-14
SLIDE 14

PREDICTORS OF PERFORMANCE MATH AND READING

FOOTER GOES HERE

9/5/19 14

Predictor Outcome Pupil always hungry at the start of school

Teacher punishes pupil for poor performance

School is in rural locality

Low language match

Predictor Outcome High fidelity to program

Enthusiasm for reading and math

Female student

Higher percent of reading teachers who are female

slide-15
SLIDE 15

PROGRAM PREDICTORS OF SUCCESS

The individual program component most significantly associated with program impact is lesson performance the extent to which teachers follow the scripted lesson plans and engage their students in doing so

  • 0.2
  • 0.1

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

Pace Lesson Performance Pupil Workbook

  • Sup. Materials

Reading Instr. Training

  • Obs. & Coaching

SBI CS Monitoring

Letter Sounds ORF

slide-16
SLIDE 16

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Low FOI Partial FOI Near-Full FOI Full FOI

Effect Size ORF

Overall fidelity to the program was

  • essential. The effect

size of the program schools with low or partial fidelity were significantly lower than those with near full or full FOI

PROGRAM PREDICTORS OF SUCCESS

slide-17
SLIDE 17

RECOMMENDATIONS

9/5/19 17 Outcomes

Ø Program lesson plans and teacher guides are helpful to teachers and are associated with success Ø For full programmatic scale up, MOE and GES management should integrate more accountability mechanisms that ensure greater fidelity

  • f

implementation Ø Teacher training curriculum should encourage teachers to give positive rather than negative reinforcement to students. Ø Research how to overcome the gap between the large conceptual gains and the smaller procedural gains we see on math performance

slide-18
SLIDE 18

Thank you