in Low- and Middle-Income Countries to Measure Cognitive Outcomes - - PowerPoint PPT Presentation

Overview of Basic Education Instruments Commonly Used in Low- and Middle-Income Countries to Measure Cognitive Outcomes

Session 1. Measuring Student Learning Outcomes Marguerite Clarke, PhD The World Bank Group

Overview

2

• 1. Cognitive outcomes
• 2. Common instruments
• 3. Technical issues
• 4. Basic steps to keep in mind

Learning domains – ‘cognitive’ and ‘non- cognitive’ outcomes

Source: Learning Metrics Task Force (2013) http://www.brookings.edu/about/centers/universal-education/learning-metrics-task-force-2

3

Common instruments for measuring cognitive outcomes

4

Pros Cons Best for… EGRA Granular data useful for informing daily classroom practice Not optimized for aggregate reporting Targeting classroom practices NLSA Representative data linked to nationally relevant standards Technical quality and comparability issues Tracking policy reforms and system quality ILSA Comparable data based

• n internationally

relevant standards Typically not treatment sensitive Tracking policy reforms and system quality Bespoke/adapted Optimized for relevance to particular context/project Technical quality and comparability issues Determining effectiveness of specific intervention

Common instruments for measuring cognitive outcomes - EGRA

5

Pros Cons Best for… EGRA Granular data useful for informing daily classroom practice Not optimized for aggregate reporting Targeting classroom practices

EGRA description

6

• Speeded measure of early reading skills (Grades 1-3)
• One-on-one oral delivery
• Supply-type items
• No background questionnaire
• 8 subscale scores (speeded)

– Letter name knowledge – Phonemic awareness – Letter sound knowledge – Familiar word reading – Unfamiliar non-word reading – Passage reading and comprehension – Listening comprehension – Dictation

https://www.eddataglobal.org/reading/index.cfm

Best suited for

7

 Informing daily classroom practice by teachers  Measuring impact of targeted interventions on classroom practices and student learning  Monitoring status and growth at individual and group level over short time frame on specific subscales

Common instruments for measuring cognitive outcomes – Bespoke/adapted

8

Pros Cons Best for… Bespoke/adapted Optimized for relevance to particular context/project Technical quality and comparability issues Determining effectiveness of specific intervention

Bespoke/adapted general description

9

• Typically based on existing NLSA or

ILSA (any grade/age)

• Domain-specific knowledge and

skills

• Publicly released items
• Written, group-administered,

standardized

• Selection- and supply-type items
• Background questionnaire
• Subscale and overall scores

Best suited for

10

 Informing policy decisions  Measuring impact of specific interventions

• n student learning

 Comparisons with other/external groups at item level

Example: IEA tests adapted for Benin

11

• Purpose:
• Baseline in math and reading for schools in RCT
• Sample:
• 144 schools, 3,817 students (grade 5)
• Instrument development:
• Released items and materials from previous IEA
• Developed in collaboration with local experts
• Mathematics: 30 items, all multiple choice
• Reading: 25 items; docs, narrative/expository prose
• Background questionnaire

12

13

14

15

Technical issues

16

1. Treatment sensitivity/information value 2. Reliability 3. Comparability

Treatment sensitivity (1)

17

Treatment sensitivity (2)

18

Treatment sensitivity (3)

19

Reliability

20

Suitable for EGRA

21

Comparability (1)

22

23

24

Comparability (2)

Basic steps to keep in mind

Volume 1: Assessing National Achievement Levels in Education Volume 2: Developing Tests and Questionnaires Volume 3: Implementing a National Assessment Volume 4: Analyzing Data from a National Assessment Volume 5: Using the Results of a National Assessment

http://go.worldbank.org/M2O1YDQO90

Thank you

Marguerite Clarke mclarke2@worldbank.org World Bank Group