Unpacking the Learner- Selection Suitcase : A Synthesis of - - PowerPoint PPT Presentation

Unpacking the Learner- Selection Suitcase: A Synthesis of Evaluation Findings from Learner – Directed Educational Improvement Initiatives.

Benita Williams Mobile: +27 82 772 9709 bwilliams@feedbackpm.com

“Case” Study Questions

1) You have to select 100 learners for a two year FET Saturday school programme running in the district. A third of the “top learners” score below 30% on a standardized assessment of Grade 10 Maths knowledge. Do you select them? 2) You want to select learners for a post-matric programme. You have WAIS III scores, scholastic test scores in Maths and English, past matric results, and face to face interview notes, How do you combine all of these to make a recommendation? Do you think all of these tests are necessary? 3) You want to select learners with the best chance of success in an enrichment programme. Do you rather select a learner with past performance of 30% or 60%?

Three “Case” Studies…

 Case 1: Supplementary ICT Education Initiative –

Using School Assessment Results to Select Learners

Case 1: Supplementary ICT Education Initiative

 Target - “learners with potential”  Activities

 1) Mathematics Saturday classes,  2) Mid-week sessions using ICT technology to do project work

intended to enrich their understanding of the Mathematics and Physical Science content taught in class, and

 3) mid-week sessions for learners engaging in self-paced

learning with online content intended to further learners’ understanding of basic Mathematics and Physical Science concepts

Case 1: Supplementary ICT Education Initiative

 Objectives

 Develop independent, self driven, self motivated learners, with

good quality Mathematics and Physical Science passes.

 Learners who emerge from the programme would be inspired

to pursue careers in the Mathematics and Science fields,

 Learners will be emotionally and academically prepared to

enter tertiary institutions.

 Selection Tools

 Districts Select Schools from townships that are functional and

have access to ICT labs

 Schools select “top” learners based on school results – must

have maths and science and pass both learning areas

 Exclude learners involved in other projects

Case 1: Supplementary ICT Education Initiative

 Selection Result

 Selected the learners, and tested them independently at the

Saturday school almost all got 25% - 35%

 Facilitators of ICT skills classes reported that the learners able

to learn the ICT skills, but that they are worried that the programme content are pitched at learners who are supposed to have a basic understanding of maths and science concepts

 Some of the learners could not do the self paced learning in

the ICT lab, because their school did not have a lab

 Some of the learners dropped in and out – depending on their

participation from other schools

10 20 30 40 50 60 70 80 90 100 1 2 3 4 5 6 7 8 9 Learner Marks

Range of Individual Learner's T

• tal

T est Marks

Boitumelong Bokomoso Eketsang Katlehong Masisebenze Mpilisweni Tembisa Learners With Potential- Strict Definition Learners With Potential- Expanded Definition School A School B School C School D School E School F School G School H

Case 1: Lessons Learnt

 Drawing learners from different schools into a

programme, requires a comparable measure across schools.

 If the programme is targeted at a specific performance

level, the measurement needs to exclude those who do not qualify

 Practical constraints have a bigger impact on project

success than previous performance (

 Participation a big issues.

Case 2: Saturday School Programme

 for 400 learners in four non-urban sites – Using

Standardised Maths and Science tests to Select Learners

Case 2: Saturday School Programme

 Target:

 “Bright Minds” from rural schools in a 60km radius from a

Saturday School facility

 Learners with 40% - 60% in Grade 10

 Activities:

 T

wo years of Saturday School and holiday tuition in

 Maths  Physical Science  English  Career Guidance  Life Skills  Computer Technology Training

Case 2: Saturday School Programme

 Objectives

 Increase the number of good quality passes in maths and

science

 Open pipeline for entrance to further studies in SET careers

 Selection Tools

 T

ested 700 learners from about 76 schools identified by the DBE

 Used a standardised test administered by an external service

provider

 Selected “top” 100 learners from each district

y = 1.1831x + 13.14 R² = 0.1377 y = 0.8624x + 32.259 R² = 0.1045 10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100 Mathematics Matric Exam Mark Mathematics Selection Test Mark

Mathematics Results for the Control and Experimental Groups

Control Experimental Linear (Control) Linear (Experimental)

CutOff = 22 % MatricDistinctions

Case 2: Saturday School Programme

20 40 60 80 100 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 Test Result

Distinction Candidates - Selection test and Matric results

Selection Test Matric Exam

• 20.0
• 10.0

0.0 10.0 20.0 30.0 40.0

0 to 70% 71% to 75% 76% to 80% 81% to 85% 85% to 90% 90% to 95% 96% to 99% 100%

Change in mark from pre to post-test Attendance Rate

Maths Physical Science English Linear (Maths) Linear (Physical Science) Linear (English)

Case 2: Lessons Learnt

 The selected participants weren’t what they looked for, but

the programme could be adapted - Good idea!

 If the donor kept to the original criteria, a number of learners

who was able to produce good quality passes, would have been excluded

 Still, a number of learners who could have produced good

quality passes were excluded

 Misclassification is more likely if you use only one test result.

Case 3: Post-matric programme

 T

• help learners improve their maths results –

Using a battery of psychological and scholastic tests.

Case 3: Post-matric programme

 Target:

 Learners under the age of 21 years  Have completed the Grade 12 year of schooling  Have previously sat the matric examinations  Have mathematics, and Physical Science and/or accounting as

Grade 12 school subjects

 Be able to attend the Programme full time for one year  Be able to obtain a Grade 12 matriculation by the end of the

programme

Case 3: Post-matric programme

 Activities:

 8 months of tuition 08:00 to 17:00 Monday to Friday  Maths & Physical Science / Accounting IEB exam preparation  Career Guidance, Life Skills  Computer T

echnology

 English for non-exam purposes  Assistance with bursary applications, study applications, job

applications

 Objectives

 Improve number of good quality passes  Emotionally prepared for further studies and world of work

Case 3: Post-matric programme

 Selection Tools

 Initial Application form  “Mental Alertness” test  Assessment Centre

 WAIS III  Clinical Observations  Thematic Apperception Test & Draw a Person Test  Grade 12 Reading Comprehension Test (old Grade 12 exam paper)  A spelling test (Called the “Spelling Test for Psychologists  A Mathematical Test developed by a school teacher to be

representative of a typical Grade 12 Mathematics Assessment.

 Interactive Group Assessment – Panel

 Selection – March Exams: Final Selection

Case 3: Post-matric programme

 Selection Result  Not necessarily above average in terms of aptitude  Excludes learners who are decidedly below average ito aptitude.  Not very well prepared scholastically – even learners who have

relatively higher aptitude, perform poorly on the scholastic tests.

 Do not have insurmountable emotional challenges,  Able to express themselves at least at a very basic level in English  Likely to come from impoverished backgrounds, but at least able

to function socially and occupationally

 Able to negotiate their social environments. Those who are unable

to adapt to the programme demands, or able to reconcile their home circumstances with the performance drops out

 Average difference between the pre and post maths marks

was 9.9%.

 Best predictor of improvement – previous performance. The

weaker, the bigger improvement

0% 20% 40% 60% 80% 100% 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53

2009 Cohort Pre and Post Maths Marks

Pre Post

Points to ponder

 Why Select?

 Is selection necessary – What are the Programme pitch or

financial considerations?

 Even if the donor can afford to “select” everyone, remember

there are costs of participation for beneficiaries

 Are you selecting “In” or screening “out”  What counts more – past performance or current

participation?

 You may select, but the participants still get to elect to

participate or not.

Points to ponder

 Select with which tools?

 Choose a selection strategy, test it, and refine it based on the

programme requirements.

 Select Whom?

 Learners with potential  “T

• p learners”

 “Bright Minds”