2019 019 summer er ser eries es Sally y Collier lier - - PowerPoint PPT Presentation

■ ■ ■ ■ ■ ■

2019 019 summer er ser eries es

Sally y Collier lier Chief Regulator

Evaluation uation of re refor

• rm

m

Jul ulie e Swan n Executive Director, General Qualifications Janet net Holloway lloway Associate Director, Standards for Design, Development & Evaluation of General Qualifications Be Beth h Bl Black k Director, Research and Analysis

■ ■ ■

■ Generally satisfied – outcomes mainly as expected ■ Command words and mark scheme expectations ■ Comparison with sample assessment materials ■ Timing

There were some clever discriminating questions that identified the students who really understood the science and the task (GCSE chemistry teacher) Overall, for GCSE middle and upper band students, it is more challenging and more interesting (GCSE PE teacher) This was a tough but great

• paper. It was accessible for all

but had plenty of scope to show what they could do (GCSE history teacher)

■ Tier entry and accountability ■ Aspirational target grades ■ Perceptions and understanding ■ Progression to A level (entry requirements) ■ Mixed tier It feels like a pressure cooker for getting grades out in terms of measuring progress and the accountability measures (GCSE science teacher) Exam boards and Ofqual could do more to inform school leadership teams about the overlapping grades – the messaging around grade 5 students being better suited to FT needs to be louder/stronger (GCSE science teacher)

The mathematical skills were shoehorned in and not necessarily driven by the subject – artificially mathematical (GCSE science teacher) Questions that assess practical skills are much better than controlled assessment (GCSE science teacher) So little of the fieldwork done was actually assessed (GCSE geography teacher)

■ ■ ■ ■

13

Evaluatio uation n & Res esea earch rch

Beth th Black Di Direct ector

• r,

, Research arch & Anal alysi ysis

■

□ □ □

■

□ □ □

■

14

Ev Evaluating uating the impact ct of A l f A level el qua ualification ification refo form rm on n science ence practical ctical skills ills

Practical work directly assessed by teachers throughout the A level course Assessment of a minimum of 12 practical assignments (per subject) Performance assessed against ‘Common Practical Assessment Criteria’ (CPAC) Students receive a separate result (Pass/Unclassified)

• Min. 15% of marks in written exams allocated

to indirect assessment of practical skills

The he ass ssessm essment ent of p f pract ctical ical work k in new A l A levels els

16

The he ass ssessm essment ent of p f pract ctical ical work k in new A l A levels els

17

18

Ofqual’s research programme

• Study 1: Teacher interviews – Perspectives on A level reform after one year
• Study 2: Pre and Post reform evaluation of practical ability – A comparison of science

practical skills in pre and post reform cohorts of undergraduate students

• Study 3: Valid discrimination in practical skills assessment – An exploration of classification

reliability when assessing the performance of practical skills

• Study 4: Technical functioning of assessment – An analysis of A level examination items that

assess science practical skills

19

Ofqual’s research programme for A level science pr pract ctic icals als

• Study 1: Teacher interviews – Perspectives on A level reform after one year
• Study 2: Pre and Post reform evaluation of practical ability – A comparison of science

practical skills in pre and post reform cohorts of undergraduate students

• Study 3: Valid discrimination in practical skills assessment – An exploration of classification

reliability when assessing the performance of practical skills

• Study 4: Technical functioning of assessment – An analysis of A level examination items that

assess science practical skills

20

St Study dy 2: 2: Rese searc arch h with th univer versities sities

Research question: What impact have changes to A level science had on the practical skills obtained by students? Method: Quasi-experiment which compares the practical skills of two groups: a. New undergraduates who studied pre-reform A-level science (2016) b. New undergraduates who studied post-reform A-level science (2017, 2018)

• Data collected at universities
• Practical Skills Measure (PSM)
• Biology, Chemistry and Physics

21

St Study dy 2: 2: The PSM carousel

• ‘Carousel’ of practical tasks (15 mins for each)
• Observed and assessed by university lecturer or

PhD student

• Focus is on direct assessment of practical skills
• Participants are ‘marked’ against binary

assessment criteria

• Questionnaire about A-level attainment, level of

confidence, and experience with practical work Subject Unis 2016 2017 2018 Biology 6 138 298 278 Chemistry 4 109 185 202 Physics 5 293 225 131 TOTAL 15 540 708 611

22

St Study dy 2: 2: Bi Biology

• gy PSM

SM st stati tions

• ns

Task 3: Determining concentration of an unknown from a standard curve Task 4: Aseptic technique - streaking plates with mock culture

23

St Study dy 2: 2: Overall all pe perfo formance rmance by ta task sk - Phy hysics sics

Tasks

1) Use of an oscilloscope to measure the EMF of a battery 2) Micrometre and Vernier caliper 3) Measuring resistance with a voltmeter and an ammeter 4) Preparation of a circuit 5) Measurement and timing with a pendulum 6) Use of a signal generator and an oscilloscope

24

St Study dy 2: 2: Overall all pe perfo formance rmance by ta task sk - Ch Chemistry istry

Tasks

1) Setting up a burette 2) Thin layer chromatography 3) Setting up a reflux and distillation 4) Making up a standard solution 5) Iodine clock (kinetics)

25

St Study dy 2: 2: Overall all pe perfo formance rmance by ta task sk - Bi Biolo logy gy

Tasks

1) Making a standard solution and 10 fold dilution 2) Using a compound high power microscope 3) Determining concentration of unknown from standard curve 4) Aseptic technique - streaking plates with mock culture 5) Using an eyepiece graticule 6) Field survey skills

■ Reasons to be optimistic: □ No evidence for a decline in practical skills □ Some tentative evidence for an increase in skills in biology in 2017 & 2018 □ Self-report evidence of an increase in frequency of practical work ■ Caveats: □ There are limitations to the sample and methodology □ Reformed qualifications – 4th cohort starts in September… things may still ‘bed in’.

26

27

Evaluatio uation n of modu dular lar ver ersus s linea ear

Multi-stranded ■ Literature review of the impact of structure on teaching and learning ■ Account of the policy history across qualifications ■ Qualitative research talking to teachers before and after teaching the new linear GCSEs ■ Quantitative analysis of the impact of structure on results ■ Impact on ability to set and maintain standards ■ Economic impact – on schools and colleges, exam boards and the wider system

28

29

Linear gives more time to develop Linear demands long term retention Students switched on by regular modules Motivation from success in early modules Demotivatio n from resit availability Weaker students

• verwhelmed by

linear content Stress from regular modules Linear stressful Linear supports deep learning Linear allows more teaching time Modular exams logistically burdensome Linear level playing field Modular flexible in meeting needs e.g. SEND

Pros & Cons

30

■ ■ ■ ■

Subject Coefficient Standard error Effect size English

• 0.001

0.002 0.001 Mathematics

• 0.034

0.002 0.032 Science 0.001 0.003 0.001 Separate 0.087 0.003 0.091

31

32

GCSE E mathem hematics atics and pr d probl

• blem

em solving ing

■ ■

□

33

■ ■

34

■ ■ ■

□ □ □ □

35

Sample assessments 2017 assessments 36

■ ■ ■

37

A small decrease (2-3 %) in marking consistency with increase problem solving content. Have paid a tiny bit in reliability, in order to increase validity and value of the qualification.

■

□ □ □

■

38

Qu Ques estion tions? s?

41

Summer er award rding ing

Ca Cath h Jadha hav Director, Standards and Comparability

A level entries up in: Biology, chemistry, physics, psychology, sociology Also, computing, politics and Spanish Down in: English subjects and maths

■ ■ ■ ■

■ ■ ■ ■

■ ■ □ □ ■ ■

Combined science tier split Foundation tier: 63% (55% in 2018) Higher tier: 37% (45% in 2018) Largest % increases in entries

• 1. Spanish
• 2. Computer science
• 3. History
• 4. Combined science
• 5. French

■ ■ ■ ■ ■ ■

■ ■ □ □ □ ■ ■ ■

■ ■ ■ □ □ □ ■

■ ■ ■ ■ ■

■ ■ ■

• Centre variability in results
• Results by county
• Grade distributions (9 to 1)

Updated with 2019 on results days Analytics.ofqual.gov.uk

Historic results over time, by subject, for all students, male/female, separate age groups…

Qu Ques estion tions? s?

■

□ □ □ □

■

□ □ □ □ □ □

56

Roger er Taylo ylor Ofqual Chair