KS2 Mathematics Content Validation Study Paul E. Newton 5 - - PowerPoint PPT Presentation

KS2 Mathematics Content Validation Study

Paul E. Newton 5 December 2017

Chronology of my involvement Test Preparation (IOE-Nuffield) ■ November 2013

□ joint applicant on Melanie’s successful bid to Nuffield

■ October 2014

□ switched role to ‘adviser’

Content Validation (Ofqual) ■ October 2014

□ joined Ofqual

■ Early 2016

□ began to plan content validation study in advance of new KS2 tests

Why was Ofqual interested in the new KS2 tests? Ofqual has two statutory objectives in relation to national assessments:

• 1. Assessments standards

□ to promote their development and implementation (NAs are sufficiently valid)

• 2. Public confidence

□ to promote public confidence in them (NAs are trusted)

The Ofqual study had a different focus from the IOE-Nuffield study Test Preparation Teachers’ approaches to preparing pupils for KS2 mathematics tests ■ perceived patterns, over time, in test questions set; and ■ teachers’ responses to them Content Validation STA’s approach to domain sampling in the new suite of mathematics tests ■ actual patterns, during 2016, in test questions set; and ■ their relation to the new NC

Derives from DfE’s National Curriculum Mediated by STA’s Test Framework

NC Framework = point of reference for STA’s approach to domain sampling

General aims Specific teaching (learning) objectives

NC Framework = point of reference for STA’s approach to domain sampling

Test Framework re-represents NC’s 173 bullet points as 195 requirements

Content domain with 9 ‘strands’ (straight from the NC) Cognitive domain with 5 ‘strands’ (developed by STA for the new maths tests)

Test Framework specifies two domains (for the purpose of sampling)

Number, Ratio and Algebra

1. Number (N) – number and place value 2. Number (C) – addition & subtraction, multiplication & division 3. Number (F) – fractions 4. Ratio and proportion (R) 5. Algebra (A)

Measurement, Geometry and Statistics

6. Measurement (M) 7. Geometry (G) – properties of shapes 8. Geometry (P) – position and direction 9. Statistics (S)

1. Depth of understanding 2. Computational complexity 3. Spatial reasoning & data interpretation 4. Response strategy

Calculation Geometry – properties and shapes

Questions differ in terms of the subject content they test

It’s reasonably straightforward to distinguish between, say, a calculation question and a geometry question.

Calculation Calculation

Questions also differ in terms of the cognitive demands they make

Although it’s far less obvious what kind of features distinguish a less demanding calculation question from a more demanding one.

Rationale underlying the STA’s cognitive domain ■ Questions differ in terms of the nature, and/or the level, of the cognitive challenges (i.e. demands) that they pose. ■ It’s just as important that the test samples adequately from the cognitive domain as from the content domain. ■ The four cognitive domain strands aim to capture the most important kinds of cognitive demands (for KS2 maths questions).

Depth of Understanding (as defined by STA) ■ This strand is used to assess the demand associated with recalling facts and using procedures to solve problems.

□ Questions requiring less depth of understanding require simple procedural knowledge, such as the quick and accurate recall of mathematical facts or the application of a single procedure to solve a problem. □ At intermediate levels of demand, a question may require the interpretation of a problem or application of facts and procedures. However, the component parts of these questions are simple and the links between the parts and processes are clear. □ At a high level of demand, a greater depth of understanding is expected. Questions may require that facts and procedures will need to be used flexibly and creatively to find a solution to the problem.

Lowest Level of (DoU) Demand High Level of (DoU) Demand

Depth of Understanding

LOWEST = SOME (not none)

Test Framework sampling requirements for the cognitive domain

Test Framework sampling requirements for the content domain

Principal Research Questions

• 1. Do items from the KS2 test embody content and cognitive demands that are

consistent with the national curriculum (item relevance)?

• 2. Does the KS2 test effectively sample those content and cognitive demands

adequately (test representativeness)? … in terms of how STA experts rated the

■ content demands of each item from each of the three papers ■ cognitive demands of each item (for each strand) from each of the three papers

… in terms of how 12 independent experts rated the

■ content demands of each item from each of the three papers ■ cognitive demands of each item (for each strand) from Papers 2 and 3

Test Framework sampling requirements for the cognitive domain

Test Framework sampling requirements for the content domain

What about lower level sampling expectations for the content domain?

Independent SMEs’ views on content demands

What the results looked like (e.g. Paper 3)

Independent SMEs’ views on cognitive demands We’d like you to rate each item (from Paper 2 and 3) ■ using the STA definition for each of the four cognitive demand strands

□ DO your best to understand exactly what STA means by demand (for each strand)

■ using your own sense of how demanding the item is (in terms of each strand)

□ DON’T necessarily try to ‘replicate’ STA application of 4-point-scale (for each strand) □ USE scale to indicate your opinion of the level of demand of the item (for each strand)

What the results looked like (Paper 2 and Paper 3)

What the results looked like (Paper 2 and Paper 3)

Conclusions

• 1. In terms of how the STA

(i) represented domain sampling requirements for the mathematics curriculum, and (ii) judged the content and cognitive demands of items from the 2016 test

• a. the items were all relevant to the NC, and
• b. the test overall was representative of the NC
• 2. Our SMEs largely agreed with how STA judged the content and cognitive

demands of items from the 2016 test, which

• a. provided additional support for conclusion 1, and
• b. supported the plausibility of the innovative cognitive domain strands

STA’s approach to domain sampling compares favourably with approaches adopted for similar tests, internationally.

However… Still room for debate over the ‘proper’ interpretation of domain sampling requirements for the mathematics curriculum ■ STA’s atomistic approach to domain sampling tends to focus attention more

• n the NC’s (bullet point) teaching requirements than on its Aims

■ STA’s guidelines require only a small number of high cognitive demand items

Consider how the depth of understanding strand relates to problem solving The Test Framework distinguishes between rating points 3 and 4 on the depth of understanding strand: 3 use facts and procedures to solve more complex problems 4 understand and use facts and procedures creatively to solve complex or unfamiliar problems (Bear in mind that STA rated 5/110 marks from the 2016 test at DoU 4) The Aims of the NC for mathematics include: ■ can solve problems by applying their mathematics to a variety of routine and non-routine problems with increasing sophistication, including breaking down problems into a series of simpler steps and persevering in seeking solutions.

For full details of this research…