Approach to Comparability as Virginia Moved to Online Testing - - PowerPoint PPT Presentation

Approach to Comparability as Virginia Moved to Online Testing

National Conference on Student Assessment

June 2018

1

Introduction

2

 Virginia first investigated mode comparability in fall 2001.  Three End-of-Course tests  Algebra I  Earth Science  English Reading, Literature, and Research  The philosophical approach was to examine whether the tests

were of comparable difficulty in paper and online modes. If so, a common scale score table could be used for both modes.

The first study – Fall 2001

 School divisions were invited to participate in the study after fall

2001 operational testing.

 Each district was certified to have adequate infrastructure to

deliver the online tests.

 Approximately 2200 students participated.  Students were assigned by each school to take either a paper

version or an online version of the same test form.

3

The first study – Fall 2001

 Paper versions of the EOC tests were administered statewide

during the fall.

 Approximately one week later, participating students were

administered an alternate form either on paper or online.

 Students were able to keep the higher score from the study test

• r the operational test.

4

The first study – Fall 2001

5

Form 1 Paper Form 2 Paper Form 2 Computer

Live fall testing Comparability Study Randomly equivalent groups

The first study – Fall 2001

6

EOC Test Condition Computer Paper N of test takers N of test takers English: RLR 301 268 Algebra I 398 365 Earth Science 465 409 Total 1164 1042

The first study – Fall 2001

7

EOC Test Condition Mean Median Standard Deviation Minimum Maximum English: RLR Paper 421.2 426 55.5 252 562 Computer 445.1 445 54.3 285 592 Algebra I Paper 411.4 408 37.0 328 547 Computer 411.9 408 40.3 321 600 Earth Science Paper 402.8 394 45.3 276 600 Computer 409.3 403 48.8 314 554

Comparison of fall operational test scores in the two study groups

The first study – Fall 2001

Comparison of study test raw scores in the two groups

8

EOC Test Condition Mean Median Standard Deviation Minimum Maximum English: RLR Paper 22.98 23 8.41 39 Computer 26.78 27 7.00 9 42 Algebra I Paper 26.03 25 8.35 48 Computer 26.38 26 8.01 5 46 Earth Science Paper 27.70 27 8.39 48 Computer 29.04 28 8.97 49

The first study – Fall 2001

 The computer and paper groups were similar but not as

comparable as was hoped.

 Mean scores on the common live paper test were different for

the two groups.

 Survey responses indicated possible differential motivation in the

two groups.

9

The first study – Fall 2001

Do you consider this an extra opportunity to pass the EOC test?

10

No Yes Computer N 154 754 % 17.0 84.0 Paper N 265 627 % 29.7 70.3

The first study – Fall 2001

Compared to your last testing opportunity, how motivated were you to take this test?

11

Less Motivated 2 About the Same 4 More Motivated Computer N 91 91 372 225 128 % 10.0 10.0 41.0 24.8 14.1 Paper N 220 131 372 100 75 % 24.5 14.6 41.4 11.1 8.4

The first study – Fall 2001

 Performance differences between the study groups on the

• perational fall tests and the study tests, as well as apparent

motivational differences, made it difficult to conclude that scores were comparable in the two modes.

 The focus shifted to adjusting scores for possible mode effects.  Equate the computer form to its previous administration in paper.

12

Fall 2001 – Revised Approach

13

Spring 2000

Form 2 Paper

Fall 2001

Form 1 Paper

Fall 2001

Live Testing Comparability Study

Fall 2001

Form 2 Paper Form 2 Computer Common Persons Common Items

Fall 2001 – Revised Approach

14

Spring 2000

Form 2 Paper

Fall 2001

Form 2 Computer Common items, non-equivalent groups

Fall 2001 – Revised Approach

 Mode effects may or may not be present.  If they are, the scores can be adjusted by equating the computer

version to the paper version of the test.

 Mode effects will be reflected by the magnitude of differences in

item parameter estimates between the two modes.

15

Fall 2001 – Revised Approach

16

 The computer version is equated to the paper version through a

set of linking items.

 Linking items with large displacements are dropped from the

linking set.

 A difference in the average difficulty of non-linking items suggest a

mode effect.

The Second study – Spring 2002

 Two End-of-Course tests  Algebra II – 1,305 students  Biology – 1,882 students  Divisions volunteered to participate.  No assignment of students to conditions.

17

The Second study – Spring 2002

 Students took a computer version of a different form than they

had taken during the operational spring 2002 administration.

 The study form was administered operationally on paper in other

divisions during spring 2002.

 Students were able to keep the higher score from the study test

• r the live test.

18

The Second study – Spring 2002

 The computer version of the test was equated to the operational

paper version.

 Linking items with large differences in item parameters were

• mitted from the anchor set.

 Resulting score tables showed differences between paper and

computer raw scores of 0 to 1 scale score points for Biology and 1 to 2 scale score points for Algebra II.

19

The model for moving tests online

 Schools are given a choice of testing on paper or online.  The first online form of a test is equated to a paper form.  Subsequent new forms administered online are equated to a

previous online form.

 Dual mode system

 New forms administered on paper are equated to a previous paper form  New forms administered online are equated to a previous online form

20

Timeline for bringing SOL tests online

21

Session SOL Assessments Added Fall 2001 Algebra I, Earth Science, English: Reading Spring 2002 Algebra II, Biology Fall 2002 VA & US History, World History I, World History II Spring 2003 World Geography, Chemistry Spring 2004 Geometry Spring 2005 Begin middle school Spring 2006 Begin elementary school Spring 2013 Paper tests are accommodated forms

Usability and Comparability of Different Devices

Virginia has conducted or participated in various device studies over the past several years

 Spring 2012 – Writing Test Cognitive Lab

 External and on-screen keyboards with tablets

 Fall 2012 – Tablet Usability “Think-Aloud” Study

 Specific technology enhanced item types on tables with 10” and 7” screens

 Spring 2013 – Quantitative Written Composition Study

 Student writing on laptop, tablet, or tablet with external keyboard

 Spring 2014 – Quantitative study of Reading, Math, and Science

 Students were randomly assigned to take a test on computer or tablet

22