the New Frontier for Assessment Development: Rich New Soil or a - - PowerPoint PPT Presentation

Process Data — the New Frontier for Assessment Development:

Stephen Provasnik National Center for Education Statistics, U.S. Department of Education 6 May 2019

Rich New Soil or a Quixotic Quest?

Overview

• What we discussed at the ETS symposium on

Process Data in Washington, DC last December

• What I think we might be able to agree on in regards

to uses for Process Data

• Where we could go by venturing into this new land

Logfiles vs. process data

• Logfiles - everything captured in a digital-based

assessment (DBA)

– from the order and speed of inputs (e.g., clicks and keystrokes) to the VPN of the device used to take the assessment.

• Process Data - the empirical data that reflect the

process of working on a test question

– reflecting cognitive and noncogitive, particularly psychological, constructs.

TIMSS Video Studies (late 1990s)

Recordings of selected classroom lessons were coded to indicate (among other many things):

• the assigned type of work, with categories of “whole-class

work,” “individual work,” “pair/partner work,” and “small-group work”

• the “number of words” the classroom teacher used in a ratio

to the number of words students used, when talking to the whole class

• the proportion of the lesson spent on review of previous

content vs. spent explaining new content

Conclusions from the last December’s symposium

• 1. Develop a systemic approach to logfiles—to answer

the question of what exactly logfiles should capture

• 2. Develop a theory for Process Data—to answer the

question of how to use process data

• 3. Develop guidelines and standards for how to convert

logfiles into process data

Spandrels of San Marco

Spandrels of San Marco

Spandrel 1 Spandrel 2

Spandrels of San Marco

Dome Arch 2 Arch 1 Arch 3

Source: https://www.cell.com/current-biology/pdf/S0960-9822(08)00371-0.pdf

Larynx

Source: https://www.cell.com/current-biology/pdf/S0960-9822(08)00371-0.pdf

Larynx

Ongoing Evolution in Assessment

Past Present Future

Item Development Labor Intensive Labor Intensive Automatized Item types Generic Enhanced Real-life Test design Static Semi-static Data-driven Test assembly Labor Intensive Semi-automatized Automatized Accessibility Limited Universal design Adaptive Timing Not measurable Measured Used Pathways Not observable Observable Modeled Validity Content/core-based Construct based Process based Feedback Summative Summative Diagnostic

Diagnostic or forensic applications

These include using logfiles and process data to improve data quality

• by helping understand how items function and what variables

make items more difficult or more reliable items

• by distinguishing among “missing” answers which are

– “not reached” (never seen) – “omitted” (seen, taken time over, but ultimately skipped) – “not attempted” (seen, but not time taken before being skipped)

• by identifying student guessing or cases that are outliers, which

may indicate possible cases of cheating, or cases of programming error

Visualization of NAEP reading patterns from sampled logfiles

Each sampled student represented by a blue dot. Ten test questions represented by “buckets” Pages of text represented by “roofs” indicating which page being looked at.

Visualization of NAEP reading patterns from sampled logfiles

Diagnostic or forensic applications

These include using logfiles and process data to improve data quality

• by helping understand how items function and what variables

make items more difficult or more reliable items

• by distinguishing among “missing” answers which are truly “not

reached” (never seen), which should be “omitted” (seen, taken time over, but ultimately skipped), and which are “not attempted” (seen, but not time taken before being skipped)

• by identifying student guessing or cases that are outliers, which

may indicate possible cases of cheating, or cases of programming error

Research into understanding respondent behaviors and cognitive strategies

For example

• to improve teaching and learning with specific information on

how different students think/perform

• to better understand factors that distinguish high- and low-

performers, or expert from novice strategies

• to better understand the relationship of motivation and

performance.

Use of Process Data from NAEP Writing

Essay Length by Writing Time

Expanded Use of Process Data

Essay Length by Writing Time

Expanded Use of Process Data

Essay Length by Writing Time

Expanded Use of Process Data

Essay Length by Writing Time

Expanded Use of Process Data

Essay Length by Writing Time

NCES Example of Process Data Analysis

Test Development Before DBA

23

Cog lab or piloting of items Cog lab or piloting of items Main Study data collection Main Study data collection Make final booklets Make final booklets Select final item pool Select final item pool Review Item stats and parameters Review Item stats and parameters Score results Score results Field Test data collection Field Test data collection Framework Framework Item Writing Item Writing Score results Score results Release dataset Release dataset Analysis and final report Analysis and final report IRT scaling, weighting IRT scaling, weighting Create FT booklets Create FT booklets

24

Cog lab or piloting of items Cog lab or piloting of items Main Study data collection Main Study data collection Make final booklets Make final booklets Select final item pool Select final item pool Review Item stats and parameters Review Item stats and parameters Score results Score results Field Test data collection Field Test data collection Framework Framework Item Writing Item Writing Score results Score results Release dataset Release dataset Analysis and final report Analysis and final report IRT scaling, weighting IRT scaling, weighting Create FT booklets Create FT booklets

Test Development for DBA

24

Coders render item Coders render item Device management Device management and logfiles and logfiles

Device management Device management

Review logfiles and extract process data Review logfiles and extract process data Coders program final instruments and logfiles Coders program final instruments and logfiles Analyze process data for reporting Analyze process data for reporting Anonymize process data for release in dataset Anonymize process data for release in dataset Use process data for scaling Use process data for scaling Coders program items and testlets Coders program items and testlets

Release dataset Release dataset

Test Development for DBA

25

25

Cog lab or piloting of items Cog lab or piloting of items Main Study data collection Main Study data collection Make final booklets Make final booklets Select final item pool Select final item pool Review Item stats and parameters Review Item stats and parameters Score results Score results Field Test data collection Field Test data collection Framework Framework Item Writing Item Writing Score results Score results Release dataset Release dataset Analysis and final report Analysis and final report IRT scaling, weighting IRT scaling, weighting Create FT booklets Create FT booklets

Coders render item Coders render item Device management Device management and logfiles and logfiles

Device management Device management

Review logfiles and extract process data Review logfiles and extract process data Coders program final instruments and logfiles Coders program final instruments and logfiles Analyze process data for reporting Analyze process data for reporting Anonymize process data for release in dataset Anonymize process data for release in dataset Use process data for scaling Use process data for scaling Coders program items and testlets Coders program items and testlets

Release dataset Release dataset

26

Cog lab or piloting of items Cog lab or piloting of items Main Study data collection Main Study data collection Make final booklets Make final booklets Select final item pool Select final item pool Review Item stats and parameters Review Item stats and parameters Score results Score results Field Test data collection Field Test data collection Framework Framework Item Writing Item Writing Score results Score results Release dataset Release dataset Analysis and final report Analysis and final report IRT scaling, weighting IRT scaling, weighting Create FT booklets Create FT booklets

Coders render item Coders render item Device management Device management and logfiles and logfiles

Device management Device management

Review logfiles and extract process data Review logfiles and extract process data Coders program final instruments and logfiles Coders program final instruments and logfiles Analyze process data for reporting Analyze process data for reporting Anonymize process data for release in dataset Anonymize process data for release in dataset Use process data for scaling Use process data for scaling Coders program items and testlets Coders program items and testlets

Release dataset Release dataset

Process Data Inputs and Outputs

26

Based on theoretical understanding of cognitive processes Based on theoretical understanding of cognitive processes Define processes intended to be measured by items Define processes intended to be measured by items Coding to capture target processes Coding to capture target processes Paradata for monitoring assessment integrity Paradata for monitoring assessment integrity Requires a priori definitions of process data Requires a priori definitions of process data Process data for forensic purposes Process data for forensic purposes Process data for research Process data for research Process data for scoring input Process data for scoring input Process data as performance measure Process data as performance measure

Shaping Process Data

27

Theoretical understanding of cognitive processes Theoretical understanding of cognitive processes Definition of processes intended to be measured by items Definition of processes intended to be measured by items Coding to capture target processes Coding to capture target processes A priori definitions of process data A priori definitions of process data

Framework Framework Item Writing Item Writing Coders program items and testlets Coders program items and testlets Review logfiles and extract process data Review logfiles and extract process data

Process Data Inputs and Outputs

28

Paradata for monitoring assessment integrity Paradata for monitoring assessment integrity Process data for forensic purposes Process data for forensic purposes Process data for research Process data for research Process data for scoring input Process data for scoring input Process data as performance measure Process data as performance measure Device management Device management Logfiles Logfiles Review logfiles and extract process data Review logfiles and extract process data Use process data for scaling Use process data for scaling IRT scaling IRT scaling

Inappropriate uses of process data

For example,

• Overgeneralizing from one item to all items, or one

process to many processes

• Concluding that strategies associated with higher

performance are the strategies that all students should be taught

• Making classroom and formative assessments turn
• n process data in such a way that students lose

unstructured opportunities to try out new ways of thinking and doing

Thank you

Stephen Provasnik National Center for Education Statistics, U.S. Department of Education Stephen.Provasnik@ed.gov