A Framework for Analyzing the Effects of Interruptions of Online - - PowerPoint PPT Presentation

June 22, 2016 1

A Framework for Analyzing the Effects of Interruptions of Online Assessment

Joseph Martineau Senior Associate

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 2

Based on a Paper Prepared for CCSSO

Available on the web from:

– CCSSO – The Center for Assessment

Martineau, J., Domaleski, C., Egan, K., Patelis, T., & Dadey, N. (2015, October). Recommendations for Addressing the Impact of Test Administration Interruptions and

• Irregularities. Washington, DC: Council of

Chief State School Officers and Center for Assessment.

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 3

Why Plan for Interruptions?

• Online test administration is incredibly complex, involving the

interaction of…

– Complex Test Registration and Deliver Applications (among others) – Capacity and Efficiency of Application, Database, and Web Servers – Varying capacity and quality of…

• National, state, and local electrical grids
• National, state, and local Internet backbones
• Vendor, state, and regional agency/district/school internet service providers
• District/school Ethernet and/or Wi-Fi infrastructure

– Various brands and versions of…

• Devices (e.g., brands and versions of desktops, laptops, thin clients, tablets)
• Operating systems (e.g., versions of Windows, Linux, OS X, Chrome, Android, iOS)

– Vendor, state, regional agency, district, and school personnel – A wide variety of personnel roles in both technical and administrative functions – Students – Nefarious internal and/or external actors

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 4

A Possibly Naïve Representation of the Degree of Complexity In Computer-Based Test Administration

• But seriously, what could possibly go wrong?

From Dubberly, H. (2002). Using Concept maps in product development: Preparing to redesign java.sun.com. Retrieved February 18, 2016 from http://www.dubberly.com/wp- content/uploads/2009/12/ddo_article_java_map.pdf.

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 5

Key Student-Level Causal Question

• What would a student’s score have been if he or she had not been

interrupted?

– We don’t have an observable counterfactual – A causal modeling framework may be useful for analyzing the data, but the analyses do not produce causal effects

• Three approaches (all involve matching)

– Between-students comparison of those who experienced an interruption vs. those who did not (gives estimates of mean effects of interruptions) – Between-student comparison of students that experienced an interruption vs. those that did not controlling for degree of interruption (gives estimates of mean effects of varying degrees of interruptions) – Within-student comparison of achievement estimates before and after interruption (gives estimates of effects of interruptions on individual student scores).

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 6

Key Aggregate-Level Causal Question

• What would a classroom/school/district aggregate score have

been had

– No students been interrupted during testing? – Completion of testing not been affected by other students’ interruptions? – Participation in testing not been affected by other students’ interruptions?

• We don’t have observable counterfactuals for these questions

– Question 1 can be analyzed using a pseudo-causal modeling framework, but the results do not produce a causal effect. – Questions 2 and 3 are more problematic

• Our recommended approach

– Flag aggregation units that show practically significant deviations from historical completion/participation rates – Note the number of students experiencing each type of interruption that are included in aggregate scores.

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 7

Framework to Analyze & Respond to Interruptions

Item Score Analyses Estimate effects of slowdowns

• n item scores

Student Score Analyses Aggregate Entity Score Analyses Cleanse data and create data elements to represent instances of each class of interruption and degree of test completion Describe prevalence of each class

• f slowdown by demographic

group and educational entity Establish effects of slowdowns

• n aggregate completion rates

Estimate effects of slowdowns

• n student scores

Estimate effects of slowdowns on aggregate entity scores Make judgment whether to invalidate, flag, or adjust student scores (or do nothing) Make judgment whether to invalidate, flag, or adjust aggregate entity scores (or do nothing) Give cautions for use and interpretation on score reports and in data files for affected student scores Give cautions for use and interpretation on score reports and in data files for affected aggregate entity scores 4A 1 2 3 4B 4C 5B 6B Make judgment whether to invalidate or flag item scores (or do nothing) 5A 5C Give cautions for use and interpretation of item scores in score reports and data files for affected item scores 6A 6C

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 8

Framework to Analyze & Respond to Interruptions

Item Score Analyses Estimate effects of slowdowns

• n item scores

Student Score Analyses Aggregate Entity Score Analyses Cleanse data and create data elements to represent instances of each class of interruption and degree of test completion Describe prevalence of each class

• f slowdown by demographic

group and educational entity Establish effects of slowdowns

• n aggregate completion rates

Estimate effects of slowdowns

• n student scores

Estimate effects of slowdowns on aggregate entity scores Make judgment whether to invalidate, flag, or adjust student scores (or do nothing) Make judgment whether to invalidate, flag, or adjust aggregate entity scores (or do nothing) Give cautions for use and interpretation on score reports and in data files for affected student scores Give cautions for use and interpretation on score reports and in data files for affected aggregate entity scores 4A 1 2 3 4B 4C 5B 6B Make judgment whether to invalidate or flag item scores (or do nothing) 5A 5C Give cautions for use and interpretation of item scores in score reports and data files for affected item scores 6A 6C

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 9

Framework to Analyze & Respond to Interruptions

Item Score Analyses Estimate effects of slowdowns

• n item scores

Student Score Analyses Aggregate Entity Score Analyses Cleanse data and create data elements to represent instances of each class of interruption and degree of test completion Describe prevalence of each class

• f slowdown by demographic

group and educational entity Establish effects of slowdowns

• n aggregate completion rates

Estimate effects of slowdowns

• n student scores

Estimate effects of slowdowns on aggregate entity scores Make judgment whether to invalidate, flag, or adjust student scores (or do nothing) Make judgment whether to invalidate, flag, or adjust aggregate entity scores (or do nothing) Give cautions for use and interpretation on score reports and in data files for affected student scores Give cautions for use and interpretation on score reports and in data files for affected aggregate entity scores 4A 1 2 3 4B 4C 5B 6B Make judgment whether to invalidate or flag item scores (or do nothing) 5A 5C Give cautions for use and interpretation of item scores in score reports and data files for affected item scores 6A 6C

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 10

Framework to Analyze & Respond to Interruptions

Item Score Analyses Estimate effects of slowdowns

• n item scores

Student Score Analyses Aggregate Entity Score Analyses Cleanse data and create data elements to represent instances of each class of interruption and degree of test completion Describe prevalence of each class

• f slowdown by demographic

group and educational entity Establish effects of slowdowns

• n aggregate completion rates

Estimate effects of slowdowns

• n student scores

Estimate effects of slowdowns on aggregate entity scores Make judgment whether to invalidate, flag, or adjust student scores (or do nothing) Make judgment whether to invalidate, flag, or adjust aggregate entity scores (or do nothing) Give cautions for use and interpretation on score reports and in data files for affected student scores Give cautions for use and interpretation on score reports and in data files for affected aggregate entity scores 4A 1 2 3 4B 4C 5B 6B Make judgment whether to invalidate or flag item scores (or do nothing) 5A 5C Give cautions for use and interpretation of item scores in score reports and data files for affected item scores 6A 6C

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 11

Framework to Analyze & Respond to Interruptions

Item Score Analyses Estimate effects of slowdowns

• n item scores

Student Score Analyses Aggregate Entity Score Analyses Cleanse data and create data elements to represent instances of each class of interruption and degree of test completion Describe prevalence of each class

• f slowdown by demographic

group and educational entity Establish effects of slowdowns

• n aggregate completion rates

Estimate effects of slowdowns

• n student scores

Estimate effects of slowdowns on aggregate entity scores Make judgment whether to invalidate, flag, or adjust student scores (or do nothing) Make judgment whether to invalidate, flag, or adjust aggregate entity scores (or do nothing) Give cautions for use and interpretation on score reports and in data files for affected student scores Give cautions for use and interpretation on score reports and in data files for affected aggregate entity scores 4A 1 2 3 4B 4C 5B 6B Make judgment whether to invalidate or flag item scores (or do nothing) 5A 5C Give cautions for use and interpretation of item scores in score reports and data files for affected item scores 6A 6C

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 12

Framework to Analyze & Respond to Interruptions

Item Score Analyses Estimate effects of slowdowns

• n item scores

Student Score Analyses Aggregate Entity Score Analyses Cleanse data and create data elements to represent instances of each class of interruption and degree of test completion Describe prevalence of each class

• f slowdown by demographic

group and educational entity Establish effects of slowdowns

• n aggregate completion rates

Estimate effects of slowdowns

• n student scores

Estimate effects of slowdowns on aggregate entity scores Make judgment whether to invalidate, flag, or adjust student scores (or do nothing) Make judgment whether to invalidate, flag, or adjust aggregate entity scores (or do nothing) Give cautions for use and interpretation on score reports and in data files for affected student scores Give cautions for use and interpretation on score reports and in data files for affected aggregate entity scores 4A 1 2 3 4B 4C 5B 6B Make judgment whether to invalidate or flag item scores (or do nothing) 5A 5C Give cautions for use and interpretation of item scores in score reports and data files for affected item scores 6A 6C

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 13

Framework to Analyze & Respond to Interruptions

Item Score Analyses Estimate effects of slowdowns

• n item scores

Student Score Analyses Aggregate Entity Score Analyses Cleanse data and create data elements to represent instances of each class of interruption and degree of test completion Describe prevalence of each class

• f slowdown by demographic

group and educational entity Establish effects of slowdowns

• n aggregate completion rates

Estimate effects of slowdowns

• n student scores

Estimate effects of slowdowns on aggregate entity scores Make judgment whether to invalidate, flag, or adjust student scores (or do nothing) Make judgment whether to invalidate, flag, or adjust aggregate entity scores (or do nothing) Give cautions for use and interpretation on score reports and in data files for affected student scores Give cautions for use and interpretation on score reports and in data files for affected aggregate entity scores 4A 1 2 3 4B 4C 5B 6B Make judgment whether to invalidate or flag item scores (or do nothing) 5A 5C Give cautions for use and interpretation of item scores in score reports and data files for affected item scores 6A 6C

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 14

• Classes of interruption (see Nathan Dadey’s presentation)
• Types of interruption (direct vs. indirect)
• At least three analytical approaches

– Descriptive analysis by group (e.g., interrupted vs. not interrupted, crossed with demographic groupings).

• Least rigorous
• Easiest to communicate

– Separate inferential analyses to isolate each effect of interest.

• More rigorous, but does not allow for controlling for one set of effects in

modeling another

• A compromise between rigor and ease of communication

– Unitary statistical model were all effects of interest can be modeled simultaneously and the structure of the model reflects the structure of the data.

• Most rigorous
• Most difficult to communicate
• Demographics!

Estimate effects of slowdowns on student scores

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 15

Framework to Analyze & Respond to Interruptions

Item Score Analyses Estimate effects of slowdowns

• n item scores

Student Score Analyses Aggregate Entity Score Analyses Cleanse data and create data elements to represent instances of each class of interruption and degree of test completion Describe prevalence of each class

• f slowdown by demographic

group and educational entity Establish effects of slowdowns

• n aggregate completion rates

Estimate effects of slowdowns

• n student scores

Estimate effects of slowdowns on aggregate entity scores Make judgment whether to invalidate, flag, or adjust student scores (or do nothing) Make judgment whether to invalidate, flag, or adjust aggregate entity scores (or do nothing) Give cautions for use and interpretation on score reports and in data files for affected student scores Give cautions for use and interpretation on score reports and in data files for affected aggregate entity scores 4A 1 2 3 4B 4C 5B 6B Make judgment whether to invalidate or flag item scores (or do nothing) 5A 5C Give cautions for use and interpretation of item scores in score reports and data files for affected item scores 6A 6C

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 16

Framework to Analyze & Respond to Interruptions

Item Score Analyses Estimate effects of slowdowns

• n item scores

Student Score Analyses Aggregate Entity Score Analyses Cleanse data and create data elements to represent instances of each class of interruption and degree of test completion Describe prevalence of each class

• f slowdown by demographic

group and educational entity Establish effects of slowdowns

• n aggregate completion rates

Estimate effects of slowdowns

• n student scores

Estimate effects of slowdowns on aggregate entity scores Make judgment whether to invalidate, flag, or adjust student scores (or do nothing) Make judgment whether to invalidate, flag, or adjust aggregate entity scores (or do nothing) Give cautions for use and interpretation on score reports and in data files for affected student scores Give cautions for use and interpretation on score reports and in data files for affected aggregate entity scores 4A 1 2 3 4B 4C 5B 6B Make judgment whether to invalidate or flag item scores (or do nothing) 5A 5C Give cautions for use and interpretation of item scores in score reports and data files for affected item scores 6A 6C

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 17

Framework to Analyze & Respond to Interruptions

Item Score Analyses Estimate effects of slowdowns

• n item scores

Student Score Analyses Aggregate Entity Score Analyses Cleanse data and create data elements to represent instances of each class of interruption and degree of test completion Describe prevalence of each class

• f slowdown by demographic

group and educational entity Establish effects of slowdowns

• n aggregate completion rates

Estimate effects of slowdowns

• n student scores

Estimate effects of slowdowns on aggregate entity scores Make judgment whether to invalidate, flag, or adjust student scores (or do nothing) Make judgment whether to invalidate, flag, or adjust aggregate entity scores (or do nothing) Give cautions for use and interpretation on score reports and in data files for affected student scores Give cautions for use and interpretation on score reports and in data files for affected aggregate entity scores 4A 1 2 3 4B 4C 5B 6B Make judgment whether to invalidate or flag item scores (or do nothing) 5A 5C Give cautions for use and interpretation of item scores in score reports and data files for affected item scores 6A 6C

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 18

Framework to Analyze & Respond to Interruptions

Item Score Analyses Estimate effects of slowdowns

• n item scores

Student Score Analyses Aggregate Entity Score Analyses Cleanse data and create data elements to represent instances of each class of interruption and degree of test completion Describe prevalence of each class

• f slowdown by demographic

group and educational entity Establish effects of slowdowns

• n aggregate completion rates

Estimate effects of slowdowns

• n student scores

Estimate effects of slowdowns on aggregate entity scores Make judgment whether to invalidate, flag, or adjust student scores (or do nothing) Make judgment whether to invalidate, flag, or adjust aggregate entity scores (or do nothing) Give cautions for use and interpretation on score reports and in data files for affected student scores Give cautions for use and interpretation on score reports and in data files for affected aggregate entity scores 4A 1 2 3 4B 4C 5B 6B Make judgment whether to invalidate or flag item scores (or do nothing) 5A 5C Give cautions for use and interpretation of item scores in score reports and data files for affected item scores 6A 6C

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 19

Framework to Analyze & Respond to Interruptions

Item Score Analyses Estimate effects of slowdowns

• n item scores

Student Score Analyses Aggregate Entity Score Analyses Cleanse data and create data elements to represent instances of each class of interruption and degree of test completion Describe prevalence of each class

• f slowdown by demographic

group and educational entity Establish effects of slowdowns

• n aggregate completion rates

Estimate effects of slowdowns

• n student scores

Estimate effects of slowdowns on aggregate entity scores Make judgment whether to invalidate, flag, or adjust student scores (or do nothing) Make judgment whether to invalidate, flag, or adjust aggregate entity scores (or do nothing) Give cautions for use and interpretation on score reports and in data files for affected student scores Give cautions for use and interpretation on score reports and in data files for affected aggregate entity scores 4A 1 2 3 4B 4C 5B 6B Make judgment whether to invalidate or flag item scores (or do nothing) 5A 5C Give cautions for use and interpretation of item scores in score reports and data files for affected item scores 6A 6C

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 20

Framework to Analyze & Respond to Interruptions

Item Score Analyses Estimate effects of slowdowns

• n item scores

Student Score Analyses Aggregate Entity Score Analyses Cleanse data and create data elements to represent instances of each class of interruption and degree of test completion Describe prevalence of each class

• f slowdown by demographic

group and educational entity Establish effects of slowdowns

• n aggregate completion rates

Estimate effects of slowdowns

• n student scores

Estimate effects of slowdowns on aggregate entity scores Make judgment whether to invalidate, flag, or adjust student scores (or do nothing) Make judgment whether to invalidate, flag, or adjust aggregate entity scores (or do nothing) Give cautions for use and interpretation on score reports and in data files for affected student scores Give cautions for use and interpretation on score reports and in data files for affected aggregate entity scores 4A 1 2 3 4B 4C 5B 6B Make judgment whether to invalidate or flag item scores (or do nothing) 5A 5C Give cautions for use and interpretation of item scores in score reports and data files for affected item scores 6A 6C

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 21

Framework to Analyze & Respond to Interruptions

Item Score Analyses Estimate effects of slowdowns

• n item scores

Student Score Analyses Aggregate Entity Score Analyses Cleanse data and create data elements to represent instances of each class of interruption and degree of test completion Describe prevalence of each class

• f slowdown by demographic

group and educational entity Establish effects of slowdowns

• n aggregate completion rates

Estimate effects of slowdowns

• n student scores

Estimate effects of slowdowns on aggregate entity scores Make judgment whether to invalidate, flag, or adjust student scores (or do nothing) Make judgment whether to invalidate, flag, or adjust aggregate entity scores (or do nothing) Give cautions for use and interpretation on score reports and in data files for affected student scores Give cautions for use and interpretation on score reports and in data files for affected aggregate entity scores 4A 1 2 3 4B 4C 5B 6B Make judgment whether to invalidate or flag item scores (or do nothing) 5A 5C Give cautions for use and interpretation of item scores in score reports and data files for affected item scores 6A 6C

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 22

Framework to Analyze & Respond to Interruptions

Item Score Analyses Estimate effects of slowdowns

• n item scores

Student Score Analyses Aggregate Entity Score Analyses Cleanse data and create data elements to represent instances of each class of interruption and degree of test completion Describe prevalence of each class

• f slowdown by demographic

group and educational entity Establish effects of slowdowns

• n aggregate completion rates

Estimate effects of slowdowns

• n student scores

Estimate effects of slowdowns on aggregate entity scores Make judgment whether to invalidate, flag, or adjust student scores (or do nothing) Make judgment whether to invalidate, flag, or adjust aggregate entity scores (or do nothing) Give cautions for use and interpretation on score reports and in data files for affected student scores Give cautions for use and interpretation on score reports and in data files for affected aggregate entity scores 4A 1 2 3 4B 4C 5B 6B Make judgment whether to invalidate or flag item scores (or do nothing) 5A 5C Give cautions for use and interpretation of item scores in score reports and data files for affected item scores 6A 6C

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 23

Framework to Analyze & Respond to Interruptions

Item Score Analyses Estimate effects of slowdowns

• n item scores

Student Score Analyses Aggregate Entity Score Analyses Cleanse data and create data elements to represent instances of each class of interruption and degree of test completion Describe prevalence of each class

• f slowdown by demographic

group and educational entity Establish effects of slowdowns

• n aggregate completion rates

Estimate effects of slowdowns

• n student scores

Estimate effects of slowdowns on aggregate entity scores Make judgment whether to invalidate, flag, or adjust student scores (or do nothing) Make judgment whether to invalidate, flag, or adjust aggregate entity scores (or do nothing) Give cautions for use and interpretation on score reports and in data files for affected student scores Give cautions for use and interpretation on score reports and in data files for affected aggregate entity scores 4A 1 2 3 4B 4C 5B 6B Make judgment whether to invalidate or flag item scores (or do nothing) 5A 5C Give cautions for use and interpretation of item scores in score reports and data files for affected item scores 6A 6C

2016 CCSSO NCSA Presentation on Test Interruptions June 22, 2016 24

Contact Information

Joseph Martineau Senior Associate jmartineau@nciea.org