Test Security: Its Different for State, Interim, and Classroom - - PowerPoint PPT Presentation

Test Security: It’s Different for State, Interim, and Classroom Assessment

Steve Ferrara Measured Progress June 27, 2018 In S. Valenzuela (Organizer), What’s Yours is Mine and What’s Mine is Yours, presentation in the National Conference on Student Assessment, San Diego

Overview

• Address Sally’s requirements
• Best practices: standardization, monitoring,

forensics, investigations

• Also: Resolution
• Background
• PDIR framework
• Are all shared item banks equal?

2

PDIR framework

• Prevention, Detection, Investigation,

Resolution

• Goals of test security
• No security breaches
• Completely resolve breaches that do occur
• Evidence for the validity argument for intended

score interpretations and uses (SIUs)

• Support the data integrity claim so that you can

support those SIUs

Ferrara (2017)

3

PDIR framework

• How are we (state assessment programs)

doing on PDIR?

• Some states are at risk
• States and districts not well equipped to

carry out effective investigations

• Resolution may be spotty (we just don’t

know for sure)

Ferrara (2017)

4

Shared item banks

• State accountability assessments
• E.g., PARCC, Smarter, ELPA 21, WIDA, MSAA
• A primary intended SIU: accounting for

percentages of students at/above Proficient

• Commercial interim assessment products
• E.g., MAP, iReady, eMPower
• A primary SIU: achievement growth toward a goal
• Classroom assessment item banks and test

assembly support

• A primary intended SIU: student mastery of

instructional content

5

Shared item banks

• Advantages
• Economic and operational
• Risks
• A security weakness in one state or locale is a breach

for everyone

• A breach in one state or locale is a breach for

everyone

• Consequences of security breaches
• State accountability ***validity of SIUs, loss of assets
• Interim *** validity of SIUs, loss of assets
• Classroom assessment IBs *** validity of SIUs, loss of

assets

6

Best practices: State accountability assessments

• We organize our practices around the PDIR

framework…

• Within the limits of the RFP specifications

What Can States Do?

• Identify the highest risks for a specific

program

• Decide how best to mitigate those risks
• Implement and enforce the elements in the

PDIR framework

7

Best practices: State accountability assessments

• Risk
• Biggest threats appear to be when print

materials are in schools and during administration for both print and CBT

• Exposure of content, help to examinees
• Mitigation—by the state
• Training
• Forensics, web patrolling
• Monitoring (e.g., peer review citations)

8

Best practices: Commercial interim

• Risk
• Probably the same as state accountability, but

with less intensity

• But the cost of losing secure content is just as

high

• And the impact of undermining validity of score

interpretations is higher (on students)

• Mitigation—by the vendor
• Training
• Forensics, web patrolling
• Monitoring?

9

Best practices: Classroom assessment item banks

• Risk
• Content is always available, discussing items with
• ther teachers and students is part of the

teaching-learning process

• Is item exposure in one classroom, school, etc. a

problem for other classrooms, etc.?

• Mitigation
• Training in effective classroom assessment

practices

• Other, TBD
• (Sorry)

10

Best practices: Detection

• Accountability, interim, classroom
• DIY or contracted services
• Your psychometrics vendor
• Any test security companies that may exist
• Lots of research on detection statistics
• Multiple statistical approaches for the

same security risk

• Some holes (e.g., constructed responses,

next generation assessments)

11

Best practices: Investigation

• Accountability, interim, classroom
• Educators are not trained to conduct

effective, professional investigations

• To the same degree that professional

investigators are not trained as educators

• At least for big, scary situations
• Would a commercial interim provider

investigate a suspected security breach?

• A commercial classroom provider?

12

Best practices: Resolution

• Accountability, interim, classroom
• Needs
• Policies so that everyone knows the rules and

consequences

• Support from Department, Board, Legislature
• Courage and will

13

Reference

Ferrara, S. (2017). A comprehensive framework for policies and practices to improve test security programs: Prevention, detection, investigation, and resolution (PDIR). Educational Measurement: Issues and Practice, 36(3), 5-23.

14

It’s all about student learning.

Period.

Thank you.

ferrara.steve@measuredprogress.org +1 603-749-9102, ext. 7065