Co mme nts o n Asse ssme nt Co nso rtia T e st Se c urity Pre se - PowerPoint PPT Presentation

Co mme nts o n Asse ssme nt Co nso rtia T e st Se c urity Pre se ntatio ns Steve Ferrara Presented in J. Steedle (Organizer), Test Security for Common Core Consortia Assessments , a session in the National Conference on Student Assessment June 21, 2016

Ove rvie w  Frameworks to guide my comments ◦ PDIR ◦ Threats to security  Comments on each paper  Concluding comments Test Security for Common Core Assessment Consortia 2

F rame wo rk fo r c o mpre he nsive te st se c urity syste ms  Prevention, Detection, Investigation, Resolution (PDIR)  Progress on Prevention and Detection  Investigation ◦ (Mostly from media) Making progress on investigations ◦ May be too much conflict of interest in local investigation  Resolution ◦ Much of the evidence is not accessible ◦ Evidence we do have is… Test Security for Common Core Assessment Consortia 3

Some threats require Some threats require human vigilance for human vigilance for T hre ats to te st se c urity prevention, detection, prevention, detection, and investigation and investigation Before Test Administration During Test Administration After Test Administration Examinees Acquiring test content Copying or supplying Divulging content, failing to answers, impersonating report violations Test Administrators Divulging, teaching, Providing answers, allowing Changing answers, tampering inappropriate test prep inappropriate conditions, with answer files, failing to inadequate protection report Other Testing Site Staff Failing to train, failing to Failing to monitor Failing to report, tampering monitor security administrations with answers and files Program Mangers and Operations Vendors Failing to publicize Failing to observe Failing to report, failing to expectations, failing to train administrations and protection account for materials From Ferrara, 2016, Table 1; adapted from Fremer & Ferrara, 2013, Table 2.1 Test Security for Common Core Assessment Consortia 4

Je ff Ste e dle o n PARCC fo re nsic s  Statistical plus non ‐ statistical approaches  Addresses several threats   Web crawling to detect sharing of content ◦ PARCC states follow internal breach procedures ◦ Is this adequate to protect current content and discourage future breaches via rigorous resolution procedures?  Answer changing via points gained approach ◦ Logically similar to WTR answer changing methods ◦ Now, an automated system to detect changes for constructed response items! Test Security for Common Core Assessment Consortia 5

PARCC fo re nsic s (c o nt.)  Plagiarism for prose constructed response via latent semantic analysis ◦ Great to see PARCC exploring this approach ◦ Found very few flagged pairs; LSA adequately sensitive and specific? ◦ Relatively high numbers of flags for Narrative Writing Task grades 5 (37) and 7 (127)  Hypothesis: Retelling from a character’s perspective —Plausible? Conditional on PCR score or theta?  If think of plagiarism within schools, numbers of pairings may be manageable Test Security for Common Core Assessment Consortia 6

PARCC fo re nsic s (c o nt.)  Aberrant response patterns  Modified caution index (MCI) ◦ Findings consistent with the literature?  Standardized log ‐ likelihood person fit index ◦ More sensitive in simulated data ◦ Specificity/false negatives? Its performance conditional on theta/theta ranges? Test Security for Common Core Assessment Consortia 7

PARCC fo re nsic s (c o nt.)  Longitudinal performance modeling ◦ Detect unusual performance changes via cumulative logit regression  Clark et al. found it detects “test misconduct” with good power and conservative false positive flags ◦ Two consecutive years—Could find downward spikes as well as expected upward spikes ◦ Have to determine practical significance awa statistical  Not yet used operationally ◦ Look forward to hearing results from 2015 ‐ 2016 investigation of cumulative logit regression Test Security for Common Core Assessment Consortia 8

PARCC le sso ns le arne d  Will continue response change analyses—good ◦ Considering rules for flagging score increases—good  Plagiarism/copying on constructed responses—tough detection problem ◦ Approach: States will request focus on worrisome schools (”Known unknowns”) ◦ What about schools that are new to cheating? (“Unknown unknowns”)  Add answer copying/plagiarism for short responses, where answer copying/dictating responses is easier Test Security for Common Core Assessment Consortia 9

Brandt Re dd o n Smarte r Balanc e d vie ws o n te st se c urity  “Security goals should benefit students” ◦ Of course ◦ Test security systems also should serve our responsibilities to the public and our state and federal sponsors: data integrity (Ferrara, 2012; USDE, 2013)  Open source secure browsers—and roadmap to a common industry solution ◦ Great way to pursue our responsibilities to federal sponsors Test Security for Common Core Assessment Consortia 10

Smarte r Balanc e d vie ws (c o nt.)  Security and CAT ◦ 20:1 ratio of available and presented items; suspend exposed items ◦ Would like to see that ratio within standards x scale locations matrix—“redundant items”  Efforts to protect security and enable assistive technology—laudable  Test administration policies, training, monitoring, and statistical forensics—laudable  What about human vigilance to protect and detect other security threats? Test Security for Common Core Assessment Consortia 11

Mark Hanse n o n Smarte r Balanc e d fo re nsic s fo r CAT  Aberrant response patterns  Project W from R, L, S ◦ What were the conditions for the 3% significant differences? (e.g., Theta = ‐ 2) ◦ Same question for second illustration ◦ Rudner, Bracey, & Skaggs asked that question 20 years ago  Glad to hear Mark say bigger risk may be with constructed response items; our field not doing enough here yet Test Security for Common Core Assessment Consortia 12

I n c lo sing  Glad to see this work going on in the consortia ◦ Next year: Why not invite WIDA, ELPA21, NCSC, and DLM?  Would like to see big effort on forensics for security threats for constructed response items  I’ll summarize “empty cells” in threats x detection methods matrix ◦ Source for considering next method studies Test Security for Common Core Assessment Consortia 13

T hanks! Steve Ferrara sferrara1951@gmail.com Test Security for Common Core Assessment Consortia 14

Re fe re nc e s Ferrara, S. (2012 February 28). Investigation and response: Experiences and reflections of a former state assessment director. Invited panel presentation and discussion in Testing Integrity Symposium , sponsored by the National Center for Education Statistics, Washington, DC. Ferrara, S. (2016). A framework for policies and practices to improve test security systems: Prevention, detection, investigation, and resolution (PDIR) . Manuscript in preparation. Fremer, J. J., & Ferrara, S. (2013). Security in large scale, paper and pencil testing. In J. A. Wollack & J. J. Fremer (Eds.), Handbook of test security (pp. 17 ‐ 37). New York: Routledge. US Department of Education. (2013). Testing integrity symposium: Issues and recommendations for best practice. See http://nces.ed.gov/pubs2013/2013454.pdf Test Security for Common Core Assessment Consortia 15

Aberrant Aberrant Aberrant Aberrant Web Web Web Web Plagiarism Plagiarism Plagiarism Plagiarism responses responses responses responses PARCC re spo nse s to te st se c urity thre ats crawling crawling crawling crawling Answer Answer Before Test Administration During Test Administration After Test Administration Answer Answer changing changing changing changing Examinees Acquiring test content Copying or supplying Divulging content, failing to answers, impersonating report violations Test Administrators Divulging, teaching, Providing answers, allowing Changing answers, tampering inappropriate test prep inappropriate conditions, with answer files, failing to inadequate protection report Other Testing Site Staff Failing to train, failing to Failing to monitor Failing to report, tampering monitor security administrations with answers and files Program Mangers and Operations Vendors Failing to publicize Failing to observe Failing to report, failing to Performance Performance Performance Performance expectations, failing to train administrations and protection account for materials modeling modeling modeling modeling  Test Security for Common Core Assessment Consortia 16