Discussion and Demonstration of New Digitally Based Assessments for - PowerPoint PPT Presentation

Discussion and Demonstration of New Digitally Based Assessments for the National Assessment of Educational Progress (NAEP) CCSSO 2019 NCSA June 24-26 1

Presenters Dr. Amy Dresher Shannon Richards NAEP Deputy Director NAEP Assessment Development Educational Testing Service DBA Process Manager adresher@ets.org Educational Testing Service srichards@ets.org Jeff Ackley Dr. Angela Hemingway Science Director Executive Director Center for Science & Idaho STEM Action Center and Digital Assessment Design STEM Action Center Foundation Educational Testing Service Executive Office of the Governor jackley@ets.org Angela.Hemingway@stem.idaho.gov 2

Agenda • Introduction to NAEP • Affordances of Digitally Based Assessments • Demonstration of Discrete Items Types in Math and Science Assessments • Overview of Scenario-based Tasks in Science Assessments • Affordances of DBA – a State’s Perspective • Q&A and Discussion 3

Introduction to NAEP

What is NAEP? • NAEP is an assessment that measures what the student in the United States know and can do in various subjects across the nation, states, and in some urban districts. • NAEP administered its first national assessment 50 years ago , serving as the longest running national assessment available. • A NAEP assessment is comprised of a variety of forms across several subjects. Individual students complete only a subset of item in the assessment and each student represents hundreds of students. 5

Digital Transformation and Delivery • NAEP’s history of digitally based assessments o 2009: science interactive computer tasks o 2011: writing o 2014: technology and engineering literacy (TEL) o 2015: first pilot of digitally based assessment o 2017: DBA in mathematics and reading o 2018: DBA in U.S history, civics, and geography o 2019: DBA in mathematics, reading, and science 6

NAEP in the Field 7

Contextual Information • NAEP collects contextual information about students' demographic characteristics, opportunities to learn in and outside of the classroom, and educational experiences. • Additional questions about technology usage at home and in schools was collected starting in 2017 • 85% of students have access to computer at home • 99% of students have access to digital devices in schools 8

NAEP by Numbers • In total, since NAEP has been administering digital assessments, we have assessed over 1.78 million students on laptops or tablets • 2019 was the largest DBA administration in NAEP’s history 9

Affordances of Digitally Based Assessments

Digital Assessments • Include interactive item types that are engaging and allow for better measurement of the constructs • Increase the opportunity to machine score student responses • Provide assistive technologies not available in paper-based assessments (PBAs) that enhance accommodations • Parallel similar learning environments in the classrooms and digital environments familiar to students outside the classroom • Collect new and more meaningful data about what students know and can do, including how students approach problem solving 11

DBA Example Question ID: 2017-8M9 #4 D2357MS 12

PBA Example Question ID: 2013-8M3 #3 M150001 13

DBA Tools • Universal design tools available to all students • Help • Change theme • Zoom • Read aloud • Scratchwork • Other tools available per subject, form, or item type • Calculator • Equation editor • Bilingual toggle 14

DBA Process Data Collected • Timing data • How long did students spend per item, item type, section, and form? • Tool usage • Which tools did students use, how often, and for which items? • Navigation • Do students start with the first item and then work sequentially, or do they answer questions out of order? If out of order, is there a pattern? • Cognitive processes • Do students partially work through the question and then revise their response? 15

Demonstration of Discrete Item Types in Math and Science Assessments

Released Content • Information about NAEP DBA can be found at: https://nces.ed.gov/nationsreportcard/dba/ • To view sample DBA items: o Go to https://nces.ed.gov/NationsReportCard/nqt/Search/SearchOptions o Select subjects Mathematics or Reading o Select a grade(s) of your choice o Select year 2017 for DBA items o Select OK • Example Items o Matching Item o Grid Item o Fill in the Blank Item 17

Calculators Grade 8 and 12 Grade 4 18

Demonstration of Other Item Types • Presentation of additional item types available in NAEP DBA. 19