ASSESSMENT OF EDUCATIONAL PROGRESS GRADE 4 WRITING ASSESSMENT NCSA - - PowerPoint PPT Presentation

DEVELOPING ACHIEVEMENT LEVELS FOR THE 2017 NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS GRADE 4 WRITING ASSESSMENT

NCSA Presentation – June 2018 Tim O’Neil, Ed.D. ALS Project Director Pearson

1

NGSS Design Challenges: One State’s Design

Setting Achievement Levels for NAEP Writing

Background

Current NAEP Writing Framework: Writing on Computer

• Third NAEP Writing Framework since 1990; adopted in 2007
• Writing assessments under this framework administered in

grades 8 and 12 in 2011

• Achievement levels for grades 8 and 12 set in 2012
• 2017 is first administration of this framework at grade 4

Pearson contracted with the National Assessment Governing Board to set achievement levels for the grade 4 NAEP writing assessment

2

Setting Achievement Levels for NAEP Writing

ALS High Level Design

Achievement level setting methodology

• Body of Work (BoW) method
• Appropriate for the item types and scaling
• Completed in a reasonable amount of time
• Governing Board has experience with the method
• Supported by the measurement field

3

Setting Achievement Levels for NAEP Writing

Body of Work Method

• Panelists work directly with actual student responses and

judge them in relation to the NAEP grade 4 writing Achievement Level Descriptions

• Panelist judgements combined with known student

performance (scores) allow for the derivation of cut scores for each panelist

• Panelists judge each student body of work (BoW) as

belonging to one of the defined achievement levels (e.g. Basic, Proficient, Advanced)

• Panelists engage in multiple rounds of judgement

informed by summary information and group discussion

4

Setting Achievement Levels for NAEP Writing

Moodle

We created a user interface within Moodle (open-source learning platform) with similar functionality of BoWTIE (software used for the 2011 grade 8 and 12 ALS). ‒ Solution was used prior to and throughout meetings for sharing of pre- meeting materials, panelist review of BoWs and entering ratings, extraction of rating data for analysis, providing panelist feedback, completing questionnaires, etc. ‒ Was used successfully for PARCC standard setting meetings as well as

• thers (Bookmark and Modified Angoff)

‒ Has proven effective with pre-meeting training, materials sharing, form/survey completion, presentation of items/responses, capture of panelist ratings, presentation of feedback data, etc.

5

Setting Achievement Levels for NAEP Writing

6

Setting Achievement Levels for NAEP Writing

ALS Process Overview

Panelists:

• Take a NAEP grade 4 writing test
• Receive information about the rubric used to score

responses from the NAEP grade 4 writing assessment

• Receive training on understanding the NAEP Writing

Framework

• Develop a shared understanding of each achievement

level description (ALD)

• Receive training on and implement the ALS method

known as Body of Work

• Recommend exemplars for use in reporting
• Respond to process evaluations throughout the study

7

Setting Achievement Levels for NAEP Writing

ALS Activities Conducted within Moodle

• Pre-meeting: introductory video, panelist information,

materials review (ALDs, Briefing Booklet, Frameworks, NDA)

• ALD practice classification activity
• BoW practice
• BoW judgments all rounds
• BoW feedback all rounds
• Exemplar BoW selection
• Process evaluations throughout – from pre-meeting through

wrap up. Every major task (11 overall)

8

Setting Achievement Levels for NAEP Writing

Pre Meeting

9

Setting Achievement Levels for NAEP Writing

Facilitator Materials Section

10

Prompt 1.pdf Prompt 2.pdf Prompt 3.pdf Training Exemplar 1.pdf Training Exemplar 2.pdf Training Exemplar 3.pdf Video Prompt 3.pdf 223_Prompt 1.pdf 223_Prompt 2.pdf 2854_Prompt 1.pdf 2854_Prompt 2.pdf

Setting Achievement Levels for NAEP Writing

Judgment Task

11

Prompt 1 A25_Prompt 1 Prompt 2 A25_Prompt 2

Setting Achievement Levels for NAEP Writing

Complexity of Judgment Task Design

12

Form 1 Form 2 Form 3 Form 4 Form 5 Form 6 Form 7 A1 A2 A3 A4 C1 C2 C3 Explain Explain Persuade Convey Explain Convey Convey A5 A6 A7 A8 C4 C5 C6 Convey Convey Explain Persuade Convey Explain Persuade Form 8 Form 9 Form 10 Form 11 Form 5 Form 6 Form 7 B1 B2 B3 B4 C1 C2 C3 Explain Explain Persuade Persuade Explain Convey Convey B5 B6 B7 B8 C4 C5 C6 Convey Convey Explain Convey Convey Explain Persuade Note: 7 BoWs selected per form (one common form has 8), results in each panel reviewing 50 BoWs (78 total) BoWs selected so that they are of roughly equal overall difficulty across panels 1 2 1 2 B A Unique Forms Common Forms Prompt Panel

Setting Achievement Levels for NAEP Writing

Composition of Tables

13

GENERAL PUBLIC TEACHER TEACHER TEACHER NON- TEACHER GENERAL PUBLIC TEACHER TEACHER TEACHER NON- TEACHER GENERAL PUBLIC TEACHER TEACHER NON- TEACHER NON- TEACHER GENERAL PUBLIC GENERAL PUBLIC GENERAL PUBLIC GENERAL PUBLIC TEACHER TEACHER TEACHER TEACHER TEACHER TEACHER NON- TEACHER GENERAL PUBLIC GENERAL PUBLIC TEACHER TEACHER TEACHER NON- TEACHER

Setting Achievement Levels for NAEP Writing

Feedback

14

Setting Achievement Levels for NAEP Writing

Reference Materials

15

Title Response Title Response 1 A25 Ylksu sl Ofnsl A25_Ylksu sl Ofnsl Napoidjf Umndp A25_Napoidjf Umndp 2 A12 Wldi fj Mhapvuh A12_Wldi fj Mhapvuh Radk Padfunv A12_Radk Padfunv 3 C4 Napoidjf Umndp C4_Napoidjf Umndp Ylksu sl Ofnsl A22_Ylksu sl Ofnsl 4 A15 Radk Padfunv A15_Radk Padfunv Radk Padfunv A15_Radk Padfunv 5 C16 Wldi fj Mhapvuh C16_Radk Padfunv Ylksu sl Ofnsl C16_Ylksu sl Ofnsl 6 A22 Ylksu sl Ofnsl A22_Ylksu sl Ofnsl Napoidjf Umndp A22_Napoidjf Umndp 7 A9 Radk Padfunv A9_Radk Padfunv Napoidjf Umndp A9_Napoidjf Umndp 8 A5 Napoidjf Umndp A5_Napoidjf Umndp Wldi fj Mhapvuh A5_Wldi fj Mhapvuh 9 A17 Wldi fj Mhapvuh A17_Wldi fj Mhapvuh Ylksu sl Ofnsl A17_Ylksu sl Ofnsl 10 C15 Radk Padfunv C15_Radk Padfunv Napoidjf Umndp C15_Napoidjf Umndp BoW Order BoW ID Prompt #1 Prompt #2

Setting Achievement Levels for NAEP Writing

Real Time Review of Survey Results

16

Setting Achievement Levels for NAEP Writing Prompt 1 C18_Prompt 1 Video 1 Prompt 2 C18_Prompt 2

Exemplar Selection

17

Setting Achievement Levels for NAEP Writing

Lessons Learned

Pros

• Computer-based platform is a practical necessity for managing today’s

standard settings

• Flexibility of Moodle has proven to be effective in handling many

methodologies and design complexities (as demonstrated)

• Efficiencies in terms of digital versus paper, distribution of materials,

collection of judgments/ratings, and real time oversight cannot be

• verstated

Cons

• Not all users comfortable with computers
• Reliance on multiple computers and management of multiple logins

can be cumbersome

• Security needs are greater (than paper based)
• Some interface constraints can mean limitations to functionality

18

Questions?

tim.oneil@pearson.com