Smarter Balanced Achievement Level Setting Dallas, TX, October - - PowerPoint PPT Presentation

▶
smarter balanced
SMART_READER_LITE
LIVE PREVIEW

Smarter Balanced Achievement Level Setting Dallas, TX, October - - PowerPoint PPT Presentation

Dec 29, 2022 139 likes •439 views

Smarter Balanced Achievement Level Setting Dallas, TX, October 2014 Developing the Expertise to Make Content-Based Achievement Level Recommendations Day 1 Activities that must be completed before panelists are qualified to set

slide-1
SLIDE 1

Smarter Balanced Achievement Level Setting

Dallas, TX, October 2014

slide-2
SLIDE 2

Developing the Expertise to Make Content-Based Achievement Level Recommendations

  • Day 1 Activities that must be completed before

panelists are qualified to set achievement levels using the Bookmark Procedure: – Study content standards – Study achievement level descriptors – Understand what the tests measure

  • Take practice test
  • Study ordered item booklets using item maps
slide-3
SLIDE 3

Ordered Item Books (OIBs)

  • Virtual OIB (online)
  • One item per page
  • Easiest item first
  • Items ascend by difficulty
  • Hardest item last

3

22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

Ordered Item Book

slide-4
SLIDE 4

4

slide-5
SLIDE 5

5

slide-6
SLIDE 6

6

slide-7
SLIDE 7

Study the Items in the OIB

As you study each item in the OIB, discuss two questions with your fellow panelists:

22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

Ordered Item Book

  • 1. What do you know about a student who responds

successfully to this item; that is, what skills must a student have in order to know the correct answer?

  • 2. What makes this item more

difficult than preceding items?

This activity helps you acquire the knowledge to make content-based cut score recommendations.

slide-8
SLIDE 8

Study the Items in the OIB

As you study each item in the OIB, discuss two questions with your fellow panelists:

22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

Ordered Item Book

  • 1. What do you know about a student who responds

successfully to this item; that is, what skills must a student have in order to know the correct answer? .

  • 2. What makes this item more

difficult than preceding items?

slide-9
SLIDE 9

9

slide-10
SLIDE 10
slide-11
SLIDE 11
slide-12
SLIDE 12
slide-13
SLIDE 13

Passages and Other Materials

Item deleted to maintain test security

slide-14
SLIDE 14

Grade Content Area Room Set Up

Content Facilitator Tables of panelists and Table Leader Research Facilitator One Table Facilitator at each table

slide-15
SLIDE 15

The Bookmark Procedure

Smarter Balanced Achievement Level Setting Dallas, TX, October 2014

slide-16
SLIDE 16

Bookmark Overview

  • The Bookmark Procedure is a process used to set cut

scores that define achievement levels by comparing and aligning items and content to achievement level descriptors

  • It is so named because you place bookmarks in an
  • rdered item booklet of test items to make your cut

score recommendations

slide-17
SLIDE 17

Threshold Student

  • We recommend cut scores for Threshold Students—the

students with a level of achievement that just barely qualifies them to be in the achievement level

– These are the skills of the student just entering the achievement level

Level 3 Cut Score Threshold Level 3 Student Mid-level Level 3 Student High-achieving Level 3 Student

17

Level 4 Cut Score

slide-18
SLIDE 18

Three Threshold Students

Level 2 Cut Score Level 4 Cut Score Level 2 Threshold Students Level 3 Threshold Students

18

Level 3 Cut Score Level 4 Threshold Students

slide-19
SLIDE 19

Three Cut Scores Define Four Achievement Levels

Level 2 Cut Score Level 4 Cut Score Level 1 Students Level 2 Students Level 3 Students

19

Level 3 Cut Score Level 4 Students

Level 2 Threshold Students Level 3 Threshold Students Level 4 Threshold Students

slide-20
SLIDE 20

1 4 5 6 7 3 8

Locating a Chess Player on a Scale

20

9

95% chance

2

Lower ability Higher Ability

82% chance 71% chance 64% chance 50% chance 32% chance 24% chance 11% chance 1% chance 50% chance

slide-21
SLIDE 21

1

Locating the Threshold Student on the Scale

21

95% chance

Easier items Harder Items

82% chance 74% chance 61% chance 50% chance 43% chance 34% chance 21% chance 14% chance 50% chance

1 2 3 4 5 6 7 8 9

slide-22
SLIDE 22

1

Locating the Threshold Student on the Scale

22

>50% chance

Easier items Harder Items

>50% chance >50% chance >50% chance 50% chance <50% chance <50% chance <50% chance <50% chance <50% chance 50% chance

1 2 3 4 5 6 7 8 9 10

slide-23
SLIDE 23

Bookmark Placement Instructions

  • Ask yourself: Would a student at the

threshold have at least a 50% chance of earning this point?

– Yes: Move on to the next item. – No: Place your bookmark here.

slide-24
SLIDE 24

22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6

24

Items 6 – 22 reflect content for which the threshold Level 3 student would have less than a 50% chance

  • f success.

L3 5 4 3 2 1

Ordered Item Book

Bookmark on Page 6

Items 1-5 reflect content for which the Level 3 threshold student would have at least a 50% likelihood of success. (with about a 50% chance of success on the item just before the bookmark)

It may be easiest for you to first identify a range of items Ask yourself: Would a student at the threshold have at least a 50% chance of earning this point? Yes: Move on to the next item. No: Place your bookmark here.

slide-25
SLIDE 25

Level 3 Bookmark Placement

  • When you place your Level 3 bookmark, think about a student at

the threshold of Level 3 based on the achievement level descriptors – Place your Level 3 bookmark at the point in the OIB such that

– A threshold Level 3 student would be have at least a 50% likelihood of success on each of the items before the bookmark (and about a 50% chance of success on the item just before the bookmark) – A threshold Level 3 student would have less than a 50% likelihood of success on the items from the bookmark on – It may be easiest for you to first identify a range of items that begins

where threshold Level 3 students begin to be challenged and end where they would definitely be challenged. Then pick the best page in that range.

25

slide-26
SLIDE 26

22 21 20 19 18 L3 17 16 15 14 13 12 11 10 9 8 7 6

26

Items 18 – 22: less than a 50% chance of success. Items 1-17: At least 50% chance of success

5 4 3 2 1

Ordered Item Book

Bookmark on Page 18

Ask yourself: Would a student at the threshold have at least a 50% chance of earning this point? Yes: Move on to the next item. No: Place your bookmark here. No: Place your bookmark here.

slide-27
SLIDE 27

22 21 20 19

L4

18 17 16 15 14 13 12 11 10

L3

9 8 7 6 5

L2

4 3 2 1

Ordered Item Book

27

Three bookmarks define four achievement levels.

slide-28
SLIDE 28

Setting Bookmarks

28

slide-29
SLIDE 29

Agenda

Day 1

  • Morning: Study CCSS and ALDs
  • Afternoon: Study the Ordered Item Booklet

Day 2

  • Morning: Bookmark training and Round 1

– Make bookmark recommendations individually

  • Afternoon: Round 2

– Discuss differences in Round 1 bookmark placements at each table – Place Round 2 bookmarks individually

Day 3

  • Morning: Round 3

– Discuss Round 2 bookmark placements for the entire room (all panelists at all tables) – View supporting data based on Round 2 bookmarks (e.g., impact data, online panel results) – Place Round 3 bookmarks individually – Review final recommendations – Evaluate the process

29