Setting Performance Standards for Statewide Kindergarten Assessments - - PowerPoint PPT Presentation

Setting Performance Standards for Statewide Kindergarten Assessments

1

National Conference on Student Assessment, Orlando, FL

June 24, 2019

Presentation Outline

• KEEP (General) Standard Setting Process
• Method
• Lessons Learned
• KEEP Alternate Standard Setting Process
• Method
• Lessons Learned

4/1/2019 www.nciea.org 2

KEEP (General) Standard Setting Process

KEEP Entry (Fall 2017)

1. Crowd-sourced Modified-Angoff survey (virtual) 2. PLD development meeting (virtual) 3. Modified Body of Work workshop (in-person)

KEEP Exit (Spring 2018)

1. PLD development meeting (virtual) 2. Modified Body of Work workshop (in-person)

4/6/2019 www.nciea.org www.schools.utah.gov 3

Crowd-Sourced Modified-Angoff Survey

• Leveraged existing KEEP training session to recruit kindergarten

teachers for this web-based standard setting activity

• Each teachers was asked via a web-based survey to enter the

expected score on each KEEP item for an incoming kindergarten student who is “just ready” for her or his kindergarten classroom.

• Results were used to inform the score ranges for the student profiles

used in the Body of Work standard setting workshop.

4 www.nciea.org www.schools.utah.gov 4/6/2019

Note that for this round, the intention was to have just one cut score. In the modified BOW meeting, two cut scores were recommended

4/6/2019 www.nciea.org www.schools.utah.gov 5

Crowdsourced recommendations to bound work in live modified body of work meeting

PLD Development Meeting

• USBE and the Center worked with Utah kindergarten experts to

develop recommend performance level labels, generate draft policy descriptors and PLDs.

• Prior to the meeting, USBE decided on three performance levels for

KEEP, indicating different degrees of readiness for kindergarten (KEEP Entry) or 1st grade (KEEP Exit).

• Separate PLDs were written for KEEP Entry and Exit on two different
• ccasions but by a similar group of experts.
• PLDs were reviewed, updated and finalized based on feedback from

the in-person standard setting workshop.

4/6/2019 www.nciea.org www.schools.utah.gov 6

KEEP PLD Development Template

7 www.nciea.org www.schools.utah.gov 4/6/2019

Modified Body of Work (BOW) Workshop

• 1. Review PLDs
• 2. Experience the KEEP assessment
• 3. Participate in rounds of judgment
• 4. Consider feedback data and discussions between rounds
• 5. Articulate cut scores across content areas and finalize PLD

Constraint – time allotted for in-person meetings:

• 6 hours for KEEP Entry and 4 hours for KEEP Exit

8

!!!

www.nciea.org www.schools.utah.gov 4/6/2019

Online Standard Setting Tool

Excel-based spreadsheet used to:

• Create initial profile cards and rating forms (based on real student

responses)

• Generate round feedback data (summary of ratings, impact data)
• Produce profile cards and rating forms for subsequent rounds

(dynamically)

• Determine cut scores (using non-parametric approach based on

maximizing classification consistency)

9 www.nciea.org www.schools.utah.gov 4/6/2019

Modified BOW Cut-Score Recommendation Process

• Panelists are provided with packets of profile cards.
• Two rounds of judgments in which panelist rate (or sort) the profile

cards into one of three performance categories based on the PLDs.

• Round 1: range-finding
• Round 2: pinpointing
• 10
• As a committee, panelists can

make adjustments (within boundaries) in the final articulation round.

www.nciea.org www.schools.utah.gov 4/6/2019

Example Profile Card Section for KEEP Exit ELA

4/1/2019 www.nciea.org 11

• Yellow shows how students responded for a polytomously scored item or whether the student responded

accurately to a dichotomously scored item.

• Greyed-out cells show other ways student responses could have been scored had the student responded

differently.

Another Section of an Example Profile Card

4/1/2019 www.nciea.org 12

13 www.nciea.org www.schools.utah.gov 4/6/2019

Part of the Round-1 Feedback

• Panelists were given the

profile cards in a random

• rder without any score

provided on the profile cards because we wanted them to focus closely on the body of work shown in the profile card.

• Panelists could calculate

raw scores if they wanted, and some did.

• This feedback is the first

time panelists see what the scores are for each profile they classified.

4/6/2019 www.nciea.org www.schools.utah.gov 14

More Round-1 Feedback

• Profiles are ranked from lowest score to

highest score, with profile IDs in the leftward blue cells.

• The numbers in the grey cells show which

level the panelist assigned to each profile.

• The blue numbers on the right show areas
• f disagreement for the two cut scores.
• The red-outlined rows show the median

recommended cut score of the full panel

• This focused panelist discussion on areas
• f disagreement before round 2.

4/6/2019 www.nciea.org www.schools.utah.gov 15

Identifying Recommended Cut Scores

• With 40 possible scores points, we

create 40 hypothetical raters, each

• f whom is assigned to classify all

profiles with a score lower than their assigned score as not meeting the cut score, and all others as meeting the cut score (i.e., “Guttman panelists”)

• Identifying each real panelist’s cut

score is done by counting the number of her inconsistencies with each of the hypothetical Guttmann

• panelists. The score assigned to the

hypothetical panelist with which the real rater has the fewest inconsistencies becomes the recommended cut score for the real panelist.

• (Inconsistency lines were smoothed

using a period-5 rolling mean)

4/6/2019 www.nciea.org www.schools.utah.gov 16

Round-2 Results for Pinpointing Ranges

Round 3

4/6/2019 www.nciea.org www.schools.utah.gov 17

Directly adjusting the recommended cut scores from round 2 with the benefit of having the profile cards, impact data, and a histogram showing the frequencies of student scores.

Lessons Learned

• Paper management and manual data entry slowed down the process

during the KEEP Entry standard setting meeting.

• For KEEP Exit, participants brought their own digital devices, viewed all

materials (as PDF) on at their devices and provided ratings via SurveyMonkey.

• Process evaluation survey results indicated:
• Participants responded favorably (agree or strongly agree) to the

standard setting workshop.

• Participants felt that the KEEP PLDs and cut scores appropriately

reflected the expectations of kindergarten students in Utah.

4/6/2019 www.nciea.org www.schools.utah.gov 18

KEEP Alternate Standard Setting

Goal

• Establish performance levels for status and growth on KEEP Alternate

Participants

• Kindergarten special education experts and stakeholders within Utah

Steps

1. Presentation of empirical data 2. Development of performance level descriptors (PLDs) 3. Recommendation of cut scores

4/1/2019 www.nciea.org 19

Presentation of Empirical Data: Process

• Goal is to give the participants a sense of how students with SCDs

were performing on KEEP Alternate as context for subsequent tasks.

• Participants first reviewed KEEP Alternate rubric and shared

experience administering the assessment.

• Statewide KEEP Alternate results from 2017-18 were presented. The

results, shown separately for Entry and Exit, included:

• Distribution and descriptive statistics of the total raw score for each

KEEP Alternate section, and

• Average ratings for each strand.

4/1/2019 www.nciea.org 20

Presentation of Empirical Data: Observations

KEEP Alternate Entry

• Generally low performance in both literacy and numeracy.
• Most students were rated as 1 (Not Yet Emerging) and 2 (Emerging) on

the strands in each section. There were very few 4’s (At Target) and 5’s (Advanced).

KEEP Alternate Exit

• The total raw scores and average ratings for each strand increased

noticeably compared to Entry

• There were still students with ratings of 1’s across all strands, but more

students received ratings of 2’s and 3’s, and some received 4’s and 5’s.

4/1/2019 www.nciea.org 21

Development of PLDs: Determining Performance Levels

• KEEP Alternate descriptors are needed for
• Status-based performance levels for Entry (preparedness for kindergarten)
• Status-based performance levels for Exit (readiness for 1st grade)
• Growth-based performance level (progress made during kindergarten year)
• The committee first determined the number of performance levels

and performance level labels for each of the measures, using the TAC’s recommendations as starting point

• Entry: “Entry A1” and “Entry A2”
• Exit: “Exit A1”, “Exit A2” and ‘Exit A3”
• Growth: “No/Minimal Growth”, “Sufficient Growth” and “Substantial Growth”

4/1/2019 www.nciea.org 22

Development of PLDs: Drafting Descriptors

The committee then generated descriptors for each performance level. Examples:

Entry A1 The entering kindergarten student with significant cognitive disability does not demonstrate the knowledge or skills specified in the Utah Essential Elements (Alternate Achievement Standards) for kindergarten in literacy or numeracy and will continue to need significant support to acquire the skill sets during kindergarten.

4/1/2019 www.nciea.org 23

Development of PLDs: Drafting Descriptors

Exit A2 The exiting kindergarten student with significant cognitive disability demonstrates adequate knowledge or skills specified in the Utah Essential Elements Alternate Achievement Standards) for kindergarten in literacy or numeracy and is approaching target for the Utah Essential Elements in first grade. Substantial Growth The student with significant cognitive disability demonstrated substantial progress in knowledge or skills specified in the Utah Essential Elements (Alternate Achievement Standards) for kindergarten in literacy or numeracy.

4/1/2019 www.nciea.org 24

Recommendation of Cut Scores: Data Provided

The committee was provided with two pieces of data:

• TAC recommendations
• Impact data

TAC recommendations:

• For Entry, cuts at total raw score points of 14 for Literacy and 12 for

Numeracy, which represented an average score of 2 (“Emerging”) for each strand;

• For Exit, cuts at the average score of 3 (“Approaching Target”) and

average of score of 4 (“At Target”) for each section to distinguish between Exit A1/Exit A2 and Exit A2/Exit A3, respectively.

4/1/2019 www.nciea.org 25

Recommendation of Cut Scores: Impact Data

4/1/2019 www.nciea.org 26

Impact data based on TAC recommendations:

Recommendation of Cut Scores: Growth

• After considering the cut scores in light of the KEEP Alternate rubric

and draft PLDs, and evaluating the reasonableness of the impact data, the committee agreed with the TAC’s recommendations.

• The committee than examined the possible combination of Entry/Exit

performance levels (see below) and assigned one growth-based performance levels to each cell based on the growth-based PLDs.

4/1/2019 www.nciea.org 27

Recommendation of Cut Scores: Growth

Impact data were provided to help committee evaluate the reasonableness of their recommendations.

4/1/2019 www.nciea.org 28

Lessons Learned

• The KEEP Alternate standard setting process was completed in half a

day due in large part to:

• Involvement of educators who are familiar with KEEP Alternate
• Use of TAC recommendations as starting point for status cut scores
• Simplification of performance assumptions across sections and strands
• The committee raised a concern of the possibility for a student to get

a lower total raw score on Exit than on Entry but still be classified as “Sufficient Growth” or “Substantial Growth”.

• USBE should consider including additional criteria for growth-based

classifications.

4/1/2019 www.nciea.org 29

www.nciea.org

30

www.schools.utah.gov