Setting Performance Standards for Statewide Kindergarten Assessments National Conference on Student Assessment, Orlando, FL June 24, 2019 1
Presentation Outline • KEEP (General) Standard Setting Process • Method • Lessons Learned • KEEP Alternate Standard Setting Process • Method • Lessons Learned 4/1/2019 www.nciea.org 2
KEEP (General) Standard Setting Process KEEP Entry (Fall 2017) 1. Crowd-sourced Modified-Angoff survey (virtual) 2. PLD development meeting (virtual) 3. Modified Body of Work workshop (in-person) KEEP Exit (Spring 2018) 1. PLD development meeting (virtual) 2. Modified Body of Work workshop (in-person) 4/6/2019 www.nciea.org www.schools.utah.gov 3
Crowd-Sourced Modified-Angoff Survey • Leveraged existing KEEP training session to recruit kindergarten teachers for this web-based standard setting activity • Each teachers was asked via a web-based survey to enter the expected score on each KEEP item for an incoming kindergarten student who is “just ready” for her or his kindergarten classroom. • Results were used to inform the score ranges for the student profiles used in the Body of Work standard setting workshop. 4/6/2019 www.nciea.org www.schools.utah.gov 4
Crowdsourced recommendations to bound work in live modified body of work meeting Note that for this round, the intention was to have just one cut score. In the modified BOW meeting, two cut scores were recommended 4/6/2019 www.nciea.org www.schools.utah.gov 5
PLD Development Meeting • USBE and the Center worked with Utah kindergarten experts to develop recommend performance level labels, generate draft policy descriptors and PLDs. • Prior to the meeting, USBE decided on three performance levels for KEEP, indicating different degrees of readiness for kindergarten (KEEP Entry) or 1 st grade (KEEP Exit). • Separate PLDs were written for KEEP Entry and Exit on two different occasions but by a similar group of experts. • PLDs were reviewed, updated and finalized based on feedback from the in-person standard setting workshop. 4/6/2019 www.nciea.org www.schools.utah.gov 6
KEEP PLD Development Template 4/6/2019 www.nciea.org www.schools.utah.gov 7
Modified Body of Work (BOW) Workshop 1. Review PLDs 2. Experience the KEEP assessment 3. Participate in rounds of judgment 4. Consider feedback data and discussions between rounds !!! 5. Articulate cut scores across content areas and finalize PLD Constraint – time allotted for in-person meetings: • 6 hours for KEEP Entry and 4 hours for KEEP Exit 4/6/2019 www.nciea.org www.schools.utah.gov 8
Online Standard Setting Tool Excel-based spreadsheet used to: • Create initial profile cards and rating forms (based on real student responses) • Generate round feedback data (summary of ratings, impact data) • Produce profile cards and rating forms for subsequent rounds (dynamically) • Determine cut scores (using non-parametric approach based on maximizing classification consistency) 4/6/2019 www.nciea.org www.schools.utah.gov 9
Modified BOW Cut-Score Recommendation Process • Panelists are provided with packets of profile cards. • Two rounds of judgments in which panelist rate (or sort) the profile cards into one of three performance categories based on the PLDs. • Round 1: range-finding • Round 2: pinpointing • • As a committee, panelists can make adjustments (within boundaries) in the final articulation round. 4/6/2019 www.nciea.org www.schools.utah.gov 10
Example Profile Card Section for KEEP Exit ELA • Yellow shows how students responded for a polytomously scored item or whether the student responded accurately to a dichotomously scored item. • Greyed-out cells show other ways student responses could have been scored had the student responded differently. 4/1/2019 www.nciea.org 11
Another Section of an Example Profile Card 4/1/2019 www.nciea.org 12
Part of the Round-1 Feedback • Panelists were given the profile cards in a random order without any score provided on the profile cards because we wanted them to focus closely on the body of work shown in the profile card. • Panelists could calculate raw scores if they wanted, and some did. • This feedback is the first time panelists see what the scores are for each profile they classified. 4/6/2019 www.nciea.org www.schools.utah.gov 13
More Round-1 Feedback • Profiles are ranked from lowest score to highest score, with profile IDs in the leftward blue cells. • The numbers in the grey cells show which level the panelist assigned to each profile. • The blue numbers on the right show areas of disagreement for the two cut scores. • The red-outlined rows show the median recommended cut score of the full panel • This focused panelist discussion on areas of disagreement before round 2. 4/6/2019 www.nciea.org www.schools.utah.gov 14
Identifying Recommended Cut Scores • With 40 possible scores points, we create 40 hypothetical raters, each of whom is assigned to classify all profiles with a score lower than their assigned score as not meeting the cut score, and all others as meeting the cut score (i.e., “Guttman panelists”) • Identifying each real panelist’s cut score is done by counting the number of her inconsistencies with each of the hypothetical Guttmann panelists. The score assigned to the hypothetical panelist with which the real rater has the fewest inconsistencies becomes the recommended cut score for the real panelist. • (Inconsistency lines were smoothed using a period-5 rolling mean) 4/6/2019 www.nciea.org www.schools.utah.gov 15
Round-2 Results for Pinpointing Ranges 4/6/2019 www.nciea.org www.schools.utah.gov 16
Round 3 Directly adjusting the recommended cut scores from round 2 with the benefit of having the profile cards, impact data, and a histogram showing the frequencies of student scores. 4/6/2019 www.nciea.org www.schools.utah.gov 17
Lessons Learned • Paper management and manual data entry slowed down the process during the KEEP Entry standard setting meeting. • For KEEP Exit, participants brought their own digital devices, viewed all materials (as PDF) on at their devices and provided ratings via SurveyMonkey. • Process evaluation survey results indicated: • Participants responded favorably (agree or strongly agree) to the standard setting workshop. • Participants felt that the KEEP PLDs and cut scores appropriately reflected the expectations of kindergarten students in Utah. 4/6/2019 www.nciea.org www.schools.utah.gov 18
KEEP Alternate Standard Setting Goal • Establish performance levels for status and growth on KEEP Alternate Participants • Kindergarten special education experts and stakeholders within Utah Steps 1. Presentation of empirical data 2. Development of performance level descriptors (PLDs) 3. Recommendation of cut scores 4/1/2019 www.nciea.org 19
Presentation of Empirical Data: Process • Goal is to give the participants a sense of how students with SCDs were performing on KEEP Alternate as context for subsequent tasks. • Participants first reviewed KEEP Alternate rubric and shared experience administering the assessment. • Statewide KEEP Alternate results from 2017-18 were presented. The results, shown separately for Entry and Exit, included: • Distribution and descriptive statistics of the total raw score for each KEEP Alternate section, and • Average ratings for each strand. 4/1/2019 www.nciea.org 20
Presentation of Empirical Data: Observations KEEP Alternate Entry • Generally low performance in both literacy and numeracy. • Most students were rated as 1 (Not Yet Emerging) and 2 (Emerging) on the strands in each section. There were very few 4’s (At Target) and 5’s (Advanced). KEEP Alternate Exit • The total raw scores and average ratings for each strand increased noticeably compared to Entry • There were still students with ratings of 1’s across all strands, but more students received ratings of 2’s and 3’s, and some received 4’s and 5’s. 4/1/2019 www.nciea.org 21
Development of PLDs: Determining Performance Levels • KEEP Alternate descriptors are needed for • Status-based performance levels for Entry (preparedness for kindergarten) • Status-based performance levels for Exit (readiness for 1 st grade) • Growth-based performance level (progress made during kindergarten year) • The committee first determined the number of performance levels and performance level labels for each of the measures, using the TAC’s recommendations as starting point • Entry: “Entry A1” and “Entry A2” • Exit: “Exit A1”, “Exit A2” and ‘Exit A3” • Growth: “No/Minimal Growth”, “Sufficient Growth” and “Substantial Growth” 4/1/2019 www.nciea.org 22
Development of PLDs: Drafting Descriptors The committee then generated descriptors for each performance level. Examples: Entry A1 The entering kindergarten student with significant cognitive disability does not demonstrate the knowledge or skills specified in the Utah Essential Elements (Alternate Achievement Standards) for kindergarten in literacy or numeracy and will continue to need significant support to acquire the skill sets during kindergarten. 4/1/2019 www.nciea.org 23
Recommend
More recommend
