School Accountability Systems Chris Domaleski Center for Assessment - - PowerPoint PPT Presentation

Establishing Performance Standards for School Accountability Systems

Chris Domaleski Center for Assessment National Conference on Student Assessment San Diego, CA June 27, 2018

Accountability Ratings: Meaning Matters

• How can we help stakeholders understand, interpret,

and make good decisions based on school ratings?

• Whether these ratings are points, stars, letters, or

something else, establishing and communicating the ratings thoughtfully is an important part of accountability.

Performance Standards in Accountability CCSSO NCSA 2018 2

Everyone knows what an ‘A’ means, right?

• Credit Ratings

– S&P describes an “A” as, “strong capacity to meet financial commitments” as distinguished from AAA “extremely strong” or AA “very strong” – In August 2011 S&P downgraded the U.S. from AAA to AA+ stating in part, “the downgrade reflects our view that the effectiveness, stability, and predictability of American policymaking and political institutions have weakened.” – Does the letter grade alone tell you what you need to know to inform a buy or sell decision?

• New York Department of Health restaurant grades

– “A letter grade of A indicates the a restaurant received 13 or fewer health code violations in their last inspection…violations are likely to be technicalities or non-critical infractions” – Does the letter grade alone provide the information you need to make a decision about where to dine?

Performance Standards in Accountability CCSSO NCSA 2018 3

The meaning of a rating matters

• Other interesting ratings:

– Rotten Tomatoes Movie ratings:

• Fresh = Tomatometer of at least 60% positive reviews
• Certified Fresh = 75% positive reviews.

– Hurricanes:

• Saffir-Simpson Hurricane Scale rates a hurricane's intensity using wind speed and storm surge.
• Category 5 is characterized by 155+ wind speeds and storm surges above 18 feet

– Collectables:

• Typical scale: mint, near mint, excellent, very good, good, poor.
• For example, a “very good” baseball card can have scratches, stains, creases, and fading.
• The point is: meanings matter!

– Should you spend your money on a movie that isn’t ‘Certified Fresh’? – Should you go outside in a category 3 hurricane? – Can you expect a baseball card in ‘good’ condition to have value?

Performance Standards in Accountability CCSSO NCSA 2018 4

Performance Standards in Accountability

• Given the importance of accountability ratings, it is

reasonable to require evidence that the rating has a high degree of validity for the intended uses and purposes

• A substantial part of that validity argument is the design

and implementation of sound process for establishing standards that credibly reflects the state’s vision and policy priorities

• While there is a research base for assessment standard

setting, accountability standards are often overlooked and are defined arbitrarily or tautologically

• Ultimately, the intent is to improve the clarity and

usability for stakeholders

5 Performance Standards in Accountability CCSSO NCSA 2018

Principles

• Rationale, procedure, and results should be well

documented

• The process should be designed to allow well-

qualified policy-makers, experts, and/or stakeholders an opportunity to inform the decisions

• Participants should be provided information

associated with relevant consequences and criteria

6 Performance Standards in Accountability CCSSO NCSA 2018

How to define ‘Good Enough’ Performance?

• Decisions of policy-makers and leaders, which are often

reflected in law, rules, guidance, or vision statement

– SEA long-term goals and measures of interim progress – ESSA high school graduation rate criterion of 67%

• Based on normative data

– E.g. top 10% receive an ‘A’ – Analyses of legacy data to determine rate of improvement that can be expected

• Expert judgment
• Relationship to external criteria

Performance Standards in Accountability CCSSO NCSA 2018 7

Promising Practices

• Develop policy descriptors for each level
• Develop school performance level descriptors
• Convene panel or other group (e.g. task force, advisory group)

to:

– Evaluate school profiles with respect to descriptors – Review data on achievement patterns – Set Indicator thresholds and overall thresholds

• Evaluate
• Document
• Produce material to support interpretation and use

8 Performance Standards in Accountability CCSSO NCSA 2018

State Illustrations

• Today we will hear from two states that implemented

a systematic process to set meaningful performance expectations for their school accountability system

• The intent is to learn from their experiences to

identify challenges and opportunities to inform the work in your state

9 Performance Standards in Accountability CCSSO NCSA 2018