In a State of Flux: The State of State Assessments DO STATES REMAIN - - PowerPoint PPT Presentation

1

In a State of Flux: The State of State Assessments

DO STATES REMAIN COMMITTED TO HIGHER QUALITY SUMMATIVE TESTS?

PRESENTATION TO NATIONAL CONFERENCE ON STUDENT ASSESSMENT| JUNE 2018

2

“What a long, strange trip it’s been.…”

2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 USED awards grant to WIDA (2011) and ELPA21 (2012) to develop an English proficiency assessment ready for use by the 2015-16 school year Next Generation Science Standards (NGSS) are published and ready for state adoption Every Student Succeeds Act (ESSA) passes; maintains the requirement that states test and report annually against college- and career-ready standards but creates more flexibility for states to use other measures of student learning in addition to summative testing USED awards grants to PARCC and Smarter Balanced to develop and implement next- generation assessment systems by the 2014-15 school year PARCC and Smarter Balanced report having 26 and 31 state members respectively USED change in leadership 48 states adopt Common Core State Standards (CCSS)

Source: Headline adapted from The Grateful Dead.

Every year between 2013 and 2015, 5 to 6 states left PARCC and 3 states left Smarter Balanced; The non-test participation or “opt

• ut” movement

peaks in many states

What’s next?

3

Given these changes, in 2017 the Hewlett Foundation asked Ed First to document the “state of state assessments” for the field

92%

Our response rate from states to confirm data and fill in gaps was 47 out

• f 51 states/DC (four that did not verify:

DC., Kentucky, New Jersey, Tennessee)

1 See separate PPT file where all 51 state profile slides are compiled

We also interviewed leaders in the field to identify cross-state developments and areas of need

We are grateful to these individuals for their time and contributions to this research and analysis

What we did:

2. 3.

We compiled a dossier on every state https://goo.gl/pBJJUC and a summary of trends https://tinyurl.com/yct5sabk

Summer/ Fall 2017: Compiled summative assessment information

1.

4

Here were objectives of the Ed First research—and for the presentation and discussion today…

 What are “high-quality” state summative assessments—and what do we know about which tests are high quality?  What choices have states made about PARCC, Smarter Balanced and other testing vendors?  What factors may either limit or empower state leaders to make high- quality state tests a more consistent priority?  What do recent trends, development and lessons learned suggest for a path forward, to continuing improving test quality?

5

What is a high-quality test? In 2014, CCSSO articulated criteria for state leaders to use in making sure their assessments matched the depth, breadth and rigor of new state academic standards

CCSSO encourages state officials to use these criteria as “they develop procurements and evaluate options for high-quality state summative assessments aligned to college- and career-readiness standards”

Meet Overall Assessment Goals and Ensure Technical Quality Align to Standards – English Language Arts/Literacy Align to Standards - Mathematics Yield Valuable Reports

• n Student Progress and

Performance

A B C D

Adhere to Best Practices in Test Administration State Specific Criteria (as desired)

E F

6

Why care? Summative tests are one important tool for state and education leader to help schools reset their expectations and ensure students are learning what they need for success

Ideally, state tests should:  Signal what is important  Track how schools and students are progressing  Flag where more support or attention may be needed for student success  Provide complementary information to local assessments And so states have trying to upgrade their tests to prioritize the skills and knowledge that matter most today…

Deeper understanding Innovating Knowledge application Problem solving Synthesizing Writing

Source: High Quality Assessment Project (2017) adapted from Georgetown Public Policy Institute; The Noun Project (2018)

7

Common Core standards—and PARCC and Smarter Balanced tests built to measure them—met the goal of asking more from students

• 11 states fully

participated in Smarter Balanced: California, Connecticut, Delaware, Hawaii, Idaho, Montana, Nevada, Oregon, South Dakota, Vermont, Washington

• 1 state populated its test

with Smarter Balanced items: Michigan

• Looking ahead, this year

a state selected a vendor that will use Smarter Balanced items: Indiana

Smarter Balanced states are BLUE

COLOR KEY: Maps on the following slides show which states used consortia items in the 2017- 18 school year

8

Common Core standards—and PARCC and Smarter Balanced tests built to measure them—met the goal of asking more from students

• 5 states used PARCC test forms:

District of Columbia, Illinois, Maryland, New Jersey, New Mexico

• 3 states (Colorado, Louisiana and

Massachusetts) populated their assessments with PARCC items to varying degrees

• Although Rhode Island may still be

using some PARCC items, state leaders say they have left PARCC (so we don’t count it)

• Looking ahead, the Illinois SEA is

re-bidding its state assessment (but will still use PARCC items), and the new governor of New Jersey has stated he intends to end PARCC testing

New Meridian (PARCC) states are ORANGE

COLOR KEY: Maps on the following slides show which states used consortia items in the 2017- 18 school year

9

With No Child Left Behind pressures for testing in most grades and state budget pressures, state tests in 2010 had lots of room for improvement; PARCC and Smarter Balanced raised the bar

Sources: Data in chart includes two studies by the RAND Corporation on quality of state tests, AP and other common summative assessments (2011 and 2012), as well as research by the Fordham Institute (2016) and HumRRO (2016).

Fordham Institute and HumRRO review of 4 state tests in 2016 RAND review of 17 state tests in 2011

10

While teachers in many states objected to the increased rigor and time expectations of new tests, teachers who had a chance to examine them more closely were impressed

When I had the chance to sit down and compare content side-by-side, it became evident the substance of the PARCC and Smarter Balanced tests outshined the material from the old tests in several ways. —Josh Parker Maryland State Teacher of the Year, and reviewer in NNSTOY study The most important takeaway for parents, students and teachers is that these newer state tests can be useful tools to help us all understand how our children are thinking. —Maryann Woods-Murphy New Jersey State Teacher of the Year, and reviewer in NNSTOY study As a teacher, I sympathize with parents who have grown frustrated with the number of tests their children face. But I can say with confidence these new assessments are the kind we should want our kids to take. —Pam Reilly Illinois State Teacher of the Year, and reviewer in NNSTOY study For example, the National Network of State Teachers of the Year convened panels of teachers to study and compare PARCC and Smarter Balanced assessments to assessments formerly used by states: Its first study compared new tests to previous tests from Delaware, Illinois, New Hampshire and New Jersey (2015), and a second study compared them to previous tests in Nevada and Oregon (2016)

Sources: NNSTOY, Right Trajectory and Still on the Right Trajectory

11

• Maryland’s state test—based on PARCC—was the first state

to “fully meet” requirements.

• 85% of consortia states (11 of 13 states) still in the peer

review process substantially met assessment requirements (reviewers asked for more information on some aspects)

• 44% of non-consortia states (10 of 23 states) still in the peer

review process substantially met assessment requirements

• 7 states have assessment systems that meet all requirements

(3 consortia and 4 other)

• 88% of all consortia states that started peer review in 2016 or

2017 substantially meet or have met requirements, while 52% of all other states have met these highest rating levels

USED state assessment peer reviews confirm variations still exist: As best current proxy for quality, reviews are finding states with consortia tests are further along in meeting requirements

In examining state assessments systems, reviewers consider six criteria

1. Statewide system of standards and assessments 2. Assessment system operations 3. Technical quality– validity 4. Technical quality– other 5. Inclusion of all students 6. Academic achievement and standards reporting

Source: USDE, U.S. Department of Education Peer Review of State Assessment Systems (2015); USDE, Decision Letters on Each State’s Final Assessment System Under No Child Left Behind (NCLB).

DOES NOT MEET REQUIREMENTS PARTIALLY MEETS REQUIREMENTS Alabama (grades 3-8, ACT Aspire), Arizona, Illinois, Iowa, Kansas, Kentucky, Louisiana (grades 3-8), Maine (grades 3-8), Mississippi, New Hampshire, Ohio, Texas, Virginia, Wisconsin (HS), Wyoming SUBSTANTIALLY MEETS REQUIREMENTS Arkansas (Aspire), California, Connecticut, Delaware, DC, Florida, Georgia, Hawaii, Idaho, Indiana (grades 3- 8, ISTEP), Minnesota, Missouri, New Mexico, New York, Oregon, Rhode Island, South Carolina (grades 3- 8), South Dakota, Utah, Vermont, Washington MEETS REQUIREMENTS Colorado, Maryland, New Jersey, North Carolina, North Dakota, Pennsylvania, West Virginia

States present evidence about their assessment systems against each of the six criteria, and states not fully meeting peer review requirements did so for different reasons.

12

27 8 32 13

5 10 15 20 25 30 35

2010 2011 2012 2013 2014 2015 2016 2017

Participating States

Consortia membership in PARCC and Smarter Balanced (2010 to December 2017) 1

PARCC Smarter Balanced

However, while 46 states originally joined PARCC, Smarter Balanced or both, states’ participation has eroded since 2010

• The fastest decline in state participation

happened between 2010 and 2015. The tests were first administered in spring 2015.

• As of late 2017, only 21 states report

they are using items from either PARCC

• r Smarter Balanced.
• Both New Meridian/PARCC and Smarter

Balanced are adapting to the field and now offer states more flexible options to access items (Smarter Balanced also

• ffers a more comprehensive suite of

tools in its package)

Sources: Education Next, The Politics of the Common Core Assessments (2016); Education Week, Which states are using PARCC or Smarter Balanced? (2017) Education Week; Education Week, State Testing: An Interactive Breakdown of 2015-2016 Plans (2016); PARCC; Smarter Balanced; Education First interviews.

13

Having moved away from PARCC or Smarter Balanced, those states are creating their own grade 3-8 assessments in math and English LA/literacy… once again

As of spring 2018, four vendors outside of the consortia had the majority of contracts to create new state K-8 assessments: American Institutes for Research, Data Recognition Corporation, Pearson and Measured Progress

Sources: SEA websites; State assessment directors; Education Week, Which states are using PARCC or Smarter Balanced? (2017).

AIR (10 states) Pearson (4 states) DRC (6 states) Other: ACT, ETS, NWEA, Questar, SEA-developed, University of Iowa, and University of Kansas RFP to select new vendor (as of March 2018) Measured Progress (4 states) New Meridian

• r Smarter

Balanced Vendor uses PARCC or Smarter Balanced items

KEY: State vendors 2017-18 (and anticipated 2018-19)

14

At the high school level, an increasing number of states (including consortia members) are relying on ACT or SAT for accountability

ACT SAT ACT and SAT Do not use ACT or SAT Use ACT for HS accountability Use SAT for HS accountability

In the 2016-17 school year, the ACT or SAT assessments were mandatory for all high school students in 24

• states. In 13 of these states (including six Smarter Balanced or PARCC states), ACT or SAT results also are

used for school accountability decisions

Uses grades 3-8 PARCC or Smarter Balanced test

Sources: State assessment directors; Education Week, Which State Require Students to Take the SAT or ACT? An Interactive Breakdown of States’ 2016-17 Testing Plans (2016)

KEY: States requiring all high school students to take ACT or SAT

Note: Alabama did not use ACT during the 2017-18 school year

15

Aptitude vs. Achievement:

What is most important for your state’s high school test to measure?

ACT and SAT are great measures and incentives for college-going, but will states “lean in” to make these tests work well for the additional purpose of high school accountability?

Accessibility for Different Populations:

What policies should your state insist vendors follow to continue advancing equity?

State Control:

Are there additional safeguards needed to achieve the right amount of state authority, esp. over student data?

Fairness and Transparency:

With a national high school test, does your state need to do anything else to ensure results are valid?

Sources: Achieve and Center for Assessment, High School Assessment in a New Era: What Policymakers Need to Know (2016); Erin O’Hara, Choices and Trade-offs: Key Questions for State Policymakers when Selecting High School Assessments (2016)

State leaders should be working on tough questions in these sorts of areas if using national college admissions (as federal law now specifically allows) as the sole measure of high student performance:

16

More change coming? As of February 2018, for example, 22 states were expecting to re-bid testing contracts

Sources: SEA websites; State assessment directors; Education First analysis.

States with One or More State Contracts Ending in 2017 and 2018, as of Feb 2018

(Excluding contracts with PARCC and Smarter Balanced3)

Grades 3-8 E/LA and Math High School E/LA and Math 2017—not yet awarded (4 states total) Alabama, Missouri, Wyoming Alabama, South Carolina, Wyoming 2018 (7 states total)4 Arizona, Minnesota, Illinois (new test needs to include PARCC items), Utah Arizona, Connecticut (doesn’t use Smarter Balanced H.S. test), Illinois, Minnesota, Mississippi, Utah Renews annually (13 states total) Alaska, Arkansas, Colorado, Kansas, Maine, Ohio, Tennessee, West Virginia Alaska, Arkansas, Colorado(doesn’t use PARCC H.S. test), Delaware (doesn’t use Smarter Balanced H.S. test), Kansas, Louisiana, Maine, Missouri, Nebraska, Nevada (doesn’t use Smarter Balanced H.S. test), Ohio, Tennessee, West Virginia Did not confirm Ed First data on state DC, Kentucky, Rhode Island DC, Kentucky, Rhode Island

17

Also, many states are seeing leadership changes in governors and SEA chiefs—and more change is expected after the election this fall…

Over the next two years, newly elected governors will have authority to appoint their own, new SEA chief in at least 7 states (plus new governors in New Jersey and Virginia have already appointed new chiefs) 7 states have upcoming SEA chief elections, with 1 of these races an open seat (no incumbent in California)

Sources: National Governors Associations (2017); 270 to Win, 2018 Gubernatorial Election Map; Education First analysis.

Incumbent governor running & does not appoint SEA chief Incumbent governor running & appoints SEA chief (includes DC, which currently has interim chief) Open governor’s race, but winner does not appoint SEA chief Open governor’s race2 & winner appoints SEA chief

Gubernatorial elections (2017-2019)

Open governor’s’ race and incumbent SEA chief running Open governor’s race & open-seat SEA chief race Both incumbent governor & SEA chief running New governors & chiefs New governors but incumbent chiefs Incumbent governors & chiefs No election cycle (although Missouri currently has SEA chief vacancy)

Consortia participants (at least 6 new governors, including NJ; at least 7 new chiefs)

18

State boards of education also matter: In most states, members have final or shared authority in approving assessment vendor contracts

Sources: SEA websites; State Assessment Directors; Education First analysis.

13 states

SEA chief is elected SBE is the assessment authority or shares authority in 10 of the 13 states

18 states

Governor appoints SEA chief SBE is the assessment authority or shares authority in 10 of the 18 states

37 state boards

Are the assessment authority or share the authority with the SEA or SEA chief

20 states

SBE appoints SEA chief SBE is the assessment authority or shares authority in 17 of the 20 states

19

What do these trends mean for the future of summative testing— and how well will states’ tests reflect high-quality standards and incorporate writing, problem solving and critical thinking?

?

Glass half full…

States are committed to postsecondary readiness as a goal Consortia items and test designs are highly rated—and both orgs are looking at new ways of delivering content

• utside of “membership”

40% of states still using consortia items A significant number of states are re- bidding testing contracts in 2018--a chance to influence and support High-quality tests take up lots of instructional time—maybe too much time—in the current system design “Quality” doesn’t seem to be a criteria for some state leaders; many testing decisions are political Most states are “going it alone” again, especially for tests in grades 3-8 Consortia participation continues to decline

Glass half empty…

20

Here are some questions we need to be asking ourselves more deliberately…

Can a state successfully work alone to create its

• wn truly high-quality test? What support

would help an individual state? Is it feasible to compare assessment results if more and more states are ‘going at it alone’? Do vendors have the capacity themselves to deliver high-quality assessments to all states,

• ne by one?

Could states benefit from more external reviews

• f their testing systems (outside of federal peer

review)? Have we learned anything about the best role of teaches in designing or contributing to state tests? What are ways of reducing testing time without reducing quality—or accountability? What do newly appointed or elected chiefs most need to know about their state tests in 2018 and 2019? How can we create more of a demand among elected officials, educators and parents for higher-quality tests?

21

Thank you! www.education-first.com

• William Porter, partner (bporter@education-first-com)
• Kathleen Callahan, analyst (kcallahan@education-first.com)

We compiled a dossier on every state… https://goo.gl/pBJJUC …and a summary of trends https://tinyurl.com/yct5sabk