Fleshing out Systems of Assessment in the Context of the Next Generation Science Standards Nathan Dadey Center for Assessment National Conference On Student Assessment San Diego, CA This work is licensed under June 27 th , 2018 a Creative Commons Attribution 4.0 International License.
` 1 Systems of Assessment • Definition & Context • Re-framing 6/27/2018 2
Why? 6/27/2018 3
Why? Why systems of assessment? 6/27/2018 4
Too many assessments given by too many people for too many purposes, all providing too little information. The alternative? Develop a system of assessments. ^Work towards 6/27/2018 Definition & Context | Re-framing 5
One Starting Point Call for “coordinated systems of multiple assessments that work together, along with curriculum and instruction, to promote learning” (p. 242). Pellegrino, Chudowsky & Glaser, 2001 6/27/2018 Definition & Context | Re-framing 6
Conceptual work defining “balanced”, “next generation” or “comprehensive” assessment systems followed. 6/27/2018 Definition & Context | Re-framing 7
Conceptual work defining “balanced”, “next generation” or “comprehensive” assessment systems followed. Which led to the National Research Council Report, Developing Assessments for the Next Generation Science Standards. 6/27/2018 Definition & Context | Re-framing 8
Systems & the NGSS A system of assessments will be needed to measure the NGSS performance expectations and provide students, teachers, administrators, policy makers, and the public with the information each needs about student learning. National Research Council, 2014 6/27/2018 Definition & Context | Re-framing 9
Systems & the NGSS A system of assessments will be needed to measure “It will not be feasible to assess all of the NGSS performance the performance expectations for a expectations and provide given grade level during a single students, teachers, assessment occasion. Students will need administrators, policy multiple—and varied—assessment makers, and the public with opportunities” the information each needs about student learning. National Research Council, 2014 6/27/2018 Definition & Context | Re-framing 10
So what exactly is a system of assessments? Here’s my working definition. 6/27/2018 11
Multiple assessments with potentially different designs , sponsored by different people, who are at different levels of control Coordinated by a common theory of learning Working together to meet a specific use or uses. 6/27/2018 12
Despite all of this work, much of the vision laid out by the NRC report and others has not been realized. 6/27/2018 13
A Response • Instead of working from the perspective that a system is tightly defied and purpose built, • Consider all of the assessments students take as a system – Then consider the ways which the levels, or layers, of that system can be made to work together, i.e., complement one another 6/27/2018 Definition & Context | Re-framing 14
Operationalizing in Terms of Layers Let’s consider one way to visualize the layers of an assessment system. 6/27/2018 Definition & Context | Re-framing 15
Operationalizing in Terms of Layers Purpose Level or Layer Timing 6/27/2018 Definition & Context | Re-framing 16
Operationalizing in Terms of Layers Purpose Level or Layer There is no one “system.” Instead, within each layer decision makers build, or at least implement, their own set of assessments. Timing 6/27/2018 Definition & Context | Re-framing 17
Operationalizing in Terms of Layers Purpose Level or Layer Thus each classroom’s, school’s or district’s layer may look different. Meaning within any given state, there are numerous “systems,” when taken as a whole. Timing 6/27/2018 Definition & Context | Re-framing 18
Operationalizing in Terms of Layers Purpose Level or Layer By juxtaposing the layers, hopefully we can start thinking about ways the layers can complement one another. Timing 6/27/2018 Definition & Context | Re-framing 19
This approach doesn’t fully meet the vison for a system of assessments, but it is a practical start. 6/27/2018 Definition & Context | Re-framing 20
By showing what is actually going on, we can then start to think about ways in which each layer can be made more complementary, along the lines of comprehensiveness, coherence and continuity. 6/27/2018 Definition & Context | Re-framing 21
A Starting Point • Define the state layer as concretely as possible, then examine where and how the district layer can be made more complementary (and vice-versa). • Consider key chunks of each layer: – Use – Assessment Design – Assessment Framework (i.e., Domain) – Measurement 6/27/2018 Definition & Context | Re-framing 22
A Starting Point • Define the district layer as concretely as possible, then examine where and how the district layer can be made more complementary (and vice-versa). • Consider key chunks of each layer: – Use – Assessment Framework (i.e., Domain) – Assessment Design – Measurement 6/27/2018 Definition & Context | Re-framing 23
2 The District Layer • Assessment Framework • Assessment Design 6/27/2018 24
Assessment Framework What does the assessment measure and how does it do so? Possible focuses of district assessment: • The same domain as the state-level… – exactly – at a lesser degree of complexity • A different domain at the state level… – progressions or precursors to the state-level domain – parts of the standards not covered by the domain 6/27/2018 25
Complementarity District State The Same Domain Model 6/27/2018 26
Complementarity District State The Different Domain Model 6/27/2018 27
Assessment Design What does the assessment(s) look like? Often, sensitivity to curriculum and instruction is paramount for district-level assessments. One way to make assessments better match local curriculum and instruction is through modular design. 6/27/2018 28
Fixed Designs Block Designs Modular Designs Multiple A single Multiple assessment that assessments, each assessments, each measures the measuring a chunk measuring a very of the domain entire domain small chunk of the domain Fixed Modular 6/27/2018
Supplemental Slides 6/28/2018 3D Science Assessment Reporting 30
The State Layer • Uses • Assessment Framework • Measurement 6/27/2018 31
Uses How are the assessment results used? • For better or worse, state-level assessments are often “one use, and one use only” – As part of a system of school identification • Unlike Math and Reading/English Language Arts, Science is not a mandatory part of state accountability plans 1 under ESSA. – And, possibly, to signal the importance of a given set of standards • Any other uses are likely to not be directly supported via the state-layer alone. – The massive use of assessments by districts is testament to this. 1 Although an initial look at state plans did suggest many states were including science in their accountability systems . 6/27/2018 32
Uses How are the assessment results used? • For any intended use, a key question is on sufficiency: – Is the use of the state-level assessment results likely to lead to the intended outcome (hopefully when couched in terms of a theory of action)? 6/27/2018 33
Defining the use should help us determine what we want to measure, and vice versa. 6/27/2018 34
Assessment Framework What does the assessment measure and how? • At the state-level, the domain is almost always premised on an approximation of the domain sampling approach • Even under this approach, defining the domain in any subject requires a judgmental process to translate standards into assessment specifications – As a reflection of the nature of the NGSS, the judgements for the NGSS seem particularly complex 6/27/2018 35
Assessment Framework In terms of the NGSS, much of this translation NGSS Specific involves the Performance Expectations (PEs), including whether: • The state considers the PEs as the standards or as examples of valuable intersections – If the latter, will new PEs be defined? • All PEs vs. Full Sampling (aka Matrix, within or across years) vs. a Limited Sample are assessed 6/27/2018 36
Assessment Framework Also key is in which the PEs are mapped to NGSS Specific items, and item clusters, including – Whether and how PEs are bundled to create item clusters – Phenomenon also play a substantial role in cluster design 6/27/2018 37
These types of questions help better define the domain. A detailed domain definition comes from a back and forth between design choices and domain definition. 6/27/2018 38
These types of questions help better define the domain. A detailed domain definition comes from a back Design Design Domain Domain and forth between design choices and domain Definition Definition Decisions Decisions definition. 6/27/2018 39
Recommend
More recommend
