Rethinking our Alignment Obsession: Can a Generalizability - - PowerPoint PPT Presentation

Rethinking our Alignment Obsession: Can a Generalizability Perspective Help?

Scott Marion, Center for Assessment

CCSSO National Conference on Student Assessment Aligned to What: Complex Content Standards as Targets for Assessment Design and Alignment Evaluation

June 29, 2018

Early Days of Standards-Based Reform

Assessments

Performance Standards

Content Standards

2

• Marion. Generalizability approach to alignment_June 29, 2018

Let’s not forget “School Delivery” or “Opportunity to Learn” standards

Alignment was the promissory note:

“We’ll assess only the stuff you’re expected to teach.”

Alignment Made Sense in the Early Days

– Make content and performance standards explicit – Base assessments on those standards – Fair to educators and would lead to improved learning

3

• Marion. Generalizability approach to alignment_June 29, 2018
• We were moving out of an era of norm-

referenced tests

• Theory of action behind standards-based

systems:

From Proxy to Criterion

• Alignment started off a proxy for content-related

validity evidence

• Once certain methodologies (e.g., Webb) became

popularized and used in USED standards and assessment reviews, alignment became reified as the criterion

• While embracing the fairness concerns that alignment

was intended to address, we now are experiencing some important unintended negative consequences?

– Negative or positive might depend on your view of curriculum and assessment

4

• Marion. Generalizability approach to alignment_June 29, 2018

Complex Content Standards

Increasing the complexity of the domain representation found in the newer standards (e.g., CCSS, NGSS) leads to worries about fairly and appropriately representing the curriculum (standards) targets on the assessment

5

• Marion. Generalizability approach to alignment_June 29, 2018

Core Ideas Practices Crosscutting Concepts

Sparse Content/Curriculum Coverage

6

• Marion. Generalizability approach to alignment_June 29, 2018

Science and Engineering Practices Middle School

AQDP DUM PCOI AID UMCT CEDS EAE OECI PE x DCI Totals

Life Sciences

LS1, From Molecules to Organisms: Structures and Processes LS1.A, Structure and Function

MS-LS1-2 MS-LS1-1 MS-LS1-3

3

LS1.B, Growth and Development of Organisms

MS-LS1-5 MS-LS1-4

2 8

LS1.C, Organization for Matter and Energy Flow in Organisms

MS-LS1-7 MS-LS1-6

2

LS1.D, Social Interactions and Group Behavior

MS-LS1-8

1

LS2, Ecosystems: Interactions, Energy, and Dynamics LS2.A, Interdependent Relationships in Ecosystems

MS-LS2-1 MS-LS2-2

2

LS2.B, Cycles of Matter and Energy Transfer in Ecosystems

MS-LS2-3

1 5

LS2.C, Ecosystem Dynamics, Functioning, and Resilience

MS-LS2-4 MS-LS2-5

2

LS2.D, Social Interactions and Group Behavior LS3, Heredity: Inheritance and Variation of Traits LS3.A, Inheritance of Traits

MS-LS3-1 MS-LS3-2

2 2(2)

LS3.B, Variation of Traits

MS-LS3-1 MS-LS3-2

2

LS4, Biological Evolution: Unity and Diversity LS4.A, Evidence of Common Ancestry

MS-LS4-1 MS-LS4-3 MS-LS4-2

3

LS4.B, Natural Selection

MS-LS4-4 MS-LS4-5

2 6

LS4.C, Adaptation

MS-LS4-6

1

LS4.D, Biodiversity and Humans

PE x SEP Totals 5 1 3 1 5 4 2 21

From Brian Gong (2018). Paper under development. www.nciea.org. My colleague Brian Gong has been concerned with various aspects of construct underrepresentation in the NGSS. This shows the sparse coverage of the Performance Expectations (PE) in middle school life science. Remember PEs are not all possible intersections of the three dimensions.

Sparse Assessment Coverage

• Even if the state has adopted

the PEs as the “standards,” there is still way too much to cover well on a 2-3 hour exam

• If you are assessing only at

grade spans, then the problem is even worse

7

• Marion. Generalizability approach to alignment_June 29, 2018

Finally, I know many are enamored with item clusters, but I question the fidelity to the Frameworks and NGSS

What Do We Care About?

• Kids doing real “minds on” science
• Posing questions about phenomena,

designing studies, & reasoning with evidence in light of big ideas of science

8

• Marion. Generalizability approach to alignment_June 29, 2018

From Alignment to Generalizability

• But what about the crushing weight of

alignment requirements?

9

• Marion. Generalizability approach to alignment_June 29, 2018
• Would a

generalizability framework be more useful?

Generalizability and Transfer

• Generalizability is the measurement analogue of

learning transfer

• This requires us to be clear about the target of

generalization and the factors that would influence

• r threaten such generalizations

– See TJ’s “Jill and Jack” – Yes, this is hard!

• We need to focus on providing credible evidence

that the student really knows what is being claimed the student knows based on the assessment scores.

10

• Marion. Generalizability approach to alignment_June 29, 2018

Generalizability and Validity

• Both alignment and generalizability are key components
• f a validity argument, but generalizability applies more

directly especially when using Kane’s argument-based approach

• Marion. Generalizability approach to alignment_June 29, 2018

11

Focus of argument

*Thanks to Brian Gong for the diagram

So What Could This Look Like?

• Good—a set of rich performance tasks to be administered
• ver a defined period of time or a set of performance

assessments supplemented by more easily administered and scored items

• Better—as part of a full system of assessments
• Best—as part of a full system of assessments with tasks

embedded as part of defined curricular units

12

• Marion. Generalizability approach to alignment_June 29, 2018

Typical Approach to Assessment Systems

13

• Marion. Generalizability approach to alignment_June 29, 2018

Large Scale

Assessments

District required common assessments Classroom and/or school based assessment

This is assuming we have a system!

Controlling the conversation

What If We Flipped the System?

14

• Marion. Generalizability approach to alignment_June 29, 2018

Evidence of student learning from classroom assessments and other artifacts

State and District Required Embedded Assessments

Large scale performance assessments

Focus of generalizability analyses

Yeah, yeah…

• I know all too well about the challenges with reliability,

field testing, equating, comparability, etc., but I contend that in chasing alignment, we’re running the risk of sacrificing rich science learning…

• Misplaced concern about student-level reliability

– Few, if any, statewide science assessments are used for student-level decisions—rather, they are used at the school- level if at all – Hill & DePascale (2003) convincingly demonstrated that we can achieve respectable school-level reliabilities (e.g., .90+) even with student-level reliabilities around .65 (assuming approx. 50 students in the school)

15

• Marion. Generalizability approach to alignment_June 29, 2018

Focusing on the Real Thing!

The business of schools is to invent tasks, activities, and assignments that the students find engaging and that bring them into profound interactions with content and processes they will need to master to be judged well educated.

16 Marion & Shepard, iNACOL Leadership Webinar June 15, 2017

Schlechty, P.C. (2001) Shaking up the schoolhouse. San Francisco: Jossey-Bass.

For more information: Center for Assessment www.nciea.org smarion@nciea.org

• Marion. Generalizability approach to alignment_June 29, 2018

17