Multid Multidimens imensional ional Science Asse Science - - PowerPoint PPT Presentation

Adva Advancing ncing Multid Multidimens imensional ional Science Asse Science Assessment ssment Des Design: A V ign: A View thr iew through

• ugh

Two Lenses

• Lenses

I-SMAR SMART T an and d SCIL SCILLS LSS Friday riday, , Jun une 29 e 29, , 20 2018 18 11 11:00 :00 AM AM – 12 12:00 :00 PM PM Aqua Aqua Salon Salon D (Le D (Level 3) el 3)

Strengthening Cla laims-based In Interpretations and Uses of f Lo Local and La Large-scale Science Assessment Scores (S (SCILLSS)

Using Principled-Design to Support Coherence in State and Local Assessment Systems Presentation for the National Conference on Student Assessment June 29, 2018

2

Overview

➢The SCILLSS Project: Purpose, Players, and Products ➢Coherence-based Principled-design

– Large-scale science assessments – Classroom-based, instructionally-embedded assessments (not a focus of this presentation) – Theory of Action – Self-evaluation Protocols: Reflecting on and evaluating assessment systems – Digital Workbook on Educational Assessment Design and Evaluation

➢SCILLSS Implementation in Nebraska ➢Project Impact

3

About SCIL ILLSS

• One of two projects funded by the US Department
• f Education’s Enhanced Assessment Instruments

Grant Program (EAG), announced in December, 2016

• Collaborative partnership including three states,

four organizations, and 10 expert panel members

• Nebraska is the grantee and lead state; Montana

and Wyoming are partner states

• Four year timeline (April 2017 – December 2020)

4

SCIL ILLSS Project Goals ls

• Create a science assessment design model that

establishes alignment with three-dimensional standards by eliciting common construct definitions that drive curriculum, instruction, and assessment

• Strengthen a shared knowledge base among

instruction and assessment stakeholders for using principled-design approaches to create and evaluate science assessments that generate meaningful and useful scores

• Establish a means for state and local educators to

connect statewide assessment results with local assessments and instruction in a coherent, standards-based system

5

SCILLSS Partner States, , Organizations, , and Staff

6

Co-Principal Investigators: Ellen Forte and Chad Buckendahl Project Director: Liz Summers Deputy Project Director: Erin Buchanan Psychometric Leads: Andrew Wiley and Susan Davis-Becker Principled-Design Leads: Howard Everson and Daisy Rutstein

Project Deli liverables

7

Year 1 Year 2 Year 3 Year 4

1 - Project Foundations

• SCILLSS website
• Theory of Action for the project

and for each state

• Local and state needs assessment

tools

• Assessment literacy module 1
• Three prioritized science claims

2 - Large-scale assessment resources

• Three sets of claim-specific

resources:

• PLDs
• measurement targets, task

models, and design patterns

• sample items
• Assessment literacy modules 2-5

3 - Classroom-based assessment resources

• Six task models
• Six tasks
• Six sets of student artifacts

4 - Reporting and Dissemination

• Database of student artifacts

corresponding to the performance levels

• Post-project survey
• Post-project action

plans for each state

• Final project report

Overview

➢The SCILLSS Project: purpose, players, and products ➢Coherence-based Principled-design

– Benefits and Phases of a Principled-design approach – Theory of Action – Self-evaluation Protocols: Reflecting on and evaluating assessment systems – Digital Workbook on Educational Assessment Design and Evaluation

➢SCILLSS Implementation in Nebraska ➢Project Impact

8

Benefits of f a Pri rincipled-Design Approach

9

• Principled articulation and alignment of design

components

• Articulation of a clear assessment argument
• Reuse of extensive libraries of design templates
• For accountability

– Clear warrants for claims about what students know and can do – Build accessibility into design of tasks (not retrofitted into tasks) – Cost v. scale

Three It Iterative Evidence-Centered Design Phases

10

Phase 1: Domain Analysis What do we intend to measure?

Representations of the three dimensions in the NGSS

Phase 2: Domain Modeling What does that look like in an assessment context?

Articulation of how the construct should manifest in the assessment

Phase 3: Task Development and Implementation Build and Implement the Assessment

Task models → items Items → tests

Adapted from Huff, Steinberg, & Matts, 2010

Increases in Specificity

Theory ry of f Actio ion Purpose

The purpose of a Theory of Action is to:

• Articulate the claims and assumptions that must hold true

to support the interpretation(s) and use(s) of assessment scores;

• Articulate how assessment claims connect with, and are

supported by, test scores and other sources of evidence;

• Strengthen both the validity and coherence of an

assessment system; and

• Provide stakeholders with ample documentation of design

and development logic and decisions, which can be used for future learning, evaluations, and development projects.

11

Theory ry of f Actio ion Components

12

Statewide Assessment System Design

• What are the

assessment system claims?

• How is the

assessment system designed?

• How must the

assessment system function to provide interpretable and usable scores? System Setting and Use

• How are

stakeholders meant to use assessment information?

• What are some
• f the

conditions that must be in place for the assessment system to function as intended? Teacher Actions

• What activities

are expected of teachers?

• How do

teachers interact with students in the classroom?

• How do

teachers use student work to track progress? Student Actions

• What activities

are expected of students?

• How do

students interact with teachers and

• ther students?
• How do

students track their progress? Student Outcomes

• What are the

intended student goals,

• utcomes, or

consequences

• f the

assessment system (e.g., for students, teachers, instruction)?

13

Self-Evaluation Protocol Purpose

14

The local and state self-evaluation tools are frameworks to support state and local educators in reflecting upon and evaluating the assessments they use. Local or District State

• Designed to focus on assessments

that districts or schools require

• Usually used for lower-stakes

decisions – Curriculum reviews – Malleable instructional decisions – Monitoring student progress proximally

• Focused on large-scale

assessments required statewide

• Some assessments have high

stakes – Accountability for students – Accountability for educators – Accountability for schools, districts, programs

Self-Evaluation Protocol Steps

15

Synthesize

Synthesize results from the initial steps to determine an appropriate path forward

Evaluate

Evaluate the data and evidence available for each assessment to support the program goals and objectives and to address four fundamental validity questions

Identify

Identify all current and planned assessments

Articulate

Articulate the primary goals and objectives of your assessment program

Vali lidity Questions

16

1. Construct Coherence

To what extent has the assessment been designed and developed to yield scores that can be interpreted in relation to the target domain?

2. Comparability

To what extent does the assessment yield scores that are comparable across students, sites, time, forms?

3. Accessibility and Fairness

To what extent are students able to demonstrate what they know and can do in relation to the target knowledge and skills on the test in a manner that can be recognized and accurately scored?

4. Consequences

To what extent does the test yield information that can be and is used appropriately to achieve specific goals?

Dig igit ital Work rkbook Purp rpose

The digital workbook includes five assessment literacy modules designed to:

• Inform state and local educators and other stakeholders on

the purposes of assessments;

• Ensure a common understanding of the purposes and uses of

assessment scores, and how those purposes and uses guide decisions about test design and evaluation;

• Complement the needs assessment by providing background

information and resources for educators to grow their knowledge about foundational assessment topics; and

• Address construct coherence, comparability, accessibility and

fairness, and consequences.

17

Dig igit ital Work rkbook Module Topics

18

1 Validity, validity evidence, and the assessment life cycle (design and development, administration, scoring, analysis, reporting, score use) 2 Construct Coherence: To what extent do the test scores reflect the knowledge and skills we’re intending to measure, for example, those defined in the academic content standards? 3 Comparability: To what extent are the test scores reliable and consistent in meaning across all students, classes, and schools? 4 Accessibility and Fairness: To what extent does the test allow all students to demonstrate what they know and can do? 5 Consequences: To what extent are the test scores used appropriately to achieve specific goals?

Overview

➢The SCILLSS Project: Purpose, Players, and Products ➢Coherence-based Principled-design

– Large-scale science assessments – Theory of Action – Self-evaluation Protocols: Reflecting on and evaluating assessment systems – Digital Workbook on Educational Assessment Design and Evaluation

➢SCILLSS Implementation in Nebraska ➢Project Impact

19

SCIL ILLSS Im Implementation in in Nebraska

• Theory of Action

–Lead to visioning and prioritizing for our Nebraska Science Assessment System

• Local Needs Assessment

– Lead to realization that districts needed more

assessment knowledge and resources.

• Assessment Literacy Modules (ALMs) need to be more

comprehensive for each validity question and include examples and links to resources.

• 5 Assessment Literacy Modules available first 2 years.

20

SCIL ILLSS Im Implementation in in Nebraska

• Evidence-Centered Assessment Design

–Provides language, concepts, and knowledge representations for designing and delivering a state assessment system.

• Domain Analysis

–Provided an opportunity to reflect on our state vision and assessment system model to more clearly articulate what we want and don’t want. –The process of engaging in the work to develop the products has been most meaningful and valuable.

21

SCIL ILLSS Im Implementation in in Nebraska

• Increased discussions with other science

stakeholders.

–Collaboration with others wrestling with the same issues.

22

Overview

➢The SCILLSS Project: Purpose, Players, and Products ➢Coherence-based Principled-design

– Large-scale science assessments – Theory of Action – Self-evaluation Protocols: Reflecting on and evaluating assessment systems – Digital Workbook on Educational Assessment Design and Evaluation

➢SCILLSS Implementation in Nebraska ➢Project Impact

23

What?

• Strengthen a shared

knowledge base among instruction and assessment stakeholders for using principled-design approaches to create and evaluate science assessments that generate meaningful and useful scores

• Establish a means for state

and local educators to connect statewide assessment results with local assessments and instruction in a coherent, standards-based system

How?

• Model a replicable, scalable

process for designing state and classroom-based assessment tasks aligned to three-dimensional science standards and for eliciting common construct definitions that drive curriculum, instruction, and assessment

– Develop detailed process documentation of the design process, including the

• pportunities and challenges

represented by our work, and recommendations for addressing them – Develop an exemplar set of Evidence Centered Design (ECD) resources

24

SCIL ILLSS Project Im Impact

References

25

Huff, K., Steinberg, L., & Matts, T. (2010). The promises and challenges

• f implementing evidence-centered design in large-scale
• assessment. Applied Measurement in Education, 23(4), 310-324.

Inno Innova vation tions s in Scienc in Science e Ma Map, , As Asse sess ssme ment nt, , an and d Rep epor

• rt

t Tec echn hnolo

• logie

gies s (I (I-SMAR SMART) T)

Mar arsi sie e Tor

• rcho

hon Ma Maryl yland and S Sta tate D te Depa epartment tment of

• f E

Educ ducati tion

• n

Ga Gail il Tieman Tiemann Un Univ iver ersi sity ty of

• f Kan

Kansas sas - ATLAS TLAS NC NCSA SA Jun une e 29 29, , 20 2018 18

I-SMAR SMART T Pur Purpose pose

Improve achievement of multidimensional science standards for students with and without disabilities through accessible, learning map model-based assessments and reporting tools.

I-SMAR SMART T Goals Goals

Goal 1 - Develop & evaluate science learning map Goal 2 – Design, develop & evaluate assessments Goal 3 – Design, develop & evaluate a dashboard

Goal 4 - Dissemination

Sta State P te Par artner tners

• Maryland – Lead State
• Missouri
• New Jersey
• New York
• Oklahoma

I-SMAR SMART T Team eam

• ATLAS - KU
• CAST
• Bruce Yelton, evaluator

Goal Goal 1 Science Science Lear Learning M ning Map Mod p Model el De Development elopment and R and Review view

Map Model Segment from Neighborhood Essential Element EE.MS.LS2-2: Use models of food chains/webs to identify producers and consumers in aquatic and terrestrial ecosystems. Disciplinary Core Idea: LS2.B Cycle of Matter and Energy Transfer in Ecosystems. Science and Engineering Practice (Blue Circles): Developing and using models

I-SMAR SMART T Review view Panel anel Pr Process

• cess
• Onsite panel review with

educators

• Panel review methodology, &

post-panel review process

• External and internal

evaluation of process

Goa Goal l 2 Science Science Testlet estlet De Development elopment

Pr Prototype T

• totype Testlet

estlet De Development elopment

• Narrative, inquiry-based approach
• Incorporating principles of UDL
• Items developed to assess nodes in the map
• Items grouped into testlets built on "mini-progressions" of connected map

nodes used as assessment targets in an essential element neighborhood

• Each mini-progression used as an assessment target includes DCI and

SEP nodes

Narrative-Based Includes Engagement Activities

Upc Upcoming

• ming te

test stlet let de develop elopmen ment t ac activities tivities

• Item tryouts (cognitive labs)
• Item-writing event with educators from states
• External review
• Pilot

Goal 3 Goal 3 Repor eporting Dashboar ting Dashboard d – Desi Design gn Focus Gr

• cus Groups
• ups

Repor eporting Dashboa ting Dashboards ds

Provide immediate access to student data in an accessible, attractive, and engaging way that facilitates comprehension, insight, and instructional design-making.

Dashboar Dashboard d Design Design Cadr Cadre

• Focus groups of educators from states
• Began with needs assessment
• Included iterative process of design,

feedback, updated designs

UDL-based - multiple ways to access information One view of many dashboard screens.

I-SMAR SMART T Sta State te Per erspe spectiv ctive

I-SMAR SMART T Sta State P te Per erspectiv spective

• Maryland science assessment involvement
• Participation of educators across state partners

Maryland’s Science Assessment De Development elopment

2013: Maryland adopts the Next Generation Science Standards (NGSS) 2015-2016: Students with significant cognitive disabilities take Alt-MSA Science 2016 – 2017: General Education students in grades 5 and 8 take the first year of Maryland Integrated Science Assessment (Alt-MISA). 2017-2018: MSDE adopts the Essential Elements 2017: Students with significant cognitive disabilities take Dynamic Learning Maps (DLM)

Acr Across

• ss Sta

State P te Par artner tners

• Educators have:
• participated in the I-SMART review panel
• participated in the dashboard design cadres
• Educators will:
• participate in cognitive labs (occurring in MD and MO)
• participate in pilot activities
• participate in item writing activities

Questions? Questions?

Contact Contact Us Us…

ismart@ku.edu ismart.works 785-864-7093

The contents of this presentation were developed under a grant from the Department of Education. However, those contents do not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal Government.