ESSA Innovative Assessment Designs: Examples from Science Brian - - PowerPoint PPT Presentation

ESSA Innovative Assessment Designs: Examples from Science

Brian Gong, Center for Assessment April McCrae, Delaware Department of Educa;on Michelle Center, California Department of Educa;on Karen Kidwell, Kentucky Department of Educa;on

CCSSO Na;onal Conference on Student Assessment June 28, 2017 Aus;n, TX

Assessment Challenges and Solution Approaches for Next Generation Science Standards: Domain de<inition, claims, evidence speci<ications

Brian Gong

Center for Assessment

Presenta;on in the session on “ESSA Innova;ve Assessment Designs: Examples from Science” CCSSO Na;onal Conference on Student Assessment June 28, 2017 Aus;n, TX

Overview

• Descrip;on of NGSS and assessment context
• Three issues, some possible solu;on approaches
• Summary

NGSS assessment issues & approaches - 6/28/17 3

Next Generation Science Standards

• Designed to replace previous na;onal/widely adopted

science standards (Na;onal Science Educa;on Standards, 1996; Science for All Americans, 1989)

• Framework for K-12 Science Educa:on (NRC, 2012)
• NGSS Standards, including Performance Expecta;ons

(NGSS, 2013 [Achieve and 26 states])

• Other materials, e.g., Developing Assessments for the

Next Genera:on Science Standards (NRC, 2014); Assessment Framework/aligned test item cluster (WestEd for CCSSO Science Item Assessment Collabora;ve, 2015)

NGSS assessment issues & approaches - 6/28/17 4

Summary of NGSS structure and content

• Grades K-high school
• Three dimensions

– Science and Engineering Prac;ces (SEP)

• 1) Asking ques;ons (for science) and defining problems (for engineering); 2) Developing

and using models; 3) Planning and carrying out inves;ga;ons; 4) Analyzing and interpre;ng data; 5) Using mathema;cs and computa;onal thinking; 6) Construc;ng explana;ons (for science) and designing solu;ons (for engineering); 7) Engaging in argument from evidence; 8) Obtaining, evalua;ng, and communica;ng informa;on

– Disciplinary Content Ideas (DCI)

• 11 Core ideas (4 in life sciences; 4 in physical sciences; 3 in earth and space sciences)
• 39 sub-ideas
• Each sub-idea is elaborated in a list of what students should understand about that sub-

idea at the end of grades 2, 5, 8, and 12.

– Cross-Cugng Concepts (CCC)

• 1) Paherns; 2) Cause and effect: Mechanism and explana;on; 3) Scale, propor;on, and

quan;ty; 4) Systems and system models; 5) Energy and maher: Flows, cycles, and conserva;on; 6) Structure and func;on; 7) Stability and change

NGSS assessment issues & approaches - 6/28/17 5

NGSS structure and content - PEs

• Performance Expecta;ons developed as part of NGSS

to define assessment targets in terms of what students should know and be able to do

• Each PE combines selected DCI, SEP, and CCC
• PE developed within life science, physical science,

and earth & space science content areas; oken mul;ple PE for each discipline area by grade

NGSS assessment issues & approaches - 6/28/17 6

Sample Performance Expectation

NGSS assessment issues & approaches - 6/28/17 7

Source: hhps://www.nextgenscience.org/resources/how-read-next-genera;on-science-standards

Assessment context of NGSS

• Adopted by 17 states and DC thus far (2017). Several
• ther states using related content frameworks.
• Four states have launched opera;onal NGSS-aligned

tests (IL, KA, NV, DC). A few NGSS states are field- tes;ng new assessments in 2016-17 and/or 2017-18; most states are in assessment design stage.

• Federal Every Student Succeeds Act (ESSA, 2015)

requires states to assess science at least once in grade spans 3-6, 6-9, and 10-12. Assessment must be aligned to state’s content standards—no federal men;on of NGSS.

NGSS assessment issues & approaches - 6/28/17 8

ESSA <lexibility supports possible NGSS assessment designs

• ESSA gives states authority for several flexibili;es that are relevant to possible NGSS

assessment designs

– ESSA allows states to choose their content standards; USED may allow states to assess a subset of the “instruc;onal learning content standards” as PE are a subset of the possible combina;ons of the NGSS SEP/DCI/CCC – ESSA explicitly states that “porqolios, performance events…” may be used in state assessments; while not prohibited before, this may be read as some encouragement – ESSA requires states to have a valida;on argument; not clear how recep;ve USED/Peer Review might be to valida;on arguments around more limited (less generalizable) claims. These may be necessary to address the sparse sampling in the NGSS PE – ESSA allows “banking” of math scores of grade 8 students who take a state’s high school math test – ESSA allows states to use interim assessments to produce a summa;ve score, and to use a “locally selected, na;onally recognized high school assessment” in lieu of the state

• assessment. These may provide models for how to handle innova;ve assessments that have

similar innova;ve aspects (e.g., through-course; mul;ple, non-strictly comparable assessments) – ESSA allows a Demonstra;on Authority around competency-based assessments, which may provide guidance for how to handle similar innova;ve NGSS assessments, such as ones based

• n claims of criterion mastery and individualized administra;on condi;ons (e.g., ;ming:

“move on when ready”)

NGSS assessment issues & approaches - 6/28/17 9

Rest of presentation

• Present an evidence-centered design approach to

test development to define terms and show rela;onships of parts to each other and to valida;on argument

• Discuss three domain defini;on issues posed by the

NGSS, and some possible solu;on approaches in terms of domain defini;ons, claims, and evidence

• For selected possible solu;on approaches, discuss a

few measurement challenges and possible solu;ons

NGSS assessment issues & approaches - 6/28/17 10

Evidence-Centered Design summary

• Assessment is intended, by design, to gather evidence to

support making a claim about student performance in rela;on to something

– Define the “something”—the domain/”construct” – Define the claim(s) – Define how the claims will be reported – Define what evidence is necessary to support the claims – Define what informa;on will be gathered and how it will be processed to become evidence – Design how the informa;on-gathering is comprehensive, prac;cal, fair, etc.

• In prac;ce, usually itera;ve

NGSS assessment issues & approaches - 6/28/17 11

Issue 1: Very large, complex domain

• NGSS defines

– 8 Science and Engineering Prac;ces, 39+ Disciplinary Core Idea sub-parts, 7 Cross-Cugng Concepts = 2,184 possible combina;ons [616 if restricted to 11 DCI]

• Challenge: The possible domain defined by the

complete crossings of SEP x DCI x CCC is too large to assess conven;onally (and probably too large to learn in current condi;ons)

• Solu;on approach 1: Reduce domain for learning

– 1A: Eliminate elements or combina;ons (many states: use PEs) – 1B: Combine elements (e.g., “Inquiry” from several SEP)

NGSS assessment issues & approaches - 6/28/17 12

Large, complex domain – 2

• Solu;on approach 2: Reduce domain for assessment

– 2A: Eliminate elements or combina;ons (NGSS = PEs) – 2B: Combine elements, e.g., combine 8 SEP into 2 categories for repor;ng and test design

• Solu;on approach 3: Reduce assessment

– 3A: Assess a limited por;on of the domain (e.g., assess DCI, not SEP)

• Solu;on approach 4: Spread out assessment

– 4A: Rotate coverage over years/forms – 4B: Distribute coverage over forms, matrix sample over students – 4C: Assess over mul;ple years

NGSS assessment issues & approaches - 6/28/17 13

Issue 2: NGSS PEs weak for domain de<inition

• Performance Expecta;ons developed as part of NGSS

to define what should be assessed and specify what evidence (i.e., what students should be able to do)

• Challenge 1: The set of NGSS PEs are so sparsely

sampled it is difficult to have enough evidence to generalize about individual PEs or their componen;al DCI/SEP/CCC within or across grades

• Challenge 2: The set of NGSS PEs do not have an
• bvious or documented structure of how the

combina;on of them relate to a claim about “science learning” in a single grade or across grades

NGSS assessment issues & approaches - 6/28/17 14

PE coverage of SEP x DCI content area, grades 3-5

NGSS assessment issues & approaches - 6/28/17 15

Grade AQDP DUM PCOI AID UMCT CEDS EAE OECI Grade 3 PS X X LS X X X X ESS X X X Grade 4 PS X X X X LS X X ESS X X X X Grade 5 PS X X X X LS X X ESS X X X X 3-5 ETS X X X

NGSS Scientific &Engineering Practices (SEP) Addressed in the Performance Expectations

Scien;fic and Engineering Prac;ces 1. Asking ques;ons (for science) and defining problems (for engineering); 2. Developing and using models; 3. Planning and carrying out inves;ga;ons; 4. Analyzing and interpre;ng data; 5. Using mathema;cs and computa;onal thinking; 6. Construc;ng explana;ons (for science) and designing solu;ons (for engineering); 7. Engaging in argument from evidence; 8. Obtaining, evalua;ng, and communica;ng informa;on Disciplinary Core Idea Areas 1. PS = Physical Sciences 2. LS = Life Sciences 3. ESS = Earth and Space Sciences 4. ETS = Engineering, Technology, and Society

PE coverage of SEP x DCI, grade 5

• When the full

DCI subareas and SEP are listed, the sampling by PE is extremely sparse and a ra;onale for the sampling is not evident (to me)

NGSS assessment issues & approaches - 6/28/17 16

• f Energy and Energy

• f Planet Earth

Earth and Space Sciences Earth’s Place in the Universe

• The Universe and

Its Stars (EAE)

• Earth and the

Solar System (AID)

Domain and claims

• It is not clear how the PEs were intended to support a claim about a

domain at the grade level or grade span.

• States that want to make a claim will need to come up with one on

their own; we could not find any claims that require the combina;on of PEs as evidence in the NGSS documents.

• Some illustra;ve possible claims:
• “The grade 3 student can do W (SEP) with X DCI; the grade 4 student can do Y (SEP)

with Z DCI.”

• “The grade 3 student can do W (SEP) with X DCI; the grade 4 student can do W and

X (SEP) with Y and Z DCI, assuming no forgegng.”

• “If the grade 3 student can do W (SEP) with X DCI, we will assume the student can

do all 8 SEP with all 15 DCI because all SEPxDCI were taught equally well.”

• “If the grade 4 student can do X (SEP) with Z DCI, we will assume student can also

do W (SEP) with Y DCI because these PE are in a developmental learning progression that requires WxY in order to do XxZ.”

NGSS assessment issues & approaches - 6/28/17 17

Domain and claims - 2

• The PEs do not appear selected to support reliable claims

about the whole domain (no maher how reliably any one PE is assessed).

• If the DCI at the level of LS/ES/PS were subscores, the
• pportuni;es for evidence would range from 2-4 (out of 8

possible) for each row in ay single grade

• If the SEP were subscores, no more than half would have more

than one opportunity for evidence (PE) in any single grade

NGSS assessment issues & approaches - 6/28/17 18

Number of SEP (out of 8) with number of :mes evidence is called for in NGSS domains (out of 3) 1 2 3 Grade 3 1 5 2 Grade 4 1 4 3 Grade 5 1 3 2 2

PEs and domain de<inition – 3

• Solu;on approach 1: Limit claims to assessed PEs; do not

generalize beyond

– 1A: Treat more like discrete criterion-referenced assessments than broad construct assessments – 1B: Make ALDs and other suppor;ng documents appropriately specific

• Solu;on approach 2: Say PEs provide evidence for claims

broader than specifically assessed content/skills

– 2A: Treat PEs as culmina;ng assessments of learning progressions, i.e., provide evidence of mastery of preceding content/skills – 2B: Say PEs assess narrowly, but what was learned was likely broader because instruc;on was broader

• Solu;on approach 3: Modify the PEs (assessment

targets) to support more generalized claims

NGSS assessment issues & approaches - 6/28/17 19

Reports

• States will need to decide what to report about the NGSS

as the total and/or “subscores”

– Each of three dimensions separately – Priori;ze two dimensions – Priori;ze one dimension – Report all three dimensions as a whole

• States will need to decide how to “scale”

– Uni-dimensional – Mul;-dimensional – Mul;ple separate dimensions and scales – Ver;cally; grade-spans; content clusters; grade-level

NGSS assessment issues & approaches - 6/28/17 20

Issue 3: Evidence collection

• NGSS call for performance assessments; most

science educators highly suppor;ve of performance assessments

• Challenge 1: Difficult to gather enough evidence in

rich instruc;onal context

– Solu;on approach 1: Provide (proxy for) rich instruc;onal contexts

• 1A: Simula;ons and other technology-enhanced assessment tools

(e.g., NAEP)

• 1B: Use curriculum/classroom-embedded assessments

NGSS assessment issues & approaches - 6/28/17 21

Evidence collection - 2

• Challenge 2: Evidence may be gathered under less

standardized condi;ons

– Solu;on approach 1: Work to standardize condi;ons

• 1A: Provide generally applicable rubrics
• 1B: Intensive scorer training, monitoring, modera;on

– Solu;on approach 2: Provide assurances of comparability through external evidence

• 2A: Use comparability on common tasks to support claim of

comparability on non-common tasks

– Solu;on approach 3: Make less standardized condi;ons part of the construct and/or claim

• 3A: Aspects of standardized test administra;on do not maher, e.g.,

Competency-based and combining mul;ple interim assessments involve non-standardiza;on of ;ming of gathering evidence— criterion-referenced mastery construct says ;ming of performance does not maher (e.g., ESSA grade 8 advanced math score banking)

NGSS assessment issues & approaches - 6/28/17 22

Evidence collection - 3

• Challenge 3: Performance assessments may not

generalize across content, skill, context (e.g., Shavelson et al.,

1993, 19941)

– Solu;on approach 1: Increase instruc;on and prac;ce to support more generalized knowledge and skills – Solu;on approach 2: Incorporate more informa;on to provide more reliable score

• 2A: Assess more samples across range of replica;on condi;ons
• 2B: Incorporate informa;on from other performances (e.g., Quals +

Disserta;on)

– Solu;on approach 3: Change claims to reduce generalizability

• 3A: “Student’s best work” (like KY Wri;ng porqolios)
• 3B: “Individualized area of specializa;on” (like appren;ceship or

capstone project, like RI’s previous PBGR)

NGSS assessment issues & approaches - 6/28/17 23

1 - hhps://www.researchgate.net/profile/Richard_Shavelson/publica;on/233284543_Generalizability_of_Large-Scale_Performance_Assessments_in_Science_Promises_and_Problems/links/53d12ff80cf220632f391ecf.pdf

Summary

• NGSS present considerable challenges regarding domain defini;on and design of evidence to

support claims in a large, complex domain defined by SEP/DCI/CCC and sparsely sampled PEs combined with a call for extensive performance assessments

• There are possible solu;ons for each of the NGSS challenges discussed. There is no one

“right answer” to how to assess the NGSS. Gegng a strong answer will involve thoughqul, itera;ve work.

• ESSA provides states flexibility in several ways that support these types of possible solu;ons,

including specific approaches already being explored by states

– ESSA allows states to adopt/specify content standards and construct valida;on arguments regarding assessing claims based on the defined domain. States may interpret the NGSS to create state-specific domain defini;ons, claims (e.g., regarding generaliza;on), and evidence designs, such as ALDs and repor;ng categories – ESSA explicitly states that “porqolios, performance events…” may be used in state assessments (Several states pursuing performance assessment in various forms) – ESSA allows “banking” of math scores of grade 8 students who take a state’s high school math test. (This approach may be used in NGSS design) – ESSA allows states to use interim assessments to produce a summa;ve score, and to use a “locally selected, na;onally recognized high school assessment” in lieu of the state assessment. These may provide models for how to handle innova;ve NGSS assessments that are similar in some aspects (e.g., through-course; mul;ple, non-strictly comparable assessments). – ESSA allows a Demonstra;on Authority around competency-based assessments, which may provide guidance for how to handle similar innova;ve NGSS assessments, such as ones based on claims of criterion mastery and individualized administra;on condi;ons (e.g., ;ming: “move on when ready”).

• It will be exci;ng to see and work with states and USED to help bring high quality NGSS

assessments into opera;onal existence over the next few years!

NGSS assessment issues & approaches - 6/28/17 24

Questions? Comments?

Thank you!

NGSS assessment issues & approaches - 6/28/17 25

For more information: Center for Assessment www.nciea.org

Brian Gong bgong@nciea.org

26 NGSS assessment issues & approaches - 6/28/17