What Does Rigor Look Like? A New Lens for Examining Cognitive Rigor - - PowerPoint PPT Presentation

What Does Rigor Look Like? A New Lens for Examining Cognitive Rigor in Assessments & Curriculum

National Conference on Student Assessm ent

Sponsored by the Council of Chief State School Officers

Detroit, MI , June 2 0 -2 3 , 2 0 1 0

Karin K. Hess, Ed.D., Senior Associate

National Center for Assessm ent, NH khess@nciea.org

Presenters

 Karin Hess, NCI EA  Carol Crothers, NV State Dept. of

Education

 John W alkup, The Standards Com pany,

LLC

 Stuart Kahl, Measured Progress  Som e Additional Materials

 Paper: “W hat exactly do “few er, clearer, and

higher standards” really look like in the classroom ?”

 Handouts – a look at grade 5 CCSS standards  Last slide

Developing the Cognitive Rigor Matrix

Different states use different models to describe cognitive rigor. Each addresses something different.

 Bloom – What type of thinking

(verbs) is needed to complete a task?

 W ebb – How deeply do you have to

understand the content to successfully interact with it? How complex is the content?

Bloom’s Taxonomy [1956 ] & Bloom’s Cognitive Process Dimensions [2005]

Create -- Put elements together to form a coherent whole, reorganize elements into new patterns/ structures Evaluation -- Appraise, argue, assess, choose, compare, defend, estimate, explain, judge, predict, rate, core, select, support, value Evaluate -- Make judgments based

• n criteria, check, detect

inconsistencies/ fallacies, critique Synthesis -- Rearrange, assemble, collect, compose, create, design, develop, formulate, manage, write Analyze -- Break into constituent parts, determine how parts relate Analysis -- Analyze, appraise, explain calculate, categorize, compare, criticize, discriminate, examine Apply -- Carry out or use a procedure in a given situation; carry out or use / apply to an unfamiliar task Application -- Apply, choose, demonstrate, dramatize, employ, illustrate, interpret, practice, write Understand -- Construct meaning, clarify, paraphrase, represent, translate, illustrate, give examples, classify, categorize, summarize, generalize, predict… Com prehension -- Classify, describe, discuss, explain, express, identify, indicate, locate, recognize, report, review, select, translate Rem em ber Retrieve knowledge from long-term memory, recognize, recall, locate, identify Know ledge -- Define, duplicate, label, list, name, order, recognize, relate, recall

Webb’s Depth-of-Knowledge Levels

 DOK-1 – Recall & Reproduction - Recall of a fact,

term, principle, concept, or perform a routine procedure

 DOK-2 - Basic Application of Skills/ Concepts -

Use of information, conceptual knowledge, select appropriate procedures for a task, two or more steps with decision points along the way, routine problems,

• rganize/ display data, interpret/ use simple graphs

 DOK-3 - Strategic Thinking - Requires reasoning,

developing a plan or sequence of steps to approach problem; requires some decision making and justification; abstract, complex, or non-routine; often more than one possible answer

 DOK-4 - Extended Thinking - An investigation or

application to real world; requires time to research, problem solve, and process multiple conditions of the problem or task; non-routine manipulations, across disciplines/ content areas/ multiple sources

• Synthesize

inform ation across m ultiple sources or texts

• Synthesize

inform ation w ithin

• ne source or text
• Generate

conjectures based on

• bservations or prior

know ledge

• Brainstorm ideas

about a topic

Create

• Evaluate relevancy,

accuracy, & com pleteness of inform ation – Cite evidence and develop a logical argum ent for conjectures

Evaluate

– Analyze m ultiple sources

• Analyze

com plex/ abstract them es

• Analyze or interpret

author’s craft ( literary devices, view point, or potential bias) to critique a text – Com pare literary elem ents, term s, facts, events – analyze form at,

• rganization, & text

structures

• I dentify w hether

inform ation is contained in a graph, table, etc.

Analyze

• Devise an approach

am ong m any alternatives to research a novel problem

• Use concepts to

solve non-routine problem s – Use context to identify m eaning of w ord

• Obtain and interpret

inform ation using text features

• Use language

structure ( pre/ suffix)

• r w ord relationships

( synonym / antonym ) to determ ine m eaning

Apply

• Explain how

concepts or ideas specifically relate to

• ther content

dom ains or concepts

• Explain, generalize,
• r connect ideas

using supporting evidence ( quote, exam ple…)

• Specify, explain

relationships

• sum m arize

– identify m ain ideas

• Select appropriate

w ords to use w hen intended m eaning is clearly evident

Understand

• Recall, locate basic

facts, details, events

Rem em ber

Level 4 Extended Thinking Level 3

Strategic Thinking/ Reasoning

Level 2 Skills & Concepts Level 1 Recall & Reproduction

Depth + thinking

The Cognitive Rigor Matrix: Applies Webb’s DOK to Bloom’s Cognitive Process Dimensions

The Matrix: A Reading Example

(See CCSS Gr 5 Reading Handout)

9 I - Synthesize inform ation across m ultiple sources or texts

• Synthesize

inform ation w ithin

• ne source or text
• Generate

conjectures

• Brainstorm ideas

about a topic

Create

• Evaluate relevancy,

accuracy, & com pleteness of inform ation 8 I – Cite evidence and develop a logical argum ent for conjectures

Evaluate

9 I – Analyze m ultiple sources 9 L - Analyze com plex/ abstract them es 4 L, 6 I , 8 I - Analyze

• r interpret author’s

craft ( literary devices, view point, or potential bias) to critique a text 3 L – Com pare literary elem ents, term s, facts, events 5 L, 5 I – analyze form at, organization, & internal text structures

• I dentify w hether

inform ation is contained in graphics

Analyze

• Devise an approach

am ong m any alternatives to research a novel problem 4 l & 4 I – Use context to identify m eaning of w ords 7 L - Obtain and interpret inform ation using text features

• Use language

structure ( pre/ suffix)

• r w ord relationships

( synonym / antonym ) to determ ine m eaning

Apply

• Explain how

concepts or ideas specifically relate to

• ther content

dom ains or concepts 1 L, 1 I - Explain, generalize, or connect ideas using supporting evidence ( quote, exam ple…) 2 L – identify them es 6 L – connect point of view + interpretation 3 I – Specify & explain relationships 2 L, 2 I - sum m arize 2 I – identify m ain ideas & supporting details

• Select appropriate

w ords to use w hen intended m eaning is clearly evident

Understand

7 I – Recall, locate basic facts, details, events

Rem em ber

Level 4 Extended Thinking Level 3

Strategic Thinking/ Reasoning

Level 2 Skills & Concepts Level 1 Recall & Reproduction

Depth + thinking

The Matrix: A Math Example

(See CCSS Gr 5 Math Handout)

Create Evaluate

• Analyze m ultiple

sources of evidence

• Analyze and draw

conclusions from data, citing evidence

• Generalize a pattern

4 G – Classify figures based on characteristics 3 AO- Extend pattern 2 G - Retrieve inform ation from a table

• r graph to answ er a

question

Analyze

• Select or devise

approach am ong m any alternatives to solve a problem

• Use concepts to

solve non-routine problem s 5 MD; 4 & 7 F - Solve routine problem applying m ultiple concepts or decision points 4 & 6 NO; 4 MD - Calculate, m easure, apply a rule 5 NO; 4 & 5 MD; 1 , 2 , 4 , 6 & 7 F- Apply algorithm

• r form ula

1 MD - Conversions- custom ary and m etric m easures

Apply

• Develop

generalizations of the results obtained and the strategies used and apply them to new problem situations

• Make and justify

conjectures 5 F - Specify and explain relationships 7 NO; 5 F- Explain reasoning 6 & 7 NO; 1 G; 5 MD; 2 , 3 , 6 , & 7 F - Use m odels to represent

• r explain concepts

2 F - Make and explain estim ates 1 AO - Evaluate an expression 3 AO; 2 G; 2 D – Locate points on a grid or num ber line 1 & 2 AO- Represent relationships in w ords, sym bols 2 , 3 , & 7 NO - Read, w rite, com pare decim als

Understand

1 & 2 NO; 3 G; 3 MD - Recall, observe, recognize, facts, principles, properties

Rem em ber

Level 4

Extended Thinking

Level 3

Strategic Thinking / Reasoning

Level 2

Skills & Concepts

Level 1

Recall & Reproduction

Depth + thinking

Some Implications for…

 Curriculum & Instruction  School/ Classroom Assessment  Large-scale Assessment

Some Implications for Curriculum & Instruction

 Figure out where you are now and where you

want to be

 Analyze in what ways your curriculum and

instruction need to change - what do you value?

 Develop within-grade (Sept. to May) learning

progressions and use student work analysis (SWA) to validate them

 Use SWA or a “Lesson Study” model of PD to

build pedagogy and content knowledge for formative assessment use & progress monitoring

 Build supports (support programs) to students

with cognitive rigor in mind – e.g., include conceptual AND procedural understanding in mathematics; ask for students to explain how they solved a problem or to justify their thinking/ claims with evidence

A word about learning progressions…

 Not everything that is called a learning a

progression IS a learning progression – may not be the same as a curricular progression or a skills scope & sequence; probably not be linear!

 Learning progressions are

 Supported by cognitive/ empirical research  Unified by bigger ideas across grades  Validated with student work analysis (after good

instruction)

 Grain size matters – needs to match purpose &

be manageable

 A skill subset within a larger progression  W ithin grade-level - guides form ative

assessm ent use and progress m onitoring during the year

 Across grade levels – more for program

evaluation

Learning Progressions Can Link the Zones (ZPD) of ALL Students

[K. Hess, 2008]

Proficient Advanced “On track” for proficient

2%

Consistently Low Performing

1% Alt Assessment

Many (but not all) students are here.

Modify materials & response formats? Need for additional scaffolding?

What/ how can they extend?

Some Implications for School/Classroom Assessment

 Use a larger schema (e.g., learning progressions)

to develop formative assessments and monitor progress (e.g., pre-assess prerequisite skills with

• pen-ended items before teaching)

 Design assessments that balance complexity of

content with cognitive demand – can students apply higher order thinking/ reasoning to less complex & then more complex texts/ content?

 Design assessments that provide varied

scaffolding (that decreases over time)

 Assess at multiple DOK levels for the same

content and use results to target instruction – e.g., if they can’t sort by attributes because they can’t identify attributes, then target instruction at the lowest DOK level first: what are attributes?

Some Implications for Large-scale Assessment

 Assessing only at the highest DOK level

will miss opportunities to know what students do & don’t know --- how actionable is the performance data?

 Item types & test designs might have to

be re-invented:

 Performance assessments can offer varying

levels of DOK embedded within a larger task

 Ordered multiple choice items of increasing

complexity that show differing complexity (not “difficulty”) of the same content

Some Related Resources

Papers available at www.nciea.org



• Gong. (2007). Learning progressions: Sources & implications for assessment.

http: / / www.nciea.org/ publications/



Hess (Feb, 2010). Using learning progressions to monitor progress across grades: A Science Inquiry Learning Profile for PreK-4. Science & Children. NSTA.



Hess, Kurizaki, & Holt. (2009). Reflections on tools and strategies used in the Hawai` i progress maps project: http: / / www.nciea.org/ publications/ Hawaii% 20Lessons% 20Learned_KH09.pdf



Hess (2008). Developing and using learning progressions as a schema for measuring progress. http: / / www.nciea.org/ publications/ CCSSO2_KH08.pdf



Hess (2008). Science inquiry profile K-4: http: / / www.nciea.org/ publications/ ScienceProfile_KH08.pdf



Hess (2008). Analysis to action: http: / / www.nciea.org/ publications/ Analysis% 20to% 20Action_KH08.pdf



Hess, K. (2004). “Applying Webb’s Depth-of-Knowledge (DOK) Levels in reading, writing, math, science, social studies, science” [ online] :

http: / / www.nciea.org/ publications/ DOKreading_KH08.pdf http: / / www.nciea.org/ publications/ DOKsocialstudies_KH08.pdf http: / / www.nciea.org/ publications/ DOKwriting_KH08.pdf http: / / www.nciea.org/ publications/ DOKscience_KH08.pdf http: / / www.nciea.org/ publications/ DOKmath_KH08.pdf

For more information…

 About draft K-12 learning

progressions for the Common Core Standards + science

 About ongoing learning

progressions research and formative assessment use

Karin Hess, khess@nciea.org