Evaluating New Approaches To Assessing Learning Richard J. - - PowerPoint PPT Presentation

Evaluating New Approaches To Assessing Learning

Richard J. Shavelson, Min Li, Maria Araceli Ruiz-Primo & Carlos Cuauhtémoc Ayala

Keynote Address* Joint Northumbria/EARLI Assessment Conference University of Northumbria at Newcastle, Longhirst Campus 28 August 2002

* *Available at http//www.

Available at http//www.stanford stanford. .edu edu/department/SUSE/SEAL /department/SUSE/SEAL

Overview

• Evaluating the Quality of Learning

Assessments: Conceptual Framework

• Applying the Framework
• Concluding Comments

Conceptualizing Assessment: The Assessment Triangle

• Cognition: A model of that

explains how students represent knowledge and develop competence

• Observation: Tasks or

situations that prompt student to say, do, or create something to demonstrate knowledge

• Interpretation: A process for

making sense of evidence

Observation Interpretation Cognition

[Source: Pellegrino, Chudowsky & Glaser, 1999]

Evaluating Assessments: The Assessment Square Corners

• Construct: A working definition
• f what is to be measured
• Assessment: Systematic

procedure for eliciting, capturing and scoring behavior

• Observation: Collecting and

summarizing behavior in response to a task

• Interpretation: Inference from

behavior on an assessment to the construct

Construct Interpretation Assessment Observation

[Sources: Ayala, Yin, Shavelson & Vanides, 2002; Ruiz-Primo, Shavelson, Li & Schultz, 2001]

The Corners

Evaluating Assessments: The Assessment Square Analyses

• Conceptual Analysis: Identify

domain of tasks and responses from construct definition

• Logical Analysis: Logical evidence

that task will evoke in a student a problem space and response consistent with construct

• Cognitive Analysis: Empirical

evidence on cognitive activities evoked by task/response

• Statistical and/or qualitative

analysis: Bring quantitative and/or qualitative data to bear on proposed assessment interpretation.

Construct Interpretation Assessment Observation

Logical Analysis Cognitive Analysis Statistical and/

• r Qualitative

Analysis Warranted Inference? Conceptual Analysis

The Analyses

Applying Framework To TIMSS’ Achievement Test: Construct and Assessment

• The Construct: Science Achievement
• The Assessment: TIMSS Population 2

Science Test Items

– Multiple-choice – Short-answer

• A combination of logical, cognitive and

statistical analyses

[Source: Li (2001), Li & Shavelson, 2001]

Applying The Framework: Conceptual Analysis

• Declarative—knowing

that

• Procedural—knowing

how

• Schematic—knowing

why

• Strategic—knowing

when, where, and how to apply knowledge

Schematic Knowledge

Draws Upon

Declarative Knowledge Strategic Knowledge

Involved In

Procedural Knowledge

Applying The Framework: Logical Analysis of TIMSS Items

• Task Demands
• Cognitive Demands—Assume

competent 14 year old

• Item Openness
• Complexity

Logical Analysis: Item Coding System

• Task Demands: What does the item ask student to do?

– Terms, symbols, vocabulary, definition – Procedures, steps, actions, algorithms – Models, relationships, explanation, principles

• Cognitive Demands: What prior knowledge and cognitive

processes examinee may use and reason with?

– Visualize – Calculate – Perform experiment – Recall information – Reason and interpret with models and principles – Plan and monitor behavior – Guess or eliminate wrong options

• Item Openness: How free in shaping item response?

– Hands-on v. paper-and-pencil – Selected v. constructed response – Constrained v. open response – One v. multiple solution paths – Follow steps in instruction

• Complexity: How familiar, relevant, reading difficult is item?

– Textbook vs. ill-structured task – Inclusion of irrelevant background information – Long, reading demanding descriptions and complicated vocabulary – Answers contradict everyday experience

Logical Analysis: Item Coding System (Cont’d.)

Logical Analysis: Declarative Knowledge Item

• Assume competent 14 year
• ld
• Task: Response expected to

be a term, vocabulary (e.g., saliva), factual statement

• Cognitive Activity: Likely to

be recall (question similar to form in which student learned content) with minimal reasoning to organize answer

• Openness: An open-ended

question

• P6. What digestive substance is

found in the mouth? What does it do?

Logical Analysis: Schematic Knowledge Item

• Task Demands: Asks for

explanation “why”; a model can be used to answer

• Cognitive Demands: Requires

reasoning with a model (unless memorized/recalled)

• Openness: The information forms

a complete question that allows examinees to finish the item without reading alternatives

• Complexity: Reasonable reading

load

• Q11. Which statement explains

why daylight and darkness

• ccur on Earth?
• A. The Earth rotates on its axis
• B. The Sun rotates on its axis
• C. The Earth’s axis is tilted.
• D. The Earth revolves around

the sun

Logical Analysis: Procedural Knowledge Item

• P1. The graph shows the progress

made by an ant moving along a straight line.

• A. 5cm
• B. 6cm
• C. 20cm
• D. 30cm
• Task Demands: Interpret

diagram or apply algorithm

• Cognitive Demands: Apply

the formula of Speed ÷ Distance or extend line

• Openness: Constrained—

can work backwards from alternative

• Complexity: Moderate

reading

If the ant keeps moving at the same speed, how far will it have traveled at the end of 30 seconds?

Applying The Framework: Cognitive Analysis

• Assumptions:

– Information processing model – Verbalization of working memory – Cognitive activity interpretation warranted

• Steps

– Collect concurrent verbalization – Segment protocols – Code protocols – Analyze data

Cognitive Analysis: Sequence Of Think-Aloud Study

Session 1 Session 2-Session 4 Session 5 Introduction

• f the study,

thinking aloud

• n the exercise

problems-shoe tying Solving a group of multiple- choice and free- response items with thinking- aloud &

• bservations

Inter- view about solving the items Solving the two performance assessment tasks with thinking- aloud &

• bservations

Interview about solving the PA tasks and overall reflections 5-10 10-15 2-5 15-30 5-10

[Source: Li, 2001]

Cognitive Analysis: Protocol Analysis

• Collect concurrent verbalizations—participants were

instructed to verbalize anything while responding to test items

• Segment each participant’s verbal protocol—Li used the

entire response to each item or task no matter how many statements or types of knowledge

• Code participants’ segments—Li developed a system that

captured evidence the four types of knowledge.

• Examine coding consistency (reliability)
• Bring coded protocols data to bear on how participants

employed different types of knowledge to represent and solve problems

Cognitive Analysis: Link Between Logical And Cognitive Analysis

Based on the knowledge- type construct of science achievement, we expected participants’ use of knowledge inferred from the protocols (cognitive analysis) to be congruent with the knowledge- types demanded by test items (logical analysis)

Pre-classified knowledge-type Type of knowledge used (n=9)* (n=10) (n=9) (n=2) Declarative 8 11 48 Procedural 7 9 Schematic 9 16 Strategic 2 12 2

*Number of responses: 48 = 9 x 6 participants

Chi-square = 208.12, p<.001

Declarative Procedural Schematic Strategic 54 41 10

Covariance Analysis: Links With Logical & Cognitive Analyses

Declarative Knowledge

bsmsa7 e1 bsmsa9 e2 bsmsa11 e3

Schematic Knowledge

bsmsq11 e20 bsssp5 e19 bsssp2 e18 bsmsb3 e17 bsmsa12 e15 bsmsa8 e14

Procedural Knowledge

bsmsr1 e28 bsmsp7 e27 bsmsp1 e26 bsmsb5 e25 bsmsq13 e21 bsssq18 e23 bsmsr2 e24 bsmsb4 e5 bsssp3 e6 bsssp6 e8 bsmsq14 e9 bsssq17 e10 bsssr4 e12 bsssr5 e13

.51

.84

.25 .33 .08 .54

bsmsp4 e7

• .11

.12

.40 .39 .51 .47 .41

bsmsq15 e22 bsesr3 e11

.27 .04

.08

.36 .21 .50 .19

.70 .86

.26

bsmsb1 e4 .23

• .06

.06 .14

bsmsb2 e16

• .17
• .10

.63 .42 .44 .37 .69 .34 .37 .42 .42 .11 .41

.00

• .24

.14

.02

.29

Based on the knowledge-type construct of science achievement, we expected the emerged factors from the item scores (statistical analysis) to be congruent with the knowledge-types in logical analysis

Applying The Framework To Concept Maps: Construct and Assessment

• The Construct:

– Knowledge structure (declarative)

• The Assessment

– Concept-Map – A graph with nodes (concept terms) connected by labeled lines explaining relationship

[Source: Ruiz-Primo, Shavelson, Li, & Schultz, 2001]

• pposite

• f more than
• ne type of

• n the left

• r more

• f the

• f cations and

B

Atoms & Molecules: Expert Map

The Study

• Purpose:

– Compared “construct-a-map” and “fill-in-a-map” concept-mapping techniques

• Fill-in-the-nodes with structure given
• Fill-in-the-line labels with structure given

– Motivation—fill in technique quicker, easier, and less expensive to score – Do both techniques provide equally valid measures of knowledge structure?

• Domain: Chemistry—atoms and molecules
• Participants:

– Main study: 152 high-school students – Cognitive analysis expert-novice study:

• 3 low- proficient students
• 3 high-proficient students
• 2 chemistry teachers

[Source: Ruiz-Primo, Shavelson, Li, & Schultz, 2001]

Applying The Framework: Conceptual Analysis

Concept-map Assessment of Knowledge Structure:

• Assumes long-term memory operates as a network

with linked nodes

• Builds on empirical findings that experts’

knowledge, compared to novices’, is more:

– Highly structured – Scientifically justifiable

Framework For Examining “Cognitive Validity” Portion Of Assessment Square

1. Cognitive activities evoked by assessment 2. Relation between cognitive activities and performance scores 3. Impact of variation in technique on cognitive activities 4. Correlation between scores on similar and different assessments

Construct-A-Map Cognitive Processes Performance Fill-In-A-Map Cognitive Processes Performance (1) (2) (3) (4) (1) (2)

[Source: Ruiz-Primo, Shavelson, Li, & Schultz, 2001] Assessment Cognitive Analysis Observation

Logical Analysis: Concept-Map Directedness

High Low Student Provides:

• Concept terms
• Linking Lines
• Linking Explanations
• Map Structure

Assessor Provides:

• Concept terms
• Linking Lines
• Linking Explanations
• Map Structure

Construct-A- Map Fill-In-A-Map [Source: Ruiz-Primo & Shavelson, 1996]

Cognitive Analysis: Think-Aloud

• Micro-level Analysis (Average reliability = 0.86)

– Segment protocol into small units—phrases – Code phrases

• Explanations (N2O2 is a molecular compounds because they are both nonmetals)
• Monitoring (“I can’t remember exactly what this is “)
• Conceptual errors (“Molecules are atoms”)
• Inapplicable events (Reads instructions)
• Macro-level Analysis (Average: agreement 95%/Kappa .92)

– Segment protocol by entire map – Code planning from beginning of protocol – Code strategies for entire protocol attending to sequence

Comparison Of Micro-Level Cognitive Activities

Conceptual No-Code Technique n Explanation Monitoring Errors Applicable Construct-a-map 8 39.08 28.22 9.78 22.91 Fill-in-the- nodes 8 4.64 40.93 0.16 54.27 Fill-in-the-lines 8 2.84 35.22 0.14 61.80

[Source: Ruiz-Primo, Shavelson, Li, & Schultz, 2001]

Construct-A-Map Cognitive Processes Performance Fill-In-A-Map Cognitive Processes Performance (1) (1) (2) (3) (3) (4) (1) (1) (2)

Assessment Cognitive Analysis Observation

Comparison of Macro-Level Cognition: Strategy Sequence

3.1 4.2 2.1 2.1 1.1 2.4

1 2

2.1

3 to 20

2.3

24

2.2

25 26 27 28 29 30 32

4.3

33

4.1

31

3.3 4.2 2.1 1.2 2.1

1 2

2.1

3 4

2.4

5 to 7

2.4

8 22 to 23 10

2.2

27 to 28 11 12

3.3

13 15 16 17

4.2

18

2.1

14

2.4 2.1 2.2 3.3 2.2 2.2 2.4 2.2

19

2.4

20 21

2.1 2.1

24 26 25 9

2.1 3.4 3.4 3.3 3.1 3.3

2 9 32 30 31 21 to 23

3.1 4.2 2.4 2.1

High-Proficient Student Low-Proficient Student

Codes: Starting Continuing Monitoring Ending

Shaded figures represent inaccuracy of content

• Select most

general concept (1.1)

• Select concept

related to general concept (2.1)

• Monitor

performance by reading propositions (3.1)

• And so on…

[Source: Ruiz-Primo, Shavelson, Li, & Schultz, 2001]

Link Between Cognitive Activities and Performance Scores

Construct-A-Map Cognitive Processes Performance Fill-In-A-Map Cognitive Processes Performance (1) (2) (2) (3) (4) (1) (2) (2)

Assessment Cognitive Analysis Observation

Correlations Between Verbal Proportion Scores and Performance Scores by Category (n=7) * Mapping Category Technique Explanation Monitoring Conceptual Errors .33

• .29
• .27

Fill-in-nodes .12

• .31
• a

.01

• .83b
• a

* Outlier deleted on each correlation No variability in one of the variables. b Correlation is significant at the .05 level.

[Source: Ruiz-Primo, Shavelson, Li, & Schultz, 2001]

a

Construct-a-map Fill-in-lines

Comparison Of Scores On Mapping Techniques

Construct-A-Map Cognitive Processes Performance Fill-In-A-Map Cognitive Processes Performance (1) (2) (3) (4) (4) (1) (2)

Assessment Cognitive Analysis Observation

Mean Scores and Standard Deviations Across the Mapping Techniques Students Teachers Mapping Max Mean S.D. Mean S.D. Technique n = 6 n = 2 Construct

• a-map

1 0.63 0.31 0.74 0.17 Fill-in-the-nodes 12 10.67 1.97 12.00 0.00 Fill-in-the-lines 12 8.83 3.13 10.00 0.00

[Source: Ruiz-Primo, Shavelson, Li, & Schultz, 2001]

Comparison Of Scores On Mapping Techniques

Construct-A-Map Cognitive Processes Performance Fill-In-A-Map Cognitive Processes Performance (1) (2) (3) (4) (4) (1) (2)

Assessment Cognitive Analysis Observation

Mean Scores and Standard Deviations Across the Mapping Techniques Mapping CM LIN Technique

(n = 5)

Construct

• a-map
• Fill-in-the-nodes

.21 Fill-in-the-lines .91 .42

[Source: Ruiz-Primo, Shavelson, Li, & Schultz, 2001]

NOD

Concluding Comments

Construct Interpretation Assessment Observation

Logical Analysis Cognitive Analysis Statistical or Qualitative Analysis Warranted Inference? Conceptual Analysis

The assessment square:

• Provides a framework

for examining new learning assessments

• Integrates conceptual,

logical cognitive and statistical analyses of assessment- interpretation claims