Evaluating New Approaches To Assessing Learning Richard J. - PowerPoint PPT Presentation

Evaluating New Approaches To Assessing Learning Richard J. Shavelson, Min Li, Maria Araceli Ruiz-Primo & Carlos Cuauhtémoc Ayala Keynote Address * Joint Northumbria/EARLI Assessment Conference University of Northumbria at Newcastle, Longhirst Campus 28 August 2002 * Available at http//www. * Available at http//www.stanford stanford. .edu edu/department/SUSE/SEAL /department/SUSE/SEAL

Overview • Evaluating the Quality of Learning Assessments: Conceptual Framework • Applying the Framework • Concluding Comments

Conceptualizing Assessment: The Assessment Triangle • Cognition : A model of that explains how students Observation Interpretation represent knowledge and develop competence • Observation : Tasks or situations that prompt student to say, do, or create something to demonstrate knowledge Cognition • Interpretation : A process for making sense of evidence [Source: Pellegrino, Chudowsky & Glaser, 1999]

Evaluating Assessments: The Assessment Square Corners • Construct : A working definition The Corners of what is to be measured Construct Interpretation • Assessment : Systematic procedure for eliciting, capturing and scoring behavior • Observation : Collecting and summarizing behavior in response to a task • Interpretation : Inference from Assessment Observation behavior on an assessment to the construct [Sources: Ayala, Yin, Shavelson & Vanides, 2002; Ruiz-Primo, Shavelson, Li & Schultz, 2001]

Evaluating Assessments: The Assessment Square Analyses • Conceptual Analysis : Identify The Analyses domain of tasks and responses from Construct Interpretation construct definition Warranted • Logical Analysis : Logical evidence Inference? that task will evoke in a student a Conceptual problem space and response Analysis consistent with construct Statistical and/ or Qualitative • Cognitive Analysis : Empirical Logical Analysis evidence on cognitive activities Analysis evoked by task/response Cognitive • Statistical and/or qualitative Analysis analysis : Bring quantitative and/or Assessment Observation qualitative data to bear on proposed assessment interpretation .

Applying Framework To TIMSS’ Achievement Test: Construct and Assessment • The Construct: Science Achievement • The Assessment: TIMSS Population 2 Science Test Items – Multiple-choice – Short-answer • A combination of logical, cognitive and statistical analyses [Source: Li (2001), Li & Shavelson, 2001]

Applying The Framework: Conceptual Analysis • Declarative —knowing that Schematic • Procedural —knowing Knowledge how Strategic Draws Upon Knowledge • Schematic —knowing Involved In why Procedural Declarative • Strategic —knowing Knowledge Knowledge when, where, and how to apply knowledge

Applying The Framework: Logical Analysis of TIMSS Items • Task Demands • Cognitive Demands—Assume competent 14 year old • Item Openness • Complexity

Logical Analysis: Item Coding System • Task Demands : What does the item ask student to do? – Terms, symbols, vocabulary, definition – Procedures, steps, actions, algorithms – Models, relationships, explanation, principles • Cognitive Demands: What prior knowledge and cognitive processes examinee may use and reason with? – Visualize – Calculate – Perform experiment – Recall information – Reason and interpret with models and principles – Plan and monitor behavior – Guess or eliminate wrong options

Logical Analysis: Item Coding System (Cont’d.) • Item Openness : How free in shaping item response? – Hands-on v. paper-and-pencil – Selected v. constructed response – Constrained v. open response – One v. multiple solution paths – Follow steps in instruction • Complexity : How familiar, relevant, reading difficult is item? – Textbook vs. ill-structured task – Inclusion of irrelevant background information – Long, reading demanding descriptions and complicated vocabulary – Answers contradict everyday experience

Logical Analysis: Declarative Knowledge Item • Assume competent 14 year P6. What digestive substance is old found in the mouth? What does it do? • Task: Response expected to be a term, vocabulary (e.g., saliva), factual statement • Cognitive Activity: Likely to be recall (question similar to form in which student learned content) with minimal reasoning to organize answer • Openness: An open-ended question

Logical Analysis: Schematic Knowledge Item • Task Demands: Asks for Q 11. Which statement explains explanation “why”; a model can why daylight and darkness be used to answer occur on Earth? • Cognitive Demands: Requires reasoning with a model (unless A. The Earth rotates on its axis memorized/recalled) B. The Sun rotates on its axis • Openness: The information forms C. The Earth’s axis is tilted. a complete question that allows examinees to finish the item D. The Earth revolves around without reading alternatives the sun • Complexity: Reasonable reading load

Logical Analysis: Procedural Knowledge Item P1. The graph shows the progress made by an ant moving along a • Task Demands: Interpret straight line. diagram or apply algorithm • Cognitive Demands: Apply the formula of Speed ÷ Distance or extend line • Openness: Constrained— can work backwards from If the ant keeps moving at the alternative same speed, how far will it have traveled at the end of 30 seconds? • Complexity: Moderate A. 5cm reading B. 6cm C. 20cm D. 30cm

Applying The Framework: Cognitive Analysis • Assumptions: – Information processing model – Verbalization of working memory – Cognitive activity interpretation warranted • Steps – Collect concurrent verbalization – Segment protocols – Code protocols – Analyze data

Cognitive Analysis: Sequence Of Think-Aloud Study Session 2-Session 4 Session 1 Session 5 Introduction Solving a Inter- Solving the Interview of the study, group of view two about thinking aloud multiple- about performance solving the on the exercise choice and solving assessment PA tasks problems-shoe free- the items tasks with and overall tying response thinking- reflections items with aloud & thinking- observations aloud & observations 5-10 10-15 2-5 15-30 5-10 [Source: Li, 2001]

Cognitive Analysis: Protocol Analysis • Collect concurrent verbalizations—participants were instructed to verbalize anything while responding to test items • Segment each participant’s verbal protocol—Li used the entire response to each item or task no matter how many statements or types of knowledge • Code participants’ segments—Li developed a system that captured evidence the four types of knowledge. • Examine coding consistency (reliability) • Bring coded protocols data to bear on how participants employed different types of knowledge to represent and solve problems

Cognitive Analysis: Link Between Logical And Cognitive Analysis Based on the knowledge- Pre-classified knowledge-type Type of type construct of science Declarative Procedural Schematic Strategic knowledge used (n =9) * (n =10) (n =9) (n =2) achievement, we expected participants’ 48 Declarative 8 11 0 use of knowledge Procedural 0 7 9 54 inferred from the protocols (cognitive 9 16 0 41 Schematic analysis) to be congruent 10 Strategic 2 12 2 with the knowledge- types demanded by test * Number of responses: 48 = 9 x 6 participants items (logical analysis) Chi-square = 208.12, p<.001

Covariance Analysis: Links With Logical & Cognitive Analyses e1 bsmsa7 Based on the e2 bsmsa9 e3 bsmsa11 e4 bsmsb1 .39 knowledge-type .51 e5 bsmsb4 .47 .23 e6 bsssp3 .41 construct of science .51 e7 bsmsp4 .40 .69 Declarative e8 bsssp6 Knowledge .34 achievement, we -.11 e9 bsmsq14 .37 .44 .86 bsssq17 e10 .42 expected the .63 bsesr3 e11 .04 .08 bsmsb5 e25 e12 bsssr4 .19 -.24 .12 bsmsp1 e26 .50 .14 emerged factors from e13 bsssr5 Procedural .36 -.06 e27 bsmsp7 Knowledge .84 .21 .00 bsmsr1 e28 .06 e14 bsmsa8 the item scores .14 .29 -.17 .02 e15 bsmsa12 .37 .26 e16 bsmsb2 .70 (statistical analysis) .42 -.10 e17 bsmsb3 .42 e18 bsssp2 .54 to be congruent with Schematic .08 Knowledge e19 bsssp5 .33 .11 e20 bsmsq11 the knowledge-types .27 e21 .41 bsmsq13 .25 e22 bsmsq15 in logical analysis e23 bsssq18 e24 bsmsr2