Assessing NSF Programming-- -- Assessing NSF Programming Standards- -Based Reform Based Reform Standards Assessment Technologies: Assessment Technologies: The San Francisco Project The San Francisco Project Richard J. Shavelson and Maria Araceli Ruiz-Primo S tanford U niversity Performance Effectiveness Review National Science Foundation January 27, 1999 1 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Achievement Indicators: Achievement Indicators: Instructional Sensitivity Instructional Sensitivity Depth of Assessment Probe Remote : Standardized National Science Achievement Tests Distal : Large-Scale Performance Assessment from State/ National Curriculum Framework Proximal : Same Concept/Principle--New Investigation Outside School Influences Close: “Embedded” Assessments -- A Slightly More Advanced Activity in Unit Immediate: Lab Notebooks & Classroom Tests Classroom Instruction 2 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Agenda Agenda PART I: Framework for Evaluating Science Education Reform • A Working (and Evolving) Definition of Science Achievement • Linking Assessments to Components of Achievement • Multilevel Achievement Assessment PART II: The San Francisco Study • The Proximity of the Assessments • The Study • The Findings PART III: Concluding Comments • What We Have Learned • The Larger Picture 3 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Toward An Achievement Framework: Toward An Achievement Framework: Knowledge Components Knowledge Components Declarative Procedural Strategic Knowledge Knowledge Knowledge Characteristics That Vary According to Proficiency (Knowing the “ that ”) (Knowing the “ how ”) (Knowing the “ which ,” Level “ when ,” and “ why ”) Low High Extent ( How much? ) Domain-specific Production Problem schemata/ content: system-- Structure strategies/ • facts condition- ( How is it organized? ) • concepts action rules • principles operation systems Others (Precision? Efficiency? Automaticity?) Cognitive Cognitive Tools: Tools: Planning Planning Monitoring Monitoring 4 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Linking Assessments to Linking Assessments to Achievement Components Achievement Components Declarative Procedural Strategic Knowledge Knowledge Knowledge Performance Performance Extent Multiple-Choice Assessments Assessments Conceptual Models/ Procedural Structure Maps Mental Maps Maps Others 5 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Achievement Indicators: Achievement Indicators: Instructional Sensitivity Instructional Sensitivity Depth of Assessment Probe Remote : Standardized National Science Achievement Tests Distal : Large-Scale Performance Assessment from State/ National Curriculum Framework Proximal : Same Concept/Principle--New Investigation Outside School Influences Close: “Embedded” Assessments -- A Slightly More Advanced Activity in Unit Immediate: Lab Notebooks & Classroom Tests Classroom Instruction 6 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Variables Unit: Swingers Activity Variables Unit: Swingers Activity Goal: Gain experience with the concepts of system, variable, and controlling and manipulating variables Assessment Task: Same concept and (high) structure, slightly different materials and measurement method Unit Activity Close Assessment Task release position 7 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Variables Unit: Lifeboats Activity Variables Unit: Lifeboats Activity Goal: Gain experience with the concepts of system, variable, and controlling and manipulating variables Assessment Task: Same concept, different structure (low), materials, and measurement method Unit Activity Proximal Assessment Task 8 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Mixtures & Solutions Unit: Mixtures & Solutions Unit: Reaching Saturation Activity Reaching Saturation Activity Goal: Gain experience with the concepts of mixtures and solutions, concentration, saturation, and chemical reaction Assessment Task: Same concept, slightly different structure and materials and different measurement method Unit Activity Close Assessment Task 9 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Mixtures & Solutions Unit: Mixtures & Solutions Unit: Fizz Quiz Activity Fizz Quiz Activity Goal: Gain experience with the concepts of mixtures and solutions, concentration, saturation, and chemical reaction Assessment Task: Same concept, slightly different materials, different (low) structure and measurement method Unit Activity Proximal Assessment Task 10 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
CSIAC Performance Assessment: CSIAC Performance Assessment: Is All the Trash the Same? Is All the Trash the Same? Distal Assessment Task TASK 1 TASK 3 TASK 2 Trash Chart Sort your trash Identify your trash • Record the information from your • Take all of the trash items out of Bag A • Look at the “Trash Identification Table” placemat onto the chart below. Write and put then on your placemat. below. your reasons for putting the trash in • Observe each trash item. • Find a picture of each trash item that is • Sort the trash items into groups based on your placemat. each group under “Reasons.”. on characteristics the items have in • Find the number that matches each common trash item in the “Trash Identification • Using all of the trash items, make at Table.” least three groups. Do not make more • Write the number for each trash item than five groups. on your placemat next to each item. • Each trash item can be in only one • Make sure that you put a number by group every trash item. • Draw a circle around each group on your placemat Trash Identification Table 11 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Proximity Profile of the Assessments Proximity Profile of the Assessments VARIABLES MIXTURES & SOLUTIONS Proximal Proximal Distal Distal Close Close Pendulum Bottles CSIAC Solutions Mystery CSIAC Powders Purpose Content Task Directedness Materials Measurement Methods 12 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Study Questions Study Questions • Does hands-on science instruction impact students’ performance? • If so, does the magnitude of impact differ depending on the proximity of the assessment to the curriculum? • Are findings replicable across curricular units? 13 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Study Design Study Design Participants: Ten schools from a medium sized urban school district in the Bay Area. Twenty teachers and about 500 fifth-graders. Design: Replications across classrooms: Initial Student Status, Unit Implementation, Final Student Status Instrumentation: FOSS UNITS Variables Mixtures and Solutions (Fall) (Spring) Close Pendulum Saturation Proximal Bottles Mystery Powders Distal CSIAC-PA & MC 14 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Within- -Classroom Design Classroom Design Within Sequence Initial Status Implementation Final Status Pretest Posttest 1 Close Science Journals Close CSIAC-PA & MC 2 Proximal Science Journals Proximal CSIAC-PA & MC 15 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Instructional Sensitivity: Instructional Sensitivity: Preliminary Results Preliminary Results Solutions 1.4 1.2 Mean Gain in SD 1 0.8 Pendulum 0.6 Units Mystery Bottles Powders 0.4 0.2 0 Close Proximal 16 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Close Assessments: Close Assessments: Effects Across Classrooms Effects Across Classrooms 2.5 2 Mean Gain in SD Units 1.5 Pendulum Solutions 1 0.5 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 17 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Proximal Assessment: Proximal Assessment: Effects Across Classrooms Effects Across Classrooms 1 0.8 0.6 Mean Gain in SD Units 0.4 0.2 Bottles 0 Mystery Powders -0.2 -0.4 -0.6 -0.8 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 18 Stanford University Transferring New Assessment Technologies to Teachers and Other Educators (NSF-ESI- 9596080)
Recommend
More recommend
