U.S. National Assessment of Educational Progress (NAEP) Virtual - - PowerPoint PPT Presentation

U.S. National Assessment of Educational Progress (NAEP) Virtual Science Lab

Lei Liu Shu-Kang Chen Timothy Fiser Educational Testing Service William Tirre National Center for Education Statistics Presentation at NCSA 2015 San Diego, CA

Acknowledgement

Research team

• Lei Liu
• Helena Jia
• Katherine Castellano
• Elia Mavronikolas
• Sarah Ohls

Development team

• Shu-Kang Chen
• Timothy Fiser
• Raymond De Hont

Ex-external developer: eLearning Brothers, Inc. Infuse, Inc.

NAEP SAIL Executive Directors

• Joanna Gorin
• Gregory Vafis
• Andreas Oranje

Overview

• What drives this project
• Project objective
• Design principles
• The student model
• Three-phase plan and update
• What we have learned
• Next Steps

Innovation in Science Assessment

• Future science assessment must
• improve measurement of “hard-to-measure” science

constructs that are core of NGSS (e.g., science practices), and

• increase the feasibility of measuring additional

constructs including critical thinking, problem solving, metacognition, and other 21st Century skills.

Project Objective

• To develop a tablet-based virtual science laboratory

(VSL) to measure students’ complex scientific competence and provide evidence about how well they can apply content knowledge and science practices through different paths of problem solving.

VSL Design Principles

Traditional Approach Innovative Approach Follow a “cookbook” approach of replicating or verifying certain scientific phenomena Follow inquiry-based approach by providing free space for students to inquire knowledge (e.g., trying out experiments) Gather and record data without a clear sense of the purposes of their investigation Problem-based knowledge inquiry by using lab equipments to design experiments Collect data that are predefined Conduct real experiments to collect scientifically accurate data that are not predefined in any way Experiments are dependent on individual science topics thus are costly Develop modular activities to support reusable capability and to possibly reduce the cost of development

The Student Model

NGSS Science Practices Crosscutting Concepts Disciplinary Core Ideas

NAEP Science Framework Inquiry Skills Science Content

Science Practices

• Planning investigations;
• Carrying out investigations;
• Analyzing and interpreting

data;

• Communicating

Core Ideas

• Matter and its interactions
• Forces and energy
• Waves and energy

VSL Development Plan

Feasibility Phase: A proof-of- concept prototype to determine the technical needs Prototype Phase: A prototype

• f the
• verall

setting and a chemistry lab Modular Activities Phase: Prototypes

• f four

modular activities in the chemistry lab

Feasibility Demo

Workstation

• Mixing two liquids
• The simulated

chemical reaction (e.g., color change) responds to the direct input of user in real time

Prototype Phase – Six Action Areas

Action Area 1: Virtual Laboratory

• Three labs for students to

explore (as a long-term goal)

• A 3D virtual chemistry lab
• Have access to all other

action areas through touch in the lab or the GUI dashboard

• View and select

interactive objects for lab use

• View detail info

about each object

• Navigate 3D objects
• Drag and drop any
• bject from the

shelves to a lab cart

Action Area 2: Lab Supply

• Add / remove selected

lab supply objects to the lab cart

• Indicate the quantity
• f selected objects
• See detailed label

for each selected

• bject

Action Area 3: Lab Cart

• Manipulate selected objects

to conduct virtual experiments

• Interact with 3D objects

in relative scale to try

• ut relevant features
• Change view of objects

by zooming, panning, and/or rotating the viewing position

• View simulated results

when interacting with 3D objects

Action Area 4: Workstation

• Record and refine design and

results of experiments

• Identify relevant variables
• List relevant lab supplies
• Plan experimental

procedures

• Record data
• Identify

qualitative/quantitative data patterns

• Draw conclusions
• Respond to direct

assessment items embedded in the VSL

Action Area 5: Notepad

• Share products generated in

the Notepad

• Communicate design

decisions

• Communicate
• bservations (e.g., data

patterns)

• Communicate

conclusions

Action Area 6: Whiteboard

Modular Activities Phase

Mini Modular Lab Activities

• Modular Activity 1 – Planning Experiments
• selecting relevant tools
• designing lab procedures
• Modular Activity 2 – Carrying Out Experiments
• measuring and recording weight
• measuring and recording volume
• Modular Activity 3 – Analyzing & Interpreting Data
• using notebook to record and interpret data patterns
• Modular Activity 4 – Communicating
• using whiteboard / notebook to share data patterns or lab results

What has been learned?

• Both the NAEP Science framework and the NGSS call for

measures of complex scientific competencies

• The balance between user freedom and design constraints

to limit cost and timelines is difficult to achieve

• Feasibility study for proof-of-concept is useful to help

determine technical demands and vendor capability

• Developing modular lab activities will provide several key

benefits:

• reducing construct-irrelevant noises and focus on

constructs that the virtual lab affords to measure;

• improving the reusability of the lab environment;
• reducing the cost of development

Next Steps

• Usability study on the design of effective user interfaces

(Prototype Development Stage):

• Intuitive, easy-to-learn interface and navigation
• No distraction from test content
• UDC considerations
• Cog lab study on the construct-relevant issues

(Prototype Development Stage):

• What evidence can the VSL afford students to

demonstrate their integrated science learning?

• Can the VSL generate evidence to differentiate

students’ integrated knowledge?