Using ATLAST Instruments to Study the Relationships Among Teacher - - PowerPoint PPT Presentation

Using ATLAST Instruments to Study the Relationships Among Teacher Knowledge, Classroom Instruction, and Student Learning

MSP Learning Network Conference January, 2010

Sean Smith Horizon Research, Inc.

Imagine a Study

• 1,000 randomly selected middle grades

science classes.

• Valid and reliable measures of:

– Teacher content knowledge – Instructional time on topic – Student learning

• Sophisticated and appropriate analysis

techniques

Predict the Study Findings

• In terms of the relationships between:

– Teacher content knowledge and student learning – Instructional time on topic and student learning – Teacher content knowledge and student learning

• Would any of your predictions depend
• n the specific science content?

Session Overview

• Background of ATLAST
• Description of the measures
• Description of the study
• Findings
• Discussion

Implicit Theory of Action

Professional Development Increased Teacher Knowledge for Teaching Science Improved Classroom Practice in Science Increased K-12 Student Achievement in Science Science Curriculum Materials

Content Areas

• Force and motion
• Processes that shape Earth (Plate

Tectonics)

• Flow of matter and energy in living

systems

Implicit Theory of Action

Description of the Measures

Teacher Content Assessments

In a unit on force and motion, students have been pushing small carts across their tables to the right and observing the

• motion. The teacher asks the students to draw a diagram

showing all the horizontal forces on a cart once it leaves the student's hand and is rolling across the table. (A) Which one of the following force diagrams is correct?

Common Features of All Items

• All are multiple choice
• All are keyed to a specific idea
• All are set in the context of work that

teachers do

1. A force is a push or pull interaction between two objects, and has both strength and direction. 2. The force of friction acts to oppose an

• bject’s motion.

Target Ideas for Cart Item

Other Types of Items

In a lesson on plant growth, a teacher is discussing plants’ use of light energy from the Sun. During the discussion,

• ne student says, “Plants need the light to grow, but they

don’t change the light at all. It’s like when you’re reading a book, and you need the light to help you read.” Which one of the following ideas about the role of light energy in photosynthesis does the student seem to be missing? (B) 

• A. Light energy is changed into sugars in the process of

photosynthesis.

• B. Light energy is changed into another form of energy in

the process of photosynthesis.

• C. Light energy is the energy source for the process of

photosynthesis.

• D. None. The student seems to have an accurate

understanding of the role of light energy in the process

• f photosynthesis.

Level 2 Item Features

• Address teachers’ ability to analyze

student thinking using science content knowledge 

• Cannot be answered without content

knowledge

• Only one answer choice is “content-

correct” and relevant to the instructional context

• Fairly high cognitive load

Common Errors Made With Level 2 Items

• Teachers look for common student

thinking rather than the thinking of these students

• Teachers look for a correct statement
• Teachers try to answer the student item
• Teachers look for familiar wording –

e.g., “equal and opposite”

• Teachers need options that allow them

to hold naïve conceptions

Sample Item

In a class discussion, a teacher asks his students to describe Earth’s plates. One student says, “There are thousands of plates that are moving and causing changes to Earth’s surface.” Based on this statement, which one of the following should the teacher do next to further this student’s understanding

• f Earth’s plates? (B) 

A. Discuss the types of geological features plate movement can cause. B. Have students outline the boundaries of the plates on a map. C. Introduce students to the specific ways in which plates move.

• D. Demonstrate how the plates move as a result of

convection.

Level 3 Item Features

• Address teachers’ ability to make

instructional decisions using science content knowledge

• Cannot be answered without content

knowledge

• Only one answer choice is “content-

correct” and relevant to the instructional context

• High cognitive load

Common Errors Made With Level 3 Items 

• Teachers see all activities/ questions as

“best”

– Lack of content knowledge – High cognitive load

• Context is important

– Focus on logistics – Unfamiliar scenario/ equipment

• Teacher beliefs

In Summary – 3 Kinds of Items

• Knowledge of content
• Using content knowledge to analyze

student thinking

• Using content knowledge to make

instructional decisions

Student Assessments

A girl kicks a box so that it slides across the level ground. The box moves slower and slower as it slides and eventually stops. Which one of the following is the best explanation for why the box moves slower and slower? (B)

• A. The force due to the kick slowly dies out.
• B. The force of friction is acting on the box.
• C. The force due to the kick is gradually overcome by the

force of friction.

• D. The force of friction and the force due to the kick

become equal to each other.

Student Assessment Item

Validity and Reliability

Reliability

0.86 0.86 Plate Tectonics 0.75 0.85 Force and Motion 0.78 0.83 Flow of Matter and Energy in Living Systems

Student Assessm ent Teacher Assessm ent Content Area

Test-retest reliability

0.94 Plate tectonics 0.88 Force and motion 0.93 Flow of matter and energy Test-retest reliability Teacher assessment

Validity

• Cognitive interviews
• Expert review
• Factor analysis

Instructional Log

Description of the Study

Overview

• Two topics

– Force and motion – Plate tectonics

• Recruited 200 teachers for each study
• Only a handful of teachers in both

studies

Each Teacher

• Completed the teacher assessment
• Administered the student pre-test
• Taught a unit on the topic
• Logged each day of instruction
• Administered the student post-test

Classroom Instruction Variables

• So far:

– Aligned instructional time – Each study presented unique issues

• Eventually:

– Student opportunity to learn

Analysis: 3-level HLM

• Student scores at level 1
• Student demographics at level 2
• Teacher/ class variables at level 3

Findings

Force and motion

• Teacher knowledge predicted student

learning (effect size 0.12).

• Instructional time did not predict

student learning.

• No relationship between teacher

knowledge and instructional time.

Plate Tectonics

• No relationship between teacher

knowledge and student learning.

• No relationship between instructional

time and student learning.

• No relationship between teacher

knowledge and instructional time.

Nature of Instruction

• Force and motion tends to be activity

based.

• Plate tectonics tends to be textbook

based.

Possible Explanations

• The instruments are not valid.
• Instructional materials obscure the

influence of other variables.

• Relationships vary by content area.

The Big Picture

• As a field, we have notions about what

effective science teaching looks like.

• What if those principles don’t hold for

all content? What if they are important primarily for misconception-laden content?

• We are beginning to be able to answer

these questions with the kinds of instruments ATLAST has developed.

ATLAST

Assessing Teacher Learning About Science Teaching

Sean Smith, PI Horizon Research, Inc. www.horizon-research.com/ atlast/ ssmith62@horizon-research.com