Using ATLAST Instruments to Study the Relationships Among Teacher - PowerPoint PPT Presentation

Using ATLAST Instruments to Study the Relationships Among Teacher Knowledge, Classroom Instruction, and Student Learning MSP Learning Network Conference January, 2010 Sean Smith Horizon Research, Inc.

Imagine a Study • 1,000 randomly selected middle grades science classes. • Valid and reliable measures of: – Teacher content knowledge – Instructional time on topic – Student learning • Sophisticated and appropriate analysis techniques

Predict the Study Findings • In terms of the relationships between: – Teacher content knowledge and student learning – Instructional time on topic and student learning – Teacher content knowledge and student learning • Would any of your predictions depend on the specific science content?

Session Overview • Background of ATLAST • Description of the measures • Description of the study • Findings • Discussion

Implicit Theory of Action Improved Increased Increased Teacher Professional Classroom K-12 Student Knowledge for Development Practice in Achievement Teaching Science Science in Science Science Curriculum Materials

Content Areas • Force and motion • Processes that shape Earth (Plate Tectonics) • Flow of matter and energy in living systems

Implicit Theory of Action

Description of the Measures

Teacher Content Assessments

In a unit on force and motion, students have been pushing small carts across their tables to the right and observing the motion. The teacher asks the students to draw a diagram showing all the horizontal forces on a cart once it leaves the student's hand and is rolling across the table. (A) Which one of the following force diagrams is correct? 

Common Features of All Items • All are multiple choice • All are keyed to a specific idea  • All are set in the context of work that teachers do 

Target Ideas for Cart Item 1. A force is a push or pull interaction between two objects, and has both strength and direction. 2. The force of friction acts to oppose an object’s motion. 

Other Types of Items

In a lesson on plant growth, a teacher is discussing plants’ use of light energy from the Sun. During the discussion, one student says, “Plants need the light to grow, but they don’t change the light at all. It’s like when you’re reading a book, and you need the light to help you read.” Which one of the following ideas about the role of light energy in photosynthesis does the student seem to be missing? (B)  A. Light energy is changed into sugars in the process of photosynthesis. B. Light energy is changed into another form of energy in the process of photosynthesis. C. Light energy is the energy source for the process of photosynthesis. D. None. The student seems to have an accurate understanding of the role of light energy in the process of photosynthesis.

Level 2 Item Features • Address teachers’ ability to analyze student thinking using science content knowledge  • Cannot be answered without content knowledge • Only one answer choice is “content- correct” and relevant to the instructional context • Fairly high cognitive load

Common Errors Made With Level 2 Items  • Teachers look for common student thinking rather than the thinking of these students • Teachers look for a correct statement • Teachers try to answer the student item • Teachers look for familiar wording – e.g., “equal and opposite” • Teachers need options that allow them to hold naïve conceptions

Sample Item

In a class discussion, a teacher asks his students to describe Earth’s plates. One student says, “There are thousands of plates that are moving and causing changes to Earth’s surface.” Based on this statement, which one of the following should the teacher do next to further this student’s understanding of Earth’s plates? (B)  A. Discuss the types of geological features plate movement can cause. B. Have students outline the boundaries of the plates on a map. C. Introduce students to the specific ways in which plates move. D. Demonstrate how the plates move as a result of convection.

Level 3 Item Features  • Address teachers’ ability to make instructional decisions using science content knowledge • Cannot be answered without content knowledge • Only one answer choice is “content- correct” and relevant to the instructional context • High cognitive load

Common Errors Made With Level 3 Items  • Teachers see all activities/ questions as “best” – Lack of content knowledge – High cognitive load • Context is important – Focus on logistics – Unfamiliar scenario/ equipment • Teacher beliefs

In Summary – 3 Kinds of Items • Knowledge of content • Using content knowledge to analyze student thinking • Using content knowledge to make instructional decisions

Student Assessments

Student Assessment Item A girl kicks a box so that it slides across the level ground. The box moves slower and slower as it slides and eventually stops. Which one of the following is the best explanation for why the box moves slower and slower? (B) A . The force due to the kick slowly dies out. B . The force of friction is acting on the box. C . The force due to the kick is gradually overcome by the force of friction. D . The force of friction and the force due to the kick become equal to each other.

Validity and Reliability

Reliability Teacher Student Content Area Assessm ent Assessm ent Flow of Matter and Energy in Living 0.83 0.78 Systems Force and Motion 0.85 0.75 Plate Tectonics 0.86 0.86

Test-retest reliability Teacher assessment Test-retest reliability Flow of matter and 0.93 energy Force and motion 0.88 Plate tectonics 0.94

Validity • Cognitive interviews • Expert review • Factor analysis

Instructional Log

Description of the Study

Overview • Two topics – Force and motion – Plate tectonics • Recruited 200 teachers for each study • Only a handful of teachers in both studies

Each Teacher • Completed the teacher assessment • Administered the student pre-test • Taught a unit on the topic • Logged each day of instruction • Administered the student post-test

Classroom Instruction Variables • So far: – Aligned instructional time – Each study presented unique issues • Eventually: – Student opportunity to learn

Analysis: 3-level HLM • Student scores at level 1 • Student demographics at level 2 • Teacher/ class variables at level 3

Findings

Force and motion • Teacher knowledge predicted student learning (effect size 0.12). • Instructional time did not predict student learning. • No relationship between teacher knowledge and instructional time.

Plate Tectonics • No relationship between teacher knowledge and student learning. • No relationship between instructional time and student learning. • No relationship between teacher knowledge and instructional time.

Nature of Instruction • Force and motion tends to be activity based. • Plate tectonics tends to be textbook based.

Possible Explanations • The instruments are not valid. • Instructional materials obscure the influence of other variables. • Relationships vary by content area.

The Big Picture • As a field, we have notions about what effective science teaching looks like. • What if those principles don’t hold for all content? What if they are important primarily for misconception-laden content? • We are beginning to be able to answer these questions with the kinds of instruments ATLAST has developed.

ATLAST Assessing Teacher Learning About Science Teaching Sean Smith, PI Horizon Research, Inc. www.horizon-research.com/ atlast/ ssmith62@horizon-research.com