Learning Science: The Importance of Intervention Research Barbara - - PowerPoint PPT Presentation

Learning Science: The Importance

• f Intervention Research

Barbara Schneider and Joseph Krajcik

Relevant Research Roundtable-R3 Florida State University March 9, 2020

Outlin line of e of t the P e Pres esentatio ion

• Why We Need Evidence for Science Policy and Practice

. Why Interventions are Critical Principles and Criteria for Testing an Intervention effect

• How to Form Evidence on Social and Emotional Learning
• What the First Steps in Developing a Theory of Action—

Example from Crafting Engagement in Science Environment

• Why Develop a Logic Model
• Procedures, Measures, and Results
• What We are Learning

Why We Ne Need ed I Interven ention

• ns
• Worldwide policymakers have encouraged the use of

scientifically based evidence to make decisions for supporting specific educational programs and practices.

• To be able to do this information is needed about what

programs and practices do or do not work.

• To produce such evidence leads to causal questions, such as

whether particular programs and practices improve student academic achievement, social development, and pathways to further education, earnings, and civic participation.

Principles of Interventions

As scientists should adopt principles to assess the trustworthiness of evidence by making: 1) Claims supported by evidence 2) Identify fair comparisons 3) Goals, problems & designs defined 4) Produce reproductible & replication effects

1) Intervention based on theoretically grounded principles 2) Assumptions about what the impact will be (including pre-registration of the study--SREE) 3) Possible to achieve a measurable effect (Power Effect—Optimal Design Software) 4) Sound measures of face and content validity (standards –performance expectations NGSS and rubrics 5) Reliable measures and tools (assessments—NAEP and MDE) for making claims about the intervention effects 6) Rigorous Analytics including sensitivity analysis and examination of heterogeneity differences

Criteri ria f for T r Testing an Intervention Effect

Our C Challenge

Build learning environments that:

• Foster deep and integrated understanding of

important idea

• Engage students, i.e., create optimal learning

environments, in learning science

• Support students in developing important scientific

practices and 21st century competencies

• Support students to solve problems, think critically,

and innovatively

• Goal, develop and test a system for advancing science teaching and learning

that builds a vision for enacting project-based learning and meeting NGSS for High School Chemistry and Physics.

• The system includes:
• Highly developed and specified educative teacher materials
• Highly developed and specified student materials
• Professional learning supports
• 3-dimensional formative and end-of-unit assessments

Crafting En Engagement in Science En Envi vironments: The T e Trea eatmen ent

Crafting E Eng ngag aging Science ce Environments (CESE)

• NSF funded, Multi-year project
• International Collaboration: US and Finland
• Large scale study - 130 teachers, 70 schools, ~8000 students
• Goal:

“ To increase student engagement and interest in the fields of science, technology, engineering, and mathematics (STEM)”

Impor

• rtance o

ce of Building A An Inter erdisci ciplinary T Team

• You cannot know everything

Studying an Intervention requires substantive knowledge of science learning --Project Based Learning—Joseph Krajcik Expertise in content knowledge of the intervention Solid analytic expertise— Building an interdisciplinary team of Graduate students, Undergraduates, Post Doctoral Fellows, Content Specialists Willingness to share, negotiate, transparent, and open about challenges and disappointments.

Our Claims

• Our design process allows us to create PE-aligned units and

assessment tasks

• Students learning these units show growth in their ability to answer

three-dimensional assessment items

• Students show an improvement in their engagement, OLM and
• ther socio-emotional constructs

Theory of Engagement for Science Learning

Logic Model

Putting it all together

New Vision for Science Education

Project Based Learning Curriculum & Assessment Optimal Learning Moments

• How do we create units that

aligned with the new vision for science education?

• How do we create three-

dimensional assessment items?

• Do our units allow for the

development of skills and knowledge alongside students engagement?

Our Units

Unit Name Driving Question Phenomena Performance Expectations Forces and Motion “How can I design a vehicle to be safer for a passenger during a collision?” car\cart collision HS-PS2-1 HS-PS2-3 MagLev “How do mag-lev trains function without touching the track?” Magnetic Levitation HS-PS3-5 HS-PS3-2 Electric Motor “How can I make the most efficient electric motor?” Toy motors HS-PS3-1 HS-PS2-5 HS-PS3-3 Evaporation “When I am sitting by the pool, why do I feel colder when I am wet than when I am dry?” Evaporation of different liquid on the palm of your hand HS-PS1-3 HS-PS3-2 Periodic Table “Why is table salt safe to eat but the substances that forms it are explosive

• r toxic when

separated?” Reaction of Sodium with water HS-PS1-1 HS-PS1-2 Conservation of Mass “Can I make substances appear or disappear?” Flash paper, invisible ink, Al/CuCl2 reaction HS-PS1-7

Measuri ring Social and Em Emotional Lea earnin ing Value of

• f R

Rep epli licatio ion of

• f Sc

Scales

PIRE

When working on this activity…I used my imagination. When working on this activity…I solved problems that had more than one possible solution. When working on this activity…I explored different points of view on the problem or topic. When working on this activity…I had to make connections with other school subjects.

OECD

When working on this course...I have to use my imagination. When working on this course...I have to solve problems that have more than one possible solution. When working on this course...I have to explore different points of view on a problem or topic. When working on this course...I have to make connections with other school subjects.

Experience Sampling Method (ESM)

• Focus on the situational and contextual aspects
• f what happens in and out of the classroom
• Short, repeated surveys capture what students

are doing and feeling in-the-moment

(Csikszentmihalyi, 1975)

• Less opportunity for recall bias and socially

desirable answers (Hektener et al., 2007)

• Students signaled randomly during the course
• f a week or days.

Measuri ring Social and Em Emotional Lea earnin ing Value of

• f R

Rep epli licatio ion of

• f Sc

Scales

PIRE

When working on this activity…I used my imagination. When working on this activity…I solved problems that had more than one possible solution. When working on this activity…I explored different points of view on the problem or topic. When working on this activity…I had to make connections with other school subjects.

OECD

When working on this course...I have to use my imagination. When working on this course...I have to solve problems that have more than one possible solution. When working on this course...I have to explore different points of view on a problem or topic. When working on this course...I have to make connections with other school subjects.

18

Figure 1. Field Test Single Case Design Graph

New R ew Results

• Importance of Measuring Relationship of Social and

Emotional Measures for both treatment and Control Conditions-

New R ew Results

• Importance of Measuring Relationship of Social and

Emotional Measures for both treatment and Control Conditions-

1545684

Treatment students feel less successful and confident during low-challenge moments but more successful and confident during high-challenge moments

A one e standard d dev eviation increa ease i in aggr greg egated ed situational engagemen ent c t corres esponds to to a 0.15 s standard d dev eviati tion in increase in in summative c e course g e grade. e.

1545684

Treatment classrooms focused on asking questions about phenomena and exploring a relevant driving question An emphasis on scientific practices, while still challenging, offered students

• pportunities to demonstrate their competence and feel successful in science

Project-based learning may take teachers and students out of their comfort zones, but can lead to payoffs in science learning and social and emotional experiences Implications

What h has happened?

• A large main effect on science achievement for the

intervention; most valuable for low-income and

minority students

• Raised imagination and desire to take on challenging

problems to figure things out

• Increases in interest in pursuing science courses in college

and later careers

• Growth in teacher engagement with scientific practices

Networking Our professional learning workshops provide an

• pportunity for teachers across the country to connect

and discuss teaching methods, share resources, student testimonials, lesson modifications, and more.