Development of a Smarter Balanced Assessment to Assess Grade-Level - - PowerPoint PPT Presentation

Development of a Smarter Balanced Assessment to Assess Grade-Level Readiness and Summer Learning Loss in Math

Presented at NCSA June 29, 2018 by Val Verde Unified School District Presentation Team Perris, California

Presentation Team

Michael McCormick, Superintendent VVUSD mmccormick@valverde.edu Jennifer Doskocil, Coordinator, Elementary Education VVUSD jdoskocil@valverde.edu Pete Goldschmidt, PhD, Research Consultant pete.goldschmidt@csun.edu Sandy Sanford, EdD, Assessment Consultant sandy@youasksandy.com Heather Goodwin, Assessment Consultant good1hlg@gmail.com

Presentation Plan

• Slides (PPT posted on NCSA 2018 web site)
• Pete Video available by request from

sandy@youasksandy.com

• SMC – Summer Math Camp

Project Phases

Phase I—Summer Math Camp (April to August 2018) Phase II—Analysis of 2018-19 data to determine effects/adjustment

Phase I and II results Reported at NCSA 2019

Phase III—Summer Math Camp in summer 2019 Phase IV—Analysis of 2019-20 data to determine effects/adjustment

Phase III & IV results Reported at NCSA 2020

The Project Objectives

• 1. To assess grade level readiness
• 2. To measure summer learning loss

Collateral Phase I Objective

• 3. Evaluate the efficacy of the SMC Teacher Professional Development

in increasing #1 and decreasing #2

The Plan for SMC Phase I

• Determine Critical Standards for 3rd to 4th, 4th to 5th, & 5th to 6th (April)
• Recruit 32 SMC Teachers—Recruit Candidates & Randomly Choose (April)
• Create 16 item Smarter Balanced Aligned Screening Test (ST) for each Grade
• Administer ST to as many students as possible during last two weeks of May
• Use cut points established on the ST to identify Candidates for SMC (May-June)
• Analyze ST (June)
• Create 2 parallel Tests—SMC Pre-Test and a beginning of school year Post Test
• Use lottery system to choose SMC attendees
• Create Professional Development for SMC Teachers (CSUN & MindfullyU)
• SMC to include administering Pre-Test (20 to 26 July)
• Administer Post Test during 1st week of 18-19 school year
• Analyze combined data from the 3 Tests

Recruit SMC Teachers (Random) Develop/ Administer Screening Test

May June July August

Identify SMC Student Candidates Select SMC Teachers (Randomly) Analyze Screening Test Select SMC Student Candidates (Non-Random) Teachers Receive PD & Instruct SMC Develop/ Administer Pre- Test Students Attend SMC Analyze Series of 3 Tests Develop/ Administer Post Test Identify SMC Student Candidates Select Facilities & Support Prepare Facilities/ Support Prepare/Use Facilities/ Support Clean/ReConfig Facilities/ Support Liaison with CSUN & MU for PD CSUN & MU Develop PD CSUN & MU Deliver & Monitor PD CSUN & MU Provide After- Action Rpt

Research Team Analyzes Efficacy of PD

SMC 2018 in a nutshell

• 32 Teachers—3rd, 4th, 5th, & 6th
• Each teacher…
• Dedicated Classroom
• 4 to 7 students same grade level
• Students chosen from those score RS 6, 5 , 4, & 3 of Screening Test
• 20 July—Teachers meet students, activities, Pre-Test in AM
• 20 July—Teachers receive PD from CSUN & MinfullyU in PM
• 23 to 26 July-–Teacher teach students AM & receive PD PM

Development of a Smarter Balanced Assessment to Assess Grade-Level Readiness and Summer Learning Loss in Math

Pete Goldschmidt, PhD. National Conference on Student Assessment June 29, 2018

Pete Video

Development of a Smarter Balanced Diagnostic Assessment to Assess Grade-Level Readiness and Summer-Learning Loss in Math

Pete Goldschmidt, Ph.D. National Conference on Student Assessment June 29, 2018

Summer Learning Loss

• Students demonstrate consistent patterns of loss over the summer
• Less exacerbates achievement gaps because students begin the

school year behind, catch up, and subsequently do not retain what they have learned and are then starting the next year behind (again).

• Problem particularly problematic for low SES and non-native English

speakers because they tend not to have access to the same resources and/or academic content in English over the summer.

Stylized Example of Summer Learning Loss and Achievement Gap Increase

20 40 60 80 100 120 K Begi n K End 1st gr a de Beg i n 1st gr a de End 2nd gr ad e Beg i n 2nd gr ad e End 3r d gr ade Beg i n 3r d gr ade End 4t h gr ade Beg i n 4t h gr ade End Co m p ar i son Sub gr oup

Summer Learning Loss Impact

• Some students enter next grade level not prepared for grade-level

material

• Teacher spends time assessing student’s readiness to learn.
• Teacher spends time re-teaching.
• Reinforces potential for gap.

Motivation for Study

• Study based on 3rd through 6th grade students.
• Accurately assess summer Learning Loss.
• SBAC not amenable for assessing summer loss because reflects both summer

loss and school-ear gains.

• Developed short assessment given closer to end of the year and beginning of

subsequent year.

• More quickly asses grade-level readiness.
• Identify individual and group-level readiness.
• Identify students with significant loss.

Motivation for Study

• Assessments used to evaluate summer learning loss and readiness

part of item bank available for use by teachers.

• Create and evaluate summer math intervention.
• Summer Math Camp.
• Details to follow.

Study Design

• Solicit teacher volunteers for math camp.
• Randomly select teachers for treatment condition.
• Non-selected volunteers become counterfactual.
• Use Pre-test to invite student participants.
• Given too many students scored below cut-score, randomly select students

for participation.

Grade Achievement Level Scale Score Range for Standard Not Met Summer Math Camp Pre-Test Cut Score 3 2189–2380 2379 4 2204–2410 2401 5 2219–2454 2424

Research Questions

• What is the overall loss during the summer?
• What is the level of readiness?
• Do students who participated in the Summer Math Camp have a

smaller summer learning loss compared to students who did not participate?

• Are students who participate in the Summer Math Camp better

prepared for the next year compared to students who did not participate?

• Do students who participate in the Summer Math Camp perform

better on SBAC compared to students who did not participate?

• Do students who have a teacher who participated in the Summer

Math Camp perform better on SBAC than students who do not have teachers who participated in the Summer Math Camp?

• Students who participated in the Summer Math Camp.
• Students who did not participate in the Summer Math Camp.

Research Design

• Given that students are assigned to the Summer Math Camp based
• n a cut score we can use a Regression Discontinuity (RD) design to

compare students just on either side of the cut score.

• Given that student who were below the cut were randomly assigned,

we can compare students below the cut who were selected against those students who were below the cut but not selected.

• Teacher groups can also be compared directly (volunteers selected

randomly Vs. volunteers not selected)

The Pre-Test

Grade 3

a = .74

Pre-Test

Grade 4

a = .77

Pre-Test

Grade 5

a=.74

Pre-Test SBAC Correlation

Assessment Development Process

• Standards Selection
• Item Design
• Assessment Design
• Assessment Timeline

Standards Selection

www.achievethecore.org

Standards Selection

www.achievethecore.org

Standards Selection

• Backwards Map

Major standards

• Frequency of

related standards

www.achievethecore.org

Standards 3rd to 4th Gd 3OA1 3OA3 3OA5 3OA7 3NBT2 3NF1 Standards 4th to 5th Gd 3OA1* 3OA2* 3OA6* 4OA1 4OA2 4NBT4 3NF2* 4NF1 4NF4 4NF7 Standards 5th to 6th Gd 3OA5* 4OA2* 4OA4* 5OA2 5OA3 5NBT6 5NF4 5NF7 5MD2 5G2

*indicates out of sequence standards

Standards Selection

Standards Selection

Standard/Set DOK Type Claim/Target OA1 2 MS C1* OA1, OA3 2 MC 2/C OA1, OA3 2 EQ 2/C 0A1, OA3 3 EQ 2/C OA3 1 EQ 1/A OA5 1 EQ 1/B OA5 2 MC 3/D OA7 1 EQ(2) 1/C OA7 1 EQ(2) 1/C OA7 1 MS 1/C OA7 1 TM 1/C NBT2 1 TM 1/E NBT2 2 EQ 2/A NBT2 2 EQ 2/A NF1 1 MC 1/F NF1 2 TM 3/F STANDARD DOK TYPE Claim/Target 3OA1 + OA-A 2 MC 2/C 3OA1 + OA-A 2 EQ 2/C 3OA1 + OA-A 3 EQ 2/C 0A1, OA2 2 MS 4/ C OA2 1 EQ 1/A OA2 1 EQ 1/A NBT4 1 EQ 1/E NBT4 1 EQ 1/E NBT4 2 MC 2/C NF1 1 MS 1/F NF1 2 MC 3/F NF4 2 TM 3/D NF4 1 EQ 1G NF7 2 TM 1/H MD2 2 EQ 1I MD2 2 EQ 1I STANDARD DOK TYPE Claim/Target 4OA1, OA2 2 MS 3/ 4OA2 1 EQ 1/A 4OA2 1 EQ 1/A OA2 2 MC 1/A OA2 1 mc 1a OA3 2 mc 1B OA3 2 MC 1/B NBT6 1 EQ 1/D NBT6 1 mc 1d NF4 1 mc 1f NF4, NF7 2 EQ 2/ C NF7 3 EQ 2a MD2 2 eq 1/ H MD2 2 eq 1/H G2 1 mc 1/J G2 3 mc 2/C

Grade 3 Grade 5 Grade 4

Item Design

• Task Models from

Smarter Balanced Item Specifications

Item Design

Written Item Authored Item

Assessment Design

• 3 Separate Tests
• 3rd to 4th
• 4th to 5th
• 5th to 6th
• 16 items each
• 3 items are the same across two grades
• 11 items use parallel items on each Form (same Task Models)
• Emphasis on DOK 2 & 3 (start test with 3 DOK 1 items)
• Representative mix of Claims, Targets, & Task Models

Assessment Design

3 Parallel Forms for each of the 3 Tests

• Screening Test (ST) at end of 17-18 school year (Form A)
• Pre-Test at beginning of Math Camp (Form B)
• Post Test at beginning of 18-19 school year (Form C)

2018 SMC Testing Schedule Test Name Administered Measuring Purpose SMC Screening Test Last Half May CS Knowledge SMC Student Selection Summer Baseline SMC Pre-Test 1st Day of SMC July 20, 2018 CS Knowledge SMC Baseline Mid Summer Loss SMC Post Test August 20 to 24 CS Knowledge SCM to End of Summer Change Summer Loss

Assessment Timeline

Recruitment of Teachers

3rd Grade 8 Teachers 4th Grade 7 Teachers 5th Grade 10 Teachers 6th Grade 7 Teachers 11/12 ES 3/4 MS*

Recruitment of Student Candidates

Recruitment

• f Student

Candidates

Grade Level % Correct on Assessment # of Students Needed # of Students Invited 3rd 37.5 25 33 31.3 15 20 25 8 10 18.8 3 3 3rd Totals 51 66 4th 37.5 25 33 31.3 15 20 25 8 10 18.8 3 3 4th Totals 51 66 5th 37.5 35 46 31.3 21 27 25 11 14 18.8 4 5 5th Totals 71 92 Grand Totals 173 221

Recruitment of Student Candidates

Confirmation of Student Participants

Forming Classes for Summer Math Camp

Mathematical Practices and Concepts

• Ratio of 1 Teacher to approximately 4-7 Students
• Collaborative Cognitively Demanding Mathematical Tasks
• Sequenced throughout week to build on & expand students’ knowledge
• Multiple correct answers and entry points
• Opportunities for students to showcase their thinking
• Promotes growth mindset

Social Emotional Development

• “Fresh Start” morning activities to develop students’ collaboration &

team-building skills

○

Trust developed and lesson learned during “Fresh Start” activities benefit students during the cognitively demanding tasks later in the day

• Mindfulness exercises to develop students’ techniques for positive self-

talk and dealing with stress & anxiety

○

5 minute mindfulness exercises after breaks help students refocus & be ready to learn

Schedule

Questions?