Development of a Smarter Balanced Assessment to Assess Grade-Level - PowerPoint PPT Presentation

Development of a Smarter Balanced Assessment to Assess Grade-Level Readiness and Summer Learning Loss in Math Presented at NCSA June 29, 2018 by Val Verde Unified School District Presentation Team Perris, California

Presentation Team Michael McCormick, Superintendent VVUSD mmccormick@valverde.edu Jennifer Doskocil, Coordinator, Elementary Education VVUSD jdoskocil@valverde.edu Pete Goldschmidt, PhD, Research Consultant pete.goldschmidt@csun.edu Sandy Sanford, EdD, Assessment Consultant sandy@youasksandy.com Heather Goodwin, Assessment Consultant good1hlg@gmail.com

Presentation Plan • Slides (PPT posted on NCSA 2018 web site) • Pete Video available by request from sandy@youasksandy.com • SMC – Summer Math Camp

Project Phases Phase I—Summer Math Camp (April to August 2018) Phase II—Analysis of 2018-19 data to determine effects/adjustment Phase I and II results Reported at NCSA 2019 Phase III—Summer Math Camp in summer 2019 Phase IV—Analysis of 2019-20 data to determine effects/adjustment Phase III & IV results Reported at NCSA 2020

The Project Objectives 1. To assess grade level readiness 2. To measure summer learning loss Collateral Phase I Objective 3. Evaluate the efficacy of the SMC Teacher Professional Development in increasing #1 and decreasing #2

The Plan for SMC Phase I • Determine Critical Standards for 3 rd to 4 th , 4 th to 5 th , & 5 th to 6 th (April) • Recruit 32 SMC Teachers—Recruit Candidates & Randomly Choose (April) • Create 16 item Smarter Balanced Aligned Screening Test (ST) for each Grade • Administer ST to as many students as possible during last two weeks of May • Use cut points established on the ST to identify Candidates for SMC (May-June) • Analyze ST (June) • Create 2 parallel Tests—SMC Pre-Test and a beginning of school year Post Test • Use lottery system to choose SMC attendees • Create Professional Development for SMC Teachers (CSUN & MindfullyU) • SMC to include administering Pre-Test (20 to 26 July) • Administer Post Test during 1 st week of 18-19 school year • Analyze combined data from the 3 Tests

Liaison with CSUN & MU CSUN & MU CSUN & MU Research Team Analyzes Efficacy of PD CSUN & MU Deliver & Provide After- Develop PD for PD Monitor PD Action Rpt Recruit SMC Select SMC Teachers Receive Analyze Series of Teachers Teachers PD & 3 Tests (Random) (Randomly) Instruct SMC Develop/ Analyze Develop/ Develop/ Administer Screening Test Administer Pre- Administer Post Screening Test Test Test Select Identify SMC Student Identify Students Attend SMC Student Candidates SMC Student SMC (Non-Random) Candidates Candidates Prepare Clean/ReConfig Prepare/Use Select Facilities & Facilities/ Facilities/ Facilities/ Support Support Support Support May June July August

SMC 2018 in a nutshell • 32 Teachers—3 rd , 4 th , 5 th , & 6 th • Each teacher… • Dedicated Classroom • 4 to 7 students same grade level • Students chosen from those score RS 6, 5 , 4, & 3 of Screening Test • 20 July—Teachers meet students, activities, Pre-Test in AM • 20 July—Teachers receive PD from CSUN & MinfullyU in PM • 23 to 26 July-–Teacher teach students AM & receive PD PM

Development of a Smarter Balanced Assessment to Assess Grade-Level Readiness and Summer Learning Loss in Math Pete Goldschmidt, PhD. National Conference on Student Assessment June 29, 2018

Pete Video

Development of a Smarter Balanced Diagnostic Assessment to Assess Grade-Level Readiness and Summer-Learning Loss in Math Pete Goldschmidt, Ph.D. National Conference on Student Assessment June 29, 2018

Summer Learning Loss • Students demonstrate consistent patterns of loss over the summer • Less exacerbates achievement gaps because students begin the school year behind, catch up, and subsequently do not retain what they have learned and are then starting the next year behind (again). • Problem particularly problematic for low SES and non-native English speakers because they tend not to have access to the same resources and/or academic content in English over the summer.

Stylized Example of Summer Learning Loss and Achievement Gap Increase 120 100 80 60 Co m p ar i son Sub gr oup 40 20 0 K Begi n K End 1st gr a de 1st gr a de 2nd gr ad e 2nd gr ad e 3r d gr ade 3r d gr ade 4t h gr ade 4t h gr ade Beg i n End Beg i n End Beg i n End Beg i n End

Summer Learning Loss Impact • Some students enter next grade level not prepared for grade-level material • Teacher spends time assessing student’s readiness to learn. • Teacher spends time re-teaching. • Reinforces potential for gap.

Motivation for Study • Study based on 3 rd through 6 th grade students. • Accurately assess summer Learning Loss. • SBAC not amenable for assessing summer loss because reflects both summer loss and school-ear gains. • Developed short assessment given closer to end of the year and beginning of subsequent year. • More quickly asses grade-level readiness. • Identify individual and group-level readiness. • Identify students with significant loss.

Motivation for Study • Assessments used to evaluate summer learning loss and readiness part of item bank available for use by teachers. • Create and evaluate summer math intervention. • Summer Math Camp. • Details to follow.

Study Design • Solicit teacher volunteers for math camp. • Randomly select teachers for treatment condition. • Non-selected volunteers become counterfactual. • Use Pre-test to invite student participants. • Given too many students scored below cut-score, randomly select students for participation. Achievement Level Scale Score Range for Grade Summer Math Standard Not Camp Pre-Test Met Cut Score 3 2189–2380 2379 4 2204–2410 2401 5 2219–2454 2424

Research Questions • What is the overall loss during the summer? • What is the level of readiness? • Do students who participated in the Summer Math Camp have a smaller summer learning loss compared to students who did not participate? • Are students who participate in the Summer Math Camp better prepared for the next year compared to students who did not participate? • Do students who participate in the Summer Math Camp perform better on SBAC compared to students who did not participate?

• Do students who have a teacher who participated in the Summer Math Camp perform better on SBAC than students who do not have teachers who participated in the Summer Math Camp? • Students who participated in the Summer Math Camp. • Students who did not participate in the Summer Math Camp.

Research Design • Given that students are assigned to the Summer Math Camp based on a cut score we can use a Regression Discontinuity (RD) design to compare students just on either side of the cut score. • Given that student who were below the cut were randomly assigned, we can compare students below the cut who were selected against those students who were below the cut but not selected. • Teacher groups can also be compared directly (volunteers selected randomly Vs. volunteers not selected)

The Pre-Test Grade 3 a = .74

Pre-Test Grade 4 a = .77

Pre-Test Grade 5 a =.74

Pre-Test SBAC Correlation

Assessment Development Process • Standards Selection • Item Design • Assessment Design • Assessment Timeline

Standards Selection www.achievethecore.org

Standards Selection www.achievethecore.org

Standards Selection • Backwards Map Major standards • Frequency of related standards www.achievethecore.org

Standards Selection Standards Standards Standards 3 rd to 4 th Gd 4 th to 5 th Gd 5 th to 6 th Gd 3OA1 3OA1* 3OA5* 3OA3 3OA2* 4OA2* 3OA5 3OA6* 4OA4* 3OA7 4OA1 5OA2 3NBT2 4OA2 5OA3 3NF1 4NBT4 5NBT6 3NF2* 5NF4 4NF1 5NF7 4NF4 5MD2 4NF7 5G2 *indicates out of sequence standards

Standards Selection Grade 4 Grade 5 Grade 3 Standard/Set DOK Type Claim/Target STANDARD DOK TYPE Claim/Target STANDARD DOK TYPE Claim/Target OA1 2 MS C1* 3OA1 + OA-A 2 MC 2/C 4OA1, OA2 2 MS 3/ OA1, OA3 2 MC 2/C 3OA1 + OA-A 2 EQ 2/C 4OA2 1 EQ 1/A OA1, OA3 2 EQ 2/C 3OA1 + OA-A 3 EQ 2/C 4OA2 1 EQ 1/A 0A1, OA3 3 EQ 2/C 0A1, OA2 2 MS 4/ C OA2 2 MC 1/A OA3 1 EQ 1/A OA2 1 EQ 1/A OA2 1 mc 1a OA5 1 EQ 1/B OA2 1 EQ 1/A OA3 2 mc 1B OA5 2 MC 3/D NBT4 1 EQ 1/E OA3 2 MC 1/B OA7 1 EQ(2) 1/C NBT4 1 EQ 1/E NBT6 1 EQ 1/D OA7 1 EQ(2) 1/C NBT4 2 MC 2/C NBT6 1 mc 1d OA7 1 MS 1/C NF1 1 MS 1/F NF4 1 mc 1f OA7 1 TM 1/C NF1 2 MC 3/F NF4, NF7 2 EQ 2/ C NBT2 1 TM 1/E NF4 2 TM 3/D NF7 3 EQ 2a NBT2 2 EQ 2/A NF4 1 EQ 1G MD2 2 eq 1/ H NBT2 2 EQ 2/A NF7 2 TM 1/H MD2 2 eq 1/H NF1 1 MC 1/F MD2 2 EQ 1I G2 1 mc 1/J NF1 2 TM 3/F MD2 2 EQ 1I G2 3 mc 2/C

Item Design • Task Models from Smarter Balanced Item Specifications

Item Design Written Item Authored Item