What is School Quality? The Potential of EWS to Inform School - - PowerPoint PPT Presentation

What is School Quality?

The Potential of EWS to Inform School Accountability Under ESSA

Presented at NCSA June 28, 2018 by Jurupa Unified School District Presentation Team Jurupa Valley, California

Jurupa Unified School District

K-12 Enrollment = 19,112 SED = 82% English Learners = 31% Hispanic Latino = 86% White = 9% African American = 2% Asian = 1%

Presentation Team

Elliott Duchon, Superintendent JUSD, educhon@jusd.k12.ca.us Jay Trujillo, Director, Secondary Education JUSD, jtrujillo@jusd.k12.ca.us Pete Goldschmidt, PhD, Research Consultant, pete.goldschmidt@csun.edu Sandy Sanford, EdD, Assessment Consultant, sandy@youasksandy.com Heather Goodwin, Assessment Consultant, good1hlg@gmail.com

Presentation Plan

• Slides (PPT posted on NCSA 2018 web site)
• Pete Videos available by requesting from Sandy’s Email

sandy@youasksandy.com

• EWS vs. EEWS

What Is School Quality:

The Potential of EEWS to Inform School Accountability Under ESSA

Pete Goldschmidt, PhD. National Conference on Student Assessment June 28, 2018

Pete Video Here

Definition

Extended Early Warning System

• Most Early Warning Systems predominantly use data from

middle and or high school.

• Extended refers to both greater depth and breadth in

data utilized.

• Using data from elementary school
• Using student responses to a school bonding survey.

EEWS—Behind the Scenes

EEWS Data Base

Modeling Module Modeling Module Modeling Module Modeling Module Modeling Module Modeling Module Modeling Module Modeling Module

Flags VAM

User Interface Processing Information

What does EEWS User Interface Look Like?

Four Student Factors

• Student Background
• Academics
• Behavior
• Social Emotional

ü Student Information System ü Assessment Data ü Annual Student Survey

Four Student Factors—Student Background

• Characteristics such as SES, SWD, Language, etc.
• Family Support
• My family expects me to attend college
• An adult at home asks about my homework and

classwork.

Four Student Factors—Academics

• Assessment Results
• English Language Development
• Course Credits
• F Grades
• On Track

Four Student Factors—Behavior

• Absences
• Absences first month of school
• Discipline*
• Suspension

Four Student Factors—Social Emotional

• Locus of Control
• School Bonding
• School Culture

Four Student Factors—Social Emotional

• Locus of Control
• “Locus of control (LoC) orientation is a belief about

whether the outcomes of actions are contingent on what we do (internal control orientation) or on events outside

• ur personal control (external control orientation)."

(Zimbardo, 1985). Some evidence suggests that context influences LoC and that LoC is malleable.

• When I study hard, I do well.
• I am good at math
• My success in math is completely up to me.
• I can be myself at this school.

Four Student Factors—Social Emotional

• School Bonding
• My teacher cares about me
• I have friends who help me at school
• I feel respected by other students

Four Student Factors—Social Emotional

• School Culture
• I see my principal in my classroom
• I feel safe at this school
• I have been called mean names, made fun of, hit, or

pushed at school

Flags

• Markers pre-programmed on the EEWS database to compile what is

determined to be meaningful information—30 so far

• For example,

ü5 or more absences during first 30 school days üLocus of Control üSchool Year Off Track üPrior Year Fs üMiddle to High School Absence Change üTeacher Cares

How can we check to see whether or not any particular Flag is ACCURATE or MEANINGFUL?

Pete Video Here

Verifying Flags

• Literature generally applies a 75/10 “rule”.
• 75% students with the outcome have the flag.
• 10% of students identified by flag, have the outcome.
• Relative Risk.
• Conditional Effect.
• Health science literature slightly different, yet similar, approach.
• Sensitivity – Ability to test positive if have outcome (10)
• Specificity- Ability to test as outcome “free”
• PPV – Probability of outcome when test positive (75)
• If around 90% very strong Flag.
• NPV – Probability do not have outcome with no Flag

Verifying Flags

• Early Warning Flags generally do not perform as well as

medical tests (although many medical tests do not perform nearly as well as most people think).

• The relative importance of misclassifications depends on

existing/intended/(to be) developed strategies.

• e.g., School-wide strategies impact all students – both

those for whom it is intended and others Vs. Specific student centered strategies.

Classroom Impact on Math Scores

School Bonding and School Performance – Grade 6

Percent of Students at each School with Flag

0 % 1 0 % 2 0 % 3 0 % 4 0 % 5 0 % 6 0 % 7 0 % 8 0 % 9 0 % 1 0 0 %

4 6 1 4 6 1 5 4 6 1 1 4 6 1 1 5 4 6 1 2 4 6 1 2 5 4 6 1 3 4 6 1 3 5 4 6 1 4 4 6 1 4 5 4 6 1 5 4 6 1 5 5 4 6 1 6 4 6 1 6 5 4 6 1 7 4 6 1 7 5 4 6 2 4 6 2 5 4 6 2 1 4 6 3 4 6 3 5 4 6 3 1 4 6 4 4 6 4 5 4 6 4 1 4 6 4 3 4 6 5 4 6 5 2 9 4 6 6 T

• t

a l

First3 5 Flag 5 T

• tAb

sF l g An y_ Di sc Flag Susp e n _ FLG FsOff_ T rac k T y Of f_ T ra ck iStu dy_ Flag iR e ad_ Flag iM a t h_ F l a g Sc h

• l_

F l a g LoC _ Flag An y_ Flag

EEWS Supporting Jurupa Programs

The # of Students Failing and NOT Graduating was Growing

Di District Crisis

Accelerating Academic Achievement

9th

th Gr

Grade AAA AAA Ac Academy

• Unprecedented
• Unpopular
• Costly

Big Deal

Evaluation of the AAA Program

In real-time

& On-going

EEWS Support of Evaluation

• Created Meaningful Comparison Group
• Determined most effective entrance criteria
• Determine early grades predictors of AAA candidates
• Measured long-term effects (does the program have legs?)

AAA Evaluation Results

• In all measured area AAA students outdid control group
• Attendance
• Discipline
• Academics
• BUT, the improvement gap narrowed after the treatment
• year. This led to improved post treatment support.

N O M OR E D .R .I .P .

Pete Video Here

Summary

• Identify relevant outcomes
• Identify potential malleable indicators that may be related to
• utcomes.
• Examine extent to which indicators differentiate between students

with and without the flag.

• Determine whether and how to combine various indicators into a

composite.

• Examine extent to which school level composite works within

accountability system (see Handbook).

Questions?