Theories and Actions for Reporting and Reducing Achievement Gaps: - - PowerPoint PPT Presentation

Theories and Actions for Reporting and Reducing Achievement Gaps: Framework and Examples

Brian Gong, Center for Assessment Mary Batiwalla, Tennessee Department of Education Jennifer Stafford, Kentucky Department of Education

CCSSO National Conference on Student Assessment June 27, 2018 San Diego

Theories and Actions for Reporting and Reducing Achievement Gaps

Brian Gong

Drawing on work done with Leslie Keng, Erika Landl, and Juan D’Brot

Center for Assessment

Presentation in the session on “Theories and Actions for Reporting and Reducing Achievement Gaps: Framework and Examples,” CCSSO National Conference on Student Assessment June 27, 2018 San Diego, CA

Overview

Central messages

• Reports and reporting will be more effective when the

state develops and improves a compelling theory of change that coherently merges theories of action for different agencies/groups.

• Reports of achievement gaps/closure must carefully

consider the comparison group, the achievement metric, and the gap/closure criteria to be sure they fit the theory

• f change.
• A theory of change should go beyond reporting systems

that identify problems to actual provision of effective

• supports. The state must carefully identify what the state

can do to reduce achievement gaps and what others should do.

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 3

Reports and Reporting

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 4

Reporting and a theory of change/use

• Reports are a powerful, essential part of a state’s construct

definition and theory of change about improving education, e.g., achieving goals related to equity

• What is reported (and how and when), to whom, how that

information is intended to be interpreted, and especially how that information is intended to be used are all part of the state’s plan for using assessments to improve equitable outcomes

• The state should have a detailed, plausible theory of

change that specifies actions for results to achieve goals

• The state should update its theory of change regularly,

based on evidence and sound advice

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 5

Reporting system

• Reports – displays of data with intended

interpretations and uses

• Reporting – actions taken to support intended

interpretations and uses (e.g., communication plan to “tell your interpretation,” trainings to help people make appropriate use of data)

• Reporting system – like an assessment system, the

state should have a reporting system consisting of policies, plans, practices, personnel, and infrastructure to provide capacity and continuity; the effectiveness of the reporting system (e.g., on improving equity) should be evaluated to improve

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 6

Equity and achievement gaps/closure

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 7

Equity and Achievement Gaps

• Deciding why attending to equity is important
• Defining equity and achievement gaps
• Measuring and reporting achievement gaps
• Informing reduction of achievement gaps

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 8

Definitions

• Excellence

– Standards-based excellence (individual student): A rigorous performance criterion related to comprehensive, relevant content standards – Standards-based excellence (groups of students): A rigorous performance criterion involving all or virtually all students in an identified student group – Excellence involves both common (all students) and individual aspects

– Excellence may also be defined in other ways (e.g., normative)

• Equity

– Equity as fairness implies that personal or socio-economic circumstances, such as gender, ethnic origin or family background are not obstacles to educational success – Equitable education systems are fair and inclusive and support their students to reach their learning potential without either formally or informally pre-setting barriers or lowering expectations

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 9

Types of Equity and key issues

• Equitable to whom – be clear about value choices
• Equity in outcomes

– Equity in outcomes = converging on Excellence for all? vs. Relative Excellence, growth, etc.

• Equity in opportunities – providing access to opportunities

designed to lead to more equitable outcomes

– Equal vs. equitable

• Relationships of equity in opportunities to equity in outcomes

– Intermediate outcomes (pre-requisites) – Processes/actions that produce outcomes (how) – Relation with distal outcomes (goals through theory of action)

• How time and equity are related

– Often concerned about “change over time”; equal time; more time for

• pportunities

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 10

Equity in outcomes or goals

• Comparable in some dimension(s)

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 11 End point: same End time: same Slope: different End point: different End time: same Slope: same, absolute End point: same End time: different Slope: same, absolute End point: different End time: same Slope: same (same proportion, e.g., “reduce non-proficient by 10%”) End point: different End time: same Slope: same (same self- relative, e.g., “50th percentile, for own reference group”) End point: % proficient same End time: same Distribution/variance: same, less, more End point: different End time: different Slope: different

Checking for gap reduction

• Status/Improvement

– Goal-based: comparison of student group performance to goal – Group-based: comparison of one student group to another group

• 5 ways a gap may be reduced (status)

– Higher, Lower groups both increase; Lower increases more – Higher stays same (ceiling?), Lower increases – Higher declines, Lower stays the same – Higher, Lower both decline; Higher declines more – Any of above, but statistically significant change for time period occurs only for gap reduction

• Growth

– Cautions when conditional (e.g., value-added) growth – Goal-based – Group-based

• How much is “good enough”? How can you tell?

– Criteria (e.g., goals) – Statistical tests of differences

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 12

Equity as factors affecting fairness of

• pportunity and access
• Equity as opportunities and access

– Opportunities and access

• School offers/student takes
• School/district actively supports

– Intermediate outcomes (“leading indicators”)

• Example: “Pipeline of opportunity”

– Schools have 100% of all student groups meet Proficiency goals of scores on state high school math assessments à Schools offer advanced math courses necessary to prepare students for Alg. 2 EOC à Schools have teachers who are effectively teach math à Students enroll in advanced math courses proportional to student group enrollment à Students enroll in preparatory math courses proportional to student group enrollment, by grade 9 à Schools support students to be on-track each grade, etc.

• Example: “Early warning and RtI systems”

– School monitors student performance and provides effective support well before a student fails the course or drops out (e.g., Attendance/Behavior/Credits) – District provides resources equitably across schools (e.g., effective leadership, effective teachers, coherent curriculum across grades, special services, community engagement) – State helps districts implement district/school/classroom early warning and RtI systems that inform action well before state accountability system’s severe ID & consequences

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 13

Equity as underlying quality of performance

• For equity measures to be effective, they must

reflect appropriately comparable underlying quality

– Be sure users understand any normative/conditional measures – Triangulate multiple measures (especially growth and status; equity process and equity outcomes) to confirm you are getting what you really want

• Pay special care to quality of local measures
• Pay special care to quality of measures that

transform or cross lines from one main user or accountability unit to another (e.g., cross grades; from high to post-high school) (one type of “lagging indicator”)

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 14

Sample report focusing on coherence of quality (e.g.,

rigor) between local school and external measures

Sch. ID

Pct High School Gradu- ates

(2012-13)

• Avg. HS

GPA Pct enroll in college (2013-14) Pct met CCR require- ments (e.g., ACT) Pct placed in remedial course(s) Avg. credit hrs earned (met CCR) Avg. credit hrs earned (did not meet CCR)

275179

96.3 2.83 70.3 39.0 18.3 10.9

531040

96.1 3.24 63.8 44.9 23.9 3.4

535040

97.1 3.11 61.4 92.2 20.0 10.6

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 15

Local Measures External Measures Claim: “If you graduate from our school with good grades, you will be prepared to do well in college.”

Including Equity in Accountability

• Equity of outcomes

– Goals and performance and “good enough” – Disaggregated student group outcomes affect accountability ratings

• Met goals/”good enough” criteria
• If <gap measure> then cannot be higher than <certain> rating
• If <gap measure> then identified for <consequence>
• Equity of opportunity and access

– Included in School Quality/Student Success indicator

• Included in other than school accountability (e.g.,

student, teacher, district; accreditation process)

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 16

Reporting equity in addition to accountability

• Reasons to report measures associated with equity

when those measures are not included in the formal state accountability rating:

– Provides valuable information to inform analyses of “why” to inform program evaluations and associated policies by researchers, schools, districts, or state (e.g., needs analyses, improvement plans, support and assistance plans) – Provides valuable information to inform local actions based on public reporting accountability

• May spur improvement in data quality

– Can help inform theories of action at the state and local levels

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 17

Theory of Action/Change

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 18

Concepts and Definitions – theory of action

• A theory of action is a statement of how actions are

expected to lead to results to accomplish goals

• A theory of action includes:

– An empirically and logically stated argument about actions and results – A set of underlying assumptions – A testable hypothesis (revisable “theory,” given the evidence) …that outlines how and why a given assessment, system, or program, as designed, will support the achievement of specified goals. It requires specification of each component of the assessment/reporting/ evaluation system, the connection between components, and the manner in which they jointly fulfill the requirements of the system.

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 19

Concepts and Definitions – theory of change

• A theory of action focuses on one agent (e.g., state

DOE, state assessment)

• A theory of change focuses on interactive system of

theories of action by multiple agents needed to accomplish the goals

• A logic model provides specific actions for all the key

steps to produce a result

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 20

Multiple agents à actions à results à accomplish goals § state <issues reports> which the school will <use by action x> which will <lead to student results y> which will <help achieve goal z> § state assessment unit <issues assessment data> which state accountability unit uses to <issue accountability reports> which state support unit uses to <inform actions> with district/school x <to lead to changes y> which <help achieve goal z>

Example theories of change – detailed

• Goal: “All high schools will have a minimum

graduation rate of 98% by year 2030”

• Possible theories of change

– The state will help schools improve graduation rates by decreasing the number of student who dropout by <action>:

• Report to schools their graduation/dropout rates, because they

don’t know; when they know, they will act and improve the rates

• Report graduation/dropout rates publicly to galvanize

policymakers, community, parents, students to push the schools/ districts to make improving graduation rates a priority and to act

• Report to help monitor districts’ support systems (e.g., to act

more effectively by identifying, monitoring, reporting, and acting

• n research-based dropout early warning systems)

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 21

Dropout “early warning systems”

• Systems of key indicators, reports, and actions

implemented to identify students at risk of dropping out and providing appropriate supports

• ABC’s [key indicators] of Dropout Early Warning Systems

– A—Attendance Research has long shown a strong relationship between how often a student misses school and his or her probability of graduating in four years. Most students who do not regularly attend class fall behind in their coursework and consequently see their grades suffer. Poor attendance can also indicate that a student is struggling with health, family, or other issues that are distracting them from their studies. – B—Behavior incidents As few as one suspension in grade 6 may predict whether a student graduates in four years (Balfanz et al., 2007). Behavior incidents can indicate that a student is disengaged with the school environment. Suspensions often cause an additional burden on students to catch up on the material they missed. – C—Course performance Number of course failures and overall grade point average

• bviously correlate with a student’s probability of graduating in four years. If a student

fails a course, he or she will need to make up the credit outside the regularly scheduled school time to stay on track. Poor course performance can also indicate disengagement at the classroom level.

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 22

Source: Frazelle, S. & Nagel, A., 2015

Dropout “early warning systems” – 2

• Theory of action for keeping students on track to graduation/

reduce dropouts (early warning system + response to intervention)

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 23

Type of intervention Focus of intervention

Attendance Behavior Course performance

Schoolwide

(all students)

• Every absence brings a response
• Create a culture that says attending

every day matters

• Positive social incentives for good

attendance

• Data tracking by teacher teams
• Teach, model, and expect good

behavior

• Positive social incentives and

recognition for good behavior

• Advisory
• Data tracking by teacher teams
• Research-based instructional

programs

• In-classroom support to enable

active and engaging pedagogies

• Data tracking by teacher teams

Targeted

(15–20 percent of students)

• Two or more unexcused absences

in a month brings brief daily check by an adult

• Attendance team (teacher,

counselor, administrator, parent) investigates and problem solves (why isn’t student attending?)

• Two or more office referrals brings

involvement of behavior team

• Simple behavior checklist students

bring from class to class, checked each day by an adult

• Mentor assigned
• Elective extra-help courses—tightly

linked to core curriculum—preview upcoming lessons and fill in knowledge gaps

• Targeted, reduced class size for

students whose failure is rooted in social or emotional issues

Intensive

(5–10 percent of students)

• Sustained one-on-one attention

and problem solving

• Appropriate social service or

community support

• In-depth behavioral assessment

(why is student misbehaving?)

• Behavior contracts with family

involvement

• Appropriate social service or

community supports

• One-on-one tutoring

Treatment

(quality of actual implementation)

Monitoring & Improvement Diagnosis Prescription

Source: Frazelle, S. & Nagel, A., 2015

Summary

• Have reports and reporting systems
• Define excellence, equity carefully
• Choose metrics of achievement gap and gap closure

according to your theory of change (e.g., in defining and portraying gaps/closure, what is similar and what is different; why did you choose that—how will it lead to better action)

• Theory of action and theory of change: what and

who (state and others) should be done to achieve goals (diagnosis, prescription, treatment, monitoring)

Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18 24

For more information: Center for Assessment www.nciea.org

Brian Gong bgong@nciea.org

25 Reducing achievement gaps - Gong - CCSSO NCSA - 6/27/18