Final Evaluation Results of The Milwaukee Community Literacy - - PowerPoint PPT Presentation

Final Evaluation Results of The Milwaukee Community Literacy Project/ SPARK Early Literacy

SPARK/MCLP Evaluation 1

Project funded by a Department of Education Investing in Innovations (i3) grant. Contract number U396C100694.

Curtis Jones, SREed, University of Wisconsin, Milwaukee Michael Christian, Education Analytics Andrew Rice, Education Analytics

March, 3rd, 2016 Presented to the Annual Meeting of the Society for Research on Educational Effectiveness, Washington DC

Literacy Context in Milwaukee

SPARK/MCLP Evaluation 2

The need for increased literacy development opportunities for Milwaukee students is urgent. According to the 2013 National Assessment of Educational Progress (NAEP) 15% of 4th grade MPS students were At or Above Proficient in reading. This need is even more pronounced for low-income and minority students.

• 38% of 4th grade, White MPS students were proficient in reading, compared to 9% of

Black and 14% of Hispanic students.

• 11% of 4th grade low-income (free/reduced lunch participants) MPS students were

proficient in reading, compared to 39% of non-low-income students.

History of SPARK Early Literacy

2005 - SPARK was created to address this need by Boys & Girls Clubs of Greater Milwaukee and piloted at one site in Milwaukee Public Schools (MPS). 2006 - SPARK was expanded to three MPS schools with funding from the United Way and AmeriCorps. 2010 - SPARK received a Department of Education investing in innovation (i3) grant award to expand to seven more schools (10 total).

3 SPARK/MCLP Evaluation

SPARK/MCLP Evaluation 4 Program: Situation: After-School Tutoring Family Support/Collaboration Rev: 12 16 15 Assumptions Context

Typical literacy interventions only focus on skill acquisition and ignore the reasons why students fall behind. The RTI model does not work when 90% of students are below proficiency in reading. Collaboration Ongoing mtgs with teachers, district and school leadership

Participation

One-on-one tutoring is provided during the school day in a print- rich environment.

Chronic absenteeism (missing more than 10% of school days) is a serious program in MPS and other urban districts. Many parents do not understand the importance of early grade participation.

The benefits observed in other literacy interventions have typically faded over time. Boys & Girls Clubs is in a unique position, however, to continue to work with students and families beyond SPARK.

Many districts face staffing and budget shortages that prevent them from using other reading interventions.

SPARK adds capacity to a school by supplying tutors and a parents liaison to work with students.

Inputs Outputs Outcomes

Family events; parents contacts; newsletters; home visits Monthly family events; 2 home visits; 2 contacts per month; monthly newsletters Students receive 30 minutes of tutoring, three times per week, for up to two years. Family engagement manager, parent liaisons, parents

SPARK Early Literacy Logic Model

Club after-school academic and enrichment programming Students participate in Club programming throughout elementary school.

The need for increased literacy opportunities in the Milwaukee area is urgent. According to the 2013 National Assessment of Educational Progress (NAEP) 15% of 4th grade MPS students were At or Above Proficient in reading.

Boys & Girls Clubs staff and programming (e.g. KidzLit)

This need is even more pronounced for low-income and minority students. 38% of 4th grade, White MPS students were proficient in reading, compared to 9% of Black and 14% of Hispanic students. 11% of 4th grade low-income (free/reduced lunch participants) MPS students were proficient in reading, compared to 39% of non-low-income students. Short Medium Long Activities

Sustained improvement in reading achievement; School success Families more engaged in and knowledgeable about student's education. Improved reading (Running Records) Improved reading achievement SPARK site manager (certified teacher), tutors, SPARK room, materials (bins) Project director, district liaison, teachers, evaluation, school and district leadership Ongoing SPARK PD & support Tutors observed 1x/ month; Staff receive PD. Improved literacy (PALS Assessment) Improved school performance

Collaboration and Support

5 SPARK/MCLP Evaluation

Support/Collaboration

Project director, district liaison, teachers, evaluation, school and district leadership Ongoing SPARK PD & support Tutors observed 1x/ month; Staff receive PD. Collaboration Ongoing mtgs with teachers, district and school leadership

Inputs

• Certified teacher in each site supports and supervises a cadre of

about six tutors.

• Tutors include AmeriCorps members, UWM preservice

teachers, and a small number of community members.

• Tutoring is done in a print-rich “SPARK room”
• Each site also has a parent liaison who would work with

SPARK families.

6 SPARK/MCLP Evaluation

After-School Tutoring Family

SPARK site manager (certified teacher), tutors, SPARK room, materials (bins) Boys & Girls Clubs staff and programming (e.g. KidzLit) Family engagement manager, parent liaisons, parents

Activities - Tutoring

• Students are pulled out of non-core classes and taken to

SPARK room three times per week for up to two years.

• Starts with familiar activity.
• Running Records assessment every third session.
• Word play - Students receive instruction using Word Sorts and

Making Words. Word Sorts involve students sorting words into categories to increase their understanding of sounds and letters. Making Words involves students using letters to make words so students learn how the sounds of language are put together.

• Reading a book at instructional level.
• Writing sentences - Elkonin boxes are a central piece of

SPARK writing and used to help students encode words.

• End with tutor read aloud.

7 SPARK/MCLP Evaluation

Participation

One-on-one tutoring is provided during the school day in a print- rich environment.

Outputs

Family events; parents contacts; newsletters; home visits Monthly family events; 2 home visits; 2 contacts per month; monthly newsletters Students receive 30 minutes of tutoring, three times per week, for up to two years. Club after-school academic and enrichment programming Students participate in Club programming throughout elementary school.

Activities

Activities – Family Engagement

The goals of family engagement is to keep families aware of student progress in SPARK, help families promote literacy at home, and address any school attendance issues that arise during the program.

• Monthly family events,
• 2 home contacts per month (email, in person, or by phone),
• 1 home visit per year, usually over the summer,
• Monthly newsletters sent home.

8 SPARK/MCLP Evaluation

Participation

One-on-one tutoring is provided during the school day in a print- rich environment.

Outputs

Family events; parents contacts; newsletters; home visits Monthly family events; 2 home visits; 2 contacts per month; monthly newsletters Students receive 30 minutes of tutoring, three times per week, for up to two years. Club after-school academic and enrichment programming Students participate in Club programming throughout elementary school.

Activities

Evaluation Design

The primary goal of the evaluation was to design a study that would meet What Works Clearinghouse (WWC) design standards without reservations.

• Studies have to use a randomized-control framework to qualify.

 576 students were randomly assigned to either the control (290) or SPARK (286) group by the evaluation team. Assignments were stratified by school and grade level within schools. ELL and IEP students were not eligible for study.

• Studies have to use valid and reliable outcome measures.

 NWEA Measures of Academic Progress (MAP) Reading Benchmark Assessment,  Phonological Literacy Screening (PALS),  Regular-school-day attendance.

• Studies have to meet WWC attrition and differential attrition standards.

SPARK/MCLP Evaluation 9

Overall Selection & Attrition Results

576 Students selected 389 (187 Moved) Final sample

• f b/w 367

and 381

SPARK/MCLP Evaluation 10

33.9% - 36.3% Overall Attrition Rate

Missing Data

Differential Attrition Results

286 Students 195 (91Moved) b/w 185-192 (3- 10 missing data)

SPARK/MCLP Evaluation 11

1.9% to 2.3% Differential Attrition Rate

290 Students 194 (96 Moved)

b/w 182-189 (5- 12 missing data)

SPARK Participants Control Students 32.9% - 35.3% attrition 34.8% - 37.2% attrition

What Works Clearinghouse (WWC) Attrition Standards

SPARK/MCLP Evaluation 12

Final Sample

Control SPARK Total Grade Level K-1st 72 66 138 1st-2nd 63 74 137 2nd-3rd 59 55 114 Race/Ethnicity Black 149 146 295 Hispanic 29 34 63 Other 16 15 31 Gender Female 96 100 196 Male 98 95 193 F/R Lunch Eligible No 9 8 17 Yes 185 187 372 IEP No 182 184 366 Yes 12 11 23 Total 194 195 389

SPARK/MCLP Evaluation 13

Evaluation Design

The primary goal of the evaluation was to design a study that would meet What Works Clearinghouse (WWC) design standards without reservations.

• Studies have to use a randomized-control framework to qualify.

 576 students were randomly assigned to either the control (290) or SPARK (286) group by the evaluation

• team. Assignments were stratified by school and grade level within schools. ELL and IEP students were

not eligible for study.

• Studies have to use valid and reliable outcome measures.

 NWEA Measures of Academic Progress (MAP) Reading Benchmark Assessment,  Phonological Literacy Screening (PALS),  Regular-school-day attendance.

• Studies have to meet WWC attrition and differential attrition standards.

 Study attrition results were within acceptable ranges. In addition, attrition was exogenous to the program. Thus, the evaluation should meet WWC attrition standards.

• The program has to be implemented as intended.

SPARK/MCLP Evaluation 14

MCLP/SPARK Implementation -Tutoring

SPARK/MCLP Evaluation 15

• SPARK students received an intensive

amount of services across the two program years while control group students received the “business as usual” reading instruction provided by MPS.

• The average SPARK student received

122.5 tutoring sessions (SD = 27.3).

• Nearly all students received more than the

minimum dosage of 90 sessions.

MCLP/SPARK Implementation – Average Tutoring Sessions by School

SPARK/MCLP Evaluation 16

123 118.6 140.1 125.4 122.4 117.7 114.4 122.5 School 1 (n=34; SD=32) School 2 (n=25; SD=29.6) School 3 (n=21; SD=15.4) School 4 (n=24; SD=40.8) School 5 (n=34; SD=23.8) School 6 (n=32; SD=19.4) School 7 (n=25; SD=17.4) Total (n=195; SD=27.3) Tutoring Sessions

MCLP/SPARK Implementation - Family

SPARK/MCLP Evaluation 17

• All schools sent home monthly newsletters to all SPARK students.
• All schools held monthly family SPARK events.
• Parents of SPARK students attended an average of three family events (Range 0 to 10,

SD = 2.2), received 32 parent contacts (Range 0 to 69, SD = 13.9) and had 2.4 home visits (Range 0 to 8, SD = 1.4).

• 47 out of 53 interviewed parents indicated that they received regular communications

from SPARK.

• All 53 indicated that they understood the components of the SPARK lesson plan.

MCLP/SPARK Implementation - Family

SPARK/MCLP Evaluation 18

3 65 53 31 25 13 3 2 1 2 3 4 5 6 7

Number of Home Visits

MCLP/SPARK Implementation – Family Average Home Contacts per Participant by School

SPARK/MCLP Evaluation 19

33.1 35.7 31.1 23.3 22.4 47.1 29.6 32 School 1 (n=34; SD=16) School 2 (n=25; SD=11.4) School 3 (n=21; SD=10.4) School 4 (n=24; SD=9.7) School 5 (n=34; SD=8.6) School 6 (n=32; SD=13.4) School 7 (n=25; SD=5) Total (n=195; SD=13.9) Home contacts

MCLP/SPARK Implementation – Family Number of Home Visits per Participant by School

SPARK/MCLP Evaluation 20

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% School 1 School 2 School 3 School 4 School 5 School 6 School 7 Total 1 2 3 4 5 6 7

Evaluation Design

The primary goal of the evaluation was to design a study that would meet What Works Clearinghouse (WWC) design standards without reservations.

• Studies have to use a randomized-control framework to qualify.

 576 students were randomly assigned to either the control (290) or SPARK (286) group by the evaluation team. Assignments were stratified by school and grade level within schools. ELL and IEP students were not eligible for study.

• Studies have to use valid and reliable outcome measures.

 NWEA Measures of Academic Progress (MAP) Reading Benchmark Assessment,  Phonological Literacy Screening (PALS),  Regular-school-day attendance.

• Studies have to meet WWC attrition and differential attrition standards.

 Study attrition results were within acceptable ranges. In addition, attrition was exogenous to the

• program. Thus, the evaluation should meet WWC attrition standards.
• The program has to be implemented as intended.

 Nearly all students and families received an intensive amount of tutoring and family engagement.

SPARK/MCLP Evaluation 21

Outcomes/Impact Analysis

SPARK/MCLP Evaluation 22

Short-term Outcome - Engaged Families

SPARK/MCLP Evaluation 23

7 5 26 23 36 41

Thinking back to the last 30 days, how often have you or another adult in the home read with your child? Thinking back to the last 30 days, how often have you or another adult in the home taught new words to your child?

Rarely About once a week Daily

Outcomes Short Medium Long

Sustained improvement in reading achievement; School success Families more engaged in and knowledgeable about student's education. Improved reading (Running Records) Improved reading achievement Improved literacy (PALS Assessment) Improved school performance

Short-term Outcome – School Performance (Attendance)

SPARK/MCLP Evaluation 24

32.8 27.4 25 28.7 29.8 22.8 20.7 24.6 K 1st 2nd Total Control SPARK

Unadjusted number of absences per student

Outcomes Short Medium Long

Sustained improvement in reading achievement; School success Families more engaged in and knowledgeable about student's education. Improved reading (Running Records) Improved reading achievement Improved literacy (PALS Assessment) Improved school performance

Short-term Outcome – School Performance (Attendance)

SPARK/MCLP Evaluation 25

Adjusted Impact of SPARK on Attendance (Number of Absences)

𝛾 SE t-stat p-value Unstandardized K-1st

• 5.6

4.5 1.23 1st-2nd

• 5.3

3.9 1.36 2nd-3rd

• 6.5

4.2 1.55 Overall

• 5.8

2.4 2.39 <.05 Standardized K-1st

• 0.20 0.16

1.23 1st-2nd

• 0.26 0.19

1.36 2nd-3rd

• 0.36 0.23

1.55 Overall

• 0.25 0.11

2.34 <.05

𝐵 = 𝛾𝑄𝐵 + 𝛾𝑄𝐵 + 𝛾𝐺𝑚ℎ + 𝛾

𝐾 =

ℎ𝑚 + 𝛾𝑄𝐵ℎ𝑚 + 𝜁

K – 1st and 1st – 2nd models 2nd – 3rd model

The pooled interaction term of SPARK participation and baseline PALS scores was not significant (t = 1.04, p > .05) suggesting that SPARK did not differentially impact students with different starting literacy levels. 𝐵 = 𝛾𝑄𝐵 + 𝛾𝐵𝑄ℎ + 𝛾𝐵𝑄 + 𝛾

𝐾 =

ℎ𝑚 + 𝜁

Short-term Outcome – Reading Levels

SPARK/MCLP Evaluation 26

57.5% 42.5% Below reading level At reading level 18.5% 81.5% Below reading level At reading level

Baseline Follow-up

Outcomes Short Medium Long

Sustained improvement in reading achievement; School success Families more engaged in and knowledgeable about student's education. Improved reading (Running Records) Improved reading achievement Improved literacy (PALS Assessment) Improved school performance

Short-term Impact – PALS

SPARK/MCLP Evaluation 27

18 2 8 13 Below Met

Post Benchmark Status of Students who Started Below Benchmark

Control SPARK

Outcomes Short Medium Long

Sustained improvement in reading achievement; School success Families more engaged in and knowledgeable about student's education. Improved reading (Running Records) Improved reading achievement Improved literacy (PALS Assessment) Improved school performance

Short-term Impact - PALS

SPARK/MCLP Evaluation 28

𝛾 SE t-stat p-value Overall K-1st 0.34 0.14 2.45 1st-2nd 0.55 0.16 3.54 2nd-3rd 0.16 0.16 0.95 Overall 0.36 0.09 4.05 <.001 Students in lower half

• f score distribution at

baseline K-1st 0.48 0.23 2.08 1st-2nd 1.14 0.27 4.24 2nd-3rd 0.14 0.40 0.34 Overall 0.66 0.16 4.11 < .001 Students in higher half

• f score distribution at

baseline K-1st

• 0.05 0.17 -0.29

1st-2nd 0.06 0.16 0.38 2nd-3rd 0.23 0.11 2.02 Overall 0.13 0.08 1.49 > .05

Adjusted Standardized Impact of SPARK on PALS scores

K – 1st and 1st – 2nd models 2nd – 3rd model

The pooled interaction term of SPARK participation and baseline PALS scores was significant (t = 5.61, p < .001) suggesting that SPARK differentially impacted PALS scores for students with different starting literacy levels. 5𝑄𝐵 = 𝛾𝑄𝐵 + 𝛾𝑄𝐵 + 𝛾𝐵𝑄 + 𝛾

𝐾 =

ℎ𝑚 + 𝛾𝑄𝐵ℎ𝑚 + 𝛾𝐵𝑄ℎ𝑚 + 𝜁 5𝑄𝐵 = 𝛾𝑄𝐵 + 𝛾𝐵𝑄ℎ + 𝛾𝐵𝑄 + 𝛾

𝐾 =

ℎ𝑚 + 𝛾𝐵𝑄ℎℎ𝑚 + 𝛾𝐵𝑄ℎ𝑚 + 𝜁

Medium-term Impact – Reading Achievement (MAP)

SPARK/MCLP Evaluation 29

Outcomes Short Medium Long

Sustained improvement in reading achievement; School success Families more engaged in and knowledgeable about student's education. Improved reading (Running Records) Improved reading achievement Improved literacy (PALS Assessment) Improved school performance

29.5 20.3 25.6 25.4 31.7 25.1 25.5 27.4 K-1st 1st-2nd 2nd-3rd Total Control SPARK

Medium-term Impact - MAP

b SE t-stat p-value Overall K-1st 0.18 0.13 1.34 1st-2nd 0.39 0.13 3.04 2nd-3rd 0.09 0.14 0.65 Overall 0.23 0.08 2.95 <.01 Students in lower half

• f score distribution

K-1st 0.23 0.24 0.94 1st-2nd 0.58 0.23 2.54 2nd-3rd 0.19 0.28 0.68 Overall 0.36 0.14 2.49 < .05 Students in higher half

• f score distribution

K-1st 0.11 0.18 0.60 1st-2nd 0.43 0.20 2.15 2nd-3rd 0.14 0.26 0.55 Overall 0.23 0.12 1.92 < .10

SPARK/MCLP Evaluation 30

Adjusted Standardized Impact of SPARK on 2015 MAP RIT Reading scores

K – 1st and 1st – 2nd models 2nd – 3rd model

The pooled interaction term of SPARK participation and baseline MAP scores was significant (t = 2.17, p < .05) suggesting that SPARK differentially impacted MAP scores for students with different starting achievement levels. 5𝐵𝑄 = 𝛾𝑄𝐵 + 𝛾𝑄𝐵 + 𝛾𝐵𝑄 + 𝛾

𝐾 =

ℎ𝑚 + 𝛾𝑄𝐵ℎ𝑚 + 𝛾𝐵𝑄ℎ𝑚 + 𝜁 5𝐵𝑄 = 𝛾𝑄𝐵 + 𝛾𝐵𝑄ℎ + 𝛾𝐵𝑄 + 𝛾

𝐾 =

ℎ𝑚 + 𝛾𝐵𝑄ℎℎ𝑚 + 𝛾𝐵𝑄ℎ𝑚 + 𝜁

Robustness of results: Impact on absences

SPARK/MCLP Evaluation 31

Measured impact of SPARK on absences is robust to alternative absence metrics

• Linear probability model with the LHS

variable an indicator for chronic (>10%)

• r severe chronic (>20%) absence
• Indicates that SPARK has a specific

effect at margins of chronic, severe chronic absence

• Results robust to probit
• While average effect is statistically

significant, joint hypothesis that individual grade effects are all zero is not rejected regardless of LHS variable B SE t-stat p-value Number of absences K-1st

• 5.6

4.5 1.23 1st-2nd

• 5.3

3.9 1.36 2nd-3rd

• 6.5

4.2 1.55 Overall

• 5.8

2.4 2.39 <.05 Chronic absence (>10% absent) K-1st

• .10

.07 1.32 1st-2nd

• .11

.08 1.39 2nd-3rd

• .08

.08 1.04 Overall

• .10

.05 2.17 <.05 Severe chronic absence (>20% absent) K-1st

• .04

.06 0.71 1st-2nd

• .05

.04 1.30 2nd-3rd

• .08

.04 1.88 Overall

• .06

.02 2.29 <.05

Robustness of results: Impact on assessments

SPARK/MCLP Evaluation 32

Measured impact of SPARK on assessments is robust to alternative specifications, especially in the case of PALS literacy assessment

• Transforming scale of assessment to rank-

based z-scores, forcing normality

• Simpler specification with only the pretest

and treatment as RHS variables

• More complex specification including math

pretest, cubic pretest terms, interactions among linear and quadratic pretest terms, interactions between cubic pretest terms and school fixed effects MAP PALS Model B SE B SE Baseline 0.23 0.08 0.36 0.09 Baseline, assessments as rank-based z-scores 0.25 0.08 0.34 0.08 Single pretest and treatment as only RHS variables 0.13 0.09 0.31 0.08 Two pretests, school FEs, treatment as only RHS 0.15 0.07 0.31 0.08 Baseline plus math MAP pretest, cubic pretest terms (incl. interactions with each

• ther, school FEs)

0.18 0.08 0.28 0.09

Joint hypothesis testing between MAP and PALS

SPARK/MCLP Evaluation 33

Estimating models for MAP and PALS jointly as seemingly unrelated regressions (SUR) allows testing of joint hypotheses across MAP and PALS MAP and PALS models use same RHS variables, so joint estimation yields the same results as equation-by-equation

• Results still slightly different from baseline because analysis only included students

with both MAP and PALS scores, model assumes i.i.d. standard errors Substantially rejects null that effect of SPARK across MAP and PALS is zero

• b (MAP,overall) = b (PALS,overall) = 0 rejected at .0001 level
• b (MAP,K1) = b (PALS,K1) = b (MAP,12) = b (PALS,12) = b (MAP,23) = b (PALS,23) = 0 rejected at .002 level

Joint hypothesis testing between MAP and PALS (ct’d)

SPARK/MCLP Evaluation 34

Also substantially rejects null hypothesis that overall effect of SPARK across MAP and PALS is the same for high-achieving and low-achieving students

• b (MAP, high, overall) - b (MAP, low, overall) = b (PALS, high, overall) - b (PALS, low, overall) = 0 rejected at .01 level
• Analogous null hypothesis for individual grades rejected at .005 level

Interpreting standardized effects

SPARK/MCLP Evaluation 35

Standard deviation of fall MAP scale scores from SPARK sample are 15% to 25% smaller than national norms

• Effect of SPARK in nationwide standard deviation units is approximately 75% to 85%

the effect in SPARK sample standard deviation units

Standard deviation of scale scores Fall MAP pretest Spring MAP posttest SPARK Sample MAP Norm SPARK Sample MAP Norm K-1st 10.1 13.5 11.3 14.5 1st-2nd 10.9 13.1 15.4 15.2 2nd-3rd 11.9 15.5 13.6 15.1

Long-term Outcome – Sustained Reading and School Success

SPARK/MCLP Evaluation 36

Typical literacy interventions focus mostly on skill acquisition and ignore the reasons why students fall behind.

• SPARK supports the whole child and family and

thus is designed to address some of the reasons why students are struggling to learn to read. The benefits observed in literacy interventions typically fade over time (Suggate, 2016).

• By engaging the whole family and by continuing to

work with students and families beyond SPARK, BGCGM is in a unique position to maintain and expand the impact of SPARK. The evaluation will continue to work with BGCGM to follow SPARK students as they progress in their education and will measure the sustained impact of SPARK over time.

Outcomes Short Medium Long

Sustained improvement in reading achievement; School success Families more engaged in and knowledgeable about student's education. Improved reading (Running Records) Improved reading achievement Improved literacy (PALS Assessment) Improved school performance

Thank you!

For more information about the SPARK evaluation contact Curtis Jones at: jones554@uwm.edu 312.421.0277

SPARK/MCLP Evaluation 37