The Effects of the The Effects of the T T Tennessee Voluntary Pre - - PowerPoint PPT Presentation

The Effects of the The Effects of the T V l t P T V l t P Tennessee Voluntary Pre Tennessee Voluntary Pre-

• Kindergarten Program:

Kindergarten Program: Initial Results Initial Results Initial Results Initial Results

Mark Lipsey Mark Lipsey Dale Farran Dale Farran Kerry Hofer Carol Bilbrey Kerry Hofer Carol Bilbrey Nianbo Dong Nianbo Dong

Peabody Research Institute, Vanderbilt University Peabody Research Institute, Vanderbilt University Society for Research on Educational Effectiveness March 2011 Society for Research on Educational Effectiveness March 2011 Society for Research on Educational Effectiveness, March 2011 Society for Research on Educational Effectiveness, March 2011

In partnership with the Tennessee Dept. of Education, In partnership with the Tennessee Dept. of Education, Division of School Readiness and Early Learning Division of School Readiness and Early Learning Bobbi Lussier, Assistant Commissioner Bobbi Lussier, Assistant Commissioner C i C h Di t f E l Childh d P C i C h Di t f E l Childh d P Connie Casha, Director of Early Childhood Programs Connie Casha, Director of Early Childhood Programs Robert Taylor, Consultant Robert Taylor, Consultant With the invaluable assistance of our fine research team: With the invaluable assistance of our fine research team: Patricia Abelson Patricia Abelson Kurt Scheib Kurt Scheib Ri h d F ld Ri h d F ld Ilk S k Ilk S k Richard Feldser Richard Feldser Ilknur Sekmen Ilknur Sekmen Janie Hughart Janie Hughart Travis Wimsett Travis Wimsett And many more And many more And many more … And many more … Supported by a grant from the U.S. Department of Supported by a grant from the U.S. Department of Ed ti I tit t f Ed ti S i (R305E090009) Ed ti I tit t f Ed ti S i (R305E090009) Education, Institute of Education Science (R305E090009) Education, Institute of Education Science (R305E090009)

Background

Tennessee invests over $85 million a year in its

statewide Pre-K program

934 t t f d d P K l 18 000

934 state-funded Pre-K classrooms serve 18,000+

economically disadvantaged children across all 95 Tennessee counties

Support for Pre-K is based on the belief that high

quality Pre-K:

Improves at-risk children’s readiness for kindergarten Improves achievement test scores and decreases retention rates,

special education placements, and drop outs In adulthood increases employment rates and earnings and

In adulthood, increases employment rates and earnings, and

reduces welfare needs and criminal behavior

Political Controversy and Limited y Research Support

Many Tennessee legislators question the value of Pre-K,

some calling it “expensive babysitting,” and whether it warrants funding given the current budget constraints warrants funding given the current budget constraints

The evidence for long-term Pre-K effects is from small

intensive programs that are not typical of state programs

The national Head Start study provides little support for

the view that large scale preschool programs are especially effective for boosting academic performance especially effective for boosting academic performance

No research using a randomized control trial design has

studied the benefits of typical public Pre-K programs and studied the benefits of typical public Pre K programs and their effects beyond the beginning of kindergarten

Phase 1 RCT Intensive Substudy

Phase 1 in 2009

Phase 1 in 2009-

• 10 school year (Phase 2

10 school year (Phase 2 underway for 2010 underway for 2010-

• 11 school year)

11 school year)

Randomized admissions in 23 schools in

Randomized admissions in 23 schools in 14 TN school districts 14 TN school districts

907 children in full randomization (will be

907 children in full randomization (will be tracked in state EIS database) tracked in state EIS database)

303 consented children with assessment

303 consented children with assessment data, 73 no pre data, 73 no pre-

• k controls and 230 pre

k controls and 230 pre-

• k

k , p , p p

Intensive Substudy Sample

M 4 4 56% i l 44% b M 4 4 56% i l 44% b

Mean age, 4.4 yrs; 56% girls, 44% boys

Mean age, 4.4 yrs; 56% girls, 44% boys

57% white, 23% African

57% white, 23% African-

• American,

American, 20% Hi i 20% Hi i 20% Hispanic 20% Hispanic

31% language other than English in home

31% language other than English in home M di d i Hi h h l/GED M di d i Hi h h l/GED

Median parent education: High school/GED

Median parent education: High school/GED

No Pre

No Pre-

• K control childcare alternatives:

K control childcare alternatives:

11% Head Start

11% Head Start

22% Private childcare center

22% Private childcare center 51% Home with parent or other 51% Home with parent or other

51% Home with parent or other

51% Home with parent or other

16% Unknown

16% Unknown

Intensive Substudy Outcome Measures

Woodcock Johnson III Scales Woodcock Johnson III Scales

Literacy:

Literacy: Letter

Letter-

• Word Identification, Spelling

Word Identification, Spelling

L L

Language:

Language: Picture Vocabulary, Oral Comprehension

Picture Vocabulary, Oral Comprehension

Math:

Math: Applied Problems, Quantitative Concepts

Applied Problems, Quantitative Concepts

Overall WJ Composite

Overall WJ Composite

Overall WJ Composite

Overall WJ Composite

• Mean W score across all WJ scales (supported by factor analysis)

Mean W score across all WJ scales (supported by factor analysis)

Teacher Ratings Teacher Ratings Teacher Ratings Teacher Ratings

Cooper

Cooper-

• Farran Work

Farran Work-

• Related & Social scales

Related & Social scales

Academic Child Behavior Record: School Readiness, Likes

Academic Child Behavior Record: School Readiness, Likes School, & Behavior Problems School, & Behavior Problems

Complications, Part 1

I di t i d t h I di t i d t h

Indirect process required to approach

Indirect process required to approach parents for consent to assess children parents for consent to assess children resulted in low and variable consent rate resulted in low and variable consent rate resulted in low and variable consent rate resulted in low and variable consent rate

37% overall across the 23 schools

37% overall across the 23 schools

Ranged from 7.5% to 82.6%

Ranged from 7.5% to 82.6%

Ranged from 7.5% to 82.6%

Ranged from 7.5% to 82.6%

Note: Changed procedure for 2010

Note: Changed procedure for 2010-

• 11 has

11 has produced higher overall rate for Phase 2 produced higher overall rate for Phase 2

Response: Use the consent rate and its

Response: Use the consent rate and its interaction with treatment conditions as a interaction with treatment conditions as a covariate in the analysis covariate in the analysis

Complications, Part 2

Some baseline differences between conditions

Some baseline differences between conditions

Pre

Pre-

• K children: more African

K children: more African-

• Americans, lower mean

Americans, lower mean parent education parent education parent education parent education

Pretest averaged 9 days later for no Pre

Pretest averaged 9 days later for no Pre-

• K control

K control children; pre children; pre-

• post test interval average of 6.5 days shorter

post test interval average of 6.5 days shorter N i ifi ( N i ifi ( 10) diff d 10) diff d

No significant (

No significant (p<.10) differences on age, gender, <.10) differences on age, gender, proportion Hispanic, or parent report of weekend TV, proportion Hispanic, or parent report of weekend TV, home literacy support, or whether language other than home literacy support, or whether language other than English spoken in home (correlates of WJ outcomes) English spoken in home (correlates of WJ outcomes)

Response: Use all these variables as covariates

Response: Use all these variables as covariates in the analysis in the analysis

Complications Continued: Pretest p Differences Despite Prior Covariates

Outcome Measure Control Mean (N=73) Pre-K Mean (N=230) Difference as an Effect Size p-value

Overall Composite 391.0 396.0 .30* .000 p Letter-Word ID 313.6 320.6 .28* .003 Spelling 350.1 355.9 .23* .008 Picture Vocabulary 449 0 454 9 27* 002 Picture Vocabulary 449.0 454.9 .27 .002 Oral Comprehension 442.5 442.8 .02 .583 Applied Problems 384.4 393.9 .39* .000 Quantitative Concepts 406.3 408.2 .13 .071

• Response: Use the respective pretest as a

Response: Use the respective pretest as a Response: Use the respective pretest as a Response: Use the respective pretest as a covariate in each analysis. covariate in each analysis.

Analysis

W-

• scores from WJIII scales

scores from WJIII scales

Teacher rating scales

Teacher rating scales

Multilevel with children nested within

Multilevel with children nested within schools/randomized lists (blocking factor) schools/randomized lists (blocking factor) C i t f ll l d C i t f ll l d

Covariates for all analyses: age, gender,

Covariates for all analyses: age, gender, race/ethnicity, home literacy, weekend TV, parent race/ethnicity, home literacy, weekend TV, parent education, home language, response rate, education, home language, response rate, education, home language, response rate, education, home language, response rate, response rate x condition response rate x condition

Additional covariates for WJ outcomes: pretest,

Additional covariates for WJ outcomes: pretest, p , p , lag to pretest, pretest lag to pretest, pretest-

• posttest interval

posttest interval

Pre-Post Gain for WJ Composite Scale

1.0 1.2 s

Pre-K No Pre-K

0.6 0.8 Deviations

No Pre K

.35 SD

(50% increase)

0.2 0.4 tandard D .69 SD

• 0.2

0.0 St 0.2 Pretest Posttest

Pre-Post Gains and Effect Sizes for WJIII Scales

Outcome Measure

Control Gain in SD Units Pre-K Gain in SD Units Difference (Effect Size) % Improvement for Pre-K

Overall Composite .69 1.03 .35* 50% Literacy Letter-Word ID .51 1.00 .49* 96% Spelling .70 1.01 .31* 45% p g Language Picture Vocabulary .18 .43 .25* 141% Oral Comprehension 28 58 31* 110% Oral Comprehension .28 .58 .31* 110% Math Applied Problems .69 .91 .22* 32% Quantitative Concepts .64 1.05 .41* 63%

Note: All the differences between the Control and Pre-K children are statistically significant, p<.05.

Effect Sizes for Teacher Ratings

Outcome Measure Effect Size p-value Cooper-Farran Work Related .29*

.027

Cooper-Farran Social Behavior

• .04

.791

ABR Readiness for Kindergarten .36*

.006

ABR Likes School

• .09

.653

ABR Behavior Problems No/Yes .06

.468

ABR Number of Behavior Problems 00

939

ABR Number of Behavior Problems .00

.939

Note: Based on ratings by 19 teachers at 19 of the 23 schools and 203 children rated (133 T and 70 C).

Regression-Discontinuity Substudy: g y y Middle Tennessee Region

36 schools in 17 middle Tennessee school

36 schools in 17 middle Tennessee school districts districts

682 children who attended Pre

682 children who attended Pre-

• K during the

K during the 2009 2009-

• 10 school year

10 school year

676 children who were below the age cutoff

676 children who were below the age cutoff and attended Pre and attended Pre-

• K during the 2010

K during the 2010-

• 11

11 school year school year

All children assessed early in the fall of

All children assessed early in the fall of y 2010 2010

Regression-Discontinuity Sample

Mean age at time of assessment

Mean age at time of assessment

Beginning of Pre

Beginning of Pre-

• K control sample: 4.4 yrs

K control sample: 4.4 yrs

Beginning of K treatment sample: 5.4 yrs

Beginning of K treatment sample: 5.4 yrs

50% boys; 50% girls

50% boys; 50% girls

52% white, 35% African

52% white, 35% African-

• American,

American, 13% Hispanic 13% Hispanic

12% native language other than English

12% native language other than English

Urban and rural schools

Urban and rural schools

Urban and rural schools

Urban and rural schools

RD Outcome Measures

W d k J h III S l W d k J h III S l Woodcock Johnson III Scales Woodcock Johnson III Scales

Literacy

Literacy

Letter

Letter-Word Identification Word Identification

Spelling

Spelling Language

Language

Picture Vocabulary

Picture Vocabulary ctu e

• cabu a y

ctu e

• cabu a y

Oral Comprehension

Oral Comprehension Math

Math

Applied Problems

Applied Problems

Applied Problems

Applied Problems

Quantitative Concepts

Quantitative Concepts Overall WJ Composite

Overall WJ Composite

Mean W score across all WJ scales (supported by factor

Mean W score across all WJ scales (supported by factor

Mean W score across all WJ scales (supported by factor

Mean W score across all WJ scales (supported by factor analysis) analysis)

Pre-K Age Cutoff RDD: Timing of Pre K Age Cutoff RDD: Timing of Outcome Measures

T t t Year 1 (2009-10) Year 2 (2010-11) Pre-K (T) Kindergarten Treatment: First cohort (before cutoff) Pre-K Control: Second cohort (after cutoff) No Pre-K (C)

Administer Tests

i l d b i hd Entry into Pre-K Selected by Birthday

WJ test score

C

?

T No Pre-K yet; tested at beginning of pre-K year T Completed pre-K; tested at beginning f K

Born before October 1

• f K

Born after October 1

Age

Analysis

W f WJIII l f WJIII l

W-scores from WJIII scales

scores from WJIII scales

Multilevel with children nested within

Multilevel with children nested within schools/classrooms schools/classrooms schools/classrooms schools/classrooms

Covariates to account for functional form: age

Covariates to account for functional form: age (centered at cutpoint), age x condition, age (centered at cutpoint), age x condition, age squared, age cubed squared, age cubed

Additional covariates: gender, race/ethnicity, non

Additional covariates: gender, race/ethnicity, non-

• English native language school start to test lag

English native language school start to test lag English native language, school start to test lag English native language, school start to test lag

±

± 3 months around cutpoint 3 months around cutpoint

170 control cases

170 control cases

157 treatment cases

157 treatment cases

RD Effect Size Relative to Pre-K to K Gain for WJ Composite Scale

1.0 1.2 s

Total Year-to-Year Estimate w/o Pre-K

0.6 0.8 Deviations

.61 SD Effect Size (152% increase)

0.2 0.4 tandard D

.40 SD

• 0.2

0.0 St 0.2 Pre-K Kindergarten

RDD Effect Sizes for WJIII Scales (± 3-months around birth date cutoff)

Outcome Measure

• Est. 1-yr Gain

w/o Pre-K in SD Units 1-year Gain with Pre-K in SD Units Pre-K Effect Size Estimate % Improvement for Pre-K

Overall Composite .40 1.01 .61** 152% p Literacy Letter-Word ID .35 1.01 .66** 190% Spelling 44 1 13 69** 159% Spelling .44 1.13 .69 159% Language Picture Vocabulary .16 .46 .30 193% Oral Comprehension .08 .45 .37* 456% (!) Math Applied Problems .42 .76 .34* 81% Quantitative Concepts .40 .78 .39* 97%

* p<.10, ** p<.05.

Overview and Final Thoughts

Policy level research is remarkably difficult to

Policy level research is remarkably difficult to do with rigor do with rigor

Schools prize their independence and are reluctant to

Schools prize their independence and are reluctant to

Schools prize their independence and are reluctant to

Schools prize their independence and are reluctant to change procedures for the sake of research change procedures for the sake of research

Close collaboration with the TN DOE was

Close collaboration with the TN DOE was Close collaboration with the TN DOE was Close collaboration with the TN DOE was essential for obtaining the RCT and RDD essential for obtaining the RCT and RDD samples samples

Multiple substudies and the ability to

Multiple substudies and the ability to ‘triangulate’ results helps compensate for the ‘triangulate’ results helps compensate for the weaknesses of each individual study weaknesses of each individual study

More Final Thoughts

I t t fi di f I t t fi di f

Important findings so far

Important findings so far

Strong effects demonstrated for state

Strong effects demonstrated for state-

• funded Pre

funded Pre-

• K

K compared to what is otherwise available in the community compared to what is otherwise available in the community

Same pattern of effects found in both the RCT and the

Same pattern of effects found in both the RCT and the RDD samples lends credibility to the conclusion RDD samples lends credibility to the conclusion

Teacher ratings corroborate the effect and expand it to

Teacher ratings corroborate the effect and expand it to g p g p include important learning dispositions include important learning dispositions

Future work

Future work

Examine the second RCT sample for similar effects

Examine the second RCT sample for similar effects

Examine the second RCT sample for similar effects

Examine the second RCT sample for similar effects

Follow the full RCT sample into 3

Follow the full RCT sample into 3rd

rd grade

grade

Continue the RDD sweep across the state, eventually

Continue the RDD sweep across the state, eventually i l di 140 l i l di 140 l including 140 classrooms including 140 classrooms