Immediate and Longitudinal Effects of the Immediate and Longitudinal - - PowerPoint PPT Presentation

Immediate and Longitudinal Effects of the Immediate and Longitudinal Effects of the Tennessee Voluntary Pre-K Program Tennessee Voluntary Pre-K Program on

• n Student

Student Cognitive and Non-Cognitive Outcomes Cognitive and Non-Cognitive Outcomes

Mark W. Lipsey, Kerry G. Hofer, Dale C. Farran, Mark W. Lipsey, Kerry G. Hofer, Dale C. Farran, Nianbo Nianbo Dong, Carol Bilbr Dong, Carol Bilbrey ey

Peabody Research Institute Vanderbilt University SREE, Washington DC, March 2014

Research Team

• Senior Research Associate

– Kerry Hofer

• Research Associate

– Nianbo Dong

• Research Analysts

– Chris Ham – Rick Feldser – Ilknur Sekmen

• TN Education Consultant

– Bob Taylor

• Child assessors across the state
• Principal Investigators

– Mark Lipsey – Dale Farran

• TN Dept of Education Partner

– Connie Casha

• Project Manager

– Carol Bilbrey

• Project Coordinator

– Janie Hughart

• Regional Coordinator

– Diane Hughes

Funded by the Funded by the Institute for Ed Institute for Education Sciences Grant #R305E090009 ucation Sciences Grant #R305E090009

The Tennessee Voluntary Pre-K Program (TN-VPK)

• 4-year old children

– priority for children who qualify for the FRPL programs – after that, children at-risk for other reasons.

• Full day program

– 5.5 hours per day; minimum of 180 days per year.

• State licensed teachers with early childhood education

endorsements.

• Aide in each classroom; class size maximum of 20.
• Approved age-appropriate curriculum aligned with the

Tennessee ECE Developmental Standards.

• 934 pre-k classrooms in 133 of the 136 Tennessee school

systems across all 95 Tennessee counties; serves more than 18,000 children.

Two Components of the Evaluation

• RCT (Randomized Control Trial)

– Full Sample

• Randomization to wait lists in oversubscribed schools
• 2 cohorts; 3025 students across TN

– Cohort 1: Pre-k year = 2009-10; N=1764 – Cohort 2: Pre-k year = 2010-11; N=1261

• Tracking through the state data system to 3rd grade

– Intensive Substudy of Consented Children (ISS)

• Assessed each year by the research team
• 1076 students in the analysis sample
• 773 in TN-VPK; 303 not in TN-VPK (control)
• RDD (Age-Cutoff Regression Discontinuity Design)

– Probability sample of TN-VPK classrooms in 4 regional groups – Over 3400 students to date – potential of 2400 more – Directly assessed at the beginning of PK/K as applicable

Focus of this presentation

Full sample: 79 participating schools in 29 school districts across TN Intensive substudy sample: 76 schools in 21 districts

RCT Participants

Cohort 1: Pre-k year 2009-10 Cohort 2: Pre-k year 2010-11

Achievement (Cognitive) Outcomes

[Available only for the Intensive Substudy Sample]

Woodcock Johnson III Scales

• Literacy: Letter-Word Identification, Spelling
• Language: Picture Vocabulary, Oral Comprehension,

Passage Comprehension (K and 1st grade only)

• Math: Applied Problems, Quantitative Concepts, Calculation

(K & 1st grade only)

• Overall WJ Composite

– Mean W score across all WJ scales

• Composite of 6 scales used at end of pre-k and beyond
• Composite of 8 scales used at end of K and beyond

Characteristics of the Children in the ISS Analysis Sample

Charact Characteristic ristic Mean Mean Age start of pre-k year 4.3 Male 45% FRPL 100% Race/ethnicity White 57% Black 23% Hispanic 18% Asian and other 2% Non-native English speaker 20%

Preschool Experience of ISS Control Children

49% 6% 5% 15% 11% 6% 4% 5%

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

Home with parent Home with Other Homebased Care Private Child Care Head Start Mix (some formal) Mix (no formal) Unknown

Percent of Nonparticipants

60% at home 15% private childcare; 11% Head Start 10% mixed

Analysis Plan for ISS Data: Propensity Scores

• Variables included in the Propensity Score
• Propensity Score as a Level 1 covariate in all analyses
• Consent rate for T and C groups on each randomized list,

and interaction, used as Level 2 covariates

• Multiple imputation for missing data.
• Gender
• Race/ Ethnicity
• ESL
• Age at Pretest
• Test Lag
• Test Interval
• Cohort
• Working Parents
• Library Card Use
• Newspaper Subscription
• Magazine Subscription
• Mother’s Education
• Pretest

‐0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Pretest Posttest Standard Deviation TN‐VPK Participants Nonparticipants

VPK Effects at End of Pre-k on the WJIII Composite W-Scores

.33 SD (45% increase) .73 SD Control group pre- post gain in SD units VPK effect size = .33

Outcom e Measure Effect Size % Im provem ent for Pre-K Overall Com posite .33* 45% Literacy Letter-Word ID .46* 89% Spelling .25* 30% Language Picture Vocabulary .20* 83% Oral Comprehension .10* 26% Math Applied Problems .14* 21% Quantitative Concepts .31* 49%

* p<.05

VPK Effects at End of Pre-k on the WJII Subscales

Outcom e Measure PK ES K ES 1st ES Overall Com posite (6) .33* .04

• .09

Overall Com posite (8)

• .00
• .09

Literacy Letter-Word ID .46* .04

• .06

Spelling .25* .01

• .09

Language Picture Vocabulary .20* .08 .02 Oral Comprehension .10* .09

• .07

Passage Comprehension

• .11†
• .07

Math Applied Problems .14* .02

• .05

Quantitative Concepts .31*

• .07
• .21*

Calculation

• .07
• .06

* p<.05 † p<.10

Effects at End of PK, K, & 1st on WJII Subscales

WJIII Composite Achievement: Native English Speakers in Pre-K Classrooms with the Largest Gains

WJIII Composite Achievement: Native English Speakers in Pre-K Classrooms with the Smallest Gains

WJIII Composite Achievement: Non-native English Speakers in Pre-K Classrooms with the Largest Gains

WJIII Composite Achievement: Non-native English Speakers in Pre-K Classrooms with the Smallest Gains

Non-Cognitive Outcomes

[Available for ISS sample & portions of full sample]

Teacher ratings at beginning of K and end of 1st grade

• Cooper-Farran Behavioral Rating Scales (ISS sample)

– Work-Related Skills – Social Behavior

• Academic Child Behavior Record (ISS sample)

– Preparation for Grade – Peer Relations – Behavior Problems – Feelings About School

Measures from the State Education Information System

• Retention in grade
• Attendance
• Disciplinary Action
• ELL status (too early to assess; still collecting data)
• IEPs (too early to assess; still collecting data)

Outcom e Measure Beginning

• f K ES

End of 1st Grade ES CF Social Skills .30 .06 CF Work-Related Skills .37*

• .03

ACBR Preparation for Grade .44*

• .10

ACBR Peer Relations .19* .08 ACBR Behavior Problems

• .39*
• .05

ACBR Feelings About School .05

• .06

* p<.05

Effects on Teacher Ratings at Beginning

• f K and End of First Grade (ISS sample)

Grade Retention in Kindergarten

No VPK 6.2% No VPK 6.8% VPK 4.1% VPK 5.0%

0% 1% 2% 3% 4% 5% 6% 7% 8% 9% 10% ISS Sample Full Sample Percent Retained in K

(p<.05) (p<.05)

Grade and Sam ple

TN-VPK Participants Mean Days TN-VPK Nonparticipants Mean Days p for difference

K attendance, ISS sample 160.0 160.4 .534 K attendance, Full sample 158.0 156.6 .0 8 4 1st grade attendance, Cohort 1 164.0 160.6 .0 17

Attendance in Kindergarten & 1st Grade

Grade and Sam ple

TN-VPK Participants % with Record TN-VPK Nonparticipants % with Record p for difference

Kindergarten, ISS sample 0.9 0.8 .252 Kindergarten, Full sample 1.1 1.5 .196 1st grade, Cohort 1 2.6 2.2 .633

Recorded Disciplinary Action in Kindergarten & 1st Grade

Summary of TN-VPK Effects Found So Far (1)

Achievement (Cognitive) Outcomes Achievement (Cognitive) Outcomes

• Nontrivial positive effects at the end of pre-k on all

measures; largest on literacy scales

• Largest gains were for non-native English speakers
• Effects were not sustained through kindergarten and first

grade

Summary of TN-VPK Effects Found So Far (2)

Non-Cognitive Outcomes Non-Cognitive Outcomes

• Teacher ratings

– Nontrivial positive effects at beginning of K on ratings of preparedness for grade, work related skills, and classroom behavior – Effects were not sustained through end of first grade

• School records

– Fewer TN-VPK participants retained in kindergarten – More days attended by TN-VPK participants in K and 1st grade – No difference for recorded disciplinary actions in K and 1st grade – Sufficient data on ELL and IEP status are not yet available

Thanks!

More information available at:

http://peabody.vanderbilt.edu/research/pri/projects/by_content_area/tennessee_state_pre-k_evaluation/

• r Google “Peabody Research Institute”

Contact email: mark.lipsey@vanderbilt.edu