Going to Scale: Immediate and Long-Term Effects of a Randomized - PowerPoint PPT Presentation

Going to Scale: Immediate and Long-Term Effects of a Randomized Control Trial of a State Pre-K Dale C. Farran, Mark W. Lipsey, Kerry G. Hofer Dale C. Farran, Mark W. Lipsey, Kerry G. Hofer Nianbo Nianbo Dong, Carol Bilbr Dong, Carol Bilbrey ey Peabody Research Institute Vanderbilt University AERA, Philadelphia, PA, April 2014

Research Team • Principal Investigators Senior Research Associate • – Mark Lipsey – Kerry Hofer – Dale Farran Research Associate • • TN Dept of Education Partner – Nianbo Dong – Connie Casha Research Analysts • • Project Manager – Chris Ham – Rick Feldser – Carol Bilbrey – Ilknur Sekmen • Project Coordinator TN Education Consultant – Janie Hughart • – Bob Taylor • Regional Coordinator Child assessors across the state – Diane Hughes • Funded by the Funded by the Institute for Ed Institute for Education Sciences Grant #R305E090009 ucation Sciences Grant #R305E090009 April 2014 Farran AERA 2

The Tennessee Voluntary Pre-K Program (TN-VPK) • 4-year old children – priority for children who qualify for the FRPL programs – after that, children at-risk for other reasons. • Full day program – 5.5 hours per day; minimum of 180 days per year. • Program meets 9 of 10 NIEER recommendations for quality, e.g., – State licensed teachers with early childhood education endorsements. – Aide in each classroom; class size maximum of 20. – Approved age-appropriate curriculum aligned with the Tennessee ECE Developmental Standards. • 934 pre-k classrooms in 133 of the 136 Tennessee school systems across all 95 Tennessee counties; serves more than 18,000 children. April 2014 Farran AERA 3

RCT (Randomized Control Trial) • Full Sample – Randomization to wait lists in oversubscribed schools – 2 cohorts; 3025 students across TN • Cohort 1: Pre-k year = 2009-10; N=1764 • Cohort 2: Pre-k year = 2010-11; N=1261 – Tracking through the state data system to 3 rd grade • Intensive Substudy of Consented Children – Assessed each year by the research team – 1076 students in the analysis sample – 773 in TN-VPK; 303 not in TN-VPK (control) – Analyses focused on Treatment on Treated (TOT) April 2014 Farran AERA 4

Achievement (Cognitive) Outcomes [Available only for the Intensive Substudy Sample] Woodcock Johnson III Scales • Literacy : Letter-Word Identification, Spelling • Language : Picture Vocabulary, Oral Comprehension, Passage Comprehension (K and 1 st grade only) • Math : Applied Problems, Quantitative Concepts, Calculation (K & 1 st grade only) • Overall WJ Composite – Mean W score across all WJ scales • Composite of 6 scales used at end of pre-k and beyond • Composite of 8 scales used at end of K and beyond April 2014 Farran AERA 5

Characteristics of the Children in the ISS Analysis Sample Charact Characteristic ristic Mean Mean Age start of pre-k year 4.3 Male 45% FRPL 100% Race/ethnicity White 57% Black 23% Hispanic 18% Asian and other 2% Non-native English 20% speaker April 2014 Farran AERA 6

Preschool Experience of ISS Control Children 49% 50% 45% 60% at home 40% 35% Percent of Nonparticipants 30% 15% private childcare; 25% 11% Head Start 20% 15% 10% mixed 15% 11% 10% 6% 6% 5% 5% 4% 5% 0% Home with Home with Homebased Private Child Head Start Mix (some Mix (no Unknown parent Other Care Care formal) formal) 7

VPK Effects at End of Pre-k on the WJII Subscales % Im provem ent Outcom e Measure Effect Size for Pre-K Overall Com posite .33* 45% Literacy Letter-Word ID .46* 89% Spelling .25* 30% Language Picture Vocabulary .20* 83% Oral Comprehension .10* 26% Math Applied Problems .14* 21% Quantitative Concepts .31* 49% * p <.05 April 2014 Farran AERA 8

Effects at End of PK, K, & 1 st on WJII Subscales 1 st ES Outcom e Measure PK ES K ES Overall Com posite (6) .33* .04 -.09 -- .00 -.09 Overall Com posite (8) Literacy Letter-Word ID .46* .04 -.06 Spelling .25* .01 -.09 Language Picture Vocabulary .20* .08 .02 Oral Comprehension .10* .09 -.07 Passage Comprehension -- -.11† -.07 Math Applied Problems .14* .02 -.05 Quantitative Concepts .31* -.07 -.21* * p <.05 Calculation -- -.07 -.06 † p <.10 April 2014 Farran AERA 9

Non-Cognitive Outcomes [Available for ISS sample & portions of full sample] Teacher ratings at beginning of K and end of 1 st grade • Cooper-Farran Behavioral Rating Scales (ISS sample) – Work-Related Skills – Social Behavior • Academic Child Behavior Record (ISS sample) – Preparation for Grade – Peer Relations – Behavior Problems – Feelings About School Measures from the State Education Information System • Retention in grade • Attendance • Disciplinary Action Too early to • ELL status collect valid data • IEPs April 2014 Farran AERA 10

Effects on Teacher Ratings at Beginning of K and End of First Grade (ISS sample) End of 1 st Beginning Outcom e Measure of K ES Grade ES CF Social Skills .30 .06 CF Work-Related Skills .37* -.03 ACBR Preparation for Grade .44* -.10 ACBR Peer Relations .19* .08 ACBR Behavior Problems -.39* -.05 ACBR Feelings About School .05 -.06 * p <.05 April 2014 Farran AERA 11

Grade Retention in Kindergarten 10% 9% 8% Percent Retained in K No VPK 6.8% 7% No VPK 6.2% 6% VPK 5.0% 5% VPK 4.1% 4% 3% 2% 1% 0% ISS Sample Full Sample ( p <.05) ( p <.05) April 2014 Farran AERA 12

Attendance in Kindergarten & 1 st Grade TN-VPK TN-VPK Participants Nonparticipants p for Grade and Sam ple Mean Days Mean Days difference K attendance, ISS sample 160.0 160.4 .534 K attendance, Full sample 158.0 156.6 .0 8 4 1 st grade attendance, Cohort 1 164.0 160.6 .0 17 April 2014 Farran AERA 13

Summary of TN-VPK Effects Found So Far Achievement (Cognitive) Outcomes Achievement (Cognitive) Outcomes • Positive effects at the end of pre-k on all measures • Effects were not sustained through kindergarten and first grade Non-Cognit Non-Cognitive Outcomes e Outcomes • Teacher ratings – Positive effects at beginning of K on ratings of preparedness for grade, work related skills, and classroom behavior – Effects were not sustained through end of first grade • School records – Fewer TN-VPK participants retained in kindergarten but group difference in rate not sustained through 1 st grade – More days attended by TN-VPK participants in K and 1 st grade April 2014 Farran AERA 14

PUTTING TN-VPK RESULTS IN PUTTING TN-VPK RESULTS IN CONTEXT* CONTEXT* *With thanks to Greg Duncan for Meta Analysis Slides April 2014 Farran AERA 15

Average cognitiv erage cognitive im e impact at end of treatment pact at end of treatment 2.00 Perry Preschool 1.50 size in sd units ct size in sd units Abecedarian 1.00 effect erage ef Average 0.50 0.00 1950 1960 1970 1980 1990 2000 2010 -0.50

Evidentiary Basis for Scaling Up • Both Perry and Abecedarian – Implemented in the 1960s and 70s – Counterfactual much more limited than today – Small, focused, intense programs – Heavily guided by on site, invested researchers – Much more extensive • Abecedarian intervention from 6 weeks to age 5 • Full working day not school day • Nonetheless these are studies cited to justify expanding pre-k intervention for poor children though larger, more current studies do not replicate early effects April 2014 Farran AERA 17

Average cognitiv erage cognitive im e impact at end of treatment pact at end of treatment 2.00 Head Start Non Head Start Perry Preschool 1.50 ct size in sd units size in sd units Abecedarian 1.00 National Head Start effect erage ef Average 0.50 0.00 1950 1960 1970 1980 1990 2000 2010 -0.50

Average cognitiv erage cognitive im e impact at end of treatment pact at end of treatment 2.00 Head Start Non Head Start Perry Preschool 1.50 ct size in sd units size in sd units Abecedarian TN-VPK 1.00 National Head Start effect erage ef Average 0.50 0.00 1950 1960 1970 1980 1990 2000 2010 -0.50

Issues with Scaling Up Pre-K • We lack coherent vision for what Pre-K should accomplish • Variation among domains most strongly affected – Concrete school-like readiness skills most affected – Language and math skills least affected – Improving language and math requires rigorous research efforts • Programs that have minimum cost, minimum required program elements, scale up easily, but perhaps without important or lasting effects • Strong vision critical when scaling up at a statewide level – School districts are highly independent • Even for an effective program, it will be a challenge at scale to maintain consistent positive effects across all classrooms April 2014 Farran AERA 20

Conclusion As prekindergarten programs are scaled up, rigorous data should be collected to determine variations among individual classrooms with systems for improvement. As researchers and advocates, we need to be much more intentional and systematic in our efforts to help pre-k achieve its potential. April 2014 Farran AERA 21

Thanks! More information available at: http://peabody.vanderbilt.edu/research/pri/projects/by_content_area/tennessee_state_pre-k_evaluation/ or Google “Peabody Research Institute” Contact email: dale.farran@vanderbilt.edu April 2014 Farran AERA 22