Eff Effects of Prevention and Intervention Programs on t f P ti d - - PowerPoint PPT Presentation

Eff t f P ti d I t ti P Effects of Prevention and Intervention Programs on School Completion and Dropout

Results from a Systematic Review Results from a Systematic Review Sandra Jo Wilson b d h Peabody Research Institute

The Campbell Collaboration www.campbellcollaboration.org

The Campbell Collaboration www.campbellcollaboration.org

Acknowledgments

• Funding provided by the Campbell Collaboration and

the Peabody Research Institute

• My co‐authors:

– Mark W. Lipsey

Mark W. Lipsey

– Emily E. Tanner‐Smith

Chiungjung Huang

– Chiungjung Huang – Katarzyna Steinka‐Fry – Jan Morrison – Nianbo Dong

The Campbell Collaboration www.campbellcollaboration.org

Overview

• Discussion of different kinds of interventions that

target school dropout.

• Systematic review methodology.
• Preliminary results from the systematic review
• Preliminary results from the systematic review.

The Campbell Collaboration www.campbellcollaboration.org

Risk‐oriented Programming

• There are a great variety of strategies for preventing

school dropout.

• The different strategies are generally developed out
• f different theories about the causes of dropout.

p

• The biggest risk factors for school dropout are

pregnancy low school attendance and poor academic pregnancy, low school attendance, and poor academic performance.

The Campbell Collaboration www.campbellcollaboration.org

Risk‐oriented Programming

Risk Factor for Dropout Targeted Program or Strategy Teenage pregnancy Provide child care services or financial assistance to young mothers. Poor attendance Monitor and reward attendance. P d i f P id l t l d i i Poor academic performance Provide supplemental academic services. Lack of support for higher education Provide college‐oriented curricula and advising for students. g Family and personal problems Connect students with appropriate services through case management, provide mentors or counselors provide mentors or counselors. Traditional school structure doesn’t work for some students Restructure or reorganize the school day, curriculum, etc. to better fit the needs of different types of students. Students lack purpose, goals, understanding of the purposes of Create career/work oriented courses; involve students in community.

The Campbell Collaboration www.campbellcollaboration.org

g p p education y

Objectives of the Review Objectives of the Review

• Summarize the available evidence on the effects of

prevention and intervention programs aimed at increasing school completion or reducing dropout. c eas g sc oo co p et o o educ g d opout

• Primary emphasis on the comparative effectiveness of

different types of programs in the context of different different types of programs, in the context of different methodologies, different levels of implementation quality and different configurations of student quality, and different configurations of student characteristics.

The Campbell Collaboration www.campbellcollaboration.org

Inclusion Criteria

• Interventions are school‐based, school‐affiliated, or

community‐based programs.

• Outcomes must be dropout, graduation, or attendance.

p g

• Students are between the ages of 4‐18.

Recent dropouts (18 22) are eligible if school completion is

– Recent dropouts (18‐22) are eligible if school completion is

explicit goal. For students under the age of 12 dropout must be assessed

– For students under the age of 12, dropout must be assessed.

• Experimental and quasi‐experimental designs are

l d d included.

• 1985 and later, any language, any publication type.

The Campbell Collaboration www.campbellcollaboration.org

Literature Search

• Comprehensive literature search of major educational

databases, including grey literature. databases, including grey literature.

• Search of international databases not yet completed.

Id ifi d 16 962

• Identified 16,962 reports

– 2,627 reports were deemed potentially relevant

and retrieved

– 878 reports were determined to be eligible

878 reports were determined to be eligible

• 416 studies (188 with dropout; 228 with attendance)

– 164 coded studies (so far)

164 coded studies (so far)

– 450 dropout effect sizes from 302 independent study

samples

The Campbell Collaboration www.campbellcollaboration.org

Study Coding Study Coding

• Study coding included information about the programs
• Study coding included information about the programs

under evaluation, the students in those programs, and the methods used to evaluate the programs. methods used to evaluate the programs.

• Outcomes indexed using odds ratios.

Ratio of the odds of completing school for treatment

– Ratio of the odds of completing school for treatment

participants to the odds of completing school for control participants participants

– Odds ratio < 1 means control groups are favored; Odds

ratio > 1 means treatment groups are favored; Odds ratio > 1 means treatment groups are favored; Odds ratio = 1 means odds are equal for the two groups.

The Campbell Collaboration www.campbellcollaboration.org

Treatment Types yp

• The 302 independent samples were divided into two

The 302 independent samples were divided into two groups

269 general programs

– 269 general programs

• 299 effect sizes measuring dropout, graduation, GED, or

enrollment for general population & at‐risk samples. e

• e t o ge e a popu at o & at

s sa p es

– 33 teen parent (mostly teen mothers) programs

• 150 effect sizes measuring dropout, graduation, GED, or

150 effect sizes measuring dropout, graduation, GED, or enrollment for teen parents.

– NOTE: enrollment defined as not dropped out, not yet

pp , y graduated, and still attending school.

The Campbell Collaboration www.campbellcollaboration.org

How effective were dropout programs overall?

Mean OR Lower CI Upper CI nes Ksamples Q tau2 G l T

1 68 1 52 1 88 300 269 1712 3* 23

General Tx

1.68 1.52 1.88 300 269 1712.3* .23

Teen Parent Tx

1.75 1.41 2.17 150 33 191.3* .37

Random effects analysis

Mean Odds Ratios Translated to Percentages Mean Odds Ratios Translated to Percentages TX % CT %

General Tx Programs (Percent dropped out) 33% 45% Teen Parent Tx Programs (Percent enrolled) 41% 28%

The Campbell Collaboration www.campbellcollaboration.org

How effective are dropout programs overall?

• After treatment, the odds of completing school were

significantly better for students in prevention significantly better for students in prevention programs than for students who received no special programming programming.

– Treated students were less likely to drop out, and more

likely to graduate obtain a GED or stay enrolled in likely to graduate, obtain a GED, or stay enrolled in school.

Heterogeneity was evident for both the general

• Heterogeneity was evident for both the general

programs and for the teen parent programs.

• What study characteristics are associated with that

heterogeneity?

The Campbell Collaboration www.campbellcollaboration.org

Characteristics of the Research

• Literature on dropout largely unpublished; 71% of the

studies were technical reports (66%) or dissertations (5%).

• Evaluator role & routine practice

p

– Some evaluators were closely involved in the research;

• ther programs had independent evaluators.
• ther programs had independent evaluators.

– About 50% were routine practice programs.

55% of the studies used random assignment

• 55% of the studies used random assignment.

The Campbell Collaboration www.campbellcollaboration.org

Characteristics of the Students

• 88% were high school aged samples; 12% were middle

school or younger samples.

• Mixed ethnicity groups of students were common.

y g p 75% of the student samples were mostly minority students.

• With the exception of specialized programs for

teenage mothers most programs were delivered to teenage mothers, most programs were delivered to mixed gender groups. N l ll t d t t i k f d i t d

• Nearly all students were at risk for dropping out and

were from low socioeconomic status families.

The Campbell Collaboration www.campbellcollaboration.org

Characteristics of the Dropout Programs Characteristics of the Dropout Programs

• Community‐based programs make up 15% of the

sample; the remaining 85% are school‐based or school‐affiliated.

• Duration averaged 90 weeks (typical school year was

coded as 38 weeks). )

• Frequency of treatment varied, but over half of the

programs involved daily contact programs involved daily contact.

• Implementation quality also varied: 32% experienced

l bl 15% ll d d t ibl bl clear problems, 15% alluded to possible problems, and 53% experienced no problems or mentioned no bl

The Campbell Collaboration www.campbellcollaboration.org

problems.

Treatment Types: General Programs Treatment Types: General Programs

Treatment Type Description N School or class restructuring Small learning communities block schedules 109 School or class restructuring Small learning communities, block schedules, career academies, small class size. 109 Vocational training Coursework, internships, or employment oriented d k i 41 toward work or career interests. Supplemental academic services Remedial education, tutoring, homework assistance, etc. 25 Community service Programs involved planning and carrying out a community service project (commonly coupled with a weekly life skills curriculum). 24 y ) Attendance monitoring & contingencies Monitoring and services to increase attendance; some offer financial incentives. 17 C ll i t d C ll t i l ll i t d 12 College‐oriented programming College preparatory curriculum, college‐oriented academic advising. 12 Case management Programs revolved around connecting students & 11

The Campbell Collaboration www.campbellcollaboration.org

families with appropriate services.

Treatment Types: General Programs

T T D i i Treatment Type Description n Mentoring, counseling Programs provided adult mentors or trained counselors for students. Though mentors focused 11 more on career/work, both mentors and counselors dealt with students’ personal issues. Skills training, including CBT Generally oriented toward improving self‐esteem 6 g, g y p g

• r attitudes about school, or preventing drug use.

Multi‐service package Large, comprehensive programs; generally included academic vocational and case 6 included academic, vocational, and case management services (e.g., Job Corps) Family outreach School contact with parents and families 3 Other Recreational, residential services for homeless, etc. 4

The Campbell Collaboration www.campbellcollaboration.org

Treatment Types for Teen Parents

T T D i i Treatment Type Description n Attendance monitoring & contingencies Monitoring and services to increase attendance; most offer financial incentives and/or 22 contingencies tied to welfare. Case management Programs revolved around connecting teen parents with appropriate services for parents and 6 p pp p p children. Parenting skills training Parent skills training (and often child care services) for teen mothers 3 services) for teen mothers. School or class restructuring Teenage Pregnancy & Parenting Program 1 l k l h Multi‐service package Family Growth Center 1

The Campbell Collaboration www.campbellcollaboration.org

Analytic Plan Analytic Plan

• Meta‐regression models with robust standard

errors used to examine influence of study characteristics on treatment effects.

• Regression models were then used to calculate

covariate adjusted posttest effect sizes for each covariate adjusted posttest effect sizes for each treatment type, adjusting for method and subject h t i ti th t f d characteristics that may confound group differences.

• Models run separately by program groups

(general vs. teen parent)

The Campbell Collaboration www.campbellcollaboration.org

(g p )

Meta‐regression Results for General Programs

General Programs b se

Method Variables Random assignment design ‐.159 .157 Matched groups design ‐.051 .177 ( ) Attrition (%) ‐.041 .400 Data in OR adjusted ‐.300 * .113 Subject Characteristics Subject Characteristics Gender mix .049 .045 Ethnicity mostly white .059 .123 Treatment Characteristics Role of evaluator (higher=more independent) ‐.144 † .083 Implementation quality (higher better) 173 * 065 Implementation quality (higher=better) .173 * .065 Treatment duration (weeks) ‐.000 .001 Frequency of service per week (hours) ‐.004 .004

The Campbell Collaboration www.campbellcollaboration.org

00 00 nes=299; k=268

Study Characteristics associated with Treatment y Effectiveness: General Programs

• Effect sizes calculated with covariate‐adjusted data

tended to be smaller tended to be smaller.

• Study design and attrition not significantly associated

with treatment effects.

• Gender mix and ethnicity not associated with

differential effects.

The Campbell Collaboration www.campbellcollaboration.org

Study Characteristics associated with Treatment y Effectiveness: General Programs

• Studies in which evaluators were closely involved

produced larger effects produced larger effects.

• Implementation quality resulted in better outcomes.
• Treatment length and treatment frequency not

associated with effects.

The Campbell Collaboration www.campbellcollaboration.org

Treatment Family Outreach Type 3 n 3 k

0.88 (0.68, 1.14) OR (95% CI) 0.88 (0.68, 1.14) OR (95% CI)

Monitoring & Contingencies Multi‐service Package Case Management 21 8 18 17 6 11

1.09 (0.93, 1.27) 1.17 (0.67, 2.04) 1.21 (0.79, 1.87) 1.09 (0.93, 1.27) 1.17 (0.67, 2.04) 1.21 (0.79, 1.87)

Case Management Skills Training Counseling, Mentoring S l A d i T i i 18 5 13 27 11 5 11 25

( , ) 1.34 (1.07, 1.70) 1.43 (0.98, 2.09) 1 52 (1 13 2 05) ( , ) 1.34 (1.07, 1.70) 1.43 (0.98, 2.09) 1 52 (1 13 2 05)

• Suppl. Academic Training

School or Class Restructuring Other Treatments 27 122 5 25 109 4

1.52 (1.13, 2.05) 1.78 (1.54, 2.06) 2.23 (1.31, 3.81) 1.52 (1.13, 2.05) 1.78 (1.54, 2.06) 2.23 (1.31, 3.81)

Vocational Training Community Service College‐oriented Program 41 24 12 41 24 12

2.32 (1.91, 2.82) 2.57 (1.48, 4.47) 2.85 (1.69, 4.81) 2.32 (1.91, 2.82) 2.57 (1.48, 4.47) 2.85 (1.69, 4.81) Favors CT Favors TX 1 .5 1 1.5 2 2.5 3 5.1

The Campbell Collaboration www.campbellcollaboration.org

Favors CT Favors TX

Mean Adjusted Odds Ratios for General Programs

Summary for General Programs

• All general programs except family outreach had

positive effects on dropout, graduation, or continued enrollment.

– Attendance monitoring, case management, multi‐service

g, g , packages, and counseling/mentoring programs not significant.

– College‐oriented programming, community service,

vocational training, supplemental academic training, and g g school restructuring programs were the most effective, net of study characteristics in the regression models.

– Overlapping confidence intervals for program types

indicates that programs were not significantly different

The Campbell Collaboration www.campbellcollaboration.org

from each other.

Translating Adjusted Means

• Recall that our overall average odds ratio = 1.68

– Translation: 45% of control students drop out vs. 33% of

treated students

• Adjusted OR for school restructuring programs = 1.78

– Translation: 45% of control students drop out vs. 32% of

Translation: 45% of control students drop out vs. 32% of treatment students

• Adjusted OR for vocational training = 2 32
• Adjusted OR for vocational training = 2.32

– Translation: 45% of control students drop out vs. 26% of

treatment students treatment students

The Campbell Collaboration www.campbellcollaboration.org

Meta‐regression Results for Teen Parent Programs

Teen Parent Programs b se

M th d V i bl Method Variables Random assignment design ‐.596 .391 Matched groups design ‐2.051 * .490 g p g 2.051 .490 Attrition (%) 1.188 * .520 Data in OR adjusted .534 * .234 Subject Characteristics Ethnicity mostly white .043 .138 Treatment Characteristics Treatment Characteristics Role of evaluator (higher=more independent) ‐.459 * .178 Implementation quality (higher=better) .979 * .287 Treatment duration (weeks) ‐.014 * .005 Frequency of service per week (hours) ‐.005 .019 k

The Campbell Collaboration www.campbellcollaboration.org

nes=150; k=33

Study Characteristics associated with Treatment Effectiveness: Teen Parent Programs

• Effect sizes calculated with covariate‐adjusted data

tended to be smaller tended to be smaller.

• Study design and attrition both significantly associated

ith t t t ff t with treatment effects.

– Random assignment and matched designs produced

ll dd h d h d smaller odds ratios than non‐random, non‐matched designs.

• Ethnicity not associated with differential effects.

The Campbell Collaboration www.campbellcollaboration.org

Study Characteristics associated with Treatment Effectiveness: Teen Parent Programs

• Studies in which evaluators were closely involved

produced larger effects produced larger effects.

• Implementation quality resulted in better outcomes.
• Shorter treatments were associated with larger

treatment effects.

The Campbell Collaboration www.campbellcollaboration.org

Treatment Type n k OR (95% CI) OR (95% CI) School or Class Restructuring Parent Training & Child Care 4 4 3 3 1.59 (1.31, 1.93) 1.59 (1.31, 1.93) 1.59 (1.31, 1.93) 1.59 (1.31, 1.93) g Case Management 15 6 ( , ) 1.62 (1.33, 1.98) ( , ) 1.62 (1.33, 1.98) Monitoring & Contingencies 125 22 1.74 (1.56, 1.94) 1.74 (1.56, 1.94) Favors CT Favors TX 1 .5 1 1.5 2 2.5 3 d d dd f

The Campbell Collaboration www.campbellcollaboration.org

Mean Adjusted Odds Ratios for Teen Parent Programs

Summary for Teen Parent Programs

• All four treatment strategies for teen parents were

significantly better than controls for keeping students enrolled in school.

• The most common strategy (attendance monitoring

gy ( g and financial incentives) had the largest adjusted odds ratio, though no strategy was significantly better than ratio, though no strategy was significantly better than another.

The Campbell Collaboration www.campbellcollaboration.org

Final Comments Final Comments

Dropout programs are generally effective with

• Dropout programs are generally effective, with

negligible differences between different strategies h t d h t i ti t ll d when study characteristics are controlled.

• Study methods are influential.
• Implementation quality is strongly associated with

better treatment outcomes.

• Studies in which evaluators were closely involved with

treatment delivery tended to produce better treatment delivery tended to produce better

• utcomes.

The Campbell Collaboration www.campbellcollaboration.org

Sandra Jo Wilson email: sandra.j.wilson@vanderbilt.edu

The Campbell Collaboration www.campbellcollaboration.org

The Campbell Collaboration www.campbellcollaboration.org