Studying Variation in the Effect of Program Participation Stephen - - PowerPoint PPT Presentation

Studying Variation in the Effect of Program Participation

Stephen W. Raudenbush

Presentation at the “Workshop on Learning from Variation in Program Effects” Palo Alto, July 19, 2016

The research reported here was supported by a grant from the W.T. Grant Foundation to the University of Chicago entitled “Building Capacity for Evaluating Group-Level Interventions.” Thanks to Sean Reardon and Takako Nomi for their collaboration on these ideas.

Outline

1. Pervasiveness of

• Multi-site trials
• Non-compliance
• 2. Instrumental variables in a single-site study
• Under homogeneity of impact
• Under heterogeneity of impact
• Examples
• 3. Instrumental variables in multi-site studies:
• Method 1 Combine 2 ITT Analyses
• Method 2: Two-stage generalized least squares
• Method 2= “Between-Site Regression!”
• 4. Design Considerations
• 5. Modeling Program Participation and Program Impact on

Participants

• 1. Pervasiveness of Multi-Site Trials

Since 2002, IES has funded 175 group-randomized trials Vast majority are multi-site trials (Spybrook, 2013) Other recent examples * National Head Start Evaluation (US Dept of HHS, 2010) * Moving to Opportunity (Sonbanmatsu, Kling, Duncan, Brooks Gunn, 2006) * School-based lottery studies (Abdulkadiroglu, Angrist, Dynarski, Kane, and Pathak,

2009).

* Tennessee STAR (Finn and Achilles, 1990) * Ending Social Promotion (Jacob and Lefgren, 2009) * Double-Dose Algebra (Nomi and Allensworth, 2009) * Welfare to Work (Bloom, Hill, Riccio, 2003) * Small Schools of Choice (MDRC)

Some Recent MS Trials

Study Levels Assigned Units Sites Fixed or Random sites

National Head Start Eval. 2 Children 198 Program Sites Random Moving to Opportunity 2 Families 5 cities Fixed Boston Charter School Lotteries 2 Children Lottery pools Random Tennessee STAR 3 Teachers 79 Schools Random 4 R’s 3 Classrooms 18 Schools Random Double-Dose Algebra 2 Children 60 Schools Random

• 2. Estimating the Impact of Program

Participation in a One- Site Study

T=Random assignment M=Participation Y=outcome

Figure 1: Conventional Instrumental Variable Model (Homogeneous Treatment Effects)



    

/ : Effect ITT Total          so

Single site, heterogeneous treatment effects

T M Y

Person-specific Causal Model

  ) ( E   ) ( E

) , ( ) (         Cov B E B

i i i

 

“No Compliance-Covariance” Assumption is Strong!

/ ) , ( ) , ( ) (                  if Cov if Cov B E

Alternative Approach for binary M

“Local Average Treatment Effect” (LATE)

• r

“Complier Average Treatment Effect”

(Bloom, 1984; Angrist, Imbens, and Rubin, 1996)

Principal Stratification

Stratum

M(1) M(0) Г=M(1)-M(0) Y(M(1))-Y(M(0)) Fraction

• f pop

Average Effect

Compliers

1 1 Y(1)=Y(0) γcompliers δcompliers

Always- takers

1 1 Y(1)-Y(1)=0 γ always

Never- takers

Y(0)-Y(0)=0 γnever

Defiers

1

• 1

Y(0)-Y(1)

Complier-average treatment effect

(“Local average treatment effect”)

randomized is T if M

• n

ITT T M E T M E Y

• n

ITT T Y E T Y E Note so B E

compliers compliers never always compliers compliers

" " ) | ( ) 1 | ( " " ) | ( ) 1 | ( / * * ) (                              

In Sum

We can estimate the Population-Average Effect of Participating if we assume Cov(Г,Δ)=0 We can estimate LATE if we assume Pr(Г≥0)=1 The latter is a weaker assumption, but does not eliminate the selection problem!

Multiple Sites

How do we take this to multiple sites to * Estimate average Impact of Program Participation * Estimate variation in the Impact of Program Participation * Two methods using simulated data:

“Small Schools of Choice” Design (J=200, 80<n<120)

) 430 . , 194 . 1 : (

2 





  values true

Method 1: Combine 2 ITT analyses

Step 1: Estimate the Impact of Treatment Assignment on the Outcome Results

2

) ( , ) ( . ) . ( .

B j ij j ij j j ij

B Var B E e e T T B Y Y         

 

) 75 . 1 , 21 . ( 499 . * 96 . 1 770 . for interval value plausible % 95 499 . 249 . ˆ 53 . , 770 . ˆ

2 2

       

j B

B se  

Step 2: Estimate the Impact of Treatment Assignment on Program Participation Results

2

) ( , ) ( . ) . ( .

G j ij j ij j j ij

G Var G E T T G M M           

 

) 86 , 55 (. for interval value plausible % 95 078 . 0061 . ˆ 703 . ˆ

2 2

   

j G

G  

Step 3: Combine Results

 

) 248 . 1 , 267 . ( 483 . * 96 . 1 095 . 1 ) ( % 95 483 . ) 703 (. ) 078 (. ) 078 (. * ) 095 . 1 ( ) 249 (. ˆ case

• ur

In 095 . 1 703 . / 770 . ˆ / ˆ ˆ /

2 2 2 2 2 2 2 2 2 2 2

                D PV D G if case

• ur

In

D D

            

  

In sum

True Values Our estimates

430 . , 19 . 1 :

2 





  values True

487 . ˆ , 10 . 1 ˆ

2 





 

Method 2: Two-Stage Generalized Least Squares: Theoretical Model

2 2

) ( , ) ( . ) . ( . : 2 Stage ) ( , ) ( . ) . ( . : 1 Stage

D j ij j ij j j ij G j ij j ij j j ij

D Var D E M M D Y Y G Var G E T T G M M                      

Method 2 in Practice

2 * * 2 * *

) ( , ˆ ) ( ) . ( ˆ .

• riginal

using variances Recompute . 3 ) ( , ) ( ) . ˆ ( . : . 2 ) . ( ˆ . ˆ : . 1

D ij ij j ij j j ij D ij ij j ij j j ij j ij OLS j ij

D Var D E M M D Y Y M D Var D E M M D Y Y HLM Do T T G M M Compute                         

Method 2=“Between Site Regression!”

C j E j j j j j j j j j j j

G D G B which from G D G G D B                ˆ ) ( ˆ ˆ ..... ) (

)] ( ˆ [  

j j D

G E Requires

Results

) 863 . , 562 (. ) 746 . , 316 (. % 95 6968 . 4856 . ˆ 0730 . , 095 . 1 ˆ

2 2 2 2

      CI se



 

Envisioning Variation: LATE Effect

“Head Start” Design (J=200, 10<n<20) “Small Schools of Choice” Design (J=200, 80<n<120) “Welfare to Work” Design (J=60, 200<n<1400)

Program Participation Model (“LATE”)

) 430 . , 194 . 1 : ( 280 . ˆ , 007 . 1 ˆ : 487 . ˆ , 095 . 1 ˆ ˆ 986 . ˆ : ) , ( ~ . ) . ( . : ) ( , ) ( . ) . ( :

2 2 2 2 2 2

                   

     

              values true WtW SSC HS N D M M D Y Y Model Impact G Var G E v v T T G M M model ion Participat

j j ij j ij j j ij j ij j ij j j ij

Profile Likelihood for LATE: “HS” Design

0.13 0.25 0.38 0.50 0.63 0.76 0.88 1.01

• 1.00

1.00 2.00 3.00

Tau Beta

Profile Likelihood for LATE: “SSOC” Design

0.13 0.25 0.38 0.50 0.63 0.76 0.88 1.01

• 0.50

1.00 2.50

Tau Beta

Profile Likelihood for LATE: “WtW” Design

0.13 0.25 0.38 0.50 0.63 0.76 0.88 1.01

• 1.00

1.00 2.00 3.00

Tau Beta

Posterior intervals for site-specific LATE Effects

Posterior Intervals for LATE: “HS” Design

• 0.55

0.27 1.09 1.90 2.72

MHAT

50.50 101.00 151.50 202.00

Posterior Intervals for LATE: “SSC” Design

50.50 101.00 151.50 202.00

• 1.21
• 0.13

0.96 2.04 3.12

MHAT

Posterior Intervals for LATE: “W to W” Design

15.50 31.00 46.50 62.00

• 0.37

0.41 1.18 1.95 2.73

MHAT

Moving Toward Explanation modeling participation, modeling impact

Total Impact of Assignment= Impact of Assignment on Participation * Impact of participation on Outcome Within site: Which persons are most likely to participate? Which persons are most likely to benefit from participation? Between Sites: How do we improve site-average participation rate? How do we enhance average benefit of participating? Models are needed at both levels because sites vary not only in organizational effectiveness but also in client composition

j j j

D G B 