The Multi-Site RCT A fleet of RCTs! Each conducted in a different - - PowerPoint PPT Presentation

SLIDE 1

The Multi-Site RCT

• A fleet of RCTs!
• Each conducted in a different social setting
• Since 2002, IES has funded 175 randomized

trials;

• Vast majority are multi-site trials (Spybrook, 2013)

SLIDE 2

Opportunities

• Can assess generalizability of impact

– Gauge variation in effect – Model heterogeneity – Does the intervention equalize outcomes?

But we must formulate appropriate analyses. This is trickier than it might seem!

SLIDE 3

Some Recent MS Trials

Study Levels Assigned Units Sites Fixed or Random sites

National Head Start Eval. 2 Children 300+ Program Sites Random Moving to Opportunity 2 Families 5 cities Fixed Boston Charter School Lotteries 2 Children Lottery pools Random Tennessee STAR 3 Teachers Schools Random Double-Dose Algebra 2 Children Schools Random

SLIDE 4

Theoretical Model

b j j b j j j j j j T ij ij ij j j ij

b u Cov b b B u u U r r T B U Y

2 2 2

) , ( ) , ( ~ , ) , ( ~ , Sites Between ) , ( ~ , Sites Within              

SLIDE 5

• 4. Unbalanced Designs and

Targets of Inference

Two kinds of imbalance

• Unequal n per site
• Unequal propensity score (probability of

assignment to the treatment group)

SLIDE 6

Targets of Inference

        

        

           

* * * * * * * * *

1 1 1 2 1 2 1 1 1 * 2 1 * 2 1 *

/ ) )( ( / ) ( / Persons

• f

population a to Generalize ) )( ( 1 ) ( 1 1 Sites

• f
• n

a Populati to Generalize

J j j j J j j j persons b J j j J j j j persons b J j j J j j j persons j J j j sites b J j j sites b J j j sites

N B U N N B N N B N B U J B J B J            

SLIDE 7

Identification

lotteries school charter e.g., ) . | ( ), . | ( ) | ( ) | ( ) | ( 2 1

j j j j ij j ij ij j ij ij ij ij ij j j ij ij

T b E T u E about Worry T b E T T u E T T Y E e T b u T Y level and level Combine             

SLIDE 8

Parameters to be estimated Properties

a)Fixed effects β Biased if precision related to B b)Centering HLM B,Var(B) Bias in β if precision is related to B (less so than fixed effects ) c) Control propensity Score B,Var(B)?? Similar to Centering for β, d)Weighting (“IPTW”) with HLM B,Var(B),Cov(B,U0) Removes bias (but may be imprecise!)

SLIDE 9

a) Fixed Effects Model

j j j J j j j j J j j j j j ij j ij ij

Y Y B where T T n B T T n e T Y

1 1 1

ˆ ) . 1 ( . ˆ ) . 1 ( . ˆ        

 

 

  

SLIDE 10

Potential Bias if heterogeneous impact, and unbalanced design

average weighted precision A T T n B T T n E T B E

j J j j j j j J j j j

                 

 

 

) . 1 ( . ) . 1 ( . ) , | ˆ (

1 1



SLIDE 11

b) Centering the Predictor AND the Outcome (“FIRC” Model)

) , ( ~ ) . (

2 b j j j ij j ij j j ij

N b b B e T T B Y Y        

SLIDE 12

Robustness (compared to fixed effects)

. ) . 1 ( . , , ) . 1 ( . ) . 1 ( . ) , , , | ˆ (

2 1 1 2 2 1 1 2 2 2 2

decreases T T n

• n

reliance increases increases ity heterogene As T T n B T T n E T E

j j j b J j j j j b j J j j j j b b

                               

 

   

        

SLIDE 13

Estimation of heterogeneity

. ) . 1 ( . , , ) . 1 ( . ] ) ˆ [( ) . 1 ( .

2 1 2 ) 1 ( 2 2 ) 1 ( 2 ) 1 ( 1 2 ) 1 ( 2 2 ) 1 ( 2 ) (

decreases T T n

• n

reliance increases increases ity heterogene As T T n V B T T n

j j j b J j j j j m m b j m j J j j j j m m b m b

                   

 

        

      

SLIDE 14

c) Control the Propensity Score

ij ij j j ij

e T B U Y Level    1

2

, . 2

b j j j j j

• f

estimate good no effect treatment average for centering to similar Very b B u T U Level         

SLIDE 15

Variance Estimation

   

 

   

 

                        

                   

 

) 1 ( ) 1 ( 1 ) 1 ( ) 1 ( ) 1 ( 1 1 ) 1 ( ) 1 ( 1 ) 1 ( ) 1 ( 1 1 1 ) 1 ( ) 1 ( 1 ) 1 ( ) 1 ( ) (

ˆ ˆ ) ( * ) (

m j j m m j j T m j m j J j m j m m j m J j m j m m j m m

B T U d vec vec    d d V τ V τ V τ V τ τ

SLIDE 16

Summary so far

Four commonly used options * limit what we can estimate * produce inconsistent estimates of what we can estimate.

SLIDE 17

6.HLM with Inverse Probability

• f Treatment Weighting
• IPTW
• Embedding within HLM
• Always produces consistent estimates
• May be quite imprecise

SLIDE 18

Inverse Probability of Treatment Weights (Robins and Greenland, 2000)

j ij j ij ij j ij ij j ij ij

T T T T T T w so T T w T If T T w T If            1 1 ) 1 ( ) 1 /( ) 1 ( , / , 1

SLIDE 19

Weighting Schemes for HLM

Level-2 weight Level-1 weight Result

Weights site-specific estimates of impact by Weights site-specific estimates equally

The picture can't be displayed.

j ij j ij

T T T T T T . 1 ) 1 )( 1 ( .    

           

j ij j ij j

T T T T T T n n . 1 ) 1 )( 1 ( .

1

j

n n n j

SLIDE 20

Estimation via Weighted log Likelihood

(weighted complete-data log likelihood)

, |) | 2 log( ) 2 log( / ) ( : )] ( ln[ )] | ( ln[ )] , ( ln[ ) , ( ~ ), , ( ~ ,

1 2 1 1 1 1 2 2 1 1 1 2 2 2 2 1 1 1 2

J w N w where J u u w N u Z X Y w l weighting After u p u Y f u Y h N e N u e u Z X Y

J j j J j n i ij J j j T j j J j n i rj T ij T ij ij ij J j n i ij w ij j ij j T ij T ij j

j j j

              

    

        

        

SLIDE 21

Two examples of weighting schemes

Equally weight people Equally weight sites

Level-1 weight Level-1 weight

Level-2 weight Level-2 weight 1

j ij j ij

T T T T T T     1 1 ) 1 (

J B

J j j







1

/ ˆ ˆ 

 

 



J j j J j j j

n B n

1 1 1 /

ˆ ˆ 

n n j /

 

j j J j n n b

V B J

j

  



  2 1 1 2

) ˆ ˆ ( ˆ  

 



 

  

J j j j b

V B J

1 2 1 2

) ˆ ˆ ( ˆ  

 

j j j J j n n b

C U B J

j

   



 

) ˆ ˆ )( ˆ ˆ ( ˆ

1 1

  

           

j ij j ij n n

T T T T T T

j

1 1 ) 1 (

 



 

   

J j j j j b

C U B J

1 1

) ˆ ˆ )( ˆ ˆ ( ˆ   

SLIDE 22

• 7. Discussion

A Class of Estimators…

We are approximating various balanced designs by maximizing the weighted log likelihood This gives us a family of method-of-moments estimators Hence no reliance on normality or homoscedasticity May be quite imprecise

SLIDE 23

Extensions

• Extend easily to multi-site clustered

randomized designs

• Balance propensity within covariate classes

in randomized studies

• Extend to observational studies
• Extend to weighted mediation models (Hong

et al, 2012)

• Application to local average treatment effects

(Raudenbush, Nomi, and Reardon, 2016)