Gov 2002: 3. Randomization Inference Matthew Blackwell September 10, 2015
Where are we? Where are we going? • Last week: ▶ What can we identify using randomization? ▶ Estimators were justifjed via unbiasedness and consistency. ▶ Standard errors, test, and CIs were asymptotic. ▶ Neyman’s approach to experiments • This week: ▶ Condition on the experiment at hand. ▶ Get correct p-values and CIs just relying on randomization. ▶ Fisher’s approach to randomized experiments.
Effect of not having a runoff in sub-Sarahan African related to harrassment of opposition parties ( 𝑍 𝑗 ) in sub-Sahara African countries. to suppress turnout through intimidation. courting of small parties. • Glynn and Ichino (2012): is not having a runofg ( 𝐸 𝑗 = 1 ) • Without runofgs ( 𝐸 𝑗 = 1 ), only need a plurality ⇝ incentives • With runofgs ( 𝐸 𝑗 = 0 ), largest party needs wider support ⇝
Data on runoffs 0 0 Congo 0 0 0 ? Madagascar 0 0 0 ? Central African Republic 0 No runofg? 0 ? Ghana 0 0 0 ? Guinea-Bissau 0 0 0 ? ? 0 1 Tanzania Intimidation Unit 𝐸 𝑗 𝑍 𝑗 𝑍 𝑗 (0) 𝑍 𝑗 (1) Cameroon 1 1 ? 1 Kenya 1 1 ? 1 Malawi 1 1 ? 1 Nigeria 1 1 ? 1 data? • Clear difgerence-in-means: 0.8 • Very small sample size ⇝ can we learn anything from this
CA recall election were 135 candidates. on the fjrst page and some did not. share for a candidate? • Ho & Imai (2006): 2003 CA gubernatorial recall election there • Ballot order was randomly assigned so some people ended up • Can we detect an efgect of being on the fjrst page on the vote
What is randomization inference? make inferences. • Randomization inference (RI) = using the randomization to • Null hypothesis of no efgect for any unit ⇝ very strong. • Allows us to make exact inferences. ▶ No reliance on large-sample approximations. • Allows us to make distribution-free inferences. ▶ No reliance on normality, etc. • ⇝ truly nonparametric
Brief review of hypothesis testing RI focuses on hypothesis testing, so it’s helpful to review. 1. Choose a null hypothesis: 2. Choose a test statistic. ̅ 𝑌)/(𝑡/√𝑜) 3. Determine the distribution of the test statistic under the null. should we expect? 4. Calculate the probability of the test statistics under the null. p-value ▶ 𝐼 ∶ 𝛾 = 0 or 𝐼 ∶ 𝜐 = 0 . ▶ No average treatment efgect. ▶ Claim we would like to reject. ▶ 𝑎 𝑗 = (𝑌 𝑗 − ▶ Statistical thought experiment: we know the truth, what data ▶ What is this called?
Sharp null hypothesis of no effect Sharp Null Hypothesis ∀𝑗 imply the sharp null. potential outcomes. 𝐼 ∶ 𝜐 𝑗 = 𝑍 𝑗 (1) − 𝑍 𝑗 (0) = 0 • Motto: “No efgect means no efgect” • Difgerent than no average treatment efgect, which does not • Take a simple example with two units: 𝜐 = 1 𝜐 = −1 • Here, 𝜐 = 0 but the sharp null is violated. • This null hypothesis formally links the observed data to all
Life under the sharp null CAR ? 0 Congo 0 0 0 ? Madagascar 0 0 0 ? 0 1 0 0 ? Ghana 0 0 0 ? Guinea-Bissau 0 0 0 0 Tanzania We can use the sharp null ( 𝑍 𝑗 (1) − 𝑍 𝑗 (0) = 0 ) to fjll in the missing 1 potential outcomes: No runofg? Intimidation Unit 𝐸 𝑗 𝑍 𝑗 𝑍 𝑗 (0) 𝑍 𝑗 (1) Cameroon 1 1 ? Kenya 1 1 1 ? 1 Malawi 1 1 ? 1 Nigeria 1 1 ? ?
Life under the sharp null CAR 0 0 Congo 0 0 0 0 Madagascar 0 0 0 0 0 1 0 0 0 Ghana 0 0 0 0 Guinea-Bissau 0 0 0 0 Tanzania We can use the sharp null ( 𝑍 𝑗 (1) − 𝑍 𝑗 (0) = 0 ) to fjll in the missing 1 potential outcomes: No runofg? Intimidation Unit 𝐸 𝑗 𝑍 𝑗 𝑍 𝑗 (0) 𝑍 𝑗 (1) Cameroon 1 1 1 Kenya 1 1 1 1 1 Malawi 1 1 1 1 Nigeria 1 1 1 0
evidence against them! Comparison to the average null • Sharp null allows us to say that 𝑍 𝑗 (1) = 𝑍 𝑗 (0) ▶ ⇝ impute all potential outcomes. • Average null only allows us to say that 𝔽[𝑍 𝑗 (1)] = 𝔽[𝑍 𝑗 (0)] ▶ ⇝ tells us nothing about the individual causal efgects. • Don’t need to believe either hypothesis ⇝ looking for • Stochastic version of “proof by contradiction.”
Other sharp nulls • Sharp null of no efgect is not the only sharp null of no efgect. • Sharp null in general is one of a constant additive efgect: 𝐼 ∶ 𝜐 𝑗 = 0.2 . ▶ Implies that 𝑍 𝑗 (1) = 𝑍 𝑗 (0) + 0.2 . ▶ Can still calculate all the potential outcomes! • More generally, we could have 𝐼 ∶ 𝜐 𝑗 = 𝜐 for a fjxed 𝜐 • Complications: why constant and additive?
Test statistic Test Statistic A test statistic is a known, scalar quantity calculated from the treatment assignments and the observed outcomes: 𝑢(𝐄, 𝐙) some interesting alternative hypothesis. • Typically measures the relationship between two variables. • Test statistics help distinguish between the sharp null and • Want a test statistic with high statistical power: ▶ Has large values when the null is false ▶ These large values are unlikely when then null is true. • These will help us perform a test of the sharp null. • Many possible tests to choose from!
Null/randomization disitribution null? over difgerent randomizations? doesn’t matter. 𝐄 , outcomes won’t change. possible treatment assignment vector. • What is the distribution of the test statistic under the sharp • If there was no efgect, what test statistics would we expect • Key insight of RI: under sharp null, the treatment assignment ▶ Explicitly assuming that if we go from 𝐄 to ▶ 𝑍 𝑗 (1) = 𝑍 𝑗 (0) = 𝑍 𝑗 • Randomization distribution: set of test statistics for each
Calculate p-values the sharp null holds? total number of randomizations. Pr(𝑢(𝐞, 𝐙) ≥ 𝑢(𝐄, 𝐙)|𝜐 = 0) = 𝐿 than 100𝛽% of the time. • How often would we get a test statistic this big or bigger if • Easy to calculate once we have the randomization distribution: ▶ Number of test statistics bigger than the observed divided by ∑ 𝐞∈ 𝕁(𝑢(𝐞, 𝐙) ≥ 𝑢(𝐄, 𝐙)) • These are exact tests: ▶ p-values are exact, not approximations. ▶ with a rejection threshold of 𝛽 , RI test will falsely reject less
RI guide 𝐄 , 𝐙) . 𝑈 𝐿 } . ̃ 1. Choose a sharp null hypothesis and a test statistic, 5. Repeat steps 3 and 4 for all possible randomization to get ̃ 4. Calculate 𝐄 . 2. Calculate observed test statistic: 𝑈 = 𝑢(𝐄, 𝐙) . 3. Pick difgerent treatment vector 𝑈 = 𝑢( 𝑈 = { ̃ 𝑈 , … , ̃ 6. Calculate the p-value: 𝑞 = 𝑙= 𝕁( ̃ 𝐿 ∑ 𝐿 𝑈 𝑙 ≥ 𝑈)
Difference in means 𝑂 𝑑 (1 − 𝐸 𝑗 )𝑍 𝑗 𝑗= 𝑂 relatively few outliers in the the potential outcomes. 𝑗= 𝑂 𝑂 𝑢 1 • Absolute difgerence in means estimator: 𝐸 𝑗 𝑍 𝑗 − 1 𝑈 difg = • Larger values of 𝑈 difg are evidence against the sharp null. • Good estimator for constant, additive treatment efgects and
Example inspirational stories of learning from our graduate students. • Suppose we are targeting 6 people for donations to Harvard. • As an encouragement, we send 3 of them a mailer with • Afterwards, we observe them giving between $0 and $5. • Simple example to show the steps of RI in a concrete case.
Randomization distribution 0 0 Ted 0 4 4 (4) Marco 0 0 0 (0) Scott 0 1 1 (1) (0) 1 Mailer 1 Contr. Unit 𝐸 𝑗 𝑍 𝑗 𝑍 𝑗 (0) 𝑍 𝑗 (1) Donald 3 Ben (3) 3 Carly 1 5 (5) 5 𝑈 rank = |8/3 − 5/3| = 1
Randomization distribution 0 Ted 1 4 4 (4) Marco 1 0 (0) Mailer Scott 1 1 1 (1) ̃ ̃ ̃ 0 (0) 0 0 Contr. Unit 𝐸 𝑗 𝑍 𝑗 𝑍 𝑗 (0) 𝑍 𝑗 (1) Donald 1 3 (3) 3 Carly 1 5 (5) 5 Ben 𝑈 difg = |12/3 − 1/3| = 3.67 𝑈 difg = |8/3 − 5/3| = 1 𝑈 difg = |9/3 − 4/3| = 1.67
Randomization distribution 1 0 1 0.33 0 1 0 1 1 0 1.67 0 1 0 1 0 1 1 0 1.67 0 1 1 1 0 1.67 0 0 1 1 0 1 0 1.00 0 2.33 1 1 1.00 0 0 1 0 1 3.67 0 0 0 0 1 1 1 1 1 0 0 1 0 0 1 1 0.33 0 1 1 1 1 0 1.67 0 0 1 0 𝐸 0 3.67 1 1 0 0 1 1.00 0 1 1 0 0 0 1 0 1 1 1 𝐸 𝐸 𝐸 𝐸 𝐸 |Difg in means| 1 0 1 0 0 0 1.00 1 1 1.67 0 0 1 0 0 1 1 0 0.33 0 1.67 0 1 0 1 1.00 1 0 1 1 1 1 1 0 0 0.33 1 0 0 0 1 0 2.33 1 0 1 0 3.67
In R 0 1 1 ## 4 0 1 0 0 1 0 ## 5 0 0 1 0 0 1 ## 6 0 0 0 1 0 0 0 library(ri) 1 y <- c(3, 5, 0, 4, 0, 1) D <- c(1, 1, 1, 0, 0, 0) Dbold <- genperms(D) Dbold[, 1:6] ## [,1] [,2] [,3] [,4] [,5] [,6] ## 1 1 1 1 1 1 1 ## 2 1 1 1 1 0 0 ## 3 0 T_stat <- abs(mean(y[D == 1]) - mean(y[D == 0]))
Calculate means rdist <- rep(NA, times = ncol(Dbold)) for (i in 1:ncol(Dbold)) { D_tilde <- Dbold[, i] 0])) } rdist ## [1] 1.0000000 3.6666667 1.0000000 1.6666667 ## [5] 0.3333333 2.3333333 1.6666667 0.3333333 ## [9] 1.0000000 1.6666667 1.6666667 1.0000000 ## [13] 0.3333333 1.6666667 2.3333333 0.3333333 ## [17] 1.6666667 1.0000000 3.6666667 1.0000000 rdist[i] <- abs(mean(y[D_tilde == 1]) - mean(y[D_tilde ==
Recommend
More recommend
