Experiments and Causal Inference Erik Gahner Larsen Advanced - - PowerPoint PPT Presentation
Experiments and Causal Inference Erik Gahner Larsen Advanced applied statistics, 2015 1 / 67 Articles published in APSR 10 15 0 5 Experiments in political science research 190609 191014 191519 192024 Political Science
1 / 67
Cambridge Handbook of Experimental Political Science (Druckman et al. 2011, 5)
2 / 67
3 / 67
4 / 67
▸ Causal effects ▸ Experiments and assumptions ▸ Issues in experimental research ▸ Types of experiments
5 / 67
▸ How would we test this in an observational setting? ▸ What is the problem?
6 / 67
▸ We need strong designs in order to make causal inferences
▸ Remember: Science is all about causality
▸ The issue it not the data we have . . . but the data we do not have. ▸ “Easy” to measure the factual world
▸ What about the counterfactual world?
▸ We need theoretical and statistical tools to make valid counterfactuals
7 / 67
▸ From estimation strategies (statistics) to identification strategies
▸ “Without an experiment, a natural experiment, a discontinuity, or
8 / 67
▸ We are interested in potential outcomes to define causal effects ▸ For individual i, we have a potential outcome: Yi ▸ Treatment: Wi ▸ Potential outcome given treatment treatment status: Yi(Wi) ▸ Two potential outcomes: Yi(1), Yi(0) ▸ Unit causal effect: The difference between a unit’s potential outcome
9 / 67
10 / 67
11 / 67
12 / 67
13 / 67
▸ “It is impossible to observe the value of Yi(1) and Yi(0) on the same
▸ We observe one outcome: the realised outcome
14 / 67
15 / 67
16 / 67
17 / 67
▸ Rosenbaum (2010, 35) ▸ The FPCI is a missing data problem. What is the solution?
18 / 67
19 / 67
▸ Create two groups of observations that are, in expectation, identical
▸ Create a counterfactual group. ▸ Guarantees that the treatment is prior to the outcome, avoiding
▸ P(W ) = 0.5 (coin flip)
20 / 67
▸ Pretreatment covariates, X ▸ Unconfoundedness (Y (1), Y (0), X) ⊥ W ▸ What about (Y (1), Y (0)) ⊥ W ∣X?
▸ We will address this issue next week 21 / 67
▸ What most scholars are interested in ▸ Average treatment effect:
22 / 67
▸ A collection of implied assumptions about the effect of treatments on
▸ “The potential outcomes for any unit do not vary with the treatments
23 / 67
24 / 67
▸ Rules out social interactions and general equilibrium effects. ▸ Rules out any effect of the assignment mechanism on potential
25 / 67
▸ Wi is assignment to treatment ▸ Subjects can - in many cases - decide not to comply ▸ Di: treatment status (1 if treated, 0 if not)
26 / 67
▸ Always-takers will always be treated ▸ Wi = 1, Di = 1 ▸ Wi = 0, Di = 1 ▸ Facebook example: Will use facebook independent of treatment
27 / 67
▸ Never-takers will never be treated ▸ Wi = 1, Di = 0 ▸ Wi = 0, Di = 0 ▸ Facebook example: Will not use facebook independent of treatment
28 / 67
▸ Cooperators will. . . cooperate ▸ Wi = 1, Di = 1 ▸ Wi = 0, Di = 0 ▸ Facebook example: Will only use facebook if assigned to treatment
29 / 67
▸ Defiers will. . . do the opposite ▸ Wi = 1, Di = 0 ▸ Wi = 0, Di = 1 ▸ Facebook example: Will use facebook if not assigned to treatment
30 / 67
▸ The cases whose treatment status can be changed (hint: cooperators)
31 / 67
32 / 67
▸ We only have realised outcomes (we need a counterfactual) ▸ Hard to say whether we are dealing with compliance or noncompliance ▸ Remember: Try to measure compliance!
33 / 67
▸ Our effects are often intention-to-treat (ITT) estimates. ▸ Mean difference on Y between subjects assigned to treatment and
34 / 67
▸ From Table 5.5 in Rosenbaum (2002, 182). ▸ Y = forced expiratory volume (higher numbers signifying better lung
▸ Will subject exercice with encouragement? (di(1)) ▸ Will subject exercice without encouragement? (di(0))
35 / 67
36 / 67
37 / 67
38 / 67
39 / 67
40 / 67
41 / 67
42 / 67
43 / 67
44 / 67
45 / 67
46 / 67
47 / 67
48 / 67
49 / 67
▸ Lab experiments ▸ Survey experiments ▸ Field experiments ▸ Natural experiments ▸ Quasi-experiments
50 / 67
51 / 67
52 / 67
▸ Lab: Get subjects into the lab, randomize, treatment group use
▸ Survey: Get subjects to answer a survey, randomize, treatment group
▸ Field: Get subjects to sign up, randomize, treatment group use
▸ Natural/quasi: Utilize (as-if) random variation in the access to
▸ Observational: Ask people about facebook use and life satisfaction :-(
53 / 67
54 / 67
55 / 67
Control Treatment 2 4 6 Before Now Before Now
56 / 67
57 / 67
▸ Always report the unadjusted treatment effect: “If an estimated
▸ We use experiments so we don’t have to care about covariates: “Yet,
▸ Covariates reduce noise, increases the chance that we reach statistical
▸ Positive view: Variables measured before the variables of interest was
58 / 67
▸ Remember theory. ▸ Consider – from a practical perspective – whether an experiment is
▸ Specify hypothesis/hypotheses prior to the data collection
▸ Prespecification (if you plan to publish in academic journals:
▸ What is your dependent variable? 59 / 67
▸ We have specific guidelines for reporting experimental research in
▸ See Gerber et al. (2014): Reporting Guidelines for Experimental
60 / 67
▸ External validity is all about your theory ▸ And remember: “It makes no sense to say that some empirical
61 / 67
▸ One of the biggest issues with experiments (and all research) today:
▸ Novelty bias (especially in political science!) ▸ We need more replications of existing experiments ▸ Sadly, only few examples of direct replications in political science ▸ “Indeed, few experimental literatures have generated repicable
62 / 67
▸ Make sure that your research is reproducible (STATA do-file and/or R
▸ Share your data ▸ Reproduce and replicate existing studies (great way to “learn science”) ▸ Be transparent (what did you do, how did you do it etc.)
63 / 67
▸ Cochran’s Basic Advice: “The planner of an observational study
64 / 67
▸ Think of experiments as observational studies ▸ Think of observational studies as experiments
65 / 67
66 / 67
▸ Today: Experiments ▸ Next: Matching ▸ Lecture 13 and 14: Natural experiments
▸ Regression-Discontinuity Designs ▸ Instrumental Variable Regression
▸ Lab session 6 and 7: R and matching
67 / 67