SLIDE 1 THE ESSENTIAL ROLE OF EXTERNAL AND CONSTRUCT VALIDITY FOR CAUSAL IDENTIFICATION
Kevin Esterling, School of Public Policy & Department
David Brady, School of Public Policy, UCR & WZB Berlin Social Science Center Eric Schwitzgebel, Department of Philosophy, UCR
https://www.dropbox.com/sh/ti09kj0f53e1yw2/AABmNY9SSzTDMCkeviHj_nvja?dl=0
SLIDE 2 Introductory Vignette
Gold Standard Lab (GSL) Conducts an Internally-Valid RCT to Evaluate a Voter Turnout Program
Well-Trained/Energetic Canvassers Visit Houses and Read Script Persuading Non-Voters to Vote in Minneapolis 90% of Treated Non-Voters Vote, Only 10% of Control Vote GSL Writes: “Exposure to the voter turnout script causes greater turnout.“
GSL Does Same RCT of Same Program in Atlanta
Surprisingly, No Difference Between Treated and Control Perhaps Due to Low Trust and Bad Voting System in Atlanta
- vs. High Trust and Good Voting System in Minneapolis
GSL Reruns RCT in Minneapolis with Inexperienced and Cheaper Canvassers
This Time, No Difference Between Treated & Control
SLIDE 3 Argument & Outline
We Argue Construct & External Validity Are Equally Important As Internal Validity for Causal Inference
All Three Are Jointly Required – 3 Legs of a Stool Causal Inference Always Requires Semantic Communication About a Generalizable Claim
Outline
1.
The Credibility Revolution
2.
Conceptualize Validity and Causality
3.
Essential Role of Construct Validity for Causal Inference
4.
Essential Role of External Validity for Causal Inference
SLIDE 4
The Credibility Revolution
The Rise of Internally-Valid Quantitative Designs for Identifying “Causal Effects”
Internal Validity: No Confounding of a Manipulated Cause A for Effect B in One Setting Potential Outcomes (Rubin) and Structured Causal Model (Pearl) Frameworks Lexical Priority on Internal Over Other Validities Internal Validity Has Dominated and Construct & External Have Been Subjugated in Past 30 Years
Response & Critique of Prior Verbal Justifications Common in Applied Empirical Social Sciences
Self-Validating Methodologies Eliminate Confoundedness and Displace Verbal Justifications
SLIDE 5
Validity and Causality
Credibility Revolution Presumed Internal Validity ≈ Causal Inference
Internal Validity Framed As Warrant “Local” or “Molar” or “Black Box” Causal Claim (Cook & Campbell)
A Causal Claim is VALID if the Relationship In Fact Holds in the World
Semantic Causal Relata = Actual Causal Relata
Causal Inference: The Process of Generalizing from Evidence to Valid Claim
VALIDATION Is the Epistemic Activity of Supporting the Claim With Evidence – Such That Claim is Warranted
SLIDE 6 Validity and Causality
The Three Validities Correspond to Three Potential Failures in Causal Inference
Internal: Fail Due to Unobserved Confounding Construct: Fail to Assign Correct Semantic Labels to Cause
- r/and Effect Such That Claim is False (High Quality Canvasser
Was Treatment Not Script or Perhaps Turnout Misreported) External: Fail to Identify Contextual Factors Moderating Treatment Such That Claim is False (Minneapolis vs. Atlanta Trust & Voting System)
Goal is to Infer a Valid Causal Generalization:
Internal: Identify A Caused B IN C Construct: Correctly Label A & B External: Identify Conditions C Where A Causes B
SLIDE 7
Construct Validity
Priority on Internal Validity -> Neglect of Measurement Measurement = Matter of Correlations (Cronbach) Internal Validity Advocates Assume Problem Away
Exclusion Restriction: Assume Manipulation Has Effect ONLY from Exposure to the Actual Cause Stable Unit Treatment Value Assumption (SUTVA): Assume Treatment Each Unit Receives is Not Affected by Other Units
Reality: Measures Bundle Active & Inert Ingredients
Internal Validity Alone Abdicates How to Assign Labels One Doesn’t Even Know What One Is Talking About
SLIDE 8
Construct Validity
Definition of CONSTRUCT VALIDITY: Correct Semantic Labeling of the Ontological Causal Relata
BOTH A (Labeled Cause: Script) & B (Labeled Effect: Voting)
Semantic Label Applies to Active NOT Inert Ingredient
Active is the Necessary Component of a Sufficient Cause Script in Vignette Inert is (Hopefully) Unnecessary Component Training/Enthusiasm of Canvassers in Vignette
Most Focus on Construct Validity of Outcome, But We Focus on Cause Here
SLIDE 9
Some Preliminaries for Formalization Active & Inert Ingredients of Cause (A), Effect (B), and Context (C) Formalization of Internally Valid Claim
What You Claim What You Know
SLIDE 10
Construct Validity of Cause Only
WEAK Construct in the Cause STRONG Construct in the Cause
SLIDE 11
External Validity
External Validity Traditionally Defined As Generalizability and Extrapolation Across Settings
Prioritizing Internal & Implicitly Vaguely Generalizing True External = Unreasonably High Threshold
Pervasive Low “External Validity” RCTs & Experiments
GSL Had Internal But Does Not Know Why Treatment Worked in 1st Minneapolis But NOT in Atlanta Variation in Treatment Effects Across Settings Revels Causes Are Actually Treatment*Context Moderations
Internal Validity = Very Limited Kind of Knowledge
Only a Causal Effect for a Given Setting, and Only Knowable Retrospectively; Cartwright (2011): “it works somewhere.” Actually: It WorkED for Specific People Exposed to Specific Events in a Specific Time and Place Not “It Works Widely” or “It Will Work for Us” Internal Validity Rules Out Constant/Stable Characteristics of Settings Because Not Manipulated
SLIDE 12 External Validity
Two Major Responses Don’t Really Solve Problem
1.
No Intention to Produce “Universal” Knowledge: Only Saying A Causes B in C and Cannot Claim Beyond C
- Historicist’s Refuge: Still Do Not Understand Treatment
Effect Because Do Not Know What Contextual Factors Are
2.
Go Forth Across “Range of Settings”
- Problematic “Simple Enumerative Induction”
- Knowing What Defines “Range of Settings” Presumes
Knowledge of Relevant Contextual Factors
- Contextual Factors are Unknown Confounders Without a
Self-Validating Methodology to Identify
- “Transportability” (Pearl) Through Effect Moderation in DAG
Assumes You Know Contextual Factors
- “Structured Speculation” (Banerjee et al) Relies on Same
Imputed Verbal Justification Credibility Revolution Critiqued
SLIDE 13 External Validity
CONTEXTUAL FACTORS: Non-Manipulated Conditions
- f a Setting That Augment or Undermine a Cause
Cartwright: “Helping Factors” or “Countering Causes” Other Necessary Components to a Causal Effect or “Causal Field” In Which Treatment Takes Place (Oxygen -> Fire) Vignette: Trust & Voting Systems in Minneapolis vs. Atlanta
Revised Definition of EXTERNAL VALIDITY:
Identify How Contextual Factors Interact With Treatment to Produce Causal Effects That Generalize Across Settings Specify Which Contextual Factors Define Range of Settings
Causal Inference Only Warranted With Accurate Claim
- f How to Generalize Causal Claim Across Settings
(i.e. External Validity is Essential to Causal Inference)
SLIDE 14
External Validity
WEAK External STRONG External
SLIDE 15
Conclusions
Excesses & Trappings of the Credibility Revolution Revised Definitions:
Construct: Correct Semantic Labeling of the Ontological Causal Relata; Active vs. Inert Ingredients External: Identify How Contextual Factors Interact With Treatment to Produce Causal Effects That Generalize
Formalization Clarifies the Many Hidden Confounds
Construct: Inert Ingredients External: Contextual Factors
Construct & External Validity Are Equally Important As Internal Validity for Causal Inference
Internal Validity Alone Cannot Warrant Causal Claim Advocate for Renewed Commitment to Construct & External Causal Inference Always Requires Semantic Communication About a Generalizable Claim
SLIDE 16
Bringing the Whole Formalization Together
