Designing Experiments in Political Science Delegation in - - PowerPoint PPT Presentation

▶

Jun 07, 2023 40 likes •531 views

Designing Experiments in Political Science Delegation in Bureaucracies Michael F. Stoffel University of Konstanz Department of Politics and Public Administration UniversittKonstanz5Fach9?578457Konstanz DrxMichaelHerrmann

SLIDE 1

Designing Experiments in Political Science

Delegation in Bureaucracies Michael F. Stoffel

University of Konstanz Department of Politics and Public Administration

DrxäMichaeläHerrmann FachbereichäPolitik3 und Verwaltungswissenschaft Fachä9? Universitätsstraßeä,d D378464äKonstanz Telä+49ä753,ä883477, Faxä+49ä753,ä8834?dd MichaelxHerrmann@uni3konstanzxde ??x ??x ???? UniversitätäKonstanz5äFachä9?5ä78457äKonstanz

Dagstuhl Seminar “Empirical Evaluation for Graph Drawing” 26 to 30 January 2015

SLIDE 2

Experimental Political Science

We are limited by the impossibility of

experiment. Politics is an observational,

not an experimental science ...

A. Lawrence Lowell

President of the APSA, 1910

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 3

Experimental Political Science

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 4

Examples of Applications

media effects (Iyengar and Kinder 1987) mobilization (Gerber and Green 2000) voting (Lodge, McGraw, and Stroh 1989) legislative and bureaucratic rules (Eavey and Miller 1984; Miller, Hammond, and Kile 1996) foreign policy decisionmaking (Geva and Mintz 1997) international negotiations (Druckman 1994) coalition bargaining (Riker 1967; Fr´ echette et al. 2003) electoral systems (Morton and Williams 1999)

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 5

Types of Experimental Approaches

Laboratory experiments, a.k.a controlled experiments Problem: sample often not representative Survey experiments Problem: treatment assignment can be corroborated Field experiments Problem: less control over experimental stimuli (Computer experiments)

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 6

Types of Experimental Approaches

Laboratory experiments, a.k.a controlled experiments Problem: sample often not representative Survey experiments Problem: treatment assignment can be corroborated Field experiments Problem: less control over experimental stimuli (Computer experiments)

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 7

Coffee break ...

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 8

Foundations

Potential outcomes framework (Neyman, 1923) There are two different potential outcomes for the same person depending on whether or not she receives a treatment (counterfactual).

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 9

Foundations

Potential outcomes framework (Neyman, 1923) There are two different potential outcomes for the same person depending on whether or not she receives a treatment (counterfactual). Fundamental problem of causal inference (Holland 1986) We cannot simultaneously observe a person or entity in its treated and untreated states.

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 10

Foundations

Potential outcomes framework (Neyman, 1923) There are two different potential outcomes for the same person depending on whether or not she receives a treatment (counterfactual). Fundamental problem of causal inference (Holland 1986) We cannot simultaneously observe a person or entity in its treated and untreated states. Solution: Take two groups of individuals

ne receives the treatment, the other serves as the control

calculate the mean value on the outcome variable for both groups calculate the difference between the means = ⇒ average treatment effect (ATE)

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 11

Assumptions I

Unconfoundedness Assignment to treatment and control group is independent from the potential outcomes.

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 12

Assumptions I

Unconfoundedness Assignment to treatment and control group is independent from the potential outcomes. Individuals are individual and not identical Possible characteristics of each individual may affect

the assignment to the treatment (“self-selection”) the way the treatment works

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 13

Assumptions I

Unconfoundedness Assignment to treatment and control group is independent from the potential outcomes. Individuals are individual and not identical Possible characteristics of each individual may affect

the assignment to the treatment (“self-selection”) the way the treatment works

Random assignment (Fisher 1935) Each individual has an equal probability of being in the treatment group. = ⇒ all groups have the same individual characteristics in expectation

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 14

Assumptions II

Stable Unit Treatment Value Assumption (SUTVA) “The observation on one unit should be unaffected by the particular assignment of treatments to the other units.” (Cox, 1958)

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 15

Assumptions II

Stable Unit Treatment Value Assumption (SUTVA) “The observation on one unit should be unaffected by the particular assignment of treatments to the other units.” (Cox, 1958) Outcome of individual i does not depend on whether or not individual j was given a treatment. This it typically violated under social interference, i.e., if individuals interact (broadly defined).

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 16

Internal and external validity

Internal validity means whether the design of an experiment and the causal argumentation are correct. External validity asks whether the conclusions drawn from an experiment can be generalized to a larger population.

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 17

Threats to internal validity

Failure of randomization Social interference, a.k.a. diffusion (violation of SUTVA) Confounding variables Selection-Maturation Interaction Maturation (fatigue and learning) Repeated testing Non-compliance with experimental protocol Attrition (loss of participant), e.g., through dropout, non-response, or withdrawal Reactivity, i.e., individuals alter their performance or behavior due to the awareness that they are being observed (placebo, novelty, and Hawthorne effects) Control vs. treatment group motivation (John Henry effect) Experimenter bias To be continued ...

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 18

Furthering internal validity

Make the task sufficiently absorbing that the subject finds it more interesting to concentrate on the task at hand Pilot study, pre-tests

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 19

Threats to external validity

Sample is not representative = ⇒ confounding factors may affect the way the treatment works At the most extreme: individuals that self-select into experiments might differ from those that do not Experimental situation differs from “reality” Does the experimental stimulus resemble the “true” stimulus To be continued ...

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 20

Extension: within-subjects design

Thus far: between-subject design, i.e., each individual only receives

ne treatment.

Now: within-subject design, a.k.a repeated-measures design, i.e., each individual receives more than one treatment.

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 21

Extension: within-subjects design

Thus far: between-subject design, i.e., each individual only receives

ne treatment.

Now: within-subject design, a.k.a repeated-measures design, i.e., each individual receives more than one treatment. Advantages Given the same number of subjects, statistical power

increases. Thus, the experiment can be run with fewer

subjects and is cheaper. Eliminates differences between treatment and control group

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 22

Extension: within-subjects design

Thus far: between-subject design, i.e., each individual only receives

ne treatment.

Now: within-subject design, a.k.a repeated-measures design, i.e., each individual receives more than one treatment. Advantages Given the same number of subjects, statistical power

increases. Thus, the experiment can be run with fewer

subjects and is cheaper. Eliminates differences between treatment and control group Drawbacks Being exposed to one treatment may influence the effect of another treatment (carryover effect) Fatigue and learning/practice effects

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 23

Extension: within-subjects design

If possible, use a counterbalanced measures design; that is, test the different treatments in varying order. E.g., if there are three treatments A, B, and C, there are six possible orders to test: ABC, ACB, BAC, BCA, CAB and CBA.

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 24

Extension: within-subjects design

If possible, use a counterbalanced measures design; that is, test the different treatments in varying order. E.g., if there are three treatments A, B, and C, there are six possible orders to test: ABC, ACB, BAC, BCA, CAB and CBA. You can also use Latin square or randomization to determine treatment order and only assign some treatments to each individual (especially with many treatments).

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 25

Extension: within-subjects design

If possible, use a counterbalanced measures design; that is, test the different treatments in varying order. E.g., if there are three treatments A, B, and C, there are six possible orders to test: ABC, ACB, BAC, BCA, CAB and CBA. You can also use Latin square or randomization to determine treatment order and only assign some treatments to each individual (especially with many treatments). Check after experiment, whether a specific treatment order had an effect on the outcome (it shouldn’t).

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 26

Application example: Delegation in bureaucracies

“Information Accuracy in Legislative Oversight: Theoretical Implications and Experimental Evidence” (under review, with Susumu Shikano and Markus Tepe)

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 27

Application example: Delegation in bureaucracies

“Information Accuracy in Legislative Oversight: Theoretical Implications and Experimental Evidence” (under review, with Susumu Shikano and Markus Tepe) Modern political systems rely on the division of labor, just as modern economies There are hierarchies and differing responsibilities Principals delegate tasks to agents

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 28

Application example: Delegation in bureaucracies

“Information Accuracy in Legislative Oversight: Theoretical Implications and Experimental Evidence” (under review, with Susumu Shikano and Markus Tepe) Modern political systems rely on the division of labor, just as modern economies There are hierarchies and differing responsibilities Principals delegate tasks to agents Principal-agent relationships are characterized by informational asymmetries: agents have more information about a specific topic and their task If preferences diverge, agents can exploit their informational advantage at the expense of their principal

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 29

Application example: Delegation in bureaucracies

Legislators and ministers delegate tasks to the bureaucracy Legislators can provide oversight and impose sanctions to foster compliance Legislators themselves are subject to review (through press and the public) and can be punished (denial of re-election)

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 30

Application example: Delegation in bureaucracies

Legislators and ministers delegate tasks to the bureaucracy Legislators can provide oversight and impose sanctions to foster compliance Legislators themselves are subject to review (through press and the public) and can be punished (denial of re-election) Which institutional set-ups are able to deal with such asymmetries and simultaneously combine effective democratic control with efficient task fulfillment? = ⇒ Game-theoretic model and laboratory experiment

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 31

The game

t = 1 Bureaucrat (B) decides whether to provide a high quality product (H) with some effort (e) t = 2 Legislator (L) decides whether to observe B’s work (O) with some costs (c) = ⇒ If L does not observe, the game ends t = 3 If L observes, she receives an information about B’s work (I ∈ {iH, i=H) with a specific accuracy (π) t = 4 L decides whether to punish B with some fee (f )

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 32

Game tree

❜ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ PPPPPPPPPPPPPPP P

B H ¬H

❅ ❅ ❅ ❅

L ¬O O

b−e b

r r

❅ ❅ ❅ ❅

L O ¬O

r r

b −s

♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ r ❅ ❅ ❅ ❅ ❅

N i = iH

r ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅

N i = i¬H

i = i¬H

i = iH

❅ ❅

L ¬P P

b−e b−c

b−e−f b−c−s

❅ ❅

L ¬P P

b−e b−c

b−e−f b−c−s

❅ ❅

L ¬P P

b −c−s

b−f −c+s

❅ ❅

L ¬P P

b −c−s

b−f −c+s

♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣ ♣

Figure 2: Noisy information game Γn

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 33

Model prediction

For accuracy of information 1/2 ≤ π ≤ 2/3 Bureaucrat: Pr(high quality) = 0 Legislator: Pr(observe) = 1 For accuracy of information π ≥ 2/3 Bureaucrat: Pr(high quality) = 2πs−c

(1+π)s

Decreasing control costs → high quality goods Increasing popularity impact → high quality goods Increasing information accuracy → high quality goods

Legislator: Pr(observe) =

e (2π−1)f

Decreasing effort → less control Increasing punishment fee → less control Increasing information accuracy → less control

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 34

The experiment

Original study 120 Students of University of Oldenburg 6 between-subject treatments (20 persons per treatment) We varied costs of observation, punishment fee, and information accuracy Subjects were randomly assigned a fixed role (10 for B and 10 for L) 33 periods with 3 trials in the beginning Randomized partner matching for each period Average earning: e 8.71

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 35

The experiment

Original study 120 Students of University of Oldenburg 6 between-subject treatments (20 persons per treatment) We varied costs of observation, punishment fee, and information accuracy Subjects were randomly assigned a fixed role (10 for B and 10 for L) 33 periods with 3 trials in the beginning Randomized partner matching for each period Average earning: e 8.71 Replication B.A. students of statistics lecture at University of Konstanz 6 treatments (varying # of persons per treatment) Students were not paid

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 36

Results: observation cost & punishment fee

5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Bureaucrat

Round P(high good) high cost & high fee high cost & low fee low cost & high fee low cost & low fee 5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Legislator

Round P(observe)

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 37

Results: observation cost & punishment fee

5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Bureaucrat

Round P(high good) high cost & high fee high cost & low fee low cost & high fee low cost & low fee 5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Legislator

Round P(observe)

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 38

Results: observation cost & punishment fee

5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Bureaucrat

Round P(high good) high cost & high fee high cost & low fee low cost & high fee low cost & low fee 5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Legislator

Round P(observe) high cost & high fee low cost & high fee high cost & low fee low cost & low fee

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 39

Results: observation cost & punishment fee

5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Bureaucrat

Round P(high good) high cost & high fee high cost & low fee low cost & high fee low cost & low fee 5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Legislator

Round P(observe) high cost & high fee low cost & high fee high cost & low fee low cost & low fee

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 40

Results: observation cost & punishment fee, males

5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Bureaucrat

Round P(high good) high cost & high fee high cost & low fee low cost & high fee low cost & low fee 5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Legislator

Round P(observe) high cost & high fee low cost & high fee high cost & low fee low cost & low fee

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 41

Results: observation cost & punishment fee, females

5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Bureaucrat

Round P(high good) high cost & high fee high cost & low fee low cost & high fee low cost & low fee 5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Legislator

Round P(observe) high cost & high fee low cost & high fee high cost & low fee low cost & low fee

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 42

Results: information quality treatment

5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Bureaucrat

Round P(high good) low quality info medium quality info high quality info 5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Legislator

Round P(observe)

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 43

Results: information quality treatment

5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Bureaucrat

Round P(high good) low quality info medium quality info high quality info 5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Legislator

Round P(observe)

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 44

Results: information quality treatment

5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Bureaucrat

Round P(high good) low quality info medium quality info high quality info 5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Legislator

Round P(observe) low quality info medium quality info high quality info

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 45

Results: information quality treatment

5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Bureaucrat

Round P(high good) low quality info medium quality info high quality info 5 10 15 20 25 30 0.0 0.2 0.4 0.6 0.8 1.0

Legislator

Round P(observe) low quality info medium quality info high quality info

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 46

Cheats

What we did Exclude first 10 payed rounds Control for observed individual characteristics Control for unobserved individual characteristics with mixed models Search for subjects that do not show a “reasonable” behavior = ⇒ Some subjects took the same decision throughout the entire experiment

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 47

Cheats

What we did Exclude first 10 payed rounds Control for observed individual characteristics Control for unobserved individual characteristics with mixed models Search for subjects that do not show a “reasonable” behavior = ⇒ Some subjects took the same decision throughout the entire experiment Conduct the replication study, which completely failed.

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 48

Cheats

What we did Exclude first 10 payed rounds Control for observed individual characteristics Control for unobserved individual characteristics with mixed models Search for subjects that do not show a “reasonable” behavior = ⇒ Some subjects took the same decision throughout the entire experiment Conduct the replication study, which completely failed. Why?

No payment Experimenter and instruction bias

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science

SLIDE 49

Cheats

What we did Exclude first 10 payed rounds Control for observed individual characteristics Control for unobserved individual characteristics with mixed models Search for subjects that do not show a “reasonable” behavior = ⇒ Some subjects took the same decision throughout the entire experiment Conduct the replication study, which completely failed. Why?

No payment Experimenter and instruction bias

Comment on repeated task-fulfillment: Control for clustering within the same subject

Michael F. Stoffel (U of Konstanz) Designing Experiments in Political Science