JESA and Replication Studies 2015 workshop in Experimental Methods: - - PowerPoint PPT Presentation

JESA and Replication Studies

2015 workshop in Experimental Methods: The replicability crisis in the social sciences and how to address it

Bob Slonim 27, November, 2015

Why a new journal for experimental economics?

• Demand/Growth of the field

– ESA established 1986 – Experimental Economics First Issue 1998

• A need for experimental methodology research
• A need for replication studies

10 20 30 40 50 60 50 100 150 200 250 300 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Acceptance rate Number of papers

Year Number of submissions Number of original papers published Acceptance rate

Experimental Economics 2005‐2014

5 10 15 20 25 30 35 40 45 50

75-7677-7879-8081-8283-8485-8687-8889-9091-9293-9495-9697-9899-0001-0203-0405-0607-0809-1011-1213-14

Number of published exp/al papers

Year

lab experiments field experiments

Experimental Publications in “Top‐5”: 1975‐2014

100 200 300 400 500 600 700 800 900 1000 1100

2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Year

ESA members ESA Conference Attendees

Growth of the ESA 2001‐2014

JESA: Aims & Scope

• JESA will focus on publishing shorter papers

(original articles, methodological pieces, surveys, comments on recently published experimental papers), and article types that are important yet under‐represented in the experimental literature (i.e., replications, minor extensions, robustness checks, meta‐ analyses, and good experimental designs even if obtaining null results)

JESA: Aims & Scope

• The Journal of the Economic Science Association

(JESA) is dedicated to advancing theoretical, empirical, methodological and policy‐relevant knowledge using experimental economic methods. JESA promotes research pioneering and advancing laboratory and field methods to address important economic questions that are difficult to examine using naturally occurring data. JESA is open to all areas of inquiry in economics and at the intersection of economics and other disciplines including but not limited to psychology, political science, statistics, finance, marketing, and

• rganizational behavior.

What is a replication study??????

• “Run a study the way I think it should be run”
• Consider the following seminal study conducted

by Researcher O (original):

– Between Subject Design – Two conditions: Control (C) and Treatment (T) – Theory ‐> Hypothesis:

• Mean choice in T greater than mean choice in C

– Results show significant difference at the p < .05 level

Question 1: Sample Population

• Researcher R1 (Replicator) runs the identical

study and only changes the subject population:

– Recruits from the general population – Recruits from a communist country – Recruits children – Recruits nurses – Recruit from Identical population, but the recruitment process puts more stress on monetary rewards, or helping science, or the time involved or the nature of tasks, … – Etc

Question 2: Information Presentation

• Researcher R2 (Replicator) runs the identical

study and only changes the way that payoff function is presented

– Shown as an equation – Shown in a table – Shown graphically – Shown with different color schemes – Etc

Question 3: Payoffs

• Researcher R3 (Replicator) runs the identical

study and only changes the payoffs:

– Increases all payoffs by a factor of 10 – Increases all payoffs by $1 – Identical payoffs but calls them experimental units rather than dollars – Determination of randomization (die roll, computerized, bingo cage) – Etc

Question 4: Instructions/Protocols

• Researcher R4 (Replicator) runs the identical

study and only changes the instruction protocols:

– Minor wording changes – Read out loud – Includes review questions – Includes (different) examples – Etc

It depends on …

It depends on …

Researcher Objectives

If researcher had no reason to expect a different result:

It depends on …

Researcher Objectives

If researcher had no reason to expect a different result: Then Replication Study

It depends on …

Researcher Objectives

If researcher expected a different result: Then Theoretical Contribution

Designing a Replication Study

• Behavioral Economics Replication Project:

http://sciencepredictionmarkets.com/studies.html

• “The effect in the replication is in the same direction

as in the original study, and is statistically significant with a p‐value smaller than 0.05.”

• Statement of hypothesis (Based on original study)
• Replication sample size (Based on replication study)
• Replication Power (90% and higher in this project)

based on data from Original study and sample size from replication study

Experimental Economics and Replication Study Importance

• Laboratory experimenters indeed replicate all the

time!

– To better understand why T>C, we run T and C, and then T1, T2, … to disentangle different hypotheses, to explore robustness, to stress test, … – It is part of the DNA of (lab) experimental economics

• Field experiments?

– Not common (rare?)

What should be replicated?

• Consider how many other (published) papers have

run the same or similar studies? None? Thousands? (Hello My Bayesian)

– Potential opportunities with field experiments

• Consider the relative importance (citations,

implications/policy impact)

– Consider where the paper was published

• Consider the theory

– E.g. cognitive dissonance

• Consider the empirical analyses/original data
• Consider the “smell” factor

– (you may not be alone)

Thank you