What papers should be published? Relevance, plausibility, validity, - - PowerPoint PPT Presentation

What papers should be published? Relevance, plausibility, validity, and learning

What papers should be published? Relevance, plausibility, validity, and learning

Alexander Frankel Maximilian Kasy November 20, 2017

1 / 28

What papers should be published? Relevance, plausibility, validity, and learning Introduction

Introduction

◮ Not all empirical results get published → selection. ◮ Reasons for selection:

• 1. Journal decisions (“publication bias”).
• 2. Researcher decisions (“p-hacking”).

◮ Possible motivations for selection:

• 1. Statistical significance testing.
• 2. Novelty of results.
• 3. Confirmation of priors.

◮ Consequences of selection:

• 1. Conventional estimators are biased.
• 2. Conventional confidence sets don’t control size.

2 / 28

What papers should be published? Relevance, plausibility, validity, and learning Introduction

What is to be done?

◮ Reform proposals to mitigate selection:

◮ Pre-registration plans ◮ Hypothesis registries ◮ “Data snooping” corrections ◮ Results-blind review ◮ Journal of replication studies ◮ Journal of null results

◮ We argue: No selection is not optimal, in general. ◮ Need to be careful about specifying objective!

3 / 28

What papers should be published? Relevance, plausibility, validity, and learning Introduction

Possible journal objectives

• 1. Validity:

◮ Conventional estimators are unbiased. ◮ Conventional confidence sets control size.

• 2. Relevance:

◮ Published results inform policy. ◮ Publish to maximize social welfare.

• 3. Plausibility:

◮ Maximize probability that published results are correct. ◮ Minimize distance of published results to truth.

• 4. Learning:

◮ Minimize posterior variance, ◮ given the number of publications.

4 / 28

What papers should be published? Relevance, plausibility, validity, and learning Introduction

Preview of results

◮ Optimal selectivity depends on objective:

◮ Validity: Don’t select on findings. ◮ Relevance and learning: Publish surprising findings. ◮ Plausibility: Publish unsurprising findings.

◮ Relevance can rationalize selection based on one-sided or

two-sided testing.

◮ Dynamic relevance can rationalize publication of precise null

results.

5 / 28

What papers should be published? Relevance, plausibility, validity, and learning Introduction

Literature

◮ Systematic replication studies:

Open Science Collaboration (2015), Camerer et al. (2016)

◮ Publication bias:

Ioannidis (2005), Ioannidis (2008), McCrary et al. (2016); Andrews and Kasy (2017)

◮ Reform proposals:

Olken (2015), Coffman and Niederle (2015), Christensen and Miguel (2016)

◮ Economic models of publication:

Glaeser (2006), Libgober (2015), Akerlof and Michaillat (2017)

6 / 28

What papers should be published? Relevance, plausibility, validity, and learning Introduction

Outline of talk

Introduction Static model Validity Relevance Plausibility Learning Dynamic relevance Discussion and conclusion

7 / 28

What papers should be published? Relevance, plausibility, validity, and learning Static model

Static model

Researcher submits Journal decides Policymaker chooses welfare is whether to publish policy realized X ∼ fX|θ → D = d(X) → A = a(π1) → u(A,θ)− D · c ◮ Common prior π0 for θ of journal and policymaker. ◮ Posterior:

Journal

πJ = π0 ·

fX|θ(X|·) fX(X)

Policymaker if D = 1

π1 = π0 ·

fX|θ(X|·) fX(X)

Naive policymaker if D = 0

π0 = π0

Sophisticated policymaker if D = 0

π0 = π0 · 1−E[d(X)|θ=·]

1−E[d(X)]

8 / 28

What papers should be published? Relevance, plausibility, validity, and learning Validity

Validity

Proposition

Suppose X|θ ∼ N(θ,s). The following statements are equivalent:

• 1. Bayesian validity of naive updating:

θ|d(X) = 0 ∼ π0

• 2. Frequentist unbiasedness:

E[X|θ,D = 1] = θ for all θ.

• 3. Publication probability independent of parameter:

E[d(X)|θ] is constant in θ.

• 4. Publication decision independent of estimate:

d(X) does not depend on X.

9 / 28

What papers should be published? Relevance, plausibility, validity, and learning Validity

Proof

◮ 4 ⇒ 1, 2, 3: immediate. ◮ 1 ⇒ 3:

θ|d(X) = 0 ∼ π0 · 1−E[d(X)|θ=·]

1−E[d(X)]

.

Equality to π0 is equivalent to E[d(X)|θ] = ¯ d for all θ.

◮ 2 ⇒ 3: Wlog s = 1. Unbiasedness equivalent to

0 =

• (z −θ)ϕ(z −θ)d(z)dz

= −

• ϕ′(z −θ)d(z)dz

= ∂θ

• ϕ(z −θ)d(z)dz
• = ∂θE[d(Z)|θ].

◮ 3 ⇒ 4: completeness of X for θ in the normal location family ⇒

If E[d(X)|θ] = ¯ d for all θ, then d(X) = ¯ d almost surely.

10 / 28

What papers should be published? Relevance, plausibility, validity, and learning Relevance

Relevance

◮ Policymaker observes (D,D · X), updates prior to π1

picks policy A = a(π1).

◮ Common objective of journal and policymaker:

maximize expectation of welfare u(A,θ), net of (shadow) cost of publication D · c.

◮ Expected welfare:

U(a,π) =

• u(a,θ)dπ(θ).

◮ Optimal policy choice:

a(π) = argmax

a

U(a,π).

11 / 28

What papers should be published? Relevance, plausibility, validity, and learning Relevance

The journal’s problem

◮ Denote ad = a(πd) for d = 0,1. ◮ Journal maximizes U(ad,πJ)− d · c. ◮ Thus decides to publish iff

U(a1,πJ)− U(a0,πJ) > c.

◮ Notes:

◮ This presumes no commitment: Journal takes policymaker’s

beliefs π0 as given when choosing D.

◮ Therefore takes a0 as given. ◮ π0 depend on E[d(X)|θ] for sophisticated updating! ◮ Also, since π1 = πJ,

U(a1,πJ) = max

a

U(a,πJ).

12 / 28

What papers should be published? Relevance, plausibility, validity, and learning Relevance

No commitment = commitment = planner’s solution

◮ Three related problems:

• 1. No commitment:

Bayes Nash optimal d(·). Take a0 as given.

• 2. Commitment:

Pick d(·) ex-ante. Take a0 as function of d(·).

• 3. Planner’s problem:

Pick ex-ante both d(·) and a0.

Proposition

Assuming uniqueness, all three problems have the same solution.

◮ Journal and policymaker have same objective in choosing a0. ◮ Equivalence of

• 1. Joint optimization (planner’s problem)
• 2. Concentrating out (commitment problem)
• 3. Conditionally optimizing (no commitment problem)

13 / 28

What papers should be published? Relevance, plausibility, validity, and learning Relevance

Canonical policy problem 1: Binary treatment, linear welfare

◮ a ∈ {0,1} and

u(a,θ) = a·θ.

◮ Expected welfare and optimal policy:

U(a,π) = a· Eπ[θ] a(π) = 1(Eπ[θ] > 0).

◮ Return to publishing (sophisticated updating):

U(a(π1),πJ)−U(a(π0),πJ) =

    

sign(E[θ|X]) = sign(E[θ|d(X) = 0])

|E[θ|X]|

else.

14 / 28

What papers should be published? Relevance, plausibility, validity, and learning Relevance

Canonical policy problem 2: Continuous policy, quadratic welfare

◮ a ∈ R and

u(a,θ) = −(a−θ)2.

◮ Expected welfare and optimal policy:

U(a,π) = −Varπ(θ)−(a− Eπ[θ])2 a(π) = Eπ[θ].

◮ Return to publishing (sophisticated updating):

U(a(π1),πJ)− U(a(π0),πJ) = (E[θ|d(X) = 0]− E[θ|X])2

15 / 28

What papers should be published? Relevance, plausibility, validity, and learning Relevance

Normal likelihood and prior

◮ Suppose X|θ ∼ N(θ,s2) and θ ∼ N(µ0,σ 2). ◮ Then, for κ = σ 2

s2+σ 2 ,

θ|X ∼ N(κX +(1−κ)µ0,σ 2

0 ·(1−κ)).

◮ Solutions to optimal publication problems:

◮ In the binary case: “one-sided testing,”

dR,b(x) =

  

1

• x > c−(1−κ)µ0

κ

• µ0 < 0

1

• x < −c−(1−κ)µ0

κ

• µ0 > 0.

◮ In the quadratic case: “two-sided testing,”

dR,c(x) = 1

• |x − µ0| > √

c/κ

• .

◮ Same solutions for naive and sophisticated policymaker.

16 / 28

What papers should be published? Relevance, plausibility, validity, and learning Plausibility

Plausibility

◮ Assume alternatively that journals don’t want to publish wrong

results or estimates far from the truth.

◮ For instance: Choose d to maximize expectation of

d ·(−(X −θ)2 + b).

◮ ⇒ Publish iff

E[(X −θ)2|X] = Var(θ|X = x)+(E[θ|X = x]− x)2 < b.

◮ With X|θ ∼ N(θ,s2) and θ ∼ N(µ0,σ 2),

dP(x) = 1

• |x − µ0| <

√

b−σ 2

0 ·(1−κ)

1−κ

• .

◮ → Only publish unsurprising findings!

17 / 28

What papers should be published? Relevance, plausibility, validity, and learning Learning

Learning

◮ For any publication rule d(·): what is the speed of learning? ◮ What is the expected posterior variance of θ?

E[Var(θ|D · X,D)]

=Var(θ)− Var(E[θ|D · X,D]) =Var(θ)− Var(E[θ|X])+ Var(E[θ|X]|D = 0)· E[1− D],

◮ Thus: Given publication probability E[D],

higher speed of learning ⇔ smaller variance Var(E[θ|X]|d(X) = 0).

18 / 28

What papers should be published? Relevance, plausibility, validity, and learning Learning

Normal likelihood and prior

◮ Suppose again X|θ ∼ N(θ,s2) and θ ∼ N(µ0,σ 2). ◮ Then, for cutoffs xj chosen such that E[D] = ¯

d, Var(E[θ|X]||X − ¯

θ| < x1) < Var(E[θ|X]|X < x2) < Var(E[θ|X]) < Var(E[θ|X]||X − ¯ θ| > x3).

◮ ⇒ ranking of selection rules in terms of the speed of learning:

Proposition

Relevance quadratic ≻ relevance binary ≻ validity ≻ plausibility.

19 / 28

What papers should be published? Relevance, plausibility, validity, and learning Dynamic relevance

Dynamic model

Researcher submits Journal decides Policymaker chooses whether to publish policy X1 ∼ fX|θ → D = d(X1) → A1 = a(π1) → Researcher submits, Policymaker chooses Welfare is journal publishes policy realized X2 ∼ fX|θ → A2 = a(π2) → u(A1,θ)+β · u(A2,θ)− D · c

◮ First period as in the static model. ◮ Second period result always published.

20 / 28

What papers should be published? Relevance, plausibility, validity, and learning Dynamic relevance

Choices

◮ Recall

U(a,π) =

• u(a,θ)dπ(θ),

a(π) = argmax

a

U(a,π).

◮ Optimal policy choice: at = a(πt).

Potential policy choice: ad

t = a(πd t ).

◮ Value function: Period 2 utility expected by journal in period 1,

V d = E[u(a(πd

2 ),θ)|X1]

◮ πd

2 is (naive) posterior given (D · X1,D,X2).

◮ X2 and θ are dependent given X1.

◮ Returns to publication:

• U(a1

1,πJ)− U(a0 1,πJ)

• +β ·
• V 1 − V 0

.

21 / 28

What papers should be published? Relevance, plausibility, validity, and learning Dynamic relevance

Updating

◮ Naive posteriors:

π1

1(θ) = fθ|X1(θ|X1),

π1

2(θ) = fθ|X1,X2(θ|X1,X2)

π0

1(θ) = π0(θ),

π0

2(θ) = fθ|X2(θ|X2).

◮ Consider normal prior and estimates, where

θ ∼ N(µ0,σ 2

0 ),

X1|θ ∼ N(θ,s2

1),

X2|X1,θ ∼ N(θ,s2

2).

◮ Denote ¯

X = (X1 + X2)/2 and

κ1 =

σ 2

s2

1+σ 2 0 ,

κ2 =

σ 2

s2

2+σ 2 0 ,

¯ κ =

σ 2 (s2

1+s2 2)/4+σ 2 0 .

◮ Then

θ|X1 ∼ N(κ1X1 +(1−κ1)µ0,σ 2

0 ·(1−κ1))

θ|X1,X2 ∼ N(¯ κ ¯

X +(1− ¯

κ )µ0,σ 2

0 ·(1− ¯

κ )) θ|X2 ∼ N(κ2X2 +(1−κ2)µ0,σ 2

0 ·(1−κ2)).

22 / 28

What papers should be published? Relevance, plausibility, validity, and learning Dynamic relevance

Quadratic loss, normal model and prior

◮ Canonical policy problem 2. ◮ a ∈ R, u(a,θ) = −(a−θ)2.

Proposition

The optimal publication rule in this case is given by dR2,c(x) = 1

• α0 +α1 ·(X1 − µ0)2 > c
• ,

α0 = β ·(κ2 − 1

2 ¯

κ)2(σ 2

0 ·(1−κ1)+ s2 2),

α1 = κ2

1 +β ·[κ2κ1 − ¯

κ 1

2(1+κ1)]2.

◮ Key difference to static case: intercept α0. ◮ → Publish precise null results!

23 / 28

What papers should be published? Relevance, plausibility, validity, and learning Dynamic relevance

Plotting returns to publication

◮ Plot for µ0 = 0, σ 2

0 = s2 1 = s2 2 = 1, β = 10.

◮ First period returns, second period returns, total returns. ◮ Different scales!

• 2
• 1.5
• 1
• 0.5

0.5 1 1.5 2 X1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 U( 1 1, J 1) - U( 0 1, J 1)

• 2
• 1.5
• 1
• 0.5

0.5 1 1.5 2 X1 0.5 1 1.5 2 2.5 3 V( 1 1, J 1) - V( 0 1, J 1)

• 2
• 1.5
• 1
• 0.5

0.5 1 1.5 2 X1 0.5 1 1.5 2 2.5 3 3.5 4 returns to publication

◮ Key takeaways:

◮ Quadratic total returns to publication

⇒ Interval censoring is optimal.

◮ But: Positive expected returns even for X1 = 0

⇒ Might publish everything when β high / c low / X1 precise.

24 / 28

What papers should be published? Relevance, plausibility, validity, and learning Dynamic relevance

Binary treatment, normal model and prior

◮ Canonical policy problem 1. ◮ a ∈ {0,1}, u(a,θ) = a·θ, ◮ µ0 < 0 (opposite case is symmetric). ◮ Denote

λ = 1

2 ¯

κ(1+κ1), ν = 1

2 ¯

κ

• σ 2

0 ·(1−κ1)+ s2 2,

e1 =

1

√

σ 2

0 ·(1−κ1)+s2 2

• (κ1 − 1− s2

2

σ 2

0 )µ0 −κ1X1

• ,

e2 =

1

√

σ 2

0 ·(1−κ1)+s2 2

• (κ1 − 1− s2

1+s2 2

2σ 2

0 )µ0 −(κ1 + 1)X1

• ,

Proposition

The optimal publication rule in this case is given by dR2,b(x) = 1(max(µ0 +κ1(X1 − µ0),0)+β ·∆V > c),

∆V = (Φ(e1)−Φ(e2))·((1−λ)µ0 +λX1)+ν ·(ϕ(e2)−ϕ(e1)).

25 / 28

What papers should be published? Relevance, plausibility, validity, and learning Dynamic relevance

Plotting returns to publication

◮ Plot for µ0 = −.1, σ 2

0 = s2 1 = s2 2 = 1, β = 10.

◮ First period returns, second period returns, total returns. ◮ Different scales!

• 6
• 4
• 2

2 4 6 X1 0.5 1 1.5 2 2.5 3 U( 1 1, J 1) - U( 0 1, J 1)

• 6
• 4
• 2

2 4 6 X1 0.02 0.04 0.06 0.08 0.1 0.12 V( 1 1, J 1) - V( 0 1, J 1)

• 6
• 4
• 2

2 4 6 X1 0.5 1 1.5 2 2.5 3 3.5 returns to publication

◮ Key takeaways:

◮ When c is large / β is small, one sided censoring is still optimal. ◮ Otherwise:

Might additionally publish in an interval of negative values for X1.

26 / 28

What papers should be published? Relevance, plausibility, validity, and learning Discussion and conclusion

Summary

◮ Should we aim to eliminate selectivity of publication process? ◮ Depends on objective!

◮ Relevance: Publish surprising results. ◮ Validity: Don’t select on results. ◮ Plausibility: Publish unsurprising results.

◮ For canonical cases of relevance, optimal selection looks like

• ne-sided / two-sided testing.

◮ Test for two-sided case:

◮ Centered at ¯

θ, not 0.

◮ Critical value for t-stat at √

c

• s

σ2 + 1

s

• , not 1.96.

◮ Speed of learning:

◮ Objective aligned with relevance, ◮ opposite of plausibility.

◮ Dynamic relevance additionally provides rationale for publishing

precise null results.

27 / 28

What papers should be published? Relevance, plausibility, validity, and learning Discussion and conclusion

Thanks for your time!

28 / 28