The necessity and formulation of a robust (imprecise) Bayes Factor - - PowerPoint PPT Presentation

▶
the necessity and formulation of a robust imprecise bayes
SMART_READER_LITE
LIVE PREVIEW

The necessity and formulation of a robust (imprecise) Bayes Factor - - PowerPoint PPT Presentation

Mar 04, 2023 44 likes •608 views

The necessity and formulation of a robust (imprecise) Bayes Factor Patrick Schwaferts Ludwig-Maximilians-Universit at M unchen 01. August 2018 Patrick Schwaferts (LMU) Imprecise Bayes Factor 01. August 2018 1 / 18 Introduction

slide-1
SLIDE 1

The necessity and formulation of a robust (imprecise) Bayes Factor

Patrick Schwaferts

Ludwig-Maximilians-Universit¨ at M¨ unchen

  • 01. August 2018

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

1 / 18

slide-2
SLIDE 2

Introduction

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

2 / 18

slide-3
SLIDE 3

Introduction

Reproducibility crisis in psychological research

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

2 / 18

slide-4
SLIDE 4

Introduction

Reproducibility crisis in psychological research Rise of popularity of Bayesian Statistics: promoted as being the solution

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

2 / 18

slide-5
SLIDE 5

Introduction

Reproducibility crisis in psychological research Rise of popularity of Bayesian Statistics: promoted as being the solution Bayes Factor for comparing two hypotheses (“Bayesian t-Test”)

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

2 / 18

slide-6
SLIDE 6

What is the Bayes Factor?

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

3 / 18

slide-7
SLIDE 7

What is the Bayes Factor?

Informal: A generalization of the Likelihood Ratio to include prior information.

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

3 / 18

slide-8
SLIDE 8

From Likelihood Ratio to Bayes Factor I

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

4 / 18

slide-9
SLIDE 9

From Likelihood Ratio to Bayes Factor I

Situation: Two independent groups with observations xi and yj and model Xi ∼ N(µ, σ2), i = 1, ..., n, Yj ∼ N(µ + α, σ2), j = 1, ..., m, with parameters µ, σ2, δ = α/σ.

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

4 / 18

slide-10
SLIDE 10

From Likelihood Ratio to Bayes Factor I

Situation: Two independent groups with observations xi and yj and model Xi ∼ N(µ, σ2), i = 1, ..., n, Yj ∼ N(µ + α, σ2), j = 1, ..., m, with parameters µ, σ2, δ = α/σ. Research Question: Is there a difference between both groups?

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

4 / 18

slide-11
SLIDE 11

From Likelihood Ratio to Bayes Factor I

Situation: Two independent groups with observations xi and yj and model Xi ∼ N(µ, σ2), i = 1, ..., n, Yj ∼ N(µ + α, σ2), j = 1, ..., m, with parameters µ, σ2, δ = α/σ. Research Question: Is there a difference between both groups? Hypotheses: H0 : δ = 0 vs. H1 : δ = 0

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

4 / 18

slide-12
SLIDE 12

From Likelihood Ratio to Bayes Factor II

Likelihood Ratio: L R10 = max

µ,σ2,δ f (data|µ, σ2, δ)

max

µ,σ2 f (data|µ, σ2, δ = 0)

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

5 / 18

slide-13
SLIDE 13

From Likelihood Ratio to Bayes Factor II

Likelihood Ratio: L R10 = max

µ,σ2,δ f (data|µ, σ2, δ)

max

µ,σ2 f (data|µ, σ2, δ = 0)

Law of Likelihood: The extent to which the data support one model over another (:= evidence) is equal to the ratio of their likelihoods.

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

5 / 18

slide-14
SLIDE 14

From Likelihood Ratio to Bayes Factor II

Likelihood Ratio: L R10 = max

µ,σ2,δ f (data|µ, σ2, δ)

max

µ,σ2 f (data|µ, σ2, δ = 0)

Law of Likelihood: The extent to which the data support one model over another (:= evidence) is equal to the ratio of their likelihoods. Interpretation of LR: The data is L R10 times as much evidence for the model chosen(∗) under H1 than for the model chosen(∗) under H0.

(∗): chosen refers to the max-operation

⇒ L R10 quantifies the maximum evidence for H1 (in a comparison with H0)

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

5 / 18

slide-15
SLIDE 15

From Likelihood Ratio to Bayes Factor III

Introducing Prior Probabilities: P(H1) and P(H0)

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

6 / 18

slide-16
SLIDE 16

From Likelihood Ratio to Bayes Factor III

Introducing Prior Probabilities: P(H1) and P(H0) Bayes Rule: P(H1|data) P(H0|data)

  • PosteriorOdds

= L R10 · P(H1) P(H0)

PriorOdds

The data is used to learn about P(H1) and P(H0).

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

6 / 18

slide-17
SLIDE 17

From Likelihood Ratio to Bayes Factor III

Introducing Prior Probabilities: P(H1) and P(H0) Bayes Rule: P(H1|data) P(H0|data)

  • PosteriorOdds

= L R10 · P(H1) P(H0)

PriorOdds

The data is used to learn about P(H1) and P(H0). Interpretation of Posterior Probabilities: After seeing the data, the maximum belief in H1 is P(H1|data).

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

6 / 18

slide-18
SLIDE 18

From Likelihood Ratio to Bayes Factor IV

Introducing Parameter Priors: Pµ, Pσ2 and Pδ

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

7 / 18

slide-19
SLIDE 19

From Likelihood Ratio to Bayes Factor IV

Introducing Parameter Priors: Pµ, Pσ2 and Pδ Bayesian Hypotheses: HB

0 :

µ ∼ Pµ σ2∼ Pσ2 δ = 0 vs. HB

1 :

µ ∼ Pµ σ2∼ Pσ2 δ ∼ Pδ Pδ is called test-relevant prior.

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

7 / 18

slide-20
SLIDE 20

From Likelihood Ratio to Bayes Factor IV

Introducing Parameter Priors: Pµ, Pσ2 and Pδ Bayesian Hypotheses: HB

0 :

µ ∼ Pµ σ2∼ Pσ2 δ = 0 vs. HB

1 :

µ ∼ Pµ σ2∼ Pσ2 δ ∼ Pδ Pδ is called test-relevant prior. Marginalized Likelihoods: m(data|HB

1 ) =

  • f (data|µ, σ2, δ)Pµ(µ)Pσ2(σ2)Pδ(δ)dδdσ2dµ

m(data|HB

0 ) =

  • f (data|µ, σ2, δ = 0)Pµ(µ)Pσ2(σ2)dσ2dµ

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

7 / 18

slide-21
SLIDE 21

From Likelihood Ratio to Bayes Factor V

Bayes Factor: BF10 = m(data|HB

1 )

m(data|HB

0 )

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

8 / 18

slide-22
SLIDE 22

From Likelihood Ratio to Bayes Factor V

Bayes Factor: BF10 = m(data|HB

1 )

m(data|HB

0 )

Bayes Rule: P(HB

1 |data)

P(HB

0 |data) = BF10 · P(HB 1 )

P(HB

0 )

The data is used to learn about P(HB

1 ) and P(HB 0 ). Nothing can be

learned about the parameter priors. ⇒ Pδ is part of the HB

1 -model.

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

8 / 18

slide-23
SLIDE 23

From Likelihood Ratio to Bayes Factor V

Bayes Factor: BF10 = m(data|HB

1 )

m(data|HB

0 )

Bayes Rule: P(HB

1 |data)

P(HB

0 |data) = BF10 · P(HB 1 )

P(HB

0 )

The data is used to learn about P(HB

1 ) and P(HB 0 ). Nothing can be

learned about the parameter priors. ⇒ Pδ is part of the HB

1 -model.

Interpretation of BF: The data is BF10 times as much evidence for the model behind m(data|HB

1 ) than for the model behind m(data|HB 0 ).

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

8 / 18

slide-24
SLIDE 24

What is the model behind m(data|HB

1 )?

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

9 / 18

slide-25
SLIDE 25

What is the model behind m(data|HB

1 )?

Again Bayes Rule: P(HB

1 |data) = m(data|HB 1 ) · P(H1)

P(data)

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

9 / 18

slide-26
SLIDE 26

What is the model behind m(data|HB

1 )?

Again Bayes Rule: P(HB

1 |data) = m(data|HB 1 ) · P(H1)

P(data) In order to apply Bayes Rule, m(data|HB

1 ) needs to be a likelihood, which

describes the data-generating process.

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

9 / 18

slide-27
SLIDE 27

What is the model behind m(data|HB

1 )?

Again Bayes Rule: P(HB

1 |data) = m(data|HB 1 ) · P(H1)

P(data) In order to apply Bayes Rule, m(data|HB

1 ) needs to be a likelihood, which

describes the data-generating process. So the model behind m(data|HB

1 ) models a data-generating process with

likelihood m(data|HB

1 ) =

  • f (data|µ, σ2, δ)Pµ(µ)Pσ2(σ2)Pδ(δ)dδdσ2dµ

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

9 / 18

slide-28
SLIDE 28

What is the model behind m(data|HB

1 )?

Again Bayes Rule: P(HB

1 |data) = m(data|HB 1 ) · P(H1)

P(data) In order to apply Bayes Rule, m(data|HB

1 ) needs to be a likelihood, which

describes the data-generating process. So the model behind m(data|HB

1 ) models a data-generating process with

likelihood m(data|HB

1 ) =

  • f (data|µ, σ2, δ)Pµ(µ)Pσ2(σ2)Pδ(δ)dδdσ2dµ

⇒ A model with subjective components!

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

9 / 18

slide-29
SLIDE 29

What is the model behind m(data|HB

1 )?

Again Bayes Rule: P(HB

1 |data) = m(data|HB 1 ) · P(H1)

P(data) In order to apply Bayes Rule, m(data|HB

1 ) needs to be a likelihood, which

describes the data-generating process. So the model behind m(data|HB

1 ) models a data-generating process with

likelihood m(data|HB

1 ) =

  • f (data|µ, σ2, δ)Pµ(µ)Pσ2(σ2)Pδ(δ)dδdσ2dµ

⇒ A model with subjective components! [The Bayes Factor does not directly answer: Is there an effect?]

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

9 / 18

slide-30
SLIDE 30

Necessity of properly specifying Pδ

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

10 / 18

slide-31
SLIDE 31

Necessity of properly specifying Pδ

The test-relevant prior Pδ is part of the HB

1 -model.

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

10 / 18

slide-32
SLIDE 32

Necessity of properly specifying Pδ

The test-relevant prior Pδ is part of the HB

1 -model.

Pδ need to be specified properly.

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

10 / 18

slide-33
SLIDE 33

Necessity of properly specifying Pδ

The test-relevant prior Pδ is part of the HB

1 -model.

Pδ need to be specified properly. If not: HB

1 -model misspecifies the experimental situation. BF results would

be worthless.

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

10 / 18

slide-34
SLIDE 34

How to properly specify Pδ? I

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

11 / 18

slide-35
SLIDE 35

How to properly specify Pδ? I

What is Pδ? Pδ is a probability distribution, which specifies the available knowledge and beliefs about δ prior to data collection. It should reflect the expectations about δ under HB

1 (?).

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

11 / 18

slide-36
SLIDE 36

How to properly specify Pδ? I

What is Pδ? Pδ is a probability distribution, which specifies the available knowledge and beliefs about δ prior to data collection. It should reflect the expectations about δ under HB

1 (?).

What is state of the art? Predominantly Pδ ∼ Cauchy(0, √ 2/2). Sometimes Pδ ∼ N(0, 1) or Pδ ∼ N(µδ, σ2

δ).

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

11 / 18

slide-37
SLIDE 37

The Cauchy distribution

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

12 / 18

slide-38
SLIDE 38

The Cauchy distribution

Effect sizes: δ = 0.2: small; δ = 0.5: medium; δ = 0.8: large δ = 1.8: association gender - body height

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

12 / 18

slide-39
SLIDE 39

How to properly specify Pδ? II

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

13 / 18

slide-40
SLIDE 40

How to properly specify Pδ? II

About the absurdity of the Cauchy effect size prior: Before seeing the data, the researcher is about 23.8% confident that |δ| is larger than one of the largest effect sizes in psychology.

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

13 / 18

slide-41
SLIDE 41

How to properly specify Pδ? II

About the absurdity of the Cauchy effect size prior: Before seeing the data, the researcher is about 23.8% confident that |δ| is larger than one of the largest effect sizes in psychology. I would offer bets :-)

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

13 / 18

slide-42
SLIDE 42

How to properly specify Pδ? II

About the absurdity of the Cauchy effect size prior: Before seeing the data, the researcher is about 23.8% confident that |δ| is larger than one of the largest effect sizes in psychology. I would offer bets :-) Necessity of an imprecise effect size prior: By default, precise information about δ is lacking. Else, no scientific investigation would be needed. ⇒ A proper specification of Pδ should be imprecise.

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

13 / 18

slide-43
SLIDE 43

A first imprecise Bayes Factor

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

14 / 18

slide-44
SLIDE 44

A first imprecise Bayes Factor

Test-relevant prior: δ ∼ N(µδ, σ2

δ)

with µδ ∈

  • µδ; µδ
  • ,

σ2

δ ∈

  • σ2

δ; σ2 δ

  • Patrick Schwaferts (LMU)

Imprecise Bayes Factor

  • 01. August 2018

14 / 18

slide-45
SLIDE 45

A first imprecise Bayes Factor

Test-relevant prior: δ ∼ N(µδ, σ2

δ)

with µδ ∈

  • µδ; µδ
  • ,

σ2

δ ∈

  • σ2

δ; σ2 δ

  • M :=
  • Pδ = N(µδ, σ2

δ) | µδ ∈

  • µδ; µδ
  • , σ2

δ ∈

  • σ2

δ; σ2 δ

  • Patrick Schwaferts (LMU)

Imprecise Bayes Factor

  • 01. August 2018

14 / 18

slide-46
SLIDE 46

A first imprecise Bayes Factor

Test-relevant prior: δ ∼ N(µδ, σ2

δ)

with µδ ∈

  • µδ; µδ
  • ,

σ2

δ ∈

  • σ2

δ; σ2 δ

  • M :=
  • Pδ = N(µδ, σ2

δ) | µδ ∈

  • µδ; µδ
  • , σ2

δ ∈

  • σ2

δ; σ2 δ

  • Bayesian Hypotheses:

HB

0 :

µ ∼ Pµ σ2∼ Pσ2 δ = 0 vs. HB

1 :

µ ∼ Pµ σ2∼ Pσ2 δ ∼ M

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

14 / 18

slide-47
SLIDE 47

A first imprecise Bayes Factor

Test-relevant prior: δ ∼ N(µδ, σ2

δ)

with µδ ∈

  • µδ; µδ
  • ,

σ2

δ ∈

  • σ2

δ; σ2 δ

  • M :=
  • Pδ = N(µδ, σ2

δ) | µδ ∈

  • µδ; µδ
  • , σ2

δ ∈

  • σ2

δ; σ2 δ

  • Bayesian Hypotheses:

HB

0 :

µ ∼ Pµ σ2∼ Pσ2 δ = 0 vs. HB

1 :

µ ∼ Pµ σ2∼ Pσ2 δ ∼ M Imprecise Bayes Factor: IBF10 =

  • min

Pδ∈M BF10; max Pδ∈M BF10

  • Patrick Schwaferts (LMU)

Imprecise Bayes Factor

  • 01. August 2018

14 / 18

slide-48
SLIDE 48

Example I

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

15 / 18

slide-49
SLIDE 49

Example I

δ ∼ N(µδ, σ2

δ)

with µδ ∈ [0; 0.5] , σ2

δ ∈ [0.5; 3]

  • −4

−2 2 4 6 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative probability

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

15 / 18

slide-50
SLIDE 50

Example II

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

16 / 18

slide-51
SLIDE 51

Example II

IBF10 = [1.84; 5.99]

0.5 1.0 1.5 2.0 2.5 3.0 0.0 0.1 0.2 0.3 0.4 0.5

µδ σδ

2

2 3 4 5 Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

16 / 18

slide-52
SLIDE 52

Example II

IBF10 = [1.84; 5.99]

0.5 1.0 1.5 2.0 2.5 3.0 0.0 0.1 0.2 0.3 0.4 0.5

µδ σδ

2

2 3 4 5

Interpretation: The data is between 1.84 and 5.99 times as much evidence for HB

1 than for HB 0 ,

i.e. for an effect with an effect size in accordance with the available knowledge about it than for no ef- fect. (ignoring Pµ and Pσ2)

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

16 / 18

slide-53
SLIDE 53

What is next?

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

17 / 18

slide-54
SLIDE 54

What is next?

This was only a credal set of normal effect size distributions.

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

17 / 18

slide-55
SLIDE 55

What is next?

This was only a credal set of normal effect size distributions. ⇒ p-boxes as effect size priors.

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

17 / 18

slide-56
SLIDE 56

Thank you for your Attention!

Thank you for your Attention!

Patrick Schwaferts (LMU) Imprecise Bayes Factor

  • 01. August 2018

18 / 18