Replication, Preregistration & Open Science Why most published - - PowerPoint PPT Presentation

▶

Jun 04, 2023 222 likes •326 views

Replication, Preregistration & Open Science Why most published research findings are false select random card 1 / 3 1 / 3 1 / 3 1 2 3 choose side choose side choose side 1 0 1 / 2 1 / 2 0 1 b r b r b r 0 0 1 / 3 1 / 6 1 / 6 1

SLIDE 1

Replication, Preregistration & Open Science

Why most published research findings are false

SLIDE 2

select random card

1/3 1/3 1/3

choose side choose side choose side 1

1/2 1/2

1

1/3 1/6 1/6 1/3

1 2 3 b r b b r r

SLIDE 3

P(Card = i) P(Obs = j | Card = i) P(Obs = j | Card = i)P(Card = i)

P(Card = 1 | Obs = b) = P(Obs = b | Card = 1)P(Card = 1) P(Obs = b) = P(Obs = b | Card = 1)P(Card = 1) P

i P(Obs = b | Card = i)P(Card = i)

=

1 3 1 2

= 2 3

SLIDE 4

hypothesis H

R/R+1 1/R+1

true false apply test T apply test T sign null sign null β 1 − β 1 − α α P(H) P(T | H) Positive predictive value P(H = t | T = s) = P(T = s | H = t)P(H = t) P(T = s) = R(1 − β) R(1 − β) + α

“probability that the hypothesis is true, given a significant test result”

SLIDE 5

Positive predictive value P(H = t | T = s) = P(T = s | H = t)P(H = t) P(T = s) = R(1 − β) R(1 − β) + α

“probability that the hypothesis is true, given a significant test result”

example: R = 1, β = 0.2, α = 0.05 P(H = t | T = s) = 0.8 0.85 ≈ 0.94

SLIDE 6

hypothesis H

R/R+1 1/R+1

true false apply test T apply test T sign null sign null β 1 − β 1 − α α p-hack p-hack sign null sign null u u 1 − u 1 − u

p-hacking ::: combination of design/presentation/analysis factors that favor a significant test result beyond the normal alpha level

SLIDE 7

Positive predictive value example: R = 1, β = 0.2, α = 0.05 P(H = t | T = s) = R(1 − β) + uβR R(1 − β) + uβR + α + u(1 − α)

p-hacking ::: combination of design/presentation/analysis factors that favor a significant test result beyond the normal alpha level

SLIDE 8

hypothesis H

R/R+1 1/R+1

true false apply test T apply test T sign null sign null

p-fishing ::: reporting at least

ne significant test results

from n (equally powered) studies

1 − βn βn (1 − α)n 1 − (1 − α)n

SLIDE 9

Positive predictive value example: R = 1, β = 0.2, α = 0.05 P(H = t | T = s) = R(1 − βn) R + 1 − (1 − α)n − Rβn

p-fishing ::: reporting at least

ne significant test results