administration research and what should we change to be more open? - - PowerPoint PPT Presentation

▶
administration research and
SMART_READER_LITE
LIVE PREVIEW

administration research and what should we change to be more open? - - PowerPoint PPT Presentation

Aug 17, 2023 180 likes •509 views

Dominik Vogel How open is public administration research and what should we change to be more open? @DrDominikVogel 03.02.2020 | Dominik Vogel Disclaimer 2 https://pixabay.com/photos/school-teacher-education-asia-1782427/ Stop sign:

slide-1
SLIDE 1

How open is public administration research and what should we change to be more open?

Dominik Vogel

@DrDominikVogel

slide-2
SLIDE 2

Disclaimer

https://pixabay.com/photos/school-teacher-education-asia-1782427/ Stop sign: Freepik.com

03.02.2020 | Dominik Vogel 2

slide-3
SLIDE 3

What is open science?

3 03.02.2020 | Dominik Vogel

Open peer review image [modified]: Joe The Goat Farmer - How to Grow Your Email List with A Great Newsletter, CC-BY 2.0 Logo OER: Markus Büsges (leomaria design) für Wikimedia Deutschland e. V., CC-BY-SA 4.0 via Wikimedia Commons. Open Source Software: Logo Open Source Initiative [modified] by Simon Phipps under CC-BY 2.5

Open Peer Review

…

slide-4
SLIDE 4

Open science is “the process of making the content and process of producing evidence and claims transparent and accessible to others” (Munafò et al. 2017, p. 5).

4 03.02.2020 | Dominik Vogel

slide-5
SLIDE 5
  • OK. But why?

5 03.02.2020 | Dominik Vogel

https://pixabay.com/photos/question-mark-why-problem-solution-2123967/

slide-6
SLIDE 6

Normative mative answe wer Practi ctical cal answe wer

6 03.02.2020 | Dominik Vogel

Tools photo by Haupes Co. on Unsplash

slide-7
SLIDE 7

The replication / credibility crisis in psychology

7 03.02.2020 | Dominik Vogel

6 4 2

2005 2011 2014 2017 2012 2015 2019

Bem: “Feeling the Future” Simonsohn et al: “p-Curve: The key to the file drawer” Chambers: “The Seven Deadly Sins

  • f Psychology”

Ioannidis: “Why most research findings are false” Simmons et al.: “False- positive Psychology” Open Science Collaboration: “Estimating the Reproducibility

  • f Psych

Kvarven et al.: “Comparing meta-analyses and prereg. multi-lab repl. projects

based on Spellman et al. 2017

Designed by PresentationGO.com

slide-8
SLIDE 8
  • Why do we have such low

replicability?

  • What results can we trust?

Psychologists (and other social scientists) wonder

8 03.02.2020 | Dominik Vogel

Image by Robin Higgins from Pixabay

slide-9
SLIDE 9
  • At the center: Publication

bias and the file drawer problem

The many ingredients of the replication crisis

9 03.02.2020 | Dominik Vogel

https://pixabay.com/photos/files-paper-office-paperwork-stack-1614223/

slide-10
SLIDE 10
  • Preference for novel, surprising, and significant results

sets incentives for Questionable Research Practices (QRP)

  • HARK

RKing ng: Hypothesizing after results are known

  • p-hacki

acking ng: additional analyses / data to pass p < .05

  • Conducting underpo

derpowered wered studies

  • Fraud

The many ingredients of the replication crisis

10 03.02.2020 | Dominik Vogel

Photo by Calum Lewis on Unsplash

slide-11
SLIDE 11
  • Transparency:
  • Everybody should be able to assess how results were obtained
  • Reducing researcher degrees of freedom
  • Define as much as possible in advance

The solution(?)

11 03.02.2020 | Dominik Vogel

slide-12
SLIDE 12
  • Open Data
  • Open Materials (esp. Code)
  • Reporting standards
  • Open peer review
  • Open Source Software
  • [Open Access]

The solution: Transparency

12 03.02.2020 | Dominik Vogel

Open peer review image [modified]: Joe The Goat Farmer - How to Grow Your Email List with A Great Newsletter, CC-BY 2.0 Open Source Software: Logo Open Source Initiative [modified] by Simon Phipps under CC-BY 2.5
slide-13
SLIDE 13
  • Separate exploratory from confirmatory research
  • Confirmatory: Define as much as possible in advance
  •  Less ways to (unconsciously) tweak the results in the

desired direction

  • Preregistration
  • Registered reports

The solution: Reducing researcher degrees of freedom

13 03.02.2020 | Dominik Vogel

slide-14
SLIDE 14
  • Culture needs to change
  • Incentives need to change

Tools do not magically lead to better science

14 03.02.2020 | Dominik Vogel

Image by Alexas_Fotos from Pixabay

slide-15
SLIDE 15
  • Open science badges
  • Registered reports
  • Journals value replications
  • Many Lab projects / large-scale replications
  • Error (and fraud?) detection

Changes to incentive structure

15 03.02.2020 | Dominik Vogel

slide-16
SLIDE 16

Enough psychology, I want to learn about PA

16 03.02.2020 | Dominik Vogel

Stop sign: Freepik.com

slide-17
SLIDE 17

Yes es

  • No careful assessment, yet
  • Incentives are the same as

in psychology

  • Survey research offers even

more ways for HARKing and p-hacking (control variables) No No

  • No evidence
  • Less small-n experiments

(yet?)

  • Less ways to repeat studies

to get intended results

  • More PSM of PA

researchers(?)

03.02.2020 | Dominik Vogel 17

Is there a replication crisis in PA?

Icons made by Freepik from www.flaticon.com

slide-18
SLIDE 18

03.02.2020 | Dominik Vogel 18

Does the literature on the PSM– performance relationship contain evidential value?

(Vogel & Homberg under review)

slide-19
SLIDE 19
  • Distribution of p values (p-curve) follows a predictable

pattern

  • Holds for subset of significant p values
  • Reporting of significant p values should be unbiased

p-curve method: analyze significant p values of published research

19 03.02.2020 | Dominik Vogel

slide-20
SLIDE 20

Distribution of p values without a true effect

20 03.02.2020 | Dominik Vogel

slide-21
SLIDE 21

21 03.02.2020 | Dominik Vogel

not p-hacked p-hacked

slide-22
SLIDE 22

Result of the p-curve analysis

22 03.02.2020 | Dominik Vogel

slide-23
SLIDE 23

So, no reason to worry?

23 03.02.2020 | Dominik Vogel

Photo by Lidya Nada on Unsplash

slide-24
SLIDE 24
  • We know little about the credibility of PA research
  • Even if there is no replication crisis, open science

practices help to prevent a crisis in the future

  • They help to do better science  find the truth

Why should PA adopt open science practices?

24 03.02.2020 | Dominik Vogel

slide-25
SLIDE 25
  • Reviewers are more aware of adverse effects of

underpowered studies, HARKing, and p-hacking

  • Preregistration more and more common and valued
  • New open access journals
  • Funders are pushing for open

science practices

What did already change?

25 03.02.2020 | Dominik Vogel

slide-26
SLIDE 26
  • No pre-print culture
  • No registered reports
  • Journals still closed access

What did not change?

26 03.02.2020 | Dominik Vogel

slide-27
SLIDE 27
  • “You’re doing it because you want to do

high quality work. You want to have the best possible chance of learning something True about the world and the people in it.” (Corker 2018)

  • “The first principle is that you must not

fool yourself – and you are the easiest person to fool.” – Richard Feynman

Why should I adopt open science practices?

27 03.02.2020 | Dominik Vogel

Photo of Richard Feynman by Tamiko Thiel available under a CC BY-SA 3.0 license

slide-28
SLIDE 28
  • Preregister your studies when possible (and indicate

exploratory work)

  • Publish your data, analysis code, and materials
  • When reviewing: ask for proper reporting and

transparency; be skeptical

  • Publish the accepted manuscripts of your publications
  • Publish pre-prints?

OK, you convinced me. What can I do?

28 03.02.2020 | Dominik Vogel

slide-29
SLIDE 29
  • Require proper statistical reporting
  • Enable registered reports
  • Push for open data, open materials
  • Encourage pre-prints
  • Encourage replications
  • Adopt TOP guidlines

What can journals do?

29 03.02.2020 | Dominik Vogel

slide-30
SLIDE 30
  • Value open science practices
  • Switch from traditional publishing

system to open access

What can societies do?

30 03.02.2020 | Dominik Vogel

slide-31
SLIDE 31
  • Prof. Dr. Dominik Vogel

University of Hamburg Assistant Professor of Public Management Von-Melle-Park 9 20146 Hamburg, Germany dominik.vogel-2@uni-hamburg.de Twitter: www.twitter.com/DrDominikVogel Website: https://vogel-online.info/en

31 03.02.2020 | Dominik Vogel

slide-32
SLIDE 32

Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal

  • f Personality and Social Psychology, 100(3), 407–425.

https://doi.org/10.1037/a0021524 Chambers, C. (2017). The seven deadly sins of psychology: A manifesto for reforming the culture of scientific practice. Princeton, Oxford: Princeton University Press. Corker K. (2018). Open Science is a Behavior. https://cos.io/blog/open-science-is-a-behavior/ Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124 Kvarven, A., Strømland, E., & Johannesson, M. (2019). Comparing meta-analyses and preregistered multiple-laboratory replication

  • projects. Nature Human Behaviour. Advance online publication.

https://doi.org/10.1038/s41562-019-0787-z Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Du Percie Sert, N., . . . Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1),

  • 21. https://doi.org/10.1038/s41562-016-0021

Open Science Collaboration (2015). Estimating the reproducibility

  • f psychological science. Science, 349(6251), aac4716.

https://doi.org/10.1126/science.aac4716 Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632 Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology. General, 143(2), 534–547. https://doi.org/10.1037/a0033242 Spellman, B., Gilbert, E. A., & Corker, K. S. (2017). Open Science: What, Why, and How. https://doi.org/10.31234/osf.io/ak6jr

03.02.2020 | Dominik Vogel 32

Literature