Singapore RI conference organized by A*STAR, NTU and NUS October - - PDF document

Singapore – RI conference organized by A*STAR, NTU and NUS – October 22, 2018 – 60 minutes including Q&A. 1

2

The figures concern the question ‘did you at least once in the last 3 years engage in FF / QRP ?’ and come from the highly cited meta‐analysis: Fanelli D. How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta‐Analysis of Survey Data. PLoS ONE 2009; 4(5): e5738

3

4

This is the Top 5 based on the survey performed as part of the Academic Research Climate in Amsterdam (ARCA). Based on frequency weighted impact on validity (frequency X impact on validity). These are all questionable research practices that research institutes can do something about. FFP turn up only in the bottom half of the top 60. Website: www.amsterdamresearchclimate.nl Preregistration: https://osf.io/x6t2q/

5

Many rewards in academia are linked to having positive and spectacular results as these are published more easily in high impact journals and will be cited more often. The various QRP have in common that they can effectively help to get these positive and spectacular results.

6

7

This slide shows – in a simplified way – how things can go wrong. In most disciplines the proportion of papers reporting positive results increases over

• time. Positive results are published and cited more often, and also get more media
• attention. This will probably increase the likelihood of getting grants and tenure. We

have also some evidence that conflicts of interest and sponsor interests may lead to sloppy science or worse. QRP and RM can effectively help to get (false) positive results. Negative findings are so unpopular that often these are not reported at all. This mechanism will lead to publication bias, selective reporting and selective citation. Especially small studies with positive outcomes will predominantly be chance findings. These phenomena will distort the level of truth in the published record and can explain the large replication difficulties some fields (e.g. preclinical research) experience. There is evidence for some of the relations suggested in this slide, but no or only little evidence for most of them. We really need more solid empirical research to clarify how these things work. Gaining this knowledge is important for effectively fostering RCR and preventing QRP and RM.

8

Wicherts et al ‐ Degrees of freedom ‐ checklist to avoid p‐hacking ‐ Front Psych 2016; 7 1832 This wonderful article comes from the faculty where Diederik Stapel was dean: never waste a good crisis. The idea of Researcher Degrees of Freedom indicates that sloppy science offers a lot of room to get the findings and conclusions you want. Please note: we’re talking about hypothesis testing research (confirmatory research), NOT about exploratory research. In the latter domain ‘anything goes’ as long as it’s clearly stated that exploration is at issue. See also: Wicherts – The weak spots of contemporary science (and how to fix them) ‐ Animals 2017, 7, 90; doi:10.3390/ani7120090

9

de Vries YA, Roest AM, de Jonge P, Cuijpers P, Munafò MR, Bastiaansen JA (2018). The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: the case of depression. Psychological Medicine 1–3. https://doi.org/10.1017/S0033291718001873 This example concerns the fate of an inception cohort of 105 RCTs of the efficacy of anti‐ depression drugs from the FDA database. The cohort is conplete in the sense that pharmeceutical companiet must register all trials the intend to use to obtain FDA approval before embarking on data collection. The FDA considered 50% of the trials positive after carefully looking at the results.

10

Advances in Methods and Practices in Psychological Science 2018 1‐20 (DOI: 10.1177/2515245917747646)

11

Advances in Methods and Practices in Psychological Science 2018 1‐20 (DOI: 10.1177/2515245917747646) In fact this is a replication study consisting of re‐analyses (29 times) of the same data set.

12

13

Let’s first agree that replication is essential in science, both at the basic and the more sophisticated levels. 14

15

Just two recent Nature headlines. The topic draws attention, and rightly so. Nuzzo ‐ Fooling ourselves ‐ Nature 2015;526 182‐185 Baker ‐ Is there a replicability crisis ‐ Nature 2016; 533 452‐4

16

Replication problems have been studied mainly in the social and biomedical sciences (animal research and clinical studies), but there is little reason to believe that they would not occur in other disciplinary fields. Peels R, Bouter LM. The possibility and desirability of replication in the humanities. Palgrave Communications 2018; 4: 95 See also:

Peels R, Bouter LM. R eplication is both possible and desirable in the humanities, j ust as it is in the sciences. LS E Impact Blog 2018; October 1st (http:/ / blogs.lse.ac.uk/ impactofsocialsciences/ 2018/ 10/ 01/ replication-is- both-possible-and-desirable-in-the-humanities-j ust-as-it-is-in-the-sciences/ ).

17

We need to discriminate between 3 forms of replication and 3 criteria for when a replication is successful. Nosek BA, Errington TM. Making sense of replications. eLife 2017; 6: e23383 Goodman et al ‐ What does reproducibility really mean ‐ Science Translational Medicine 2016; 8 341ps12 Munafò and Davey Smith ‐ Repeating experiments is not enough ‐ Nature 2018; 553 399‐401 PDF of recent report on replication studies by Royal Netherlands Academy of Arts and Sciences is available at: https://www.nrin.nl/wp‐content/uploads/KNAW‐Replication‐ Studies‐15‐01‐2018.pdf

18

Wicherts et al ‐ Degrees of freedom ‐ checklist to avoid p‐hacking ‐ Front Psych 2016; 7 1832 Nosek et al ‐ The preregistration revolution ‐ PNAS 2018; 115 2600‐6 Bouter ‐ Fostering responsible research practices is a shared responsibility of multiple stakeholders ‐ J Clin Epidemiol 2018; 93 143‐6

19

http://www.nature.com/articles/s41562‐016‐0021

20

Zwaan et al ‐ Making replication mainstream ‐ Behavioral and Brain Sciences 2018; 41 e120

21

Ioannidis ‐ Why replication has more scientific value than original discovery ‐ Behavioral and Brain Sciences 2018; 41 e137

22

In theory the solution is easy and takes the form of ensuring that all research findings are published and the whole process is transparent, meaning that all steps can be checked and reconstructed. Studies need to be preregistered and a full protocol must be uploaded in a repository before the start of data collection. Similarly a data‐analysis plan, syntaxes, data sets and full results need to be uploaded. Amendments and changes are possible but should always leave traces, thus enabling users to identify actions that were potentially data‐driven. While ideally these elements of transparency are publicly accessible, there are many situations where delayed, conditional or incomplete access is indicated. But that does not detract from the principle of full transparency: even the process and outcomes of highly classified research for the defence industry should if necessary be made available for a thorough check by an investigation committee that is bound by confidentiality. Bouter LM. Perverse incentives and rotten apples. Accountability in Research 2015; 22:148‐161. Bouter LM. Open data is not enough to realize full transparency. J Clin Epidemiol 2016; 70: 256‐7. Bouter LM. Fostering responsible research practices is a shared responsibility of multiple

• stakeholders. Journal of Clinical Epidemiology 2018; 96: 143‐6.

ter Riet G, Bouter LM. How to end selective reporting in animal research. In: Martic‐Kehl

23

MI, Schubiger PA, eds. Animal models for human cancer: discovery and development of novel therapeutics. First edition. Weinheim: Wiley, 2016: 61‐77. See also: Nosek BA, Ebersole CR, DeHaven AC, Mellor D. The preregistration revolution. PNAS 2018;115:2600‐6 ‐ http://www.pnas.org/content/115/11/2600

23

Nosek BA, Ebersole CR, DeHaven AC, Mellor D. The preregistration revolution. PNAS 2018;115:2600‐6. http://www.pnas.org/content/115/11/2600

24

https://osf.io/ https://figshare.com/ https://www.mendeley.com/ https://datadryad.org/ www.re3data.org

25

Chambers et al ‐ Instead of playing the game its time to change the rules ‐ registered reports ‐ AIMS Neuroscience 2014; 1 4‐17 Chambers ‐ Ten reasons why journals must review manuscripts before results are known ‐ Addiction 2015; 110 10‐11 https://cos.io/our‐services/registered‐reports

26

Kupferschmidt ‐ A recipe for rigor ‐ Science 2018; 261: 1192‐3

27

https://arxiv.org/ https://chemrxiv.org/ https://www.biorxiv.org/ https://psyarxiv.com/ http://asapbio.org/ https://pubpeer.com/ List of 46 preprint servers at https://docs.google.com/spreadsheets/d/17RgfuQcGJHKSsSJwZZn0oiXAnimZu2sZsWp8Z 6ZaYYo/edit#gid=0 YouTube video ‘What are preprints?’ (https://www.youtube.com/watch?time_continue=9&v=2zMgY8Dx9co) Bastian H. Breaking down pros and cons of preprints in biomedicine. PLoS Blog posted

• n May 1, 2016 (http://blogs.plos.org/absolutely‐maybe/2016/05/01/breaking‐down‐

pros‐and‐cons‐of‐preprints‐in‐biomedicine/) Chalmers I, Glaziou P. Should there be greater use of preprint servers for publishing reports of biomedical science? F1000Research 2016; 5: 272

• Editorial. MedRArXiv: “To boldly go where no one has gone before”—Star Trek: The Next
• Generation. Journal of Cutaneous Medicine and Surgery 2018; 22: 261–262

28

Teixeira da Silva JA et al. Problems with traditional science publishing and finding a wider niche for post‐publication peer review. Accountability in Research 2015; 22: 22‐40.

28

29

We currently perform a study on the Academic Research Climate in Amsterdam that tries to both answer scientific questions and to provide input for local measures and interventions. Website: www.amsterdamresearchclimate.nl Preregistration: https://osf.io/x6t2q/ 30

Martinson BC, Thrush CR, & Crain AL. 2013. Development and validation of the Survey of Organizational Research Climate. Sci Eng Ethics 2013; 19: 813–34 Crain AL, Martinson BC, Thrush CR. Relationships between the Survey of Organizational Research Climate and self‐reported research practices. Sci Eng Ethics 2013; 19: 835‐40 Wells JA, Thrush CR, Martinson, BC, May, TA, Stickler, M, Callahan, EC, Klomparens KL 2014. Survey of organizational research climate. J Emp Res Hum Res Ethics 2015; 9: 72–88 Tijdink JK, Smulders YM, Vergouwen AC, de Vet HC, Knol DL 2014. The assessment of publication pressure in medical science: validity and reliability of a Publication Pressure Questionnaire (PPQ). Qual Life Res 2014; 23: 2055–62 Tijdink, JK, Vergouwen AC, Smulders YM. 2013. Publication pressure and burn out among Dutch medical professors: a nationwide survey. PLoS One 2013; 8: e73381 Tijdink JK, Verbeke R, Smulders YM 2014. Publication pressure and scientific misconduct in medical scientists. J Emp Res Hum Res Ethics 2014; 9: 64–71

Article on revision of Publication Pressure Questionnaire as preprint available at https://psyarxiv.com/r5hn7

Bouter LM, Tijdink J, Axelsen N, Martinson BC, ter Riet G. Ranking major and 31

minor research misbehaviors: results from a survey among participants of four World Conferences on Research Integrity. Research Integrity and Peer Review 2016; 1: 17 (http://rdcu.be/mPZT) 31

Preprint of SOuRCe results: Haven TL et al. Perceptions of research integrity climate differ between academic ranks and disciplinary fields: results from a survey among academic researchers in Amsterdam Available at: https://psyarxiv.com/543hw/ This is food for thought for research institutions. In Amsterdam we proceeded by discussing the fact sheets with the survey results with the staff and the research leaders. 32

Preliminary unpublished results from ARCA survey. This is food for thought for research institutions. In Amsterdam. We proceeded by discussing the fact sheets with the survey results with the staff and the research leaders. 33

34

http://www.vsnu.nl/files/documents/Netherlands%20Code%20of%20Conduct%20for%2 0Research%20Integrity%202018.pdf http://www.vsnu.nl/files/documenten/Nederlandse%20gedragscode%20wetenschappel ijke%20integriteit%202018.pdf https://doi.org/10.17026/dans‐2cj‐nvwu

35

Regarding #2: Bouter LM, Hendrix S. Both whistle blowers and the scientists they accuse are vulnerable and deserve protection. Accountability in Research: Policies and Quality Assurance 2017; 24: 359‐66.

36

37

https://www.natureindex.com/news‐blog/from‐punish‐to‐empower‐a‐blame‐free‐ approach‐to‐research‐misconduct

38

Enserink ‐ Research on research ‐ Science 2018; 361 1178‐9 Here the funding agencies should take action. Examples from the Netherlands are: Replication Studies Program description: https://www.nwo.nl/en/research‐and‐ results/programmes/replication+studies Call for proposals: https://www.nwo.nl/en/documents/sgw/replication‐studies‐‐‐call‐for‐ proposals Brochure with on‐going projects: https://mediator.zonmw.nl/mediator‐special‐juni‐ 2016/ Fostering Responsible Research Practices Programe description: https://www.zonmw.nl/en/research‐and‐results/fundamental‐ research/programmas/programme‐detail/fostering‐responsible‐research‐practices/ Brochure with on‐going projects: https://mediator.zonmw.nl/mediator‐special‐juni‐ 2016/

39

Program description: https://www.nwo.nl/en/research‐and‐ results/programmes/replication+studies Call for proposals: https://www.nwo.nl/en/documents/sgw/replication‐studies‐‐‐call‐for‐ proposals Brochure with on‐going projects: https://mediator.zonmw.nl/mediator‐special‐juni‐ 2016/ Podcast (in Dutch): https://mediator.zonmw.nl/mediator‐20‐november‐2016/herhalen‐ is‐het‐hart‐van‐de‐wetenschap/ Kick off meeting (in Dutch): https://publicaties.zonmw.nl/kick‐off‐meeting‐voor‐ projectgroepen‐programmas‐bvo‐en‐replicatiestudies/ Newspaper report on one of the projects (in Dutch): https://www.volkskrant.nl/nieuws‐ achtergrond/nederlanders‐zagen‐stoelpoot‐superpsycholoog‐ariely‐af~b5d1443e/

40

Ioannidis et al. ‐ The scientists who publish a paper every five days ‐ Nature 2018; 561 167‐9

41

Ioannidis et al. ‐ The scientists who publish a paper every five days ‐ Nature 2018; 561 167‐9

42

43

Adopted at the 2nd World Conference on Research Integrity in 2010 in Singapore. It became an informal code of conduct on world level. These principles concern research integrity, but also elegantly describe the character of Tony Mayer. Tony has been and still is really important for fostering research integrity, not only in NTU and Singapore but also on a global scale. https://www.wcrif.org/documents/327‐singapore‐statement‐a4size/file

44

45