Transparency and Trust: Towards the Promise of Open Science - - PowerPoint PPT Presentation

▶
transparency and trust towards the promise of open science
SMART_READER_LITE
LIVE PREVIEW

Transparency and Trust: Towards the Promise of Open Science - - PowerPoint PPT Presentation

Feb 20, 2023 140 likes •770 views

Transparency and Trust: Towards the Promise of Open Science Professor Liz Lyon School of Information Sciences, University of Pittsburgh INCONECSS 2016, Berlin Agenda 1. In the Headlines 2. Unpacking Transparency 3. Towards Open Science

slide-1
SLIDE 1

Transparency and Trust: Towards the Promise of Open Science

Professor Liz Lyon School of Information Sciences, University of Pittsburgh INCONECSS 2016, Berlin

slide-2
SLIDE 2

Agenda

1. In the Headlines 2. Unpacking Transparency 3. Towards Open Science

– Scholarship – Stewardship

4. Making it Happen

– LIS Workforce Development – Re-engineering Research Data Service Models

slide-3
SLIDE 3

In the Headlines

slide-4
SLIDE 4
slide-5
SLIDE 5

Tensions?

slide-6
SLIDE 6

Trusted product?

slide-7
SLIDE 7

Trusted service?

slide-8
SLIDE 8

https://www.washingtonpost.com/news/morning-mix/wp/2015/11/09/scientist-falsified- data-for-cancer-research-once-described-as-holy-grail-feds-say/

Trusted data?

slide-9
SLIDE 9

US institution X experience

  • Anil Potti paper in Nature Medicine 2006
  • Independent audit of the research by

Baggerly & Coombes (bio-statisticians)

  • IRB Inquiry & Report
  • Lessons learned include (Ince 2011):

– Sloppiness in data curation & software storage – Institutional reviewers did not verify the provenance of the data – Institutional data was not released – Institutional report was not published

slide-10
SLIDE 10

Unpacking the concept: Transparency

slide-11
SLIDE 11

Open Closed

Access Participation

Lone scholar Team science Citizen science

2D Continuum of Openness

Liz Lyon (2009) Open Science at Web Scale Report

slide-12
SLIDE 12

Towards a third dimension?

Easterbrook Nature Geoscience (2014) NIST definitions of Repeatability & Reproducibility in Tech Note 1297 (1994)

slide-13
SLIDE 13

Open Science terms & definitions (1)

  • Open or Reproducible Research:

Auditable research made openly available

  • Auditable Research: Sufficient records

(including data and software) have been archived so that the research can be defended later if necessary or differences between independent confirmations resolved.

Victoria Stodden et al Setting the Default to Reproducible Workshop Report (2013)

slide-14
SLIDE 14

Open Science: terms & definitions (2) Transparency:

  • The outcome of a suite of behaviours

which characterise Reproducible Research

  • Facilitates enhanced Research

Quality, Integrity and Trust

Liz Lyon (2016) LIBER Q

slide-15
SLIDE 15

Open Closed

Access Participation

Lone scholar Team science Citizen science

3D Model of Open Science

Transparency

Liz Lyon (2016) LIBER Q

slide-16
SLIDE 16

20 Terms: What Transparency is not!

Integrity?

1. Confusing 2. Gray/grey 3. Vague 4. Unclear 5. Opaque 6. Ambiguous 7. Obscured 8. Implicit 9. Hidden

  • 10. Secret

Clarity?

  • 11. Not verified
  • 12. Not validated
  • 13. Not auditable
  • 14. Not supported
  • 15. Not described
  • 16. Not documented
  • 17. Not recorded
  • 18. Not versioned
  • 19. Not tracked
  • 20. No provenance
slide-17
SLIDE 17

https://www.flickr.com/photos/8885264

What does this mean for Libraries? ….and for Librarians?

https://www.flickr.com/photos/claudia_l/5614406866/

slide-18
SLIDE 18

Design Plan Collect, Find, Acquire Process, Visualize Analyze Store Publish, Preserve, Archive Prepare Track

Adapted from ULS RDM WG Research Data Lifecycle

Context: Research Lifecycle

slide-19
SLIDE 19

Design Plan Collect, Find, Acquire Process, Visualize Analyze Store Publish, Preserve, Archive Prepare Track

Tracking Transparency

Products Identifiers Peer Reviews Versions Workflow tools Scripts & Software Graphics Models & Simulations Data Code Samples Reagents Materials Methods Instruments Tools Subjects Metadata Annotations Formats & Standards Files Licenses Methods & Protocols Results Cloud services Field Notebooks ELN Collaboration spaces

Practice: Actions?

Proposals Templates Drafts DMPs Re-use Ratings Credits Citations Blogs Tweets

Liz Lyon Liber Q (2016)

slide-20
SLIDE 20

Open Science: terms & definitions (3)

Transparency Actions:

  • Specific interventions as components of

processes, protocols and practices

  • Applicable throughout the research lifecycle

Liz Lyon (2016) LIBER Q

slide-21
SLIDE 21

Transparency research at Pitt iSchool

  • Pilot study 2015-16: explore awareness, attitudes

and actions towards Transparency & Open Science

  • Aim: to inform LIS service development, tools, LIS

education programs, professional skills

  • Methodology: focus groups with

a) disciplinary researchers b) librarians

  • Research Lifecycle as the substrate
slide-22
SLIDE 22

Design Plan Collect, Find, Acquire Process, Visualize Analyze Store Publish, Preserve, Archive Prepare Track

Adapted from ULS RDM WG Research Data Lifecycle

Substrate: Research Lifecycle

slide-23
SLIDE 23
slide-24
SLIDE 24

Q1 How are transparency actions reflected in open scholarship?

slide-25
SLIDE 25

“ Recommendation 6

As a condition of publication, scientific journals should enforce a requirement that the data on which the argument of the article depends should be accessible, assessable, usable and traceable through information in the article.”

Science as an Open Enterprise Report, Royal Society, UK

slide-26
SLIDE 26

Journals changing (open) data policy……

  • “Data deposition

in a public repository is mandatory …”

  • A step towards

Transparency ?

slide-27
SLIDE 27

This is accepted practice in some disciplines, but in others, not so much…. this leads to issues of trust……

slide-28
SLIDE 28

GigaScience and Publons

Open peer review (CC-BY) Papers and datasets Get credit for your reviews!

http://blogs.biomedcentral.com/bmcblog/2014/06/26/gigascience-helping-reviewers-get-credit-through-publons/

slide-29
SLIDE 29

http://www.psycontent.com/content/311q281518161139/fulltext.pdf

slide-30
SLIDE 30

Reproducibility Project Psychology Results 2015 : only 39% held up

slide-31
SLIDE 31

Transparency & Openness Promotion (TOP) Guidelines

  • Center for Open Science 2015
  • Science article June 2015
  • Journal Policies and Practices
  • 8 Transparency Standards
  • Templates for 3 Levels of

each Standard

http://science.sciencemag.org/content/sci/348/6242/1422 .full.pdf?ijkey=ha1o5D9wvW4ZQ&keytype=ref&siteid=sci

slide-32
SLIDE 32

8 Transparency Standards (TOP)

  • 1. Citation
  • 2. Data transparency
  • 3. Analytic methods (code)

transparency

  • 4. Research materials transparency
  • 5. Design & analysis transparency
  • 6. Pre-registration of studies
  • 7. Registration of analysis plans
  • 8. Replication
slide-33
SLIDE 33

CISER Replication Service

http://www.dcc.ac.uk/sites/default/files/documents/IDCC16/54_Arguillas%20and%20Block%20-%20Poster%20IDCC%202016.pdf

slide-34
SLIDE 34

Reproducibility isn’t always easy… …to peer-reproduced?

Gonzalez-Beltran, Li et al 2015 PLoS ONE

From peer reviewed …..

slide-35
SLIDE 35

Q2 How are transparency actions reflected in data stewardship?

slide-36
SLIDE 36

Laboratory notebooks: 3 role models

http://mss.sagepub.com/content/8/4/422.full.pdf+html

http://darwin-online.org.uk/

http://einsteinpapers.press.princeton.edu/

slide-37
SLIDE 37

All three role models

  • Recorded thoughts,
  • bservations, ideas,

calculations

  • Demonstrated the

provenance of their conclusions

  • Allowed other scientists

to reuse their findings

  • Good practice from >

100 years ago!

slide-38
SLIDE 38

http://news.utoronto.ca/huntingtons-disease-university-toronto-researcher-first-share-lab-notes-real-time

  • Another step towards

Transparency ?

slide-39
SLIDE 39

LIS data stewardship workflows to support transparency & trust?

slide-40
SLIDE 40

http://datasealofapproval.org/en/

Certification….. Trusted

  • Data Seal of Approval for repository certification
  • Self-assessment approach with external peer review
  • DSA online tool to facilitate application process
  • DSA is based on 16 guidelines (Version 2 2013)
slide-41
SLIDE 41

Making it Happen

slide-42
SLIDE 42

Q3 How can workforce development catalyse transparency and trust?

slide-43
SLIDE 43

A family of new data science roles

(Lyon & Brenner IJDC 2015)

slide-44
SLIDE 44

Linking data roles, skills & curriculum

(Lyon et al 2016, Lyon & Mattern 2016)

  • Analysis of real-world positions for six data roles
  • Part 1: data librarian, data archivist, data steward
  • Part 2: data analyst, data engineer, data journalist
  • Map to current iSchool courses
  • Informing development of a Data Stewardship Pathway
slide-45
SLIDE 45

Methods: Data Collection

Date Range for Job Postings: Part 1 January 2014-April 2015 Part 2 October 2015 Keyword searching and visual scanning Accessed 10 full job descriptions for each role (with IASSIST postings, more abbreviated job advertisements)

slide-46
SLIDE 46

Methods: Content Analysis

Competencies: proficiency with specific tools/technologies/programming languages. Education: Academic qualifications Experience: direct, hands-on practice Knowledge: understanding

  • f/familiarity with

topics/subjects/issues Skills: ability to do an action well

Identified all requirements that appeared in at least three of the positions studied for each role and designated these as “Key Requirements” Chose not to distinguish between “essential” and “desirable” requirements

slide-47
SLIDE 47

Data Librarian

slide-48
SLIDE 48

Data Steward / Curator

slide-49
SLIDE 49

Real World Job analysis Part 1 (Lyon et al iPres Proc 2016) Promote Transparency

slide-50
SLIDE 50

Open Science: terms & definitions (4)

These new Data Science roles can act as

Transparency Agents:

  • Promote, demonstrate

and action specific behaviours and practices for Open Science

slide-51
SLIDE 51

Requirement s

Methods: Course Mapping Data Stewardship Pathway

slide-52
SLIDE 52

Course Course Course Course Data Science Position

(Data Librarian, Data Archivist, Data Curator / Steward, Data Analyst, Data Engineer, Data Journalist)

Transparency & Trust Principles

“Stepping stones” form a Course Pathway

slide-53
SLIDE 53

Transparency and Trust are in the Data Stewardship Pathway in the MLIS curriculum at Pitt iSchool

slide-54
SLIDE 54

Q4 How should Library research data service models be re-engineered to support transparency and trust?

slide-55
SLIDE 55
  • 1. Transactional delivery model
  • In the physical Library
  • Remote
  • Access & Reference
  • RDM Advocacy
  • RDM LibGuides

Lyon New Review Academic Libraries (2016) In press

https://www.flickr.com/photos/smiling-gardener

slide-56
SLIDE 56

Lyon New Review Academic Libraries (2016) In press

  • Assigned to Faculty

/ Department

  • Liaison
  • Consultancy
  • DMP
  • RDM training
  • 2. Hybrid delivery model

https://www.flickr.com/photos/brownlessbiomedicallibrary

slide-57
SLIDE 57
  • 3. Immersive delivery model –

Librarians in the Lab

  • Laboratory or clinical

setting

  • Integrated
  • Collaborative team

science

  • Data description &

curation

  • Data analysis &

visualisation

https://www.flickr.com/photos/79173425@N03/9018554012/1410324768

Lyon New Review Academic Libraries (2016) In press

Photo Credits:Flickr NASA HQ

slide-58
SLIDE 58

Economics & Business?

  • Collaborations
  • Partnerships
  • Institutes
  • Centres
  • Groups
  • Alliances
slide-59
SLIDE 59

Benefits of Re-engineering?

  • Data support at the researchers’ point-of-need

(here and now)

  • LIS professionals fully integrated at the coalface
  • (in the field, in the business, in the lab….)
  • Default listings in citations with attribution + credit

(LIS “co-authors”)

  • LIS data science roles act as transparency agents

(enhance research integrity & open science)

slide-60
SLIDE 60

Radical Re-engineering….

…our academic & research libraries

https://en.wikipedia.org/wiki/Heydar_Aliyev_Center

slide-61
SLIDE 61

Thank you…. elyon@pitt.edu

INCONECSS 2016 Professor Liz Lyon, School of Information Sciences, University of Pittsburgh