Transparency and Trust: Towards the Promise of Open Science Professor Liz Lyon School of Information Sciences, University of Pittsburgh INCONECSS 2016, Berlin
Agenda 1. In the Headlines 2. Unpacking Transparency 3. Towards Open Science – Scholarship – Stewardship 4. Making it Happen – LIS Workforce Development – Re-engineering Research Data Service Models
In the Headlines
Tensions?
Trusted product?
Trusted service?
Trusted data? https://www.washingtonpost.com/news/morning-mix/wp/2015/11/09/scientist-falsified- data-for-cancer-research-once-described-as-holy-grail-feds-say /
US institution X experience • Anil Potti paper in Nature Medicine 2006 • Independent audit of the research by Baggerly & Coombes (bio-statisticians) • IRB Inquiry & Report • Lessons learned include (Ince 2011): – Sloppiness in data curation & software storage – Institutional reviewers did not verify the provenance of the data – Institutional data was not released – Institutional report was not published
Unpacking the concept: Transparency
2D Continuum of Openness Participation Citizen science Team science Lone scholar Closed Open Access Liz Lyon (2009) Open Science at Web Scale Report
Towards a third dimension? Easterbrook Nature Geoscience (2014) NIST definitions of Repeatability & Reproducibility in Tech Note 1297 (1994)
Open Science terms & definitions (1) • Open or Reproducible Research : Auditable research made openly available • Auditable Research : Sufficient records (including data and software) have been archived so that the research can be defended later if necessary or differences between independent confirmations resolved. Victoria Stodden et al Setting the Default to Reproducible Workshop Report (2013)
Open Science: terms & definitions (2) Transparency : • The outcome of a suite of behaviours which characterise Reproducible Research • Facilitates enhanced Research Quality, Integrity and Trust Liz Lyon (2016) LIBER Q
3D Model of Open Science Participation Citizen science Transparency Team science Lone scholar Closed Open Access Liz Lyon (2016) LIBER Q
20 Terms: What Transparency is not! Clarity? Integrity? 1. Confusing 11. Not verified 2. Gray/grey 12. Not validated 3. Vague 13. Not auditable 4. Unclear 14. Not supported 5. Opaque 15. Not described 6. Ambiguous 16. Not documented 7. Obscured 17. Not recorded 8. Implicit 18. Not versioned 9. Hidden 19. Not tracked 10. Secret 20. No provenance
What does this mean for Libraries? ….and for Librarians? https://www.flickr.com/photos/8885264 https://www.flickr.com/photos/claudia_l/5614406866/
Context: Research Lifecycle Design Track Plan Publish, Collect, Find, Preserve, Acquire Archive Process, Prepare Visualize Analyze Store Adapted from ULS RDM WG Research Data Lifecycle
Practice: Actions? Proposals Templates Re-use Drafts Ratings Design Credits DMPs Citations Blogs Track Plan Tweets Data Code Tracking Products Samples Publish, Identifiers Collect, Find, Reagents Preserve, Transparency Peer Reviews Materials Acquire Archive Versions Methods Instruments Tools Process, Subjects Prepare Visualize Analyze Metadata Store Workflow tools Annotations Formats & Standards Scripts & Software Files Graphics Cloud services Licenses Models & Field Notebooks Methods & Protocols Simulations ELN Results Collaboration spaces Liz Lyon Liber Q (2016)
Open Science: terms & definitions (3) Transparency Actions : • Specific interventions as components of processes, protocols and practices • Applicable throughout the research lifecycle Liz Lyon (2016) LIBER Q
Transparency research at Pitt iSchool • Pilot study 2015-16: explore awareness, attitudes and actions towards Transparency & Open Science • Aim: to inform LIS service development, tools, LIS education programs, professional skills • Methodology: focus groups with a) disciplinary researchers b) librarians • Research Lifecycle as the substrate
Substrate: Research Lifecycle Design Track Plan Publish, Collect, Find, Preserve, Acquire Archive Process, Prepare Visualize Analyze Store Adapted from ULS RDM WG Research Data Lifecycle
Q1 How are transparency actions reflected in open scholarship?
“ Recommendation 6 As a condition of publication, scientific journals should enforce a requirement that the data on which the argument of the article depends should be accessible, assessable, usable and traceable through information in the article .” Science as an Open Enterprise Report, Royal Society, UK
Journals changing (open) data policy…… • “Data deposition in a public repository is mandatory …” • A step towards Transparency ?
This is accepted practice in some disciplines, but in others, not so much…. this leads to issues of trust……
GigaScience and Publons Open peer review (CC-BY) Papers and datasets Get credit for your reviews! http://blogs.biomedcentral.com/bmcblog/2014/06/26/gigascience-helping-reviewers-get-credit-through-publons/
http://www.psycontent.com/content/311q281518161139/fulltext.pdf
Reproducibility Project Psychology Results 2015 : only 39% held up
Transparency & Openness Promotion (TOP) Guidelines • Center for Open Science 2015 • Science article June 2015 • Journal Policies and Practices • 8 Transparency Standards • Templates for 3 Levels of each Standard http://science.sciencemag.org/content/sci/348/6242/1422 .full.pdf?ijkey=ha1o5D9wvW4ZQ&keytype=ref&siteid=sci
8 Transparency Standards (TOP) 1. Citation 2. Data transparency 3. Analytic methods (code) transparency 4. Research materials transparency 5. Design & analysis transparency 6. Pre-registration of studies 7. Registration of analysis plans 8. Replication
CISER Replication Service http://www.dcc.ac.uk/sites/default/files/documents/IDCC16/54_Arguillas%20and%20Block%20-%20Poster%20IDCC%202016.pdf
Reproducibility isn’t always easy… From peer reviewed ….. …to peer-reproduced? Gonzalez-Beltran, Li et al 2015 PLoS ONE
Q2 How are transparency actions reflected in data stewardship?
Laboratory notebooks: http://einsteinpapers.press.princeton.edu / 3 role models http://darwin-online.org.uk / http://mss.sagepub.com/content/8/4/422.full.pdf+html
All three role models • Recorded thoughts, observations, ideas, calculations • Demonstrated the provenance of their conclusions • Allowed other scientists to reuse their findings • Good practice from > 100 years ago!
• Another step towards Transparency ? http://news.utoronto.ca/huntingtons-disease-university-toronto-researcher-first-share-lab-notes-real-time
LIS data stewardship workflows to support transparency & trust?
Certification….. Trusted • Data Seal of Approval for repository certification • Self-assessment approach with external peer review • DSA online tool to facilitate application process • DSA is based on 16 guidelines (Version 2 2013) http://datasealofapproval.org/en/
Making it Happen
Q3 How can workforce development catalyse transparency and trust?
A family of new data science roles (Lyon & Brenner IJDC 2015)
Linking data roles, skills & curriculum (Lyon et al 2016, Lyon & Mattern 2016) • Analysis of real-world positions for six data roles • Part 1: data librarian, data archivist, data steward • Part 2: data analyst, data engineer, data journalist • Map to current iSchool courses • Informing development of a Data Stewardship Pathway
Methods: Data Collection Keyword searching and visual scanning Date Range for Job Postings: Part 1 January 2014-April 2015 Part 2 October 2015 Accessed 10 full job descriptions for each role (with IASSIST postings, more abbreviated job advertisements)
Methods: Content Analysis Education: Academic qualifications Experience : direct, hands-on practice Knowledge : understanding Identified all requirements that appeared of/familiarity with in at least three of the positions studied topics/subjects/issues for each role and designated these as Skills : ability to do an action well “Key Requirements” Competencies : proficiency with specific tools/technologies/programming Chose not to distinguish between languages. “essential” and “desirable” requirements
Data Librarian
Data Steward / Curator
Real World Job analysis Part 1 (Lyon et al iPres Proc 2016) Promote Transparency
Open Science: terms & definitions (4) These new Data Science roles can act as Transparency Agents : • Promote, demonstrate and action specific behaviours and practices for Open Science
Methods: Course Mapping Data Stewardship Pathway Requirement s
Data Science Position (Data Librarian, Data Archivist, Data Curator / Steward, Data Analyst, Data Engineer, Data Journalist) Transparency & Trust Principles “Stepping stones” form Course a Course Pathway Course Course Course
Recommend
More recommend
