data provenance and reproducability
play

Data Provenance and Reproducability Tim Schmidt Syeda Hiba Ahmad - PowerPoint PPT Presentation

Nov 18, 2023 •123 likes •549 views

Data Provenance and Reproducability Tim Schmidt Syeda Hiba Ahmad 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad Outline Reproducability Data Provenance

  1. Data Provenance and Reproducability Tim Schmidt Syeda Hiba Ahmad 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  2. Outline ● Reproducability ● Data Provenance ▪ ▪ Importance Importance ▪ ▪ Crisis Challenges ▪ ▪ In ML Current Standards ▪ ▪ In Companies Different Approaches ▪ ▪ What to do about it? Provenance Taxonomy ▪ Examples ● Future Work 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  3. Reproducability “A measure of whether results can be attained by a different research team, using the same methods.” 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  4. Importance of Reproducability shows that there are no confounding variables ● ○ protects against fraud ○ human error 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  5. Reproducability Crisis A crisis of repeatability: “Of these 100 studies, just 68 reproductions provided [..] results that matched the original findings.” A crisis of description: Of 400 algorithms [..] He found that only 6% [..] shared the algorithm’s code. Only a third shared the data they tested their algorithms on, and just half shared “pseudocode”. 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  6. Reproducability in ML ML swaps heuristics for blackbox for better results ● Randomness between runs (need to fix meta parameters) ● Need to store data ● 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  7. Reproducability in Companies If a researcher drops out, somebody else should be able to step in ● Cope with changed requirements or platforms ● ○ time saver in the long run It’s a lot more risky to try different variation ● 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  8. What to do about it? Science: Provide all the info: code, data, description ● Companies: No wrong incentives ● Repetition of experiments ● Keep the team educated ● 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  9. What to do about it? ML: Features causing non-deterministic results are disabled ● Versioning of models ● (Jupiter Notebooks are a nightmare) ➔ Data versioning ● 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  10. Data Provenance “Data Provenance is the documentation of data in sufficient detail to allow reproducibility of a specific dataset.” 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  11. Importance Compliance (e.g. DSGVO) ● Necessary if accused of fraud ● Prevents manual errors ● Changes in underlying database ● Data needs to be trustworthy ● Root case analysis ● 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  12. Challenges Large data sizes ● Provenance overhead ● Archiving (vs changes) ● 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  13. Archiving (vs changes) Key Added Deleted Modified 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  14. Current Standards “The PROV Family of Documents defines a model , corresponding serializations and other supporting definitions to enable the inter-operable interchange of provenance information in heterogeneous environments such as the Web.” https://www.w3.org/TR/prov-overview/ 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  15. Different Approaches Where-Provenance (Original Source), Why-Provenance (Contributing ● Source) and How-Provenance (Transformation) 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  16. Provenance Taxonomy 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  17. Use of Provenance ● Data Quality : Level of detail and error ● Audit Trail : Process with which data is produced ● Replication : Availability of similar sources ● Attribution : Pedigree (ownership) ● Informational : Metadata (descriptive) 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  18. Subject of Provenance ● Data Oriented (Explicit) Model : Metadata from source data ● Process Oriented (Indirect) : Metadata from process inputs and outputs ● Granularity : Level of detail 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  19. Provenance Representation ● Annotation : Descriptions about source data and processes ● Inversion : Reverse-engineering queries ● Contents : Of annotation and inversion methods ● Syntactic Information : The form in which data is stored ● Semantic Information : The meaning given to the data 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  20. Storing Provenance ● Tightly Coupled : Close relation with data ● Loosely Coupled : Slight relation with data ● Scalability : Growth of system ● Overhead : Management costs 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  21. Provenance Dissemination 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  22. Practical Examples DVC (for small projects) ● ○ Git expansion for data Pachyderm (for bigger projects) ● ○ Runs on Kubernetes More Informations 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  23. Data Versioning Control Data Versioning: ● DVC File 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  24. Data Piplines & Experiments Link processing steps together ● Store versions and prameters ● Compare versions ● 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  25. Pachyderm Based on Kubernetes and Docker ● 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  26. Pachyderm Data Versioning Pre1 Net Pre2 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  27. Pachyderm Containerized Analysis Pre1 Net Pre2 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  28. Pachyderm Data Pipelines Pre1 Net Pre2 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  29. Pachyderm Pre1 Net Pre2 Scalable Stages 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  30. Pachyderm Pre1 Net Pre2 Data Provenance 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  31. Google Dataset Search (GOODS) 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  32. Google Dataset Search (GOODS) 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  33. What is Next? How to rank datasets ● How to identify important datasets ● Handling missing metadata ● More work on data semantics ● Data citation ● Environment information ● Applications in ML, social media, ● block chain, cybersecurity 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  34. Questions? 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

  35. Thank you! :) 23.06.2020 | Fachbereich Informatik | Software Engineering for Artificial Intelligence| Tim Schmidt, Syeda Hiba Ahmad

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

VERSIONING, VERSIONING, PROVENANCE, AND PROVENANCE, AND REPRODUCABILITY REPRODUCABILITY

VERSIONING, VERSIONING, PROVENANCE, AND PROVENANCE, AND REPRODUCABILITY REPRODUCABILITY

VERSIONING, VERSIONING, PROVENANCE, AND PROVENANCE, AND REPRODUCABILITY REPRODUCABILITY Christian Kaestner Required reading: Halevy, Alon, Flip Korn, Natalya F. Noy, Christopher Olston, Neoklis Polyzotis, Sudip Roy, and Steven Euijong

755 views • 46 slides
Provenance of astronomical data The IVOA Provenance Working Group: Catherine Boisson Franois

Provenance of astronomical data The IVOA Provenance Working Group: Catherine Boisson Franois

Provenance Data Model Provenance of astronomical data The IVOA Provenance Working Group: Catherine Boisson Franois Bonnarel Johan Bregeon Pierre Le Sidaner Julien Lefaucheur Mireille Louys Markus Nullmeier Ana Palacios Kristin Riebe

360 views • 22 slides
Provenance from the data provider view constructing provenance information for the APPLAUSE

Provenance from the data provider view constructing provenance information for the APPLAUSE

Provenance from the data provider view constructing provenance information for the APPLAUSE archive Anastasia Galkin Ole Streicher Kristin Riebe Harry Enke EDP Forum Heidelberg 2018 6 Constructing the provenance Used Used for or the

610 views • 15 slides
Towards Semantics for Provenance Security Stephen Chong Harvard University TaPP 09

Towards Semantics for Provenance Security Stephen Chong Harvard University TaPP 09

Towards Semantics for Provenance Security Stephen Chong Harvard University TaPP 09 Provenance security Some data are sensitive Must ensure provenance does not reveal sensitive data E.g., John participated in medical study S reveals

277 views • 27 slides
Challenges in managing implicit and abstract provenance data: experiences with ProvManager

Challenges in managing implicit and abstract provenance data: experiences with ProvManager

Workshop on the Theory and Practice of Provenance 2011 Challenges in managing implicit and abstract provenance data: experiences with ProvManager Anderson Marinho, Marta Mattoso, Cludia Werner, Vanessa Braganholo, Leonardo Murta

785 views • 12 slides
Scalable Uncertainty Management 03 Provenance Rainer Gemulla May 18, 2012 Overview In this

Scalable Uncertainty Management 03 Provenance Rainer Gemulla May 18, 2012 Overview In this

Scalable Uncertainty Management 03 Provenance Rainer Gemulla May 18, 2012 Overview In this lecture Introduction to datalog What is provenance? Which types of provenance do exist? Lineage Why-provenance How-provenance How to

466 views • 43 slides
Whats the Problem? What does it mean to collect provenance when you dont control:

Whats the Problem? What does it mean to collect provenance when you dont control:

Provenance in the Wild Peter Macko, Margo Seltzer June 14, 2012 Whats the Problem? What does it mean to collect provenance when you dont control: The data (types, format, organization, structure) The operators The

445 views • 10 slides
France integration of Provenance data: the Bibale database Zakad Narodowy im. Ossoliskich ,

France integration of Provenance data: the Bibale database Zakad Narodowy im. Ossoliskich ,

France integration of Provenance data: the Bibale database Zakad Narodowy im. Ossoliskich , Wrocaw 29-30 October 2016 Monique Hulvey Federating at the national level? Research on provenance is an international project In France,

296 views • 15 slides
Secure Data Provenance in Home Energy Monitoring Networks Ming Hong Chia, Sye Loong Keoh, Zhaohui

Secure Data Provenance in Home Energy Monitoring Networks Ming Hong Chia, Sye Loong Keoh, Zhaohui

Secure Data Provenance in Home Energy Monitoring Networks Ming Hong Chia, Sye Loong Keoh, Zhaohui Tang 1 Outline Data Provenance and Smart Metering Security Threats and Requirements Proposed Architecture Threshold Cryptography

368 views • 22 slides
Fine Grain Provenance Using Temporal Databases Outline of the talk Use case: Classic

Fine Grain Provenance Using Temporal Databases Outline of the talk Use case: Classic

<Insert Picture Here> Fine Grain Provenance Using Temporal Databases Outline of the talk Use case: Classic management of patient data Data types, queries History Security and context information Fine grain provenance

401 views • 19 slides
Open data provenance and reproducibility: a case study from publishing CMS open data Tibor

Open data provenance and reproducibility: a case study from publishing CMS open data Tibor

Open data provenance and reproducibility: a case study from publishing CMS open data Tibor Simko 1 Heitor Pascoal de Bittencourt 2 Edgar Carrera 2 Clemens Lange 2 Kati Lassila-Perini 2 Lara Lloret 2 Tom McCauley 2 Jan Okraska 1 Daniel

391 views • 15 slides
A graph model for data and workflow provenance Umut Acar, Peter Buneman, James Cheney , Natalia

A graph model for data and workflow provenance Umut Acar, Peter Buneman, James Cheney , Natalia

A graph model for data and workflow provenance Umut Acar, Peter Buneman, James Cheney , Natalia Kwasnikowska, Jan van den Bussche, & Stijn Vansummeren TaPP 2010 Provenance in ... Databases Workflows Mainly for (nested)

922 views • 68 slides
Certifying Video Provenance Ashish Gehani I3P/SRI 1 INTRODUCTION : Why certify? Video

Certifying Video Provenance Ashish Gehani I3P/SRI 1 INTRODUCTION : Why certify? Video

Certifying Video Provenance Ashish Gehani I3P/SRI 1 INTRODUCTION : Why certify? Video cameras / editors ubiquitous Data originates from multiple sources Consumer trusts some producers Must be able to validate provenance

339 views • 23 slides
Provenance and data access in the context of Cherenkov astronomy C. Boisson & M. Servillat

Provenance and data access in the context of Cherenkov astronomy C. Boisson & M. Servillat

CTA Southern Hemisphere Site Rendering; credit: Gabriel Prez Diaz, IAC, SMM Provenance and data access in the context of Cherenkov astronomy C. Boisson & M. Servillat LUTh, Observatoire de Paris European Data Provider Forum, Heidelberg

332 views • 32 slides
PASS PASS Provenance-Aware Storage System Provenance-Aware Storage System Margo Seltzer, David

PASS PASS Provenance-Aware Storage System Provenance-Aware Storage System Margo Seltzer, David

PASS PASS Provenance-Aware Storage System Provenance-Aware Storage System Margo Seltzer, David Holland, Kiran-Kumar Muniswamy-Reddy, Uri Braun, and Jonathan Ledlie Harvard University What is Provenance? What is Provenance? What did the

336 views • 12 slides
Provenance in Dynamic Linked Data Marcin Wylot Linking Everything: Dynamic Graphs

Provenance in Dynamic Linked Data Marcin Wylot Linking Everything: Dynamic Graphs

Provenance in Dynamic Linked Data Marcin Wylot Linking Everything: Dynamic Graphs Integrated and summarized uncertain graph data Dynamic physical and logical network of things Necessity to established transparency 2 Data

850 views • 17 slides
Numerical Processing and Basic Data Visualization 01204111 Computers and Programmin ing Cha

Numerical Processing and Basic Data Visualization 01204111 Computers and Programmin ing Cha

Numerical Processing and Basic Data Visualization 01204111 Computers and Programmin ing Cha haip iporn Jaik Jaikaeo De Department of of Com omputer Eng ngineerin ing Kas asetsart rt Uni nivers rsity Cliparts are taken from

1.18k views • 62 slides
A new estimator for quantile-oriented sensitivity indices Thomas Browne Supervisors : J-C. Fort

A new estimator for quantile-oriented sensitivity indices Thomas Browne Supervisors : J-C. Fort

A new estimator for quantile-oriented sensitivity indices Thomas Browne Supervisors : J-C. Fort (Paris 5) & T. Klein (IMT-Toulouse) Advisors : B. Iooss & L. Le Gratiet (EDF R&D-MRI Chatou) EDF R&D-MRI Chatou - Universit Paris

518 views • 21 slides
Block I Connections between multivariate and FDA Beatriz Bueno-Larraz Real Eyes Universidad

Block I Connections between multivariate and FDA Beatriz Bueno-Larraz Real Eyes Universidad

Block I Connections between multivariate and FDA Beatriz Bueno-Larraz Real Eyes Universidad Aut onoma de Madrid IWAFDA 2019 It turns out, in my opinion, that reproducing kernel Hilbert spaces are the natural setting in which to solve

959 views • 57 slides
Hyperon puzzle and RMF models with scaled hadron masses and coupling constants Evgeni E.

Hyperon puzzle and RMF models with scaled hadron masses and coupling constants Evgeni E.

Hyperon puzzle and RMF models with scaled hadron masses and coupling constants Evgeni E. Kolomeitsev (University of Matej Bel, Slovakia) K.A. Maslov and D.N. Voskresensky (MEPhI, Moscow) Hyperon puzzle and constraints on the nuclear EoS

927 views • 41 slides
Celebration of Life Charles Avery November 10, 1930 - September 16, 2019 Club of Sioux City

Celebration of Life Charles Avery November 10, 1930 - September 16, 2019 Club of Sioux City

Celebration of Life Charles Avery November 10, 1930 - September 16, 2019 Club of Sioux City Lance Carson December 3, 1945 October 14, 2020 Club of Mitchell Richard Cutler August 26, 1941 May 8, 2019 Club of Downtown Sioux Falls

609 views • 27 slides
Detecting Abuse of Abandoned Internet Resources Tim Schmidt Betreuer: Dipl. Inf. Johann Schlamp,

Detecting Abuse of Abandoned Internet Resources Tim Schmidt Betreuer: Dipl. Inf. Johann Schlamp,

Chair for Network Architectures and Services Technische Universit at M unchen Detecting Abuse of Abandoned Internet Resources Tim Schmidt Betreuer: Dipl. Inf. Johann Schlamp, Dipl. Ing. Quirin Scheitle 12.08.2015 Chair for Network

553 views • 29 slides
The small ball inequality and binary nets Naomi Feldheim (Stanford University) j.w. Dmitriy Bilyk

The small ball inequality and binary nets Naomi Feldheim (Stanford University) j.w. Dmitriy Bilyk

The small ball inequality and binary nets Naomi Feldheim (Stanford University) j.w. Dmitriy Bilyk (University of Minnesota) IMU meeting Dead Sea, June 2016 Outline What is the Small Ball Inequality (SBI) Motivation Results: new

1.25k views • 95 slides
Resource Transfers in the RIPE NCC Service Region Marco Schmidt Policy Development O ffi cer

Resource Transfers in the RIPE NCC Service Region Marco Schmidt Policy Development O ffi cer

Resource Transfers in the RIPE NCC Service Region Marco Schmidt Policy Development O ffi cer LACNIC 24 | Bogota - Colombia | September 2015 RIPE Communitys Position 2 Arguments supporting transfers - Correct registration is the most

173 views • 16 slides

More recommend