THE 3-R'S OF DATA- THE 3-R'S OF DATA- SCIENCE: SCIENCE: - - PowerPoint PPT Presentation
06/05/2019 reveal.js The HTML Presentation Framework THE 3-R'S OF DATA- THE 3-R'S OF DATA- SCIENCE: SCIENCE: REPEATABILITY REPEATABILITY, , REPRODUCIBILITY REPRODUCIBILITY & & REPLICABILITY REPLICABILITY By Suneeta Mall
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 1/93
By Suneeta Mall
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 2/93
Industry adaptation Importance of 3-Rs Peek into Reproducibility crisis Define 3-Rs: repeatability, reproducibility & replicability Down the memory lane of confused terminology Techniques to ensure Repeatability & Reproducibility In depth review of few of the promising tools Techniques to ensure Replicability Few examples One last point!
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 3/93
“58% of respondents indicated that they were seriously building data science based solutions, with only 14% indicating no involvement just yet.
Evolving Data Infrastructure - Ben Lorica and Paco Nathan (O’Reilly, Oct 2018)
”
https://www.kdnuggets.com/2017/05/machine- learning-overtaking-big-data.html
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 4/93
“As per a survey in UK, 84%
companies preferred to build/use their own models.
David Kelnar, MMC Ventures, 2016 ” https://www.kdnuggets.com/2017/05/machine- learning-overtaking-big-data.html
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 5/93
https://xkcd.com/1838/
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 6/93
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 7/93
Know what... why ... & how ...
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 8/93
Know what... why ... & how ... recreate it ... & solve it.
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 9/93
We are continually faced by great opportunities brilliantly disguised as insoluble problems. Lee Iacocca The opportunities here are building:
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 10/93
We are continually faced by great opportunities brilliantly disguised as insoluble problems. Lee Iacocca The opportunities here are building: Reliable, & robust predictive solution
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 11/93
We are continually faced by great opportunities brilliantly disguised as insoluble problems. Lee Iacocca The opportunities here are building: Reliable, & robust predictive solution
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 12/93
“ Non-reproducible single occurrences are of no significance to science.
Popper (The logic of Scientific Discovery) ”
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 13/93
“ Non-reproducible single occurrences are of no significance to science.
Popper (The logic of Scientific Discovery) ”
Yet 70% of researchers have failed to reproduce another scientist's experiments
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 14/93
“ Non-reproducible single occurrences are of no significance to science.
Popper (The logic of Scientific Discovery) ”
Yet 70% of researchers have failed to reproduce another scientist's experiments , and > 50% have failed to reproduce their own experiments
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 15/93
“ Non-reproducible single occurrences are of no significance to science.
Popper (The logic of Scientific Discovery) ”
Yet 70% of researchers have failed to reproduce another scientist's experiments , and > 50% have failed to reproduce their own experiments
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 16/93
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 17/93
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 18/93
A reproducibility crisis
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 19/93
International Conference on Learning Representations Annual reproducibility challenge (since 2018)
lead by Dr. Joelle Pineau, an Associate Professor at McGill University and lead for Facebook’s Artificial Intelligence Research lab (FAIR)
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 20/93
Thats research and reproducibility crisis is very real! But why are we talking about it?
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 21/93
Thats research and reproducibility crisis is very real! But why are we talking about it? Industry Adaptation is making data-science accessible to people
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 22/93
Thats research and reproducibility crisis is very real! But why are we talking about it? Industry Adaptation is making data-science accessible to people Thus changing our community, and society
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 23/93
Thats research and reproducibility crisis is very real! But why are we talking about it? Industry Adaptation is making data-science accessible to people Thus changing our community, and society We have moral and social obligation to provide confident and reliable answers!
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 24/93
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 25/93
is the closeness of the agreement between the results of successive attempt of the same experiment/process carried out under the same conditions.
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 26/93
is the closeness of the agreement between the results of successive attempt of the same experiment/process carried out under the same conditions. e.g. replay, repeat
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 27/93
A simple linear regression example on Scikit diabetes dataset
diabetes = datasets.load_diabetes() test_size=0.33, random_state=None) regr = linear_model.LinearRegression() import matplotlib.pyplot as plt 1 import numpy as np 2 from sklearn import datasets, linear_model 3 from sklearn.metrics import mean_squared_error, r2_score 4 from sklearn.model_selection import train_test_split 5 6 7 diabetes_X = diabetes.data[:, np.newaxis, 9] 8 xtrain, xtest, ytrain, ytest = train_test_split( 9 diabetes_X, diabetes.target, 10 11 12 13 regr.fit(xtrain, ytrain) 14 diabetes_y_pred = regr.predict(xtest) 15
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 28/93
Run 1 Run 2
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 29/93
Linear regression example on Scikit diabetes dataset with fixed seed
diabetes = datasets.load_diabetes() test_size=0.33, random_state=32) regr = linear_model.LinearRegression() import matplotlib.pyplot as plt 1 import numpy as np 2 from sklearn import datasets, linear_model 3 from sklearn.metrics import mean_squared_error, r2_score 4 from sklearn.model_selection import train_test_split 5 6 7 diabetes_X = diabetes.data[:, np.newaxis, 9] 8 xtrain, xtest, ytrain, ytest = train_test_split( 9 diabetes_X, diabetes.target, 10 11 12 13 regr.fit(xtrain, ytrain) 14 diabetes_y_pred = regr.predict(xtest) 15
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 30/93
Repeat 1 Repeat 2
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 31/93
is the closeness of the agreement between the results of successive attempt of the same experiment/process carried out under different conditions.
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 32/93
is the closeness of the agreement between the results of successive attempt of the same experiment/process carried out under different conditions. e.g. repeat*
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 33/93
Linear regression example on Scikit diabetes dataset with different configuration
diabetes = datasets.load_diabetes() test_size=0.33, random_state=32) regr = linear_model.LinearRegression(normalize=False) import matplotlib.pyplot as plt 1 import numpy as np 2 from sklearn import datasets, linear_model 3 from sklearn.metrics import mean_squared_error, r2_score 4 from sklearn.model_selection import train_test_split 5 6 7 diabetes_X = diabetes.data[:, np.newaxis, 9] 8 xtrain, xtest, ytrain, ytest = train_test_split( 9 diabetes_X, diabetes.target, 10 11 12 13 regr.fit(xtrain, ytrain) 14 diabetes_y_pred = regr.predict(xtest) 15
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 34/93
Model fixed seed Model fixed seed with different configuration
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 35/93
is the closeness of the agreement between the results of original experiment/process to that of independent experiment/process conducted to simuate/replicate original experiement with at least similar data-set, algorithms, and conditions.
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 36/93
is the closeness of the agreement between the results of original experiment/process to that of independent experiment/process conducted to simuate/replicate original experiement with at least similar data-set, algorithms, and conditions. e.g. duplicate, copy
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 37/93
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 38/93
Stephen hawking
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 39/93
Stephen hawking Stephen hawking in Space
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 40/93
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 41/93
Plesser HE. Reproducibility vs. Replicability: A Brief History of a Confused Terminology. Front Neuroinform. 2018;11:76. Published 2018 Jan 18. doi:10.3389/fninf.2017.00076
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 42/93
Claerbout, J. F., and Karrenbach, M. (1992). Electronic documents give reproducible research a new meaning. SEG Expanded Abstracts 11, 601–604. doi: 10.1190/1.1822162
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 43/93
Drummond, C. (2009). “Replicability is not reproducibility: nor is it good science,” in Proceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML (Montreal, QC). Available online at: http://www.site.uottawa.ca/~ cdrummon/pubs/ICMLws09.pdf
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 44/93
Association for Computing Machinery, 2016, defined:
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 45/93
Association for Computing Machinery, 2016, defined:
Repeatability:same results under same conditions Replicability:same results under slightly different conditions Reproducibility:same results under very different conditions (independent simulation)
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 46/93
Association for Computing Machinery, 2016, defined:
Patil, P., Peng, R. D., and Leek, J. T. (2016). A statistical definition for reproducibility and
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 47/93
Association for Computing Machinery, 2016, defined:
Goodman, S. N., Fanelli, D., and Ioannidis, J. P. A. (2016). What does research reproducibility mean? Sci. Transl. Med. 8:341ps12. doi: 10.1126/scitranslmed.aaf5027
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 48/93
Methods reproducibility: provide enough detail so the procedures could be exactly repeated Results reproducibility: obtain similar results from an independent study with same procedures as original experiment Inferential reproducibility: draw the same conclusions from either an independent replication or a reanalysis of the original experiment
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 49/93
https://xkcd.com/1673
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 50/93
Repeat Reproduce Replicate Robust Reliable
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 51/93
Repeat Reproduce Replicate Robust Reliable
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 52/93
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 53/93
RFC 1149.5: https://xkcd.com/221
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 54/93
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 55/93
PRINCIPLE PRINCIPLE Avoid adhoc changes Reproducible randomness PROCESS PROCESS
Auditing Automate
Tracing Provenance
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 56/93
P1 P1AUTOMATE EVERYTHING
Introduce DAG pipelines that are
https://xkcd.com/242
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 57/93
P1 P1AUTOMATE EVERYTHING
Introduce DAG pipelines that are
https://xkcd.com/242
Simple, Modular,
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 58/93
P1 P1AUTOMATE EVERYTHING
Introduce DAG pipelines that are
https://xkcd.com/242
Simple, Modular, Pluggable, Scalable, & Reliable
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 59/93
Standard ML DAG flow
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 60/93
Mlflow Airflow Kubeflow Azkaban Luigi Spark Co
Data Science Pipeline AI Nextflow Snakemake DVC Delta
Lake Pachyderm Argo Digdag Bpipe
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 61/93
P2 P2FULL PROVENANCE
Maintain version control over everything - aka
https://xkcd.com/1597/
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 62/93
P2 P2FULL PROVENANCE
Maintain version control over everything - aka
https://xkcd.com/1597/
Lineage,
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 63/93
P2 P2FULL PROVENANCE
Maintain version control over everything - aka
https://xkcd.com/1597/
Lineage, Time Travel, Audit
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 64/93
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 65/93
P3 P3MODEL MANAGEMENT - AUDITING,
Manage gamut of models with: auditing, tracing & serving
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 66/93
P3 P3MODEL MANAGEMENT - AUDITING,
Manage gamut of models with: auditing, tracing & serving
Version Control Pachyderm* Mlflow Kubeflow C Pipeline AI
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 67/93
Code in Git, data and meta in object store
Version control: Model, data and config Git alike: pull, push etc. Maintains metadata in .dvc Analogous to GitOPS principles
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 68/93
DVC
Examples
dvc run -d data/feature.p \
python code/train_model.p # Step 1 1 dvc run -d code/features.py \ 2
3
4 python code/features.py 5 6 # Step 2 7 8 9
10 11
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 69/93
PROS PROS Not very dictative Independent tools, integrates easily Single source of truth with GitOPS CONS CONS DAG declaration not intutive Too much onus on user to integrate with system Relatively new - still evolving
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 70/93
Git repository for data
Data is first class citizen Like DVC automatically manages versions Containerized and platform agnostic
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 71/93
Data repos DAG Example of workflow
Pipeline Specification
"repo": "training_dat "repo": "parameters", "name": "model_train" { "input": { "union": [ { 1 "pfs": { 2 "glob": "/", 3 4 } }, { 5 "pfs": { 6 "glob": "/*.json", 7 8 } } ] }, 9 "pipeline": { 10 11 }, 12 "transform": { 13 "image": "ubuntu", 14 "cmd": ["/bin/bash"], 15
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 72/93
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 73/93
PROS PROS Treats data as first class entity Manages versioning automatically Simple pipeline constructs Scalable & Distributed Capability of long running notebooks CONS CONS Is not capable of model serving Does not simplify model tracing Still rapidly evolving
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 74/93
Provides infrastructure agnostic ML toolkit Integrates with relevant sowares for seamless, scalable processing Sources off data from
Kubernetes specific
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 75/93
ML Toolkit for Kubernetes
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 76/93
Kubeflow pipeline eg
Pipeline Specification
def PreprocessOp(...): def TrainOp(...): @dsl.pipeline( def resnet_pipeline(....): import kfp.dsl as dsl 1 2 3 return dsl.ContainerOp(..) 4 5 6 return dsl.ContainerOp(...) 7 8 9 name='resnet_cifar10_pipeline 10 ) 11 12 persistent_volume_name = '/' 13 persistent_volume_path = '/pa 14
15
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 77/93
PROS PROS Leverages kubernetes, scalable, distributed for ML compute Nicely integrates with likes of Pachyderm, Katib, ModelDB, Seldon GitOPS can be realize with ARGO CONS CONS Does not provide unified provenance Requires DSL based pipeline spec Installation may feel cumbersome Still evolving and integrating with tools
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 78/93
Promising offering for Hadoop based workloads Leverages spark for data-pipelining Manages provenance under time-travel feature Model manament/tracing via mlflow Model persistence via mleap
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 79/93
MLFlow traces
Spark Pipeline Spec
training = spark.createDataFrame(. hashingTF = HashingTF(...) pipeline = Pipeline(stages= prediction = model.transform(test) from pyspark.ml import Pipeline 1 2 3 tokenizer = Tokenizer(inputCol="tex 4
5 6 lr = LogisticRegression(maxIter=10 7 regParam=0.001) 8 9 [tokenizer, hashingTF, lr] 10 model = pipeline.fit(training) 11 12
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 80/93
Only recently launched under Apache 2: 0.1.0 24th April 2019! Proprietary version is released a year ago and in use by many organization. Very little documentation - so not enough is known Claims to have solved the provenance part right! Limited to Hadoop/Spark stack
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 81/93
Platform agnostic solution to manage end to end lifecycle of deep leaning workload Partly open source with enterprise option Provides reproducibility and model tracing Integrates with Seldon Data provenance is not managed
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 82/93
Polyaxon experiment
Polyaxon Spec
image: tensorflow/tensorflow:1.4.1 cmd: python model.py version: 1 1 2 kind: experiment 3 4 build: 5 6 build_steps: 7
8 9 run: 10 11 polyaxon run -f polyaxonfile.yaml 1
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 83/93
Throwing in 2 more Rs: Reliability and Robustness
Dataset represents right demographics Robustness through data augumentation Knowing decision boundaries of model Model testing: against adverserial attacks http://www.cleverhans.io - Blog by Goodfellow & Papernot
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 84/93
“Not make confident mistakes
(Unrestricted Adversarial Examples - Goodfellow et al.) ”
“Models generalization across architectures and training sets (Explaining and harnessing adversarial examples - Szegedy et al.) ”
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 85/93
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 86/93
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 87/93
Examples generated by .
fmodel = KerasModel(kmodel, bounds=(0, 255), preprocessing=(np.array([104, 116, 123] attack = FGSM(fmodel, criterion=Misclassification()) adversarial = attack(img, label) import foolbox 1 import numpy as np 2 from foolbox.criteria import Misclassification 3 from foolbox.models import KerasModel 4 from foolbox.attacks import FGSM 5 6 # model = <your model=""> 7 8 9 10 11 # img, label = <your image="">, <true label=""> 12 13 14 </true></your></you 15
foolbox
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 88/93
Persian Cat
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 89/93
“The price of greatness is responsibility.Winston Churchill”
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 90/93
“The price of greatness is responsibility.Winston Churchill” Do data-science responsibly!
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 91/93
Soware Link Capability DVC https://iterative.ai/ DAG, Provenance Pachyderm http://pachyderm.io DAG, Provenance Kubeflow https://github.com/kubeflow/kubeflow DAG, Model Management Delta Lake https://delta.io/ DAG, Provenance, Model Management Pipeline AI https://github.com/PipelineAI/ DAG, Model Management Polyaxon https://github.com/polyaxon/polyaxon DAG, Model Management Airflow http://airflow.apache.org/ DAG, Limited provenance via Atlas DB
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 92/93
Soware Link Capability Luigi https://github.com/spotify/luigi DAG Nextflow https://www.nextflow.io/ DAG Snakemake https://snakemake.readthedocs.io DAG Bpipe http://docs.bpipe.org/ DAG digdag https://github.com/treasure-data/digdag/ DAG Mleap https://github.com/combust/mleap Model Persistence Mlflow https://github.com/mlflow/mlflow/ Model Management Seldon https://www.seldon.io/ Model Management Modeldb https://github.com/mitdbg/modeldb Model Management Katib https://github.com/kubeflow/katib Hyper parameter tunning
06/05/2019 reveal.js – The HTML Presentation Framework localhost:8000/3Rs.html?print-pdf#/intro 93/93
QUESTIONS?? QUESTIONS??