Improving Reproducible Deep Learning Workflows with DeepDIVA M. - - PowerPoint PPT Presentation

Improving Reproducible Deep Learning Workflows with DeepDIVA

• M. Alberti1*, V. Pondenkandath1*, L. Vögtlin1, M. Würsch12, R. Ingold1, M. Liwicki13

*Equal contribution

1DIVA Group, University of Fribourg, Switzerland 2IIT, FHNW University of Applied Sciences and Arts Northwestern Switzerland, Switzerland 3EISLAB Machine Learning, Luleå University of Technology, Sweden

Reproducibility Crisis: Trust or Verify?

2

Joelle Pineau, “Reproducible, Reusable, and Robust Reinforcement Learning”, invited talk @NeurIPS 2018, Montreal, Canada

No possibility to verify No possibility to extend Lots of overhead created Leads to no trust in scientific results

Why Is This a Problem?

3

Ensure reproducibility

Of your own experiments Of other people’s experiments

Promote open-source code

Make it easy to have “good enough” code Enable code trustworthiness

How To Make Steps Forward?

4

Open-Source Python framework Built on top of PyTorch Makes your life easer for:

Reproducing your own and other people’s experiments

Provides boilerplate code for:

Common deep learning scenarios Handling time consuming everyday problems

Documentation & Tutorial available

How We Contribute: DeepDIVA

5

Reproducing Your Own Experiments

Short-term, or work in progress Long-term, or finished work

6

Kilometres of poor or incomplete log files Stochasticity in the process

Short-term Reproducibility Dangers

7

Meaningful logging

Saving all run parameters and command line args Providing concise coloured logs

Deterministic runs

Seeding the pseudo-random numbers generators: Python, Numpy and PyTorch. Disabling CuDNN (NVIDIA Deep Neural Network library) when necessary

How DeepDIVA Ensures Short-term Reproducibility

8

Poor (or non-existent!) use of version control Hard-to-die bad programming habits Silent data modifications

Long-term Reproducibility Dangers

9

Git status

Linking every run to a specific commit in Git Allowing this feature to be disabled for dev purposes

Copy code

Copying the entire running code in the output folder

Data Integrity Management

Footprint of the data in a JSON file using SHA-1 hashes

How DeepDIVA Ensures Long-term Reproducibility

10

Reproducing Other People’s Experiments

Given a paper, try to replicate the results and observations

11

In order to reproduce an experiment one needs:

Git repository URL Git commit identifier (full SHA) List of command line arguments used The data

Reproducing Other People’s Experiments

12

Productivity Out-of

• f-the-box

Making your life easier: do not reinvent the wheel!

13

“One click away” Deep Learning Scenarios

14

“when the data is ready the task is solved” Download a dataset with a click

Natural images, medical images, historical documents, …

Split your dataset

Train, Validation and Test splits

Analyse the data

Mean/std and class distributions

Ensure data integrity

Compare the footprints

Prepare Your Data

15

Real-time Visualizations

16

Tensorboard (from TensorFlow) Confusion Matrix Features Visualization Weight Histograms Performance Evaluation

Let machine learning find the best values No expensive grid or random search

Automatic Hyper-Parameter Optimization

17

Be A Part Of f It It

Getting Started With DeepDIVA

18

No Setup Time

From source on Ubuntu (or other flavours of Linux) Docker Image Coming Soon

Documentation

Online and in the code

Tutorials

Learn new features efficiently

Fork It

Extensive and modular for easy modifications

How To Use It

19

Make Your Experiment Reproducible

bit.ly/DeepDIVA

20