Distributed Deep Learning: Methods and Resources Sergey Nikolenko - - PowerPoint PPT Presentation

Distributed Deep Learning: Methods and Resources

Sergey Nikolenko

Chief Research Officer, Neuromation Researcher, Steklov Institute of Mathematics at St. Petersburg September 23, 2017, AI Ukraine, Kharkiv

Maxim Prasolov

CEO, Neuromation

Outline

• Bird’s eye overview of deep learning
• SGD and how to parallelize it
• Data parallelism and model parallelism
• Neuromation: developing a worldwide marketplace

for knowledge mining

• 10 years ago machine learning underwent a deep learning revolution
• Neural networks are one of the oldest techniques in ML
• But since 2007-2008, we can train large and deep neural networks

(in part due to distributed computations)

• And now deep NNs yield state of the art results in many fields

What is a deep neural network

• A deep neural network is a huge

composition of simple functions implemented by artificial neurons

• Usually linear combination

followed by nonlinearity, but can be anything as long as you can take derivatives

• These functions are combined into a

computational graph that computes the loss function for the model

Backpropagation

• To train the model (learn the weights),

you take the gradient of the loss function w.r.t. weights

• Gradients can be efficiently computed

with backpropagation

• And then you can do (stochastic)

gradient descent and all of its wonderful modifications, from Nesterov momentum to Adam

FEEDFORWARD NETWORKS CONVOLUTIONAL NETWORKS RECURRENT NETWORKS

Gradient descent is used for all kinds of neural networks

Distributed Deep Learning: The Problem

• One component of the DL revolution

was the use of GPUs

• GPUs are highly parallel (hundreds of cores)

and optimized for matrix computations

• Which is perfect for backprop (and fprop too)
• But what if your model does not fit on a GPU?
• Or what if you have multiple GPUs?
• Can we parallelize further?

What Can Be Parallel

• Model parallelism vs. data parallelism
• We will discuss both
• Data parallelism is much more common
• And you can unite the two:

[pictures from (Black, Kokorin, 2016)]

Examples of data parallelism

• Make every worker do its thing and

then average the results

• Parameter averaging: average w from all workers

○ but how often? ○ and what do we do with advanced SGD variants?

• Asynchronous SGD: average updates from workers

○ much more interesting without synchronization ○ but the stale gradient problem

• Hogwild (2011): very simple asynchronous SGD,

just read and write to shared memory, lock-free; whatever happens, happens

Examples of data parallelism

• FireCaffe:

○ DP on a GPU cluster ○ communication through reduction trees

Model parallelism

• In model parallelism, different weights are distributed
• Pictures from the DistBelief paper (Dean et al., 2012)
• Difference in communication:

○ DP: workers exchange weight updates ○ MP: workers exchange data updates

• DP in DistBelief: Downpour SGD
• vs. Sandblaster L-BFGS
• Now, DistBelief has been

completely replaced by...

Distributed Learning in TensorFlow

• TensorFlow has both DP (right) and MP (bottom)
• Workers and parameter servers
• MP usually works as a pipeline between layers:

First specify the structure of the cluster: Then assign (parts of) computational graph to workers and weights to parameter servers:

Example of Data Parallelism in TensorFlow

Interesting variations

• (Zhang et al., 2016) – staleness-aware SGD: add weights depending on

the time (staleness) to updates

• Elephas: distributed Keras that runs on Spark
• (Xie et al., 2015) – sufficient factor broadcasting:

represent and send only u and v

• (Zhang et al., 2017) – Poseidon: a new architecture with

wait-free backprop and hybrid communication

• Special mention: reinforcement learning; async RL is great!
• And standard (by now) DQN tricks are perfect for parallelization:

○ experience replay: store experience in replay memory and serve them for learning ○ target Q-network is separate from the Q-network which is learning now, updates are rare

Distributed reinforcement learning

Gorila from DeepMind: everything is parallel and asynchronous

Recap

• Data parallelism lets you process lots of data in parallel, copying the model
• Model parallelism lets you break down a large model into parts
• Distributed architectures are usually based on parameter servers and workers
• Especially in reinforcement learning, where distributed architectures rule
• And this all works out of the box in TensorFlow and other modern frameworks
• But how is it relevant to us? Isn’t that for the likes of Google and/or DeepMind?
• Where do we get the computational power and why do we need so much data?

Distributed deep learning works

BITCOIN OR ETHER MINING AMAZON DEEP LEARNING

$7-8 USD per DAY $3-4 USD HOUR

Chris

NOT ENOUGH LABELED DATA FOR NEURAL NETWORK TRAINING

“BOTTLENECK” OF AUTOMATION OF EVERY INDUSTRY:

BITCOIN OR ETHER MINING AMAZON DEEP LEARNING

$7-8 USD per DAY $3-4 USD HOUR

IMAGE RECOGNITION IN RETAIL

BITCOIN OR ETHER MINING AMAZON DEEP LEARNING

$7-8 USD per DAY $3-4 USD HOUR

170.000 OBJECTS

ABOUT 40 BLN IMAGES PER YEAR

TO AUTOMATE THE RETAIL INDUSTRY MUST BE RECOGNIZED ON THE SHELVES

IMAGE RECOGNITION IN RETAIL BY ECR RESEARCH:

OSA HP CONTRACTED NEUROMATION TO PRODUCE LABELED DATA AND TO RECOGNIZE

30% OF THE COST IS COMPUTATIONAL POWER.

MORE THAN 7 MLN EURO REVENUE

BITCOIN OR ETHER MINING AMAZON DEEP LEARNING

$7-8 USD per DAY $3-4 USD HOUR

Chris LABELED PHOTOS ARE REQUIRED TO TRAIN IMAGE RECOGNITION MODELS

MORE THAN

1 BLN

WHERE CAN WE GET THIS HUGE AMOUNT OF LABELED DATA?

Chris

1 MAN = 8 HOURS x 50 IMAGES, x $0.2 PER IMAGE YEARS OF MECHANICAL WORK

1 BLN LABELED PHOTOS = $240 MLN

DATA LABELING

HAS BEEN MANUAL WORK TILL NOW

WE KNOW HOW TO GENERATE SYNTHETIC LABELED DATA FOR DEEP LEARNING

• Labeled data with 100% accuracy
• Automated data generation with no limits
• Cheaper and faster than manual labor

SYNTHETIC DATA:

A BREAKTHROUGH IN DEEP LEARNING BUT REQUIRES HUGE COMPUTATIONAL POWER TO RENDER DATA AND TRAIN NEURAL NETWORKS

BITCOIN OR ETHER MINING AMAZON DEEP LEARNING

$7-8 USD per DAY $3-4 USD HOUR

BITCOIN OR ETHER MINING AMAZON DEEP LEARNING

$7-8 per DAY $3-4 per HOUR

The AI industry is ready to pay miners for their computational resources more than they can ever get from mining Ether.

KNOWLEDGE MINING IS MORE PROFITABLE. DEEP LEARNING NEEDS YOUR COMPUTATION POWER!

WE CAN BRIDGE THIS GAP

GPU

x6

GPU

BITCOIN OR ETHER MINING AMAZON DEEP LEARNING

$7-8 USD per DAY $3-4 USD HOUR

BLOCKCHAIN + DEEP LEARNING

BITCOIN OR ETHER MINING AMAZON DEEP LEARNING

$7-8 USD per DAY $3-4 USD HOUR

NEUROMATION PLATFORM

TokenAI

will combine in one place all the components necessary to build deep learning solutions with synthetic data

THE UNIVERSAL MARKETPLACE OF NEURAL NETWORK DEVELOPMENT

BITCOIN OR ETHER MINING AMAZON DEEP LEARNING

$7-8 USD per DAY $3-4 USD HOUR

• the network must be geographically distributed and keep track of

massive amounts of transactions

• the payment method for completed work should be highly liquid and

politically independent

• network nodes have to understand the model of “mining” a resource for a bounty:

transparency is required to build trust

• transactions need to be transparently auditable to prevent fraud and mitigate dispute

NEUROMATION PLATFORM will be extending Etherium with TokenAI.

Blockchain is the only technology that can realistically accomplish this. Extending Ethereum instead of building our own blockchain is an obvious first step.

Neuromation needs to quickly deploy a massive network of computation nodes (converted from crypto miners).

BITCOIN OR ETHER MINING AMAZON DEEP LEARNING

$7-8 USD per DAY $3-4 USD HOUR

DEEP LEARNING RESEARCH GRANTS

• for R&D teams and start-up’s of DL/ML

industry

• In cooperation with frontier institutions

WE ARE OPEN FOR COOPERATION

partnerships@neuromation.io

• 1000 GPUs pool (+100,000GPU are

coming)

VAST APPLICATIONS OF SYNTHETIC DATA NEUROMATION LABS:

RETAIL AUTOMATION LAB PHARMA AND BIOTECH LAB synthetic data for: ENTERPRISE AUTOMATION LAB synthetic data for:

• +170 000+ items for the

Eastern European Retail Market only

• about 50 euro per object
• contract for >7mln euro in

revenue

• medical imaging (classify

tumors and melanomas)

• health applications (smart

cameras)

• training flying drones,

self-driving cars, and industrial robots in virtual environments

• manufacturing and

supply-chain solutions

(live)

OUR TEAM:

Maxim Prasolov CEO Fedor Savchenko CTO Sergey Nikolenko Chief Research Officer Denis Popov VP of Engineering Constantine Goltsev Investor / Chairman Andrew Rabinovich Adviser Yuri Kundin ICO Compliance Adviser Aleksey Spizhevoi Researcher Esther Katz VP Communication Kiryl Truskovskyi Lead Researcher

OCTOBER 15th, 2017 Presale of TOKENAI starts NOVEMBER, 2017 Public sale starts UNKNOWN DATE Secret cap is reached, and token sale ends in 7 days Jan 1st, 2018 Token sale ends (if secret cap is not reached)

ICO ROADMAP

• Tokens Minted: 1,000,000,000
• Issued in ICO: Up to 700,000,000
• Reserve: from 300,000,000
• Price per Token: 0.001 ETH

ICO DETAILS

Jurisdiction: Estonia, EU

* Neuromation is fully compliant with Estonian crowdfunding legislation

KNOWLEDGE MINING - A NEW ERA OF DISTRIBUTED COMPUTING

THANK YOU!

neuromation.io