Distributed DeepLearning at Scale Soumith Chintala Facebook AI - - PowerPoint PPT Presentation

▶
distributed deeplearning at scale
SMART_READER_LITE
LIVE PREVIEW

Distributed DeepLearning at Scale Soumith Chintala Facebook AI - - PowerPoint PPT Presentation

Aug 26, 2022 659 likes •1.18k views

Distributed DeepLearning at Scale Soumith Chintala Facebook AI Research Overview Deep Learning Research at FAIR Deep Learning on GPUs Deep Learning at scale Emerging Trends Deep Learning Research at Facebook AI Research Image

slide-1
SLIDE 1
slide-2
SLIDE 2

Distributed DeepLearning at Scale

Soumith Chintala

Facebook AI Research

slide-3
SLIDE 3
  • Deep Learning Research at FAIR
  • Deep Learning on GPUs
  • Deep Learning at scale
  • Emerging Trends

Overview

slide-4
SLIDE 4

Deep Learning Research at Facebook AI Research

slide-5
SLIDE 5

Image Intelligence: Classification

slide-6
SLIDE 6

Image Intelligence

Language Translation from Visual Learning

slide-7
SLIDE 7

Image Intelligence : Detection

slide-8
SLIDE 8

Image Intelligence : Detection

slide-9
SLIDE 9

Image Intelligence : Detection

slide-10
SLIDE 10

Image Intelligence : Detection

VGG# 1x1# conv# 2x2# pool# # x:#3x224x224# 512x14x14# 512x7x7# 512x1x1# 1024x1x1# fsegm(x):#224x224# fscore(x):#1x1# 512x14x14# 512x1x1#

56x56#

slide-11
SLIDE 11

Image Intelligence : Detection

image scores

slide-12
SLIDE 12

Image Intelligence : Detection

image scores image scores

slide-13
SLIDE 13

Image Intelligence : Detection

slide-14
SLIDE 14

Image Intelligence

https://code.facebook.com/posts/accessibility/

slide-15
SLIDE 15

Video Intelligence

slide-16
SLIDE 16

Image and Video Generation

Predicting the Future

slide-17
SLIDE 17
  • Memory networks
  • Language Translation
  • Reading, Writing and answering Questions

Natural Language Understanding

chatbots, personal assistants

slide-18
SLIDE 18

Deep Learning at Scale

slide-19
SLIDE 19

Deep Learning at Scale

GPU-powered Convolution Neural Networks

slide-20
SLIDE 20

Deep Learning at Scale

GPU-powered Convolution Neural Networks

slide-21
SLIDE 21

Deep Learning at Scale

GPU-powered Convolution Neural Networks

Alex Khrizevsky

slide-22
SLIDE 22

Deep Learning at Scale

GPU-powered Convolution Neural Networks

Alex Khrizevsky

slide-23
SLIDE 23
  • Convolutions, GEMM take all the time
  • Faster Convolutions = faster research

Deep Learning at Scale

GPU-powered Convolution Neural Networks

slide-24
SLIDE 24

Deep Learning at Scale

GPU-powered Convolution Neural Networks

slide-25
SLIDE 25

Deep Learning at Scale

GPU-powered Convolution Neural Networks

Winograd transform based Convolutions

slide-26
SLIDE 26
  • The standard in deep learning:

Deep Learning at Scale

GPU-powered Convolution Neural Networks

NVIDIA GPUs + CUDA + CuDNN

slide-27
SLIDE 27
  • Exotic new hardware!
  • Custom chips (Yunji Chen et. al., Nervana Systems)

Deep Learning at Scale

GPU-powered Convolution Neural Networks

slide-28
SLIDE 28
  • Use multiple GPUs on single machine

Deep Learning at Scale

Multi-GPU Training

slide-29
SLIDE 29
  • Data parallel

Deep Learning at Scale

Multi-GPU Training

slide-30
SLIDE 30
  • Model parallel

Deep Learning at Scale

Multi-GPU Training

slide-31
SLIDE 31
  • Pipeline-parallel

Deep Learning at Scale

Multi-GPU Training

slide-32
SLIDE 32

Bottleneck: interconnects

Deep Learning at Scale

Multi-GPU Training

slide-33
SLIDE 33
  • Multi-machine SGD

Deep Learning at Scale

Multi-Machine Training

Send gradients

slide-34
SLIDE 34
  • Multi-machine SGD

Deep Learning at Scale

Multi-Machine Training

Send Weights

slide-35
SLIDE 35
  • Elastic Averaging SGD! (Sixin Zhang, Anna Choromanska, Yann LeCun)

Deep Learning at Scale

Multi-Machine Training

slide-36
SLIDE 36
  • Elastic Averaging SGD!

Deep Learning at Scale

Multi-Machine Training

Train synchronously Occasionally, check with master Dont go too far from everyone else

slide-37
SLIDE 37
  • Elastic Averaging SGD!

Deep Learning at Scale

Multi-Machine Training

Train synchronously Occasionally, check with neighbors Dont go too far from everyone else

slide-38
SLIDE 38
  • Elastic Averaging SGD!
  • Empirical speedup of SquareRoot(N)
  • N = number of nodes
  • No communication overhead with pre-fetching
  • 128 GPUs (32 clients * 4 GPUs)
  • Sharded parameters over 64 CPU servers
  • Tau = 10, prefetch = 5
  • zero overhead

Deep Learning at Scale

Multi-Machine Training

slide-39
SLIDE 39
  • Elastic Averaging SGD!
  • Fun fact: Trained AlexNet in 5 epochs of Imagenet data
  • Good success in training Vision and Text networks

Deep Learning at Scale

Multi-Machine Training

slide-40
SLIDE 40

Big Sur

Open Compute for Deep Learning

  • Serviceability
  • Thermal Efficiency
  • Performance
slide-41
SLIDE 41

Big Sur

Open Compute for Deep Learning

Swap PCI-e Topologies with incredible ease

Rails for in-rack servicing 2.5” drive carriers Hot swappable fan modules GPU removal using 2 thumb screws Removable motherboard tray Cables to change topologies Removable GPU baseboard

slide-42
SLIDE 42

Big Sur

PCI-e Topologies — Matter!

slide-43
SLIDE 43

Big Sur

PCI-e Topologies — Matter!

slide-44
SLIDE 44

Torch

slide-45
SLIDE 45

Emerging Trends

slide-46
SLIDE 46
  • Data / Model / Pipeline parallel seems sufficient
  • Torch (nn / autograd / distlearn)
  • Caffe

Emerging Trends

Efficient Collectives + Imperative Programs

slide-47
SLIDE 47
  • Intel CnC, Caffe, TensorFlow, MXNet, Theano
  • Graph placement hints + execution
  • DSLs to write the computation graphs

Emerging Trends

Computational Graph Toolkits

slide-48
SLIDE 48
  • Best of both worlds
  • Hard problem of automatic graph placement
  • Limited heuristic-driven success

Silver Bullet

Imperative Language + Graph Compiler

slide-49
SLIDE 49
  • Big Sur Hardware
  • Kevin Lee kevinlee@fb.com
  • Doug Wimer dwimer@fb.com
  • Soumith Chintala soumith@fb.com
  • Multi-GPU / Multi-machine Training
  • Nicolas Vasilache ntv@fb.com
  • Jeff Johnson jhj@fb.com
  • Soumith Chintala soumith@fb.com
  • Computation Graphs, Automatic Placement
  • Jeff Johnson jhj@fb.com
  • Andrew Tulloch tulloch@fb.com
  • Yangqing Jia jiayq@fb.com
  • Soumith Chintala soumith@fb.com

Presence at GTC 2016

If you want to chat in-person, drop us an email

slide-50
SLIDE 50