SLIDE 1
TensorFlow Flexible, Scalable, Portable Rajat Monga Engineering - - PowerPoint PPT Presentation
TensorFlow Flexible, Scalable, Portable Rajat Monga Engineering - - PowerPoint PPT Presentation
TensorFlow Flexible, Scalable, Portable Rajat Monga Engineering Director, TensorFlow Released in Nov. 2015 #1 repository for machine learning category on GitHub Some Stats 10,000+ commits since Nov, 2015 450+ contributors 1M+
SLIDE 2
SLIDE 3
repository
for “machine learning” category on GitHub
#1
Released in Nov. 2015
SLIDE 4
10,000+ commits since Nov, 2015 450+ contributors 1M+ binary downloads #20 most popular repository on GitHub by stars Used in ML classes at many universities: Toronto, Berkeley, Stanford, ...
Some Stats
SLIDE 5
SLIDE 6
SLIDE 7
TensorFlow powered Cucumber Sorter
From: http://workpiles.com/2016/02/tensorflow-cnn-cucumber/
SLIDE 8
Flexible
SLIDE 9
SLIDE 10
MatMul Add Relu biases weights examples labels Xent
Directed graph
SLIDE 11
Async SGD with Parameter Server
Model Workers Parameter Server p ∆p p’ = p - λ *∆p
SLIDE 12
Async SGD with TensorFlow
Model Workers PS Workers p ∆p p’ = p - λ *∆p
SLIDE 13
Sync SGD with TensorFlow
Model Workers PS Workers p ∆p delta = ∑∆p p’ = p - λ * delta
SLIDE 14
Alternate version of Parameter sharing
Workers
SLIDE 15
More ML Algorithms
- Clustering e.g. k-means
- Random Forests
- Logistic Regression
- Bayesian methods
SLIDE 16
Scalable
SLIDE 17
Deferred Execution
Operations build the dataflow graph; eval() fetches the result. import tensorflow as tf with tf.Session(): x = tf.constant([[5, 6], [7, 8]]) z = tf.matmul(x, x) + tf.matmul(x, [[1, 0], [0, 1]]) # Run graph to fetch z. result = z.eval()
MatMul const x const MatMul +
SLIDE 18
Parallelism in Op implementations
MatMul ... ...
SLIDE 19
... ... ... ... ... MatMul MatMul
Task Parallelism in DataFlow graph
SLIDE 20
Data Parallelism
MatMul Input MatMul Param ... ... MatMul MatMul ... ...
SLIDE 21
Model Parallelism
MatMul Matrix Matrix MatMul Split Concat
SLIDE 22
GPU 0 CPU Add Mul biases learning rate −= ... ...
Distribution across Devices
SLIDE 23
GPU 0 CPU Add Mul biases learning rate −= ... ...
Distribution
- TensorFlow inserts Send/Recv Ops to transport tensors across devices
- Recv ops pull data from Send ops
Send Recv
SLIDE 24
GPU 0 CPU Add Mul biases learning rate −= ... ...
Distribution
- TensorFlow inserts Send/Recv Ops to transport tensors across devices
- Recv ops pull data from Send ops
Send Recv Send Recv Send Recv Send Recv
SLIDE 25
Scale across machines
SLIDE 26
Portable
SLIDE 27
Platforms
SLIDE 28
TensorFlow Core Execution Engine CPU GPU Android iOS ...
Device Abstraction
SLIDE 29
Languages
SLIDE 30
TensorFlow Core Execution Engine CPU GPU Android iOS ...
Language Abstraction
C++ Python ...
SLIDE 31
Now what?
SLIDE 32
SLIDE 33
AutoTrash
SLIDE 34
AutoTrash
SLIDE 35
Rajat Monga
@rajatmonga