[PPT] - The Fundamentals of Deep Learning Building Blocks Theory with PowerPoint Presentation

SLIDE 1

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

The Fundamentals of Deep Learning

with Applications Jon Krohn jon@untapt.com

Chief Data Scientist at untapt

New York Open Statistical Programming Meetup January 17th, 2017 (slides available at jonkrohn.com/talks)

SLIDE 2

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Outline

1

Antecedents Case Study: A History of Biological & Artificial Vision Building Blocks

2

Theory Biological & Artificial Neurons Neural Networks Deep Neural Networks

3

Contemporary Applications Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

SLIDE 3

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Outline

1

Antecedents Case Study: A History of Biological & Artificial Vision Building Blocks

2

Theory Biological & Artificial Neurons Neural Networks Deep Neural Networks

3

Contemporary Applications Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

SLIDE 4

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Outline

1

Antecedents Case Study: A History of Biological & Artificial Vision Building Blocks

2

Theory Biological & Artificial Neurons Neural Networks Deep Neural Networks

3

Contemporary Applications Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

SLIDE 5

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Outline

1

Antecedents Case Study: A History of Biological & Artificial Vision Building Blocks

2

Theory Biological & Artificial Neurons Neural Networks Deep Neural Networks

3

Contemporary Applications Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

SLIDE 6

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

SLIDE 7

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

SLIDE 8

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Hubel & Wiesel (1959)

SLIDE 9

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

SLIDE 10

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

SLIDE 11

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

SLIDE 12

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Camera Obscura

da Vinci (15th Century)

SLIDE 13

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Block World

Larry Roberts (1965)

SLIDE 14

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Viola & Jones (2001)

SLIDE 15

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Neurocognitron

Fukushima (1980)

SLIDE 16

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

MNIST Digits & LeNet-5

LeCun, Boutou, Bengio & Haffner (1998)

SLIDE 17

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

LeNet-5

LeCun, Boutou, Bengio & Haffner (1998)

SLIDE 18

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

SLIDE 19

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

ImageNet

Fei-Fei Li et al. (2009), 14m images, 22k categories

SLIDE 20

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

ImageNet Classification Error

ILSVRC: 1.4m, 1k object classes

SLIDE 21

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

AlexNet

Krizhevsky, Sutskever & Hinton (2012)

SLIDE 22

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

SLIDE 23

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Sunspring

Sharp & Goodwin (2016)

SLIDE 24

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Sunspring

Sharp & Goodwin (2016)

[video]

SLIDE 25

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Outline

1

Antecedents Case Study: A History of Biological & Artificial Vision Building Blocks

2

Theory Biological & Artificial Neurons Neural Networks Deep Neural Networks

3

Contemporary Applications Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

SLIDE 26

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Hardware

local machine
build your own server
AWS / Google Cloud Platform
GPU(s) / TPU(s)

SLIDE 27

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Popular Libraries

based on Johnson (2016) in Stanford CS231n l.12

Caffe Torch Theano TensorFlow Language C++, Python Lua Python Python Pretrained Yes++ Yes++ Yes (Lasagne) Inception Parallel GPUs: Data Yes Yes Yes Yes Parallel GPUs: Model No Yes Experimental Yes (best) Readable Source Code Yes (C++) Yes No No Good at RNN No Mediocre Yes Yes (best) Higher-Level APIs No No Keras Keras and TFLearn

SLIDE 28

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

[Human Learning Resources]

SLIDE 29

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Outline

1

Antecedents Case Study: A History of Biological & Artificial Vision Building Blocks

2

Theory Biological & Artificial Neurons Neural Networks Deep Neural Networks

3

Contemporary Applications Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

SLIDE 30

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Biological Neuron Morphology

SLIDE 31

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Perceptron

Rosenblatt (1957)

SLIDE 32

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Biological Neuron Physiology

The Binary Action Potential

SLIDE 33

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Perceptron

Rosenblatt (1957)

SLIDE 34

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Multi-Layer Perceptron

SLIDE 35

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Multi-Layer Perceptron

SLIDE 36

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Sigmoid Neuron

SLIDE 37

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

tanh Neuron

SLIDE 38

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

ReLU: Rectified Linear Units

Nair & Hinton (2010); Maas, Hannun & Ng (2014)

SLIDE 39

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Outline

1

Antecedents Case Study: A History of Biological & Artificial Vision Building Blocks

2

Theory Biological & Artificial Neurons Neural Networks Deep Neural Networks

3

Contemporary Applications Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

SLIDE 40

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

MNIST

LeCun, Cortes & Burges

SLIDE 41

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Fully-Connected Neural Net

Single Hidden Layer

SLIDE 42

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

TensorFlow Playground

[demo]

SLIDE 43

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Outline

1

Antecedents Case Study: A History of Biological & Artificial Vision Building Blocks

2

Theory Biological & Artificial Neurons Neural Networks Deep Neural Networks

3

Contemporary Applications Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

SLIDE 44

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Deep Fully-Connected Net

3 (or more) Hidden Layers

SLIDE 45

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

TFLearn

Caffe Torch Theano TensorFlow Language C++, Python Lua Python Python Pretrained Yes++ Yes++ Yes (Lasagne) Inception Parallel GPUs: Data Yes Yes Yes Yes Parallel GPUs: Model No Yes Experimental Yes (best) Readable Source Code Yes (C++) Yes No No Good at RNN No Mediocre Yes Yes (best) Higher-Level APIs No No Keras Keras and TFLearn

SLIDE 46

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

A Simple Deep Net in TFLearn

[notebook]

SLIDE 47

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Synaptic Pruning

SLIDE 48

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

(Stochastic) Gradient Descent

Adam = AdaGrad + RMSprop

SLIDE 49

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Backpropagation

computes error & gradient of cost function

SLIDE 50

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Overfitting

...and avoiding it

L1/L2 regularization
dropout
artificial data set expansion

SLIDE 51

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Improving Neural Networks

Mostly Hyperparameter Tuning

problem simplification
number and width of layers
cost fxn: quadratic, cross-entropy, log-likelihood, &c.
more epochs, early stopping
clever initialization of weights and biases
learning rate η, variable schedule
regularization parameter λ
mini-batch size
automation, e.g., with Spearmint

[Summary Blog Post]

SLIDE 52

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Universality

Solve Any Continuous Function (Nielsen, 2015)

SLIDE 53

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Unstable Gradient

Typically Vanishes (but can Explode)

SLIDE 54

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

SLIDE 55

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Classic Deep Architectures

...introducing Convolutional Layers

SLIDE 56

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Outline

1

Antecedents Case Study: A History of Biological & Artificial Vision Building Blocks

2

Theory Biological & Artificial Neurons Neural Networks Deep Neural Networks

3

Contemporary Applications Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

SLIDE 57

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Hubel & Wiesel (1959)

SLIDE 58

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

SLIDE 59

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

SLIDE 60

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

AlexNet

Krizhevsky, Sutskever & Hinton (2012)

SLIDE 61

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

SLIDE 62

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

ConvNet Visualisation

Yosinski et al. (2015)

[video]

SLIDE 63

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Network Architectures

SLIDE 64

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

AlexNet: ILSVRC ‘12 winner

Krizhevsky et al. (2012)

[TFLearn notebook]

SLIDE 65

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

VGGNet: ILSVRC ‘14 runner-up

Simonyan & Zisserman (2015)

[TFLearn notebook]

SLIDE 66

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

ConvNet in TensorFlow

Caffe Torch Theano TensorFlow Language C++, Python Lua Python Python Pretrained Yes++ Yes++ Yes (Lasagne) Inception Parallel GPUs: Data Yes Yes Yes Yes Parallel GPUs: Model No Yes Experimental Yes (best) Readable Source Code Yes (C++) Yes No No Good at RNN No Mediocre Yes Yes (best) Higher-Level APIs No No Keras Keras and TFLearn

SLIDE 67

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

ConvNet in TensorFlow

[notebook]

SLIDE 68

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

SLIDE 69

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

ConvNet in Theano

Caffe Torch Theano TensorFlow Language C++, Python Lua Python Python Pretrained Yes++ Yes++ Yes (Lasagne) Inception Parallel GPUs: Data Yes Yes Yes Yes Parallel GPUs: Model No Yes Experimental Yes (best) Readable Source Code Yes (C++) Yes No No Good at RNN No Mediocre Yes Yes (best) Higher-Level APIs No No Keras Keras and TFLearn

SLIDE 70

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

ConvNet in Theano

[demo]

SLIDE 71

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

ConvNet in Keras

calls TensorFlow or Theano

Caffe Torch Theano TensorFlow Language C++, Python Lua Python Python Pretrained Yes++ Yes++ Yes (Lasagne) Inception Parallel GPUs: Data Yes Yes Yes Yes Parallel GPUs: Model No Yes Experimental Yes (best) Readable Source Code Yes (C++) Yes No No Good at RNN No Mediocre Yes Yes (best) Higher-Level APIs No No Keras Keras and TFLearn

SLIDE 72

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

ConvNet in Keras

calls TensorFlow or Theano

[notebook]

SLIDE 73

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

“2.5-dimension” CT Scans

Roth et al. (2015)

SLIDE 74

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Computer-Aided Detection

Shin et al. (2016); Roth et al. (2016)

SLIDE 75

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

SLIDE 76

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Kaggle

Data Science Bowl 2017

[link]

SLIDE 77

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Transfer Learning

Caffe

Caffe Torch Theano TensorFlow Language C++, Python Lua Python Python Pretrained Yes++ Yes++ Yes (Lasagne) Inception Parallel GPUs: Data Yes Yes Yes Yes Parallel GPUs: Model No Yes Experimental Yes (best) Readable Source Code Yes (C++) Yes No No Good at RNN No Mediocre Yes Yes (best) Higher-Level APIs No No Keras Keras and TFLearn

SLIDE 78

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Transfer Learning

Caffe

[Model Zoo]

SLIDE 79

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Transfer Learning

SLIDE 80

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Video Classification

[video]

SLIDE 81

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Outline

1

Antecedents Case Study: A History of Biological & Artificial Vision Building Blocks

2

Theory Biological & Artificial Neurons Neural Networks Deep Neural Networks

3

Contemporary Applications Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

SLIDE 82

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Sunspring

SLIDE 83

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Sunspring

SLIDE 84

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

SLIDE 85

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

SLIDE 86

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

RNNs; LSTM RNNs

Hochreiter & Schmidhuber (1997) Graves, ... & Schmidhuber (2009)

SLIDE 87

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Vector Space Embedding

Word2Vec: Mikolov, ... & Dean (2013)

SLIDE 88

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

t-SNE

Hinton & van der Maaten (2008)

SLIDE 89

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Word2Vec + t-SNE

SLIDE 90

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Word2Vec + t-SNE

SLIDE 91

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

‘Understand’ Language

with Word2Vec features in your model

[even with small corpora]

SLIDE 92

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Quick, Draw!

ConvNet + LSTM

[link]

SLIDE 93

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Outline

1

Antecedents Case Study: A History of Biological & Artificial Vision Building Blocks

2

Theory Biological & Artificial Neurons Neural Networks Deep Neural Networks

3

Contemporary Applications Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

SLIDE 94

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

untapt

Digital Recruitment Platform

SLIDE 95

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

untapt

SLIDE 96

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

untapt

Candidate-Side Feedback

SLIDE 97

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

untapt

Client-Side Feedback

SLIDE 98

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

untapt

Multi-Stage Bayesian Regression with PyMC3

Krohn, Rives-Corbett & Donner (2016)

SLIDE 99

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

untapt

Krohn, Rives-Corbett & Donner (2016)

SLIDE 100

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

untapt

Ensemble with Deep Neural Net

deep-orange.untapt.com

SLIDE 101

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

SLIDE 102

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Outline

1

Antecedents Case Study: A History of Biological & Artificial Vision Building Blocks

2

Theory Biological & Artificial Neurons Neural Networks Deep Neural Networks

3

Contemporary Applications Convolutional Neural Networks Long Short-Term Memory Recurrent Neural Networks Deep Learning at untapt Deep Reinforcement Learning

SLIDE 103

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

AlphaGO

Silver et al. (2016)

SLIDE 104

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

Deep Q-Learning

Mnih et al. (2015)

[Atari Games]

SLIDE 105

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement

[OpenAI Universe] [Google DeepMind Lab]

SLIDE 106

Deep Learning Antecedents

Vision Case Study Building Blocks

Theory

Neural Units Neural Nets Deep Neural Nets

Application

ConvNets LSTMs untapt Reinforcement