EE-559 Deep learning 1a. Introduction Fran cois Fleuret - - PowerPoint PPT Presentation

EE-559 – Deep learning

• 1a. Introduction

Fran¸ cois Fleuret https://fleuret.org/dlc/

[version of: June 5, 2018]

ÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE

Why learning

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 2 / 63

Many applications require the automatic extraction of “refined” information from raw signal (e.g. image recognition, automatic speech processing, natural language processing, robotic control, geometry reconstruction). (ImageNet)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 3 / 63

Our brain is so good at interpreting visual information that the “semantic gap” is hard to assess intuitively.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 4 / 63

Our brain is so good at interpreting visual information that the “semantic gap” is hard to assess intuitively. This: is a horse

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 4 / 63

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 4 / 63

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 4 / 63

>>> from torchvision import datasets >>> cifar = datasets.CIFAR10 (’./ data/cifar10/’, train=True , download=True) Files already downloaded and verified >>> x = torch.from_numpy (cifar. train_data )[43]. transpose (2, 0).transpose (1, 2) >>> x.size () torch.Size ([3, 32, 32]) >>> x.narrow (1, 0, 4).narrow (2, 0, 12) (0 ,.,.) = 99 98 100 103 105 107 108 110 114 115 117 118 100 100 102 105 107 109 110 112 115 117 119 120 104 104 106 109 111 112 114 116 119 121 123 124 109 109 111 113 116 117 118 120 123 124 127 128 (1 ,.,.) = 166 165 167 169 171 172 173 175 176 178 179 181 166 164 167 169 169 171 172 174 176 177 179 180 169 167 170 171 171 173 174 176 178 179 182 183 170 169 172 173 175 176 177 178 179 181 183 184 (2 ,.,.) = 198 196 199 200 200 202 203 204 205 206 208 209 195 194 197 197 197 199 200 201 202 203 206 207 197 195 198 198 198 199 201 202 203 204 206 207 197 196 199 198 198 199 200 201 203 204 207 208 [torch. ByteTensor

• f size 3x4x12]

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 5 / 63

Extracting semantic automatically requires models of extreme complexity, which cannot be designed by hand. Techniques used in practice consist of

• 1. defining a parametric model, and
• 2. optimizing its parameters by “making it work” on training data.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 6 / 63

Extracting semantic automatically requires models of extreme complexity, which cannot be designed by hand. Techniques used in practice consist of

• 1. defining a parametric model, and
• 2. optimizing its parameters by “making it work” on training data.

This is similar to biological systems for which the model (e.g. brain structure) is DNA-encoded, and parameters (e.g. synaptic weights) are tuned through experiences.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 6 / 63

Extracting semantic automatically requires models of extreme complexity, which cannot be designed by hand. Techniques used in practice consist of

• 1. defining a parametric model, and
• 2. optimizing its parameters by “making it work” on training data.

This is similar to biological systems for which the model (e.g. brain structure) is DNA-encoded, and parameters (e.g. synaptic weights) are tuned through experiences. Deep learning encompasses software technologies to scale-up to billions of model parameters and as many training examples.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 6 / 63

There are strong connections between standard statistical modeling and machine learning.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 7 / 63

There are strong connections between standard statistical modeling and machine learning. Classical ML methods combine a “learnable” model from statistics (e.g “linear regression”) with prior knowledge in pre-processing.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 7 / 63

There are strong connections between standard statistical modeling and machine learning. Classical ML methods combine a “learnable” model from statistics (e.g “linear regression”) with prior knowledge in pre-processing. “Artificial neural networks” pre-dated these approaches, and do not follow that

• dichotomy. They consist of “deep” stacks of parametrized processing.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 7 / 63

From artificial neural networks to “Deep Learning”

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 8 / 63

130

LOGICAL CALCULUS FOR NERVOUS ACTIVITY

b

e ~

~ 9 h

FIG~E 1

d f

Networks of “Threshold Logic Unit” (McCulloch and Pitts, 1943)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 9 / 63

1949 – Donald Hebb proposes the Hebbian Learning principle. 1951 – Marvin Minsky creates the first ANN (Hebbian learning, 40 neurons).

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 10 / 63

1949 – Donald Hebb proposes the Hebbian Learning principle. 1951 – Marvin Minsky creates the first ANN (Hebbian learning, 40 neurons). 1958 – Frank Rosenblatt creates a perceptron to classify 20 × 20 images.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 10 / 63

1949 – Donald Hebb proposes the Hebbian Learning principle. 1951 – Marvin Minsky creates the first ANN (Hebbian learning, 40 neurons). 1958 – Frank Rosenblatt creates a perceptron to classify 20 × 20 images. 1959 – David H. Hubel and Torsten Wiesel’s demonstrate orientation selectivity and columnar organization in the cat’s visual cortex.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 10 / 63

1949 – Donald Hebb proposes the Hebbian Learning principle. 1951 – Marvin Minsky creates the first ANN (Hebbian learning, 40 neurons). 1958 – Frank Rosenblatt creates a perceptron to classify 20 × 20 images. 1959 – David H. Hubel and Torsten Wiesel’s demonstrate orientation selectivity and columnar organization in the cat’s visual cortex. 1982 – Paul Werbos proposes back-propagation for ANNs.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 10 / 63

Neocognitron

195

visuo[ oreo 9l< QSsOCiQtion oreo-- lower-order

• -,. higher-order .-,. ~

.grandmother retino --,- LGB --,. simple ~ complex --,. hypercomplex hypercomplex " -- cell '~

F- 3 I-- . . . . l r I I I I 1 1

Uo ' , ~' Usl

• ---->

Ucl t~-~i Us2~ Uc2 ~ Us3----* Uc3 T

[ I L ~ L J

• Fig. 1. Correspondence

between the hierarchy model by Hubel and Wiesel, and the neural network of the neocognitron shifted in parallel from cell to cell. Hence, all the cells in a single cell-plane have receptive fields of the same function, but at different positions. We will use notations Us~(k~,n ) to represent the

• utput of an S-cell in the krth S-plane in the l-th

module, and Ucl(k~, n) to represent the output of a C-cell in the krth C-plane in that module, where n is the two- dimensional co-ordinates representing the position of these cell's receptive fields in the input layer. Figure 2 is a schematic diagram illustrating the interconnections between layers. Each tetragon drawn with heavy lines represents an S-plane or a C-plane, and each vertical tetragon drawn with thin lines, in which S-planes or C-planes are enclosed, represents an S-layer or a C-layer. In Fig. 2, a cell of each layer receives afferent connections from the cells within the area enclosed by the elipse in its preceding layer. To be exact, as for the S-cells, the elipses in Fig. 2 does not show the connect- ing area but the connectable area to the S-cells. That is, all the interconnections coming from the elipses are not always formed, because the synaptic connections incoming to the S-cells have plasticity. In Fig. 2, for the sake of simplicity of the figure,

• nly one cell is shown in each cell-plane. In fact, all the

cells in a cell-plane have input synapses of the same spatial distribution as shown in Fig. 3, and only the positions of the presynaptic cells are shifted in parallel from cell to cell.

R3 ~I

modifioble synapses ) unmodifiable synopses

Since the cells in the network are interconnected in a cascade as shown in Fig. 2, the deeper the layer is, the larger becomes the receptive field of each cell of that

• layer. The density of the cells in each cell-plane is so

determined as to decrease in accordance with the increase of the size of the receptive fields. Hence, the total number of the cells in each cell-plane decreases with the depth of the cell-plane in the network. In the last module, the receptive field of each C-cell becomes so large as to cover the whole area of input layer U0, and each C-plane is so determined as to have only one C-cell. The S-cells and C-cells are excitatory cells. That is, all the efferent synapses from these cells are excitatory. Although it is not shown in Fig. 2, we also have

• Fig. 3. Illustration showing the input interconnections

to the cells within a single cell-plane

• Fig. 2. Schematic

diagram illustrating the interconnections between layers in the neocognitron

Follows Hubel and Wiesel’s results. (Fukushima, 1980)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 11 / 63

Network for the T-C problem Trained with back-prop. (Rumelhart et al., 1988)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 12 / 63

LeNet-5

• INPUT

32x32 Convolutions Subsampling Convolutions C1: feature maps 6@28x28 Subsampling S2: f. maps 6@14x14 S4: f. maps 16@5x5 C5: layer 120 C3: f. maps 16@10x10 F6: layer 84 Full connection Full connection Gaussian connections OUTPUT 10

(leCun et al., 1998)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 13 / 63

AlexNet (Krizhevsky et al., 2012)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 14 / 63

GoogLeNet

(Szegedy et al., 2015)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 15 / 63

Resnet

7x7 conv, 64, /2 pool, /2 3x3 conv, 64 3x3 conv, 64 3x3 conv, 64 3x3 conv, 64 3x3 conv, 64 3x3 conv, 64 3x3 conv, 128, /2 3x3 conv, 128 3x3 conv, 128 3x3 conv, 128 3x3 conv, 128 3x3 conv, 128 3x3 conv, 128 3x3 conv, 128 3x3 conv, 256, /2 3x3 conv, 256 3x3 conv, 256 3x3 conv, 256 3x3 conv, 256 3x3 conv, 256 3x3 conv, 256 3x3 conv, 256 3x3 conv, 256 3x3 conv, 256 3x3 conv, 256 3x3 conv, 256 3x3 conv, 512, /2 3x3 conv, 512 3x3 conv, 512 3x3 conv, 512 3x3 conv, 512 3x3 conv, 512 avg pool fc 1000 image

(He et al., 2015)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 16 / 63

Deep learning is built on a natural generalization of a neural network: a graph

• f tensor operators, taking advantage of
• the chain rule (aka “back-propagation”),
• stochastic gradient decent,
• convolutions,
• parallel operations on GPUs.

This does not differ much from networks from the 90s

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 17 / 63

This generalization allows to design complex networks of operators dealing with images, sound, text, sequences, etc. and to train them end-to-end. (Yeung et al., 2015)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 18 / 63

CIFAR10 32 × 32 color images, 50k train samples, 10k test samples. (Krizhevsky, 2009, chap. 3)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 19 / 63

Performance on CIFAR10

75 80 85 90 95 100 2010 2012 2014 2016 2018

Krizhevsky et al. (2012) Graham (2015) Human performance Real et al. (2018)

Accuracy (%) Year

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 20 / 63

ImageNet Large Scale Visual Recognition Challenge. 1000 categories, > 1M images (http://image-net.org/challenges/LSVRC/2014/browse-synsets)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 21 / 63

ImageNet Large Scale Visual Recognition Challenge. 1000 categories, > 1M images (http://image-net.org/challenges/LSVRC/2014/browse-synsets)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 21 / 63

ImageNet Large Scale Visual Recognition Challenge. 1000 categories, > 1M images (http://image-net.org/challenges/LSVRC/2014/browse-synsets)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 21 / 63

method top-1 err. top-5 err.

VGG [41] (ILSVRC’14)

• 8.43†

GoogLeNet [44] (ILSVRC’14)

• 7.89

VGG [41] (v5) 24.4 7.1 PReLU-net [13] 21.59 5.71 BN-inception [16] 21.99 5.81 ResNet-34 B 21.84 5.71 ResNet-34 C 21.53 5.60 ResNet-50 20.74 5.25 ResNet-101 19.87 4.60 ResNet-152 19.38 4.49 Table 4. Error rates (%) of single-model results on the ImageNet validation set (except † reported on the test set).

method

top-5 err. (test) VGG [41] (ILSVRC’14) 7.32 GoogLeNet [44] (ILSVRC’14) 6.66 VGG [41] (v5) 6.8 PReLU-net [13] 4.94 BN-inception [16] 4.82 ResNet (ILSVRC’15) 3.57 Table 5. Error rates (%) of ensembles. The top-5 error is on the test set of ImageNet and reported by the test server.

(He et al., 2015)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 22 / 63

Current application domains

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 23 / 63

Object detection and segmentation (Pinheiro et al., 2016)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 24 / 63

Human pose estimation (Wei et al., 2016)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 25 / 63

Image generation (Radford et al., 2015)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 26 / 63

Reinforcement learning Self-trained, plays 49 games at human level. (Mnih et al., 2015)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 27 / 63

Strategy games March 2016, 4-1 against a 9-dan professional without handicap. (Silver et al., 2016)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 28 / 63

Translation

“The reason Boeing are doing this is to cram more seats in to make their plane more competitive with our products,” said Kevin Keniston, head of passenger comfort at Europe’s Airbus.

➙

“La raison pour laquelle Boeing fait cela est de cr´ eer plus de si` eges pour rendre son avion plus comp´ etitif avec nos produits”, a d´ eclar´ e Kevin Keniston, chef du confort des passagers chez Airbus. When asked about this, an official of the American administration replied: “The United States is not conducting electronic surveillance aimed at offices

• f the World Bank and IMF in Washington.”

➙

Interrog´ e ` a ce sujet, un fonctionnaire de l’administration am´ ericaine a r´ epondu: “Les ´ Etats-Unis n’effectuent pas de surveillance ´ electronique ` a l’intention des bureaux de la Banque mondiale et du FMI ` a Washington”

(Wu et al., 2016)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 29 / 63

Auto-captioning (Vinyals et al., 2015)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 30 / 63

Question answering

I: Jane went to the hallway. I: Mary walked to the bathroom. I: Sandra went to the garden. I: Daniel went back to the garden. I: Sandra took the milk there. Q: Where is the milk? A: garden I: It started boring, but then it got interesting. Q: What’s the sentiment? A: positive

(Kumar et al., 2015)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 31 / 63

Why does it work now?

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 32 / 63

The success of deep learning is multi-factorial:

• Five decades of research in machine learning,
• CPUs/GPUs/storage developed for other purposes,
• lots of data from “the internet”,
• tools and culture of collaborative and reproducible science,
• resources and efforts from large corporations.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 33 / 63

Five decades of research in ML provided

• a taxonomy of ML concepts (classification, generative models, clustering,

kernels, linear embeddings, etc.),

• a sound statistical formalization (Bayesian estimation, PAC),
• a clear picture of fundamental issues (bias/variance dilemma, VC

dimension, generalization bounds, etc.),

• a good understanding of optimization issues,
• efficient large-scale algorithms.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 34 / 63

From a practical perspective, deep learning

• lessens the need for a deep mathematical grasp,
• makes the design of large learning architectures a system/software

development task,

• allows to leverage modern hardware (clusters of GPUs),
• does not plateau when using more data,
• makes large trained networks a commodity.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 35 / 63

10-3 100 103 106 109 1012 1960 1970 1980 1990 2000 2010 2020 Flops per USD

(Wikipedia “FLOPS”)

TFlops (1012) Price GFlops per $ Intel i7-6700K 0.2 $344 0.6 AMD Radeon R-7 240 0.5 $55 9.1 NVIDIA GTX 750 Ti 1.3 $105 12.3 AMD RX 480 5.2 $239 21.6 NVIDIA GTX 1080 8.9 $699 12.7

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 36 / 63

103 106 109 1012 1980 1990 2000 2010 2020 Bytes per USD

(John C. McCallum) The typical cost of a 4Tb hard disk is $120 (Dec 2016).

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 37 / 63

A l e x N e t B N

• A

l e x N e t B N

• N

I N G

• g

L e N e t R e s N e t

• 1

8 V G G

• 1

6 V G G

• 1

9 R e s N e t

• 3

4 R e s N e t

• 5

R e s N e t

• 1

1 I n c e p t i

• n
• v

3

50 55 60 65 70 75 80 Top-1 accuracy [%] 5 10 15 20 25 30 35 40 Operations [G-Ops] 50 55 60 65 70 75 80 Top-1 accuracy [%]

AlexNet BN-AlexNet BN-NIN ResNet-18 VGG-16 VGG-19 GoogLeNet ResNet-34 ResNet-50 ResNet-101 Inception-v3 5M 35M 65M 95M 125M 155M

1 2 4 8 16 32 64 Batch size [ / ] 100 200 300 400 500 600 Foward time per image [ms] BN-NIN GoogLeNet Inception-v3 AlexNet BN-AlexNet VGG-16 VGG-19 ResNet-18 ResNet-34 ResNet-50 ResNet-101 1 2 4 8 16 32 64 Batch size [ / ] 5 10 20 50 100 200 500 Foward time per image [ms] BN-NIN GoogLeNet Inception-v3 AlexNet BN-AlexNet VGG-16 VGG-19 ResNet-18 ResNet-34 ResNet-50 ResNet-101

(Canziani et al., 2016)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 38 / 63

Data-set Year

• Nb. images

Resolution

• Nb. classes

MNIST 1998 6.0 × 104 28 × 28 10 NORB 2004 4.8 × 104 96 × 96 5 Caltech 101 2003 9.1 × 103 ≃ 300 × 200 101 Caltech 256 2007 3.0 × 104 ≃ 640 × 480 256 LFW 2007 1.3 × 104 250 × 250 – CIFAR10 2009 6.0 × 104 32 × 32 10 PASCAL VOC 2012 2.1 × 104 ≃ 500 × 400 20 MS-COCO 2015 2.0 × 105 ≃ 640 × 480 91 ImageNet 2016 14.2 × 106 ≃ 500 × 400 21, 841 Cityscape 2016 25 × 103 2, 000 × 1000 30

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 39 / 63

“Quantity has a Quality All Its Own.” (Thomas A. Callaghan Jr.)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 40 / 63

Implementing a deep network, PyTorch

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 41 / 63

Deep-learning development is usually done in a framework:

Language(s) License Main backer PyTorch Python BSD Facebook Caffe2 C++, Python Apache Facebook TensorFlow Python, C++ Apache Google MXNet Python, C++, R, Scala Apache Amazon CNTK Python, C++ MIT Microsoft Torch Lua BSD Facebook Theano Python BSD

• U. of Montreal

Caffe C++ BSD 2 clauses

• U. of CA, Berkeley

A fast, low-level, compiled backend to access computation devices, combined with a slow, high-level, interpreted language.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 42 / 63

We will use the PyTorch framework for our experiments. http://pytorch.org

“PyTorch is a python package that provides two high-level features:

• Tensor computation (like numpy) with strong GPU acceleration
• Deep Neural Networks built on a tape-based autograd system

You can reuse your favorite python packages such as numpy, scipy and Cython to extend PyTorch when needed.”

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 43 / 63

MNIST data-set 28 × 28 grayscale images, 60k train samples, 10k test samples. (leCun et al., 1998)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 44 / 63

class Net(nn.Module): def __init__(self): super(Net , self).__init__ () self.conv1 = nn.Conv2d (1, 32, kernel_size =5) self.conv2 = nn.Conv2d (32, 64, kernel_size =5) self.fc1 = nn.Linear (256 , 200) self.fc2 = nn.Linear (200 , 10) def forward(self , x): x = F.relu(F. max_pool2d (self.conv1(x), kernel_size =3)) x = F.relu(F. max_pool2d (self.conv2(x), kernel_size =2)) x = x.view(-1, 256) x = F.relu(self.fc1(x)) x = self.fc2(x) return x model = Net () mu , std = train_input .data.mean (), train_input .data.std () train_input .data.sub_(mu).div_(std)

• ptimizer = optim.SGD(model. parameters (), lr = 1e -1)

criterion , bs = nn. CrossEntropyLoss (), 100 model.cuda () criterion.cuda () train_input , train_target = train_input .cuda (), train_target .cuda () for e in range (10): for b in range (0, nb_train_samples , bs):

• utput = model( train_input .narrow (0, b, bs))

loss = criterion(output , train_target .narrow (0, b, bs)) model.zero_grad () loss.backward ()

• ptimizer.step ()

≃7s on a GTX1080, ≃1% test error

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 45 / 63

class Net(nn.Module): def __init__(self): super(Net , self).__init__ () self.conv1 = nn.Conv2d (1, 32, kernel_size =5) self.conv2 = nn.Conv2d (32, 64, kernel_size =5) self.fc1 = nn.Linear (256 , 200) self.fc2 = nn.Linear (200 , 10) def forward(self , x): x = F.relu(F. max_pool2d (self.conv1(x), kernel_size =3)) x = F.relu(F. max_pool2d (self.conv2(x), kernel_size =2)) x = x.view(-1, 256) x = F.relu(self.fc1(x)) x = self.fc2(x) return x model = Net () mu , std = train_input .data.mean (), train_input .data.std () train_input .data.sub_(mu).div_(std)

• ptimizer = optim.SGD(model. parameters (), lr = 1e -1)

criterion , bs = nn. CrossEntropyLoss (), 100 model.cuda () criterion.cuda () train_input , train_target = train_input .cuda (), train_target .cuda () for e in range (10): for b in range (0, nb_train_samples , bs):

• utput = model( train_input .narrow (0, b, bs))

loss = criterion(output , train_target .narrow (0, b, bs)) model.zero_grad () loss.backward ()

• ptimizer.step ()

≃7s on a GTX1080, ≃1% test error

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 45 / 63

class Net(nn.Module): def __init__(self): super(Net , self).__init__ () self.conv1 = nn.Conv2d (1, 32, kernel_size =5) self.conv2 = nn.Conv2d (32, 64, kernel_size =5) self.fc1 = nn.Linear (256 , 200) self.fc2 = nn.Linear (200 , 10) def forward(self , x): x = F.relu(F. max_pool2d (self.conv1(x), kernel_size =3)) x = F.relu(F. max_pool2d (self.conv2(x), kernel_size =2)) x = x.view(-1, 256) x = F.relu(self.fc1(x)) x = self.fc2(x) return x model = Net () mu , std = train_input .data.mean (), train_input .data.std () train_input .data.sub_(mu).div_(std)

• ptimizer = optim.SGD(model. parameters (), lr = 1e -1)

criterion , bs = nn. CrossEntropyLoss (), 100 model.cuda () criterion.cuda () train_input , train_target = train_input .cuda (), train_target .cuda () for e in range (10): for b in range (0, nb_train_samples , bs):

• utput = model( train_input .narrow (0, b, bs))

loss = criterion(output , train_target .narrow (0, b, bs)) model.zero_grad () loss.backward ()

• ptimizer.step ()

≃7s on a GTX1080, ≃1% test error

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 45 / 63

class Net(nn.Module): def __init__(self): super(Net , self).__init__ () self.conv1 = nn.Conv2d (1, 32, kernel_size =5) self.conv2 = nn.Conv2d (32, 64, kernel_size =5) self.fc1 = nn.Linear (256 , 200) self.fc2 = nn.Linear (200 , 10) def forward(self , x): x = F.relu(F. max_pool2d (self.conv1(x), kernel_size =3)) x = F.relu(F. max_pool2d (self.conv2(x), kernel_size =2)) x = x.view(-1, 256) x = F.relu(self.fc1(x)) x = self.fc2(x) return x model = Net () mu , std = train_input .data.mean (), train_input .data.std () train_input .data.sub_(mu).div_(std)

• ptimizer = optim.SGD(model. parameters (), lr = 1e -1)

criterion , bs = nn. CrossEntropyLoss (), 100 model.cuda () criterion.cuda () train_input , train_target = train_input .cuda (), train_target .cuda () for e in range (10): for b in range (0, nb_train_samples , bs):

• utput = model( train_input .narrow (0, b, bs))

loss = criterion(output , train_target .narrow (0, b, bs)) model.zero_grad () loss.backward ()

• ptimizer.step ()

≃7s on a GTX1080, ≃1% test error

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 45 / 63

class Net(nn.Module): def __init__(self): super(Net , self).__init__ () self.conv1 = nn.Conv2d (1, 32, kernel_size =5) self.conv2 = nn.Conv2d (32, 64, kernel_size =5) self.fc1 = nn.Linear (256 , 200) self.fc2 = nn.Linear (200 , 10) def forward(self , x): x = F.relu(F. max_pool2d (self.conv1(x), kernel_size =3)) x = F.relu(F. max_pool2d (self.conv2(x), kernel_size =2)) x = x.view(-1, 256) x = F.relu(self.fc1(x)) x = self.fc2(x) return x model = Net () mu , std = train_input .data.mean (), train_input .data.std () train_input .data.sub_(mu).div_(std)

• ptimizer = optim.SGD(model. parameters (), lr = 1e -1)

criterion , bs = nn. CrossEntropyLoss (), 100 model.cuda () criterion.cuda () train_input , train_target = train_input .cuda (), train_target .cuda () for e in range (10): for b in range (0, nb_train_samples , bs):

• utput = model( train_input .narrow (0, b, bs))

loss = criterion(output , train_target .narrow (0, b, bs)) model.zero_grad () loss.backward ()

• ptimizer.step ()

≃7s on a GTX1080, ≃1% test error

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 45 / 63

class Net(nn.Module): def __init__(self): super(Net , self).__init__ () self.conv1 = nn.Conv2d (1, 32, kernel_size =5) self.conv2 = nn.Conv2d (32, 64, kernel_size =5) self.fc1 = nn.Linear (256 , 200) self.fc2 = nn.Linear (200 , 10) def forward(self , x): x = F.relu(F. max_pool2d (self.conv1(x), kernel_size =3)) x = F.relu(F. max_pool2d (self.conv2(x), kernel_size =2)) x = x.view(-1, 256) x = F.relu(self.fc1(x)) x = self.fc2(x) return x model = Net () mu , std = train_input .data.mean (), train_input .data.std () train_input .data.sub_(mu).div_(std)

• ptimizer = optim.SGD(model. parameters (), lr = 1e -1)

criterion , bs = nn. CrossEntropyLoss (), 100 model.cuda () criterion.cuda () train_input , train_target = train_input .cuda (), train_target .cuda () for e in range (10): for b in range (0, nb_train_samples , bs):

• utput = model( train_input .narrow (0, b, bs))

loss = criterion(output , train_target .narrow (0, b, bs)) model.zero_grad () loss.backward ()

• ptimizer.step ()

≃7s on a GTX1080, ≃1% test error

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 45 / 63

What is really happening?

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 46 / 63

(Zeiler and Fergus, 2014)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 47 / 63

(Zeiler and Fergus, 2014)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 48 / 63

(Google’s Deep Dreams)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 49 / 63

(Google’s Deep Dreams)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 50 / 63

(Thorne Brandt)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 51 / 63

(Duncan Nicoll)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 52 / 63

(Szegedy et al., 2014) (Nguyen et al., 2015)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 53 / 63

Relations with the biology

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 54 / 63

LN LN ... LN LN ... LN LN LN LN LN LN Stimulus

a b c

Encoding Decoding Neurons Behavior RGC LGN V2 V4 V1 DOG ? ? ? PIT CIT AIT

...

Φ1 Φ2 Φk ⊗ ⊗ ⊗ Operations in linear-nonlinear layer Filter Threshold Pool Normalize ... ... ... Spatial convolution

• ver image input

100-ms visual presentation Pixels LN PIT V2 V4 V1 CIT AIT T(•)

Figure 1 HCNNs as models of sensory

• cortex. (a) The basic framework in which

sensory cortex is studied is one of encoding—the process by which stimuli are transformed into patterns of neural activity—and decoding, the process by which neural activity generates

• behavior. HCNNs have been used to make models of the encoding step; that is, they describe

6

(Yamins and DiCarlo, 2016)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 55 / 63

b

HCNN top hidden layer response prediction IT neural response Test images (sorted by category) IT site 56

a

r = . 8 7 ± . 1 5 H C N N m

• d

e l s 0.6 1.0 50 IT single-site neural predictivity (% explained variance) HMO (top hidden layer) V2-like HMAX PLOS09 SIFT V1-like Pixels Category ideal

• bserver

Categorization performance (balanced accuracy)

d

HCNN model Human IT (fMRI) Animate Human Not human Body Face Body Face Natural Artificial Inanimate

A = 0.38

1 2 3 4 1 2 3 4

c

Monkey V4 (n = 128) Monkey IT (n = 168) Ideal

• bservers

Control models HCNN layers Control models Ideal

• bservers

HCNN layers Pixels V1-like Category All variables PLOS09 HMAX V2-Like SIFT Pixels V1-Like PLOS09 HMAX V2-like SIFT 50 50 Single-site neural predictivity (% explained variance)

** **** **** **** **** ****

e

0.2 0.4 0.2 0.4 Human V1–V3 Human IT RDM voxel correlation (Kendall’s A) Scores Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6 Layer 7 Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Layer 6 Layer 7 Convolutional Fully connected

**** **** **** * **** **** **** ****

SVM Geometry- supervised

**** τ 6

(Yamins and DiCarlo, 2016)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 56 / 63

Species

• Nb. neurons
• Nb. synapses

Roundworm 302 7.5 × 103 Jellyfish 800 Sea slug 1.8 × 104 Fruit fly 1.0 × 105 1.0 × 107 Ant 2.5 × 105 Cockroach 1.0 × 106 Frog 1.6 × 107 Mouse 7.1 × 107 1.0 × 1011 Rat 2.0 × 108 4.5 × 1011 Octopus 3.0 × 108 Human 8.6 × 1010 1.0 × 1015

(Wikipedia “List of animals by number of neurons”)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 57 / 63

Device

• Nb. transistors

Intel i7 Haswell-E (8 cores) 2.6 × 109 Intel Xeon Broadwell-E5 (22 cores) 7.2 × 109 AMD Epyc (32 cores) 19.2 × 109 Nvidia GeForce GTX 1080 7.2 × 109 AMD Vega 10 12.5 × 109 NVidia GV100 21.1 × 109 (Wikipedia “Transistor count”)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 58 / 63

103 106 109 1012 1015 1018 1960 1970 1980 1990 2000 2010 2020

• Nb. human synapses
• Nb. mouse synapses
• Nb. fruit fly synapses
• Nb. Transistors

Number of transistors per CPU/GPU CPUs GPUs

(Wikipedia “Transistor count”)

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 59 / 63

Plan, pre-requisites and grading

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 60 / 63

Lecture content:

• 1. Introduction.
• 2. Standard machine-learning concepts and tools.
• 3. Multi-layer perceptrons, back-prop, stochastic gradient descent.
• 4. Convolutional networks, arbitrary graphs of operators.
• 5. Initialization, optimization, and regularization.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 61 / 63

Lecture content:

• 1. Introduction.
• 2. Standard machine-learning concepts and tools.
• 3. Multi-layer perceptrons, back-prop, stochastic gradient descent.
• 4. Convolutional networks, arbitrary graphs of operators.
• 5. Initialization, optimization, and regularization.
• 6. Going deeper.
• 7. Deep models for Computer Vision.
• 8. Analysis of deep models.
• 9. Auto-encoders, embeddings, and generative models.
• 10. Generative adversarial networks.
• 11. Recurrent models, memory networks, NLP.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 61 / 63

Lecture content:

• 1. Introduction.
• 2. Standard machine-learning concepts and tools.
• 3. Multi-layer perceptrons, back-prop, stochastic gradient descent.
• 4. Convolutional networks, arbitrary graphs of operators.
• 5. Initialization, optimization, and regularization.
• 6. Going deeper.
• 7. Deep models for Computer Vision.
• 8. Analysis of deep models.
• 9. Auto-encoders, embeddings, and generative models.
• 10. Generative adversarial networks.
• 11. Recurrent models, memory networks, NLP.
• 12. Invited speaker (Soumith Chintala, Facebook).
• 13. Invited lecture (Andreas Steiner, Google).
• 14. Invited lecture (Andreas Steiner, Google).

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 61 / 63

Pre-requisites:

• Linear algebra (vector and Euclidean spaces),
• differential calculus (gradient, Jacobian, Hessian, chain rule),
• Python programming

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 62 / 63

Pre-requisites:

• Linear algebra (vector and Euclidean spaces),
• differential calculus (gradient, Jacobian, Hessian, chain rule),
• Python programming,

... but there is more!

• basics in probabilities and statistics (discrete and continuous distributions,

law of large numbers, conditional probabilities, Bayes, PCA),

• basics in optimization (notion of minima, gradient descent),
• basics in algorithmic (computational costs),
• basics in signal processing (Fourier transform, wavelets).

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 62 / 63

The evaluation will be: 50% – One mini-project, by groups of one to three students. Group report and source code, 5 min oral for each student/project. 50% – Final written exam.

Fran¸ cois Fleuret EE-559 – Deep learning / 1a. Introduction 63 / 63

The end

References

• A. Canziani, A. Paszke, and E. Culurciello. An analysis of deep neural network models for

practical applications. CoRR, abs/1605.07678, 2016.

• K. Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of

pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4): 193–202, April 1980.

• K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CoRR,

abs/1512.03385, 2015.

• A. Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis,

Department of Computer Science, University of Toronto, 2009.

• A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional

neural networks. In Neural Information Processing Systems (NIPS), 2012.

• A. Kumar, O. Irsoy, J. Su, J. Bradbury, R. English, B. Pierce, P. Ondruska, I. Gulrajani,

and R. Socher. Ask me anything: Dynamic memory networks for natural language

• processing. CoRR, abs/1506.07285, 2015.
• Y. leCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to

document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.

• W. S. McCulloch and W. Pitts. A logical calculus of the ideas immanent in nervous activity.

The bulletin of mathematical biophysics, 5(4):115–133, 1943.

• V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves,
• M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik,
• I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis.

Human-level control through deep reinforcement learning. Nature, 518(7540):529–533,

• Feb. 2015.
• A. M. Nguyen, J. Yosinski, and J. Clune. Deep neural networks are easily fooled: High

confidence predictions for unrecognizable images. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.

• P. O. Pinheiro, T.-Y. Lin, R. Collobert, and P. Doll´
• ar. Learning to refine object segments.

In European Conference on Computer Vision (ECCV), pages 75–91, 2016.

• A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep

convolutional generative adversarial networks. CoRR, abs/1511.06434, 2015.

• D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Neurocomputing: Foundations of

Research, chapter Learning Representations by Back-propagating Errors, pages 696–699. MIT Press, 1988.

• D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche,
• J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe,
• J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu,
• T. Graepel, and D. Hassabis. Mastering the game of go with deep neural networks and

tree search. Nature, 529:484–503, 2016.

• C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus.

Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.

• C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke,

and A. Rabinovich. Going deeper with convolutions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.

• O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption
• generator. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
• S. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh. Convolutional pose machines. CoRR,

abs/1602.00134, 2016.

• Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao,
• Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser, S. Gouws,
• Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young,
• J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean.

Google’s neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144, 2016.

• D. L. K. Yamins and J. J. DiCarlo. Using goal-driven deep learning models to understand

sensory cortex. Nature neuroscience, 19:356–65, Feb 2016.

• S. Yeung, O. Russakovsky, G. Mori, and L. Fei-Fei. End-to-end learning of action detection

from frame glimpses in videos. CoRR, abs/1511.06984, 2015.

• M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In

European Conference on Computer Vision (ECCV), 2014.