ImageNet Classification with Deep Convolutional Neural Networks Alex - - PowerPoint PPT Presentation

▶

Feb 23, 2023 356 likes •710 views

ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton Presented by Tugce Tasci, Kyunghee Kim 05/18/2015 Outline Goal DataSet Architecture of the Network Reducing

SLIDE 1

ImageNet Classification with Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton Presented by Tugce Tasci, Kyunghee Kim 05/18/2015

SLIDE 2

Outline

Goal
DataSet
Architecture of the Network
Reducing overfitting
Learning
Results
Discussion

SLIDE 3

Goal

Classifica(on ¡

SLIDE 4

ImageNet

Over 15M labeled high resolution images
Roughly 22K categories
Collected from web and labeled by Amazon Mechanical

Turk

h-p://image-‑net.org/ ¡

SLIDE 5

ILSVRC

Annual competition of image classification at large scale
1.2M images in 1K categories
Classification: make 5 guesses about the image label

Appenzeller ¡ EntleBucher ¡

SLIDE 6

Convolutional Neural Networks

Model with a large learning capacity
Prior knowledge to compensate all data we do not have

SLIDE 7

ILSVRC ¡

ImageNet Classification error throughout years and groups

SLIDE 8

SuperVision (SV)

Image classification with deep convolutional neural networks

7 hidden “weight” layers
650K neurons
60M parameters
630M connections
Rectified Linear Units, overlapping pooling, dropout trick
Randomly extracted 224x224 patches for more data

h-p://image-‑net.org/challenges/LSVRC/2012/supervision.pdf ¡

SLIDE 9

Architecture

5 ¡Convolu(onal ¡Layers ¡ 3 ¡Fully ¡Connected ¡Layers ¡ 1000-‑way ¡ soLmax ¡

SLIDE 10

Layer 1 (Convolutional)

Images: 227x227x3
F (receptive field size): 11
S (stride) = 4
Conv layer output: 55x55x96

SLIDE 11

55*55*96 = 290,400 neurons
each has 11*11*3 = 363 weights and 1

bias

290400 * 364 = 105,705,600

paramaters on the first layer of the AlexNet alone!

Layer 1 (Convolutional)

SLIDE 12

Architecture

RELU Nonlinearity

Standard way to model a neuron

¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡f(x) = tanh(x) or ¡ ¡ ¡ ¡ ¡ ¡f(x) = (1 + e-x)-1 ¡ ¡ ¡ ¡ ¡ ¡ ¡Very slow to train ¡

Non-saturating nonlinearity (RELU)

f(x) = max(0, x) Quick to train

SLIDE 13

A 4 layer CNN with ReLUs (solid line) converges six times faster than an equivalent network with tanh neurons (dashed line) on CIFAR-10 dataset

Architecture

RELU Nonlinearity

SLIDE 14

intra-‑GPU ¡connec(ons ¡ inter-‑GPU ¡connec(ons ¡

GPU #1 GPU #2

Architecture

Training on Multiple GPUs

SLIDE 15

intra-‑GPU ¡connec(ons ¡ inter-‑GPU ¡connec(ons ¡

GPU #1 GPU #2

Architecture

Training on Multiple GPUs

Top-1 and Top-5 error rates decreases by 1.7% & 1.2% respectively, comparing to the net trained with one GPU and half neurons!!

SLIDE 16

Architecture

Overlaping Pooling

SLIDE 17

No need to input normalization with ReLUs.
But still the following local normalization scheme helps

generalization.

Response normalization reduces top-1 and top-5 error rates by

1.4% and 1.2% , respectively.

Architecture

Local Response Normalization

Response-‑ normalized ¡ ac(vity ¡

Ac(vity ¡of ¡a ¡neuron ¡computed ¡ by ¡applying ¡kernel ¡I ¡at ¡posi(on ¡ (x,y) ¡and ¡then ¡applying ¡the ¡ReLU ¡ nonlinearity ¡

SLIDE 18

Traditional pooling (s = z)
s < z è overlapping pooling
top-1 and top-5 error rates decrease by 0.4% and 0.3%,

respectively, compared to the non-overlapping scheme s = 2, z = 2

Architecture

Overlaping Pooling

z ¡ s ¡

SLIDE 19

Architecture

SLIDE 20

Architecture Overview

SLIDE 21

Outline

Introduction
DataSet
Architecture of the Network
Reducing overfitting
Learning
Results
Discussion

SLIDE 22

Reducing Overfitting

Data Augmentation Data Augmentation

 60 million parameters, 650,000 neurons à Overfits a lot.  Crop 224x224 patches (and their horizontal reflections.)

SLIDE 23

Reducing Overfitting

Data Augmentation Data Augmentation  At test time, average the predictions on the 10 patches.

SLIDE 24

Reducing Overfitting

 Softmax

 No need to calibrate to average the predictions over 10 patches.

L = 1 N −log e

fy j

e

f j j

∑

$ % & & & ' ( ) ) ) + λ Wk,l

2 l

∑

j ¡= ¡1…1000 ¡

P(yi | xi;W ) Likelihood ¡

Slide ¡credit ¡from ¡Stanford ¡CS231N ¡Lecture ¡3. ¡

cf. ¡ ¡ ¡SVM

L = 1 N max(0, f (xi;W ) j − f (xi;W )yi + Δ) + λ Wk,l

2 l

∑

& ' ( ) * +

j ≠yi

∑

SLIDE 25

Reducing Overfitting

Data Augmentation Data Augmentation  Change the intensity of RGB channels  add multiples of principle components

Ixy = [Ixy

R , Ixy G, Ixy B ]T

αi ~ N(0, 0.1)

SLIDE 26

Reducing Overfitting

Dr Dropout

pout

Figure ¡credit ¡from ¡Srivastava ¡et ¡al. ¡

 With probability 0.5  last two 4096 fully-connected layers.

SLIDE 27

Stochastic Gradient Descent Learning

 The training took 5 to 6 days on two NVIDIA GTX 580 3GB GPUs.

Momentum ¡Update ¡

weight ¡decay ¡ momentum(damping ¡parameter) ¡ Learning ¡rate ¡(ini(alized ¡at ¡0.01) ¡ Gradient ¡of ¡Loss ¡ ¡

w.r.t ¡weight ¡ ¡ Averaged ¡over ¡batch ¡

Batch ¡size: ¡128 ¡

¡ ¡ ¡

SLIDE 28

Results : ILSVRC-2010

SLIDE 29

Results : ILSVRC-2012

SLIDE 30

96 96 Convolutional Convolutional Ker Kernels nels

 11 x 11 x 3 size kernels.  top 48 kernels on GPU 1 : color-agnostic  bottom 48 kernels on GPU 2 : color-specific.

¡ ¡ ¡Why? ¡

SLIDE 31

Eight ILSVRC-2010 test images

SLIDE 32

Five ILSVRC-2010 test images

The output from the last 4096 fully-connected layer : 4096 dimensional feature.

SLIDE 33

Discussion

 Depth is really important. removing a single convolutional layer degrades the

performance.

K. Simonyan, A. Zisserman.

Very Deep Convolutional Networks for Large-Scale Image Recognition. Technical report, 2014. à 16-layer model, 19-layer model. 7.3% top-5 test error

n ILSVRC-2012

SLIDE 34

Discussion

 Still have many orders of magnitude to go in order to match the infero-temporal(IT) pathway of the human visual system.

¡ ¡ ¡Convolu(onal ¡Neural ¡

Networks? ¡vs. ¡ Convolutonal ¡Networks? ¡

Figure ¡adapted ¡from ¡Gross, ¡C. ¡G., ¡Rodman, ¡H. ¡R., ¡

Gochin, ¡P. ¡M., ¡and ¡Colombo, ¡M. ¡W. ¡(1993). ¡Inferior ¡temporal ¡ cortex ¡as ¡a ¡pa-ern ¡recogni(on ¡device. ¡In ¡“Computa(onal ¡ Learning ¡and ¡Cogni(on” ¡(E. ¡Baum, ¡ed.), ¡pp. ¡44–73. ¡Society ¡for ¡ Industrial ¡and ¡Applied ¡Mathema(cs, ¡Philadelphia. ¡

SLIDE 35

Discussion

 Classification on video. video sequences provide temporal structure missing

in static images.

K. Simonyan, A. Zisserman.

ImageNet Classification with Deep Convolutional Neural Networks

Outline

Goal

ImageNet

Turk

ILSVRC

Convolutional Neural Networks

ILSVRC ¡

SuperVision (SV)

Image classification with deep convolutional neural networks

Architecture

Layer 1 (Convolutional)

bias

paramaters on the first layer of the AlexNet alone!

Layer 1 (Convolutional)

Architecture

RELU Nonlinearity

¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡f(x) = tanh(x) or ¡ ¡ ¡ ¡ ¡ ¡f(x) = (1 + e-x)-1 ¡ ¡ ¡ ¡ ¡ ¡ ¡Very slow to train ¡

f(x) = max(0, x) Quick to train

A 4 layer CNN with ReLUs (solid line) converges six times faster than an equivalent network with tanh neurons (dashed line) on CIFAR-10 dataset

Architecture

RELU Nonlinearity

Architecture

Training on Multiple GPUs

Architecture

Training on Multiple GPUs

Architecture

Overlaping Pooling

Architecture

Local Response Normalization

respectively, compared to the non-overlapping scheme s = 2, z = 2

Architecture

Overlaping Pooling

Architecture

Architecture Overview

Outline

Reducing Overfitting

Data Augmentation Data Augmentation

 60 million parameters, 650,000 neurons à Overfits a lot.  Crop 224x224 patches (and their horizontal reflections.)

Reducing Overfitting

Data Augmentation Data Augmentation  At test time, average the predictions on the 10 patches.

Reducing Overfitting

 Softmax

 No need to calibrate to average the predictions over 10 patches.

L = 1 N −log e

e

∑

$ % & & & ' ( ) ) ) + λ Wk,l

∑

∑

∑

P(yi | xi;W ) Likelihood ¡

L = 1 N max(0, f (xi;W ) j − f (xi;W )yi + Δ) + λ Wk,l

∑

∑

& ' ( ) * +

∑

∑

Reducing Overfitting

Data Augmentation Data Augmentation  Change the intensity of RGB channels  add multiples of principle components

Ixy = [Ixy

αi ~ N(0, 0.1)

Reducing Overfitting

Dr Dropout

 With probability 0.5  last two 4096 fully-connected layers.

Stochastic Gradient Descent Learning

 The training took 5 to 6 days on two NVIDIA GTX 580 3GB GPUs.

Momentum ¡Update ¡

Results : ILSVRC-2010

Results : ILSVRC-2012

96 96 Convolutional Convolutional Ker Kernels nels

 11 x 11 x 3 size kernels.  top 48 kernels on GPU 1 : color-agnostic  bottom 48 kernels on GPU 2 : color-specific.

Eight ILSVRC-2010 test images

Five ILSVRC-2010 test images

Discussion

 Depth is really important. removing a single convolutional layer degrades the

performance.

Very Deep Convolutional Networks for Large-Scale Image Recognition. Technical report, 2014. à 16-layer model, 19-layer model. 7.3% top-5 test error

Discussion

Networks? ¡vs. ¡ Convolutonal ¡Networks? ¡

 60 million parameters, 650,000 neurons à Overfits a lot.  Crop 224x224 patches (and their horizontal reflections.)

Data Augmentation Data Augmentation  At test time, average the predictions on the 10 patches.

 Softmax

 No need to calibrate to average the predictions over 10 patches.

Data Augmentation Data Augmentation  Change the intensity of RGB channels  add multiples of principle components

 With probability 0.5  last two 4096 fully-connected layers.

 The training took 5 to 6 days on two NVIDIA GTX 580 3GB GPUs.

 11 x 11 x 3 size kernels.  top 48 kernels on GPU 1 : color-agnostic  bottom 48 kernels on GPU 2 : color-specific.

 Depth is really important. removing a single convolutional layer degrades the

 Classification on video. video sequences provide temporal structure missing