Lecture 7: Convolutional Networks Justin Johnson September 23, - - PowerPoint PPT Presentation

β–Ά
lecture 7 convolutional networks
SMART_READER_LITE
LIVE PREVIEW

Lecture 7: Convolutional Networks Justin Johnson September 23, - - PowerPoint PPT Presentation

Sep 10, 2022 273 likes β€’1.26k views

Lecture 7: Convolutional Networks Justin Johnson September 23, 2020 Lecture 7 - 1 Reminder: A2 Due this Friday, 9/25/2020 Justin Johnson September 23, 2020 Lecture 7 - 2 Autograder Late Tokens - Our late policy is (from syllabus): - 3

slide-1
SLIDE 1

Justin Johnson September 23, 2020

Lecture 7: Convolutional Networks

Lecture 7 - 1

slide-2
SLIDE 2

Justin Johnson September 23, 2020

Reminder: A2

Lecture 7 - 2

Due this Friday, 9/25/2020

slide-3
SLIDE 3

Justin Johnson September 23, 2020

Autograder Late Tokens

Lecture 7 - 3

  • Our late policy is (from syllabus):
  • 3 free late days
  • After that, late work gets 25% penalty per day
  • This was difficult to implement in autograder.io
  • We will keep track of free late days and penalties
  • utside of autograder.io
  • We increased autograder.io late tokens to 1000

per student; this does not mean you can turn everything in a month late!

slide-4
SLIDE 4

Justin Johnson September 23, 2020

Last Time: Backpropagation

Lecture 7 - 4

x W

hinge loss

R

+

L

s (scores)

*

Represent complex expressions as computational graphs Forward pass computes outputs Backward pass computes gradients

f

Local gradients Upstream gradient Downstream gradients

During the backward pass, each node in the graph receives upstream gradients and multiplies them by local gradients to compute downstream gradients

slide-5
SLIDE 5

Justin Johnson September 23, 2020 Lecture 7 - 5 Input image (2, 2)

56 231 24 2 56 231 24 2

Stretch pixels into column (4,)

x h

W1

s

W2 Input: 3072 Hidden layer: 100 Output: 10

f(x,W) = Wx

Problem: So far our classifiers don’t respect the spatial structure of images!

slide-6
SLIDE 6

Justin Johnson September 23, 2020 Lecture 7 - 6 Input image (2, 2)

56 231 24 2 56 231 24 2

Stretch pixels into column (4,)

x h

W1

s

W2 Input: 3072 Hidden layer: 100 Output: 10

f(x,W) = Wx

Problem: So far our classifiers don’t respect the spatial structure of images! Solution: Define new computational nodes that operate on images!

slide-7
SLIDE 7

Justin Johnson September 23, 2020

Components of a Fully-Connected Network

Lecture 7 - 7

x h s

Fully-Connected Layers Activation Function

slide-8
SLIDE 8

Justin Johnson September 23, 2020

Components of a Convolutional Network

Lecture 7 - 8

Convolution Layers Pooling Layers

x h s

Fully-Connected Layers Activation Function Normalization

! 𝑦!,# = 𝑦!,# βˆ’ 𝜈# 𝜏

# $ + 𝜁

slide-9
SLIDE 9

Justin Johnson September 23, 2020

! 𝑦!,# = 𝑦!,# βˆ’ 𝜈# 𝜏

# $ + 𝜁

Components of a Convolutional Network

Lecture 7 - 9

Convolution Layers Pooling Layers

x h s

Fully-Connected Layers Activation Function Normalization

slide-10
SLIDE 10

Justin Johnson September 23, 2020

Fully-Connected Layer

Lecture 7 - 10

3072 1

32x32x3 image -> stretch to 3072 x 1

10 x 3072 weights Output Input 1 10

slide-11
SLIDE 11

Justin Johnson September 23, 2020

Fully-Connected Layer

Lecture 7 - 11

3072 1

32x32x3 image -> stretch to 3072 x 1

10 x 3072 weights Output Input

1 number: the result of taking a dot product between a row of W and the input (a 3072- dimensional dot product)

1 10

slide-12
SLIDE 12

Justin Johnson September 23, 2020

Convolution Layer

Lecture 7 - 12

32 3

3x32x32 image: preserve spatial structure

width depth / channels height 32

slide-13
SLIDE 13

Justin Johnson September 23, 2020

Convolution Layer

Lecture 7 - 13

32 3

3x32x32 image

width depth / channels

3x5x5 filter

Convolve the filter with the image i.e. β€œslide over the image spatially, computing dot products” height 32

slide-14
SLIDE 14

Justin Johnson September 23, 2020

Convolution Layer

Lecture 7 - 14

32 3

3x32x32 image

width height depth / channels

3x5x5 filter

Filters always extend the full depth of the input volume Convolve the filter with the image i.e. β€œslide over the image spatially, computing dot products” 32

slide-15
SLIDE 15

Justin Johnson September 23, 2020

Convolution Layer

Lecture 7 - 15

32 3

3x32x32 image 3x5x5 filter

32 1 number: the result of taking a dot product between the filter and a small 3x5x5 chunk of the image (i.e. 3*5*5 = 75-dimensional dot product + bias)

π‘₯!𝑦 + 𝑐

slide-16
SLIDE 16

Justin Johnson September 23, 2020

Convolution Layer

Lecture 7 - 16

32 3

3x32x32 image 3x5x5 filter

32 convolve (slide) over all spatial locations

1x28x28 activation map

1 28 28

slide-17
SLIDE 17

Justin Johnson September 23, 2020

Convolution Layer

Lecture 7 - 17

32 3

3x32x32 image 3x5x5 filter

32 convolve (slide) over all spatial locations

two 1x28x28 activation map

1 28 1 28 28

Consider repeating with a second (green) filter:

slide-18
SLIDE 18

Justin Johnson September 23, 2020

Convolution Layer

Lecture 7 - 18

32 3

3x32x32 image

32

6 activation maps, each 1x28x28 Consider 6 filters, each 3x5x5

Convolution Layer

6x3x5x5 filters Stack activations to get a 6x28x28 output image!

slide-19
SLIDE 19

Justin Johnson September 23, 2020

Convolution Layer

Lecture 7 - 19

32 3

3x32x32 image

32

6 activation maps, each 1x28x28 Also 6-dim bias vector:

Convolution Layer

6x3x5x5 filters Stack activations to get a 6x28x28 output image!

slide-20
SLIDE 20

Justin Johnson September 23, 2020

Convolution Layer

Lecture 7 - 20

32 3

3x32x32 image

32

28x28 grid, at each point a 6-dim vector Also 6-dim bias vector:

Convolution Layer

6x3x5x5 filters Stack activations to get a 6x28x28 output image!

slide-21
SLIDE 21

Justin Johnson September 23, 2020

Convolution Layer

Lecture 7 - 21

32 3

2x3x32x32 Batch of images

32

2x6x28x28 Batch of outputs Also 6-dim bias vector:

Convolution Layer

6x3x5x5 filters

slide-22
SLIDE 22

Justin Johnson September 23, 2020

Convolution Layer

Lecture 7 - 22

W Cin

N x Cin x H x W Batch of images

H

N x Cout x H’ x W’ Batch of outputs Also Cout-dim bias vector:

Convolution Layer

Coutx Cinx Kw x Kh filters

Cout

slide-23
SLIDE 23

Justin Johnson September 23, 2020 Lecture 7 - 23

32 32 3

W1: 6x3x5x5 b1: 6

28 28 6 10 26 26

….

Stacking Convolutions

Input: N x 3 x 32 x 32 First hidden layer: N x 6 x 28 x 28

W2: 10x6x3x3 b2: 10

Second hidden layer: N x 10 x 26 x 26

Conv Conv Conv

W3: 12x10x3x3 b3: 12

slide-24
SLIDE 24

Justin Johnson September 23, 2020 Lecture 7 - 24

32 32 3

W1: 6x3x5x5 b1: 6

28 28 6 10 26 26

….

Stacking Convolutions

Input: N x 3 x 32 x 32 First hidden layer: N x 6 x 28 x 28

W2: 10x6x3x3 b2: 10

Second hidden layer: N x 10 x 26 x 26

Conv Conv Conv

W3: 12x10x3x3 b3: 12

Q: What happens if we stack two convolution layers?

slide-25
SLIDE 25

Justin Johnson September 23, 2020 Lecture 7 - 25

32 32 3

W1: 6x3x5x5 b1: 6

28 28 6 10 26 26

….

Stacking Convolutions

Input: N x 3 x 32 x 32 First hidden layer: N x 6 x 28 x 28

W2: 10x6x3x3 b2: 10

Second hidden layer: N x 10 x 26 x 26 Conv

W3: 12x10x3x3 b3: 12

Q: What happens if we stack two convolution layers? A: We get another convolution! (Recall y=W2W1x is a linear classifier) ReLU Conv ReLU Conv ReLU

slide-26
SLIDE 26

Justin Johnson September 23, 2020 Lecture 7 - 26

32 32 3

W1: 6x3x5x5 b1: 6

28 28 6 10 26 26

….

What do convolutional filters learn?

Input: N x 3 x 32 x 32 First hidden layer: N x 6 x 28 x 28

W2: 10x6x3x3 b2: 10

Second hidden layer: N x 10 x 26 x 26 Conv

W3: 12x10x3x3 b3: 12

ReLU Conv ReLU Conv ReLU

slide-27
SLIDE 27

Justin Johnson September 23, 2020 Lecture 7 - 27

32 32 3

W1: 6x3x5x5 b1: 6

28 28 6

What do convolutional filters learn?

Input: N x 3 x 32 x 32 First hidden layer: N x 6 x 28 x 28 Conv ReLU Linear classifier: One template per class

slide-28
SLIDE 28

Justin Johnson September 23, 2020 Lecture 7 - 28

32 32 3

W1: 6x3x5x5 b1: 6

28 28 6

What do convolutional filters learn?

Input: N x 3 x 32 x 32 First hidden layer: N x 6 x 28 x 28 Conv ReLU MLP: Bank of whole-image templates

slide-29
SLIDE 29

Justin Johnson September 23, 2020 Lecture 7 - 29

32 32 3

W1: 6x3x5x5 b1: 6

28 28 6

What do convolutional filters learn?

Input: N x 3 x 32 x 32 First hidden layer: N x 6 x 28 x 28 Conv ReLU First-layer conv filters: local image templates (Often learns oriented edges, opposing colors) AlexNet: 64 filters, each 3x11x11

slide-30
SLIDE 30

Justin Johnson September 23, 2020 Lecture 7 - 30

32 32 3

W1: 6x3x5x5 b1: 6

28 28 6

A closer look at spatial dimensions

Input: N x 3 x 32 x 32 First hidden layer: N x 6 x 28 x 28 Conv ReLU

slide-31
SLIDE 31

Justin Johnson September 23, 2020 Lecture 7 - 31

A closer look at spatial dimensions

7 7

Input: 7x7 Filter: 3x3

slide-32
SLIDE 32

Justin Johnson September 23, 2020 Lecture 7 - 32

A closer look at spatial dimensions

7 7

Input: 7x7 Filter: 3x3

slide-33
SLIDE 33

Justin Johnson September 23, 2020 Lecture 7 - 33

A closer look at spatial dimensions

7 7

Input: 7x7 Filter: 3x3

slide-34
SLIDE 34

Justin Johnson September 23, 2020 Lecture 7 - 34

A closer look at spatial dimensions

7 7

Input: 7x7 Filter: 3x3

slide-35
SLIDE 35

Justin Johnson September 23, 2020 Lecture 7 - 35

A closer look at spatial dimensions

7 7

Input: 7x7 Filter: 3x3 Output: 5x5

slide-36
SLIDE 36

Justin Johnson September 23, 2020 Lecture 7 - 36

A closer look at spatial dimensions

7 7

Input: 7x7 Filter: 3x3 Output: 5x5

In general: Input: W Filter: K Output: W – K + 1

Problem: Feature maps β€œshrink” with each layer!

slide-37
SLIDE 37

Justin Johnson September 23, 2020 Lecture 7 - 37

A closer look at spatial dimensions

Input: 7x7 Filter: 3x3 Output: 5x5

In general: Input: W Filter: K Output: W – K + 1

Problem: Feature maps β€œshrink” with each layer! Solution: padding Add zeros around the input

slide-38
SLIDE 38

Justin Johnson September 23, 2020 Lecture 7 - 38

A closer look at spatial dimensions

Input: 7x7 Filter: 3x3 Output: 5x5

In general: Input: W Filter: K Padding: P Output: W – K + 1 + 2P

Very common: Set P = (K – 1) / 2 to make output have same size as input!

slide-39
SLIDE 39

Justin Johnson September 23, 2020 Lecture 7 - 39

Receptive Fields

Input Output

For convolution with kernel size K, each element in the

  • utput depends on a K x K receptive field in the input
slide-40
SLIDE 40

Justin Johnson September 23, 2020 Lecture 7 - 40

Receptive Fields

Input Output

Each successive convolution adds K – 1 to the receptive field size With L layers the receptive field size is 1 + L * (K – 1)

Be careful – ”receptive field in the input” vs β€œreceptive field in the previous layer” Hopefully clear from context!

slide-41
SLIDE 41

Justin Johnson September 23, 2020 Lecture 7 - 41

Receptive Fields

Input Output

Each successive convolution adds K – 1 to the receptive field size With L layers the receptive field size is 1 + L * (K – 1) Problem: For large images we need many layers for each output to β€œsee” the whole image image

slide-42
SLIDE 42

Justin Johnson September 23, 2020 Lecture 7 - 42

Receptive Fields

Input Output

Each successive convolution adds K – 1 to the receptive field size With L layers the receptive field size is 1 + L * (K – 1) Problem: For large images we need many layers for each output to β€œsee” the whole image image Solution: Downsample inside the network

slide-43
SLIDE 43

Justin Johnson September 23, 2020 Lecture 7 - 43

Strided Convolution

Input: 7x7 Filter: 3x3 Stride: 2

slide-44
SLIDE 44

Justin Johnson September 23, 2020 Lecture 7 - 44

Strided Convolution

Input: 7x7 Filter: 3x3 Stride: 2

slide-45
SLIDE 45

Justin Johnson September 23, 2020 Lecture 7 - 45

Strided Convolution

Input: 7x7 Filter: 3x3 Stride: 2 Output: 3x3

slide-46
SLIDE 46

Justin Johnson September 23, 2020 Lecture 7 - 46

Strided Convolution

Input: 7x7 Filter: 3x3 Stride: 2 Output: 3x3

In general: Input: W Filter: K Padding: P Stride: S Output: (W – K + 2P) / S + 1

slide-47
SLIDE 47

Justin Johnson September 23, 2020

Convolution Example

Lecture 7 - 47

Input volume: 3 x 32 x 32 10 5x5 filters with stride 1, pad 2 Output volume size: ?

slide-48
SLIDE 48

Justin Johnson September 23, 2020

Convolution Example

Lecture 7 - 48

Input volume: 3 x 32 x 32 10 5x5 filters with stride 1, pad 2 Output volume size: (32+2*2-5)/1+1 = 32 spatially, so 10 x 32 x 32

slide-49
SLIDE 49

Justin Johnson September 23, 2020

Convolution Example

Lecture 7 - 49

Input volume: 3 x 32 x 32 10 5x5 filters with stride 1, pad 2 Output volume size: 10 x 32 x 32 Number of learnable parameters: ?

slide-50
SLIDE 50

Justin Johnson September 23, 2020

Convolution Example

Lecture 7 - 50

Input volume: 3 x 32 x 32 10 5x5 filters with stride 1, pad 2 Output volume size: 10 x 32 x 32 Number of learnable parameters: 760 Parameters per filter: 3*5*5 + 1 (for bias) = 76 10 filters, so total is 10 * 76 = 760

slide-51
SLIDE 51

Justin Johnson September 23, 2020

Convolution Example

Lecture 7 - 51

Input volume: 3 x 32 x 32 10 5x5 filters with stride 1, pad 2 Output volume size: 10 x 32 x 32 Number of learnable parameters: 760 Number of multiply-add operations: ?

slide-52
SLIDE 52

Justin Johnson September 23, 2020

Convolution Example

Lecture 7 - 52

Input volume: 3 x 32 x 32 10 5x5 filters with stride 1, pad 2 Output volume size: 10 x 32 x 32 Number of learnable parameters: 760 Number of multiply-add operations: 768,000 10*32*32 = 10,240 outputs; each output is the inner product

  • f two 3x5x5 tensors (75 elems); total = 75*10240 = 768K
slide-53
SLIDE 53

Justin Johnson September 23, 2020

Example: 1x1 Convolution

Lecture 7 - 53

64 56 56 1x1 CONV with 32 filters 32 56 56 (each filter has size 1x1x64, and performs a 64- dimensional dot product)

slide-54
SLIDE 54

Justin Johnson September 23, 2020

Example: 1x1 Convolution

Lecture 7 - 54

64 56 56 1x1 CONV with 32 filters 32 56 56 (each filter has size 1x1x64, and performs a 64- dimensional dot product)

Lin et al, β€œNetwork in Network”, ICLR 2014

Stacking 1x1 conv layers gives MLP operating on each input position

slide-55
SLIDE 55

Justin Johnson September 23, 2020

Convolution Summary

Lecture 7 - 55

Input: Cin x H x W Hyperparameters:

  • Kernel size: KH x KW
  • Number filters: Cout
  • Padding: P
  • Stride: S

Weight matrix: Cout x Cin x KH x KW giving Cout filters of size Cin x KH x KW Bias vector: Cout Output size: Cout x H’ x W’ where:

  • H’ = (H – K + 2P) / S + 1
  • W’ = (W – K + 2P) / S + 1
slide-56
SLIDE 56

Justin Johnson September 23, 2020

Convolution Summary

Lecture 7 - 56

Input: Cin x H x W Hyperparameters:

  • Kernel size: KH x KW
  • Number filters: Cout
  • Padding: P
  • Stride: S

Weight matrix: Cout x Cin x KH x KW giving Cout filters of size Cin x KH x KW Bias vector: Cout Output size: Cout x H’ x W’ where:

  • H’ = (H – K + 2P) / S + 1
  • W’ = (W – K + 2P) / S + 1

Common settings: KH = KW (Small square filters) P = (K – 1) / 2 (”Same” padding) Cin, Cout = 32, 64, 128, 256 (powers of 2) K = 3, P = 1, S = 1 (3x3 conv) K = 5, P = 2, S = 1 (5x5 conv) K = 1, P = 0, S = 1 (1x1 conv) K = 3, P = 1, S = 2 (Downsample by 2)

slide-57
SLIDE 57

Justin Johnson September 23, 2020

Other types of convolution

Lecture 7 - 57

So far: 2D Convolution Cin W H

Input: Cin x H x W Weights: Cout x Cin x K x K

slide-58
SLIDE 58

Justin Johnson September 23, 2020

Other types of convolution

Lecture 7 - 58

So far: 2D Convolution 1D Convolution Cin W H

Input: Cin x H x W Weights: Cout x Cin x K x K

Cin W

Input: Cin x W Weights: Cout x Cin x K

slide-59
SLIDE 59

Justin Johnson September 23, 2020

Other types of convolution

Lecture 7 - 59

So far: 2D Convolution 3D Convolution Cin W H

Input: Cin x H x W Weights: Cout x Cin x K x K

Cin-dim vector at each point in the volume W D H

Input: Cin x H x W x D Weights: Cout x Cin x K x K x K

slide-60
SLIDE 60

Justin Johnson September 23, 2020 Lecture 7 - 60

PyTorch Convolution Layer

slide-61
SLIDE 61

Justin Johnson September 23, 2020 Lecture 7 - 61

PyTorch Convolution Layers

slide-62
SLIDE 62

Justin Johnson September 23, 2020

Components of a Convolutional Network

Lecture 7 - 62

Convolution Layers Pooling Layers

x h s

Fully-Connected Layers Activation Function Normalization

! 𝑦!,# = 𝑦!,# βˆ’ 𝜈# 𝜏

# $ + 𝜁

slide-63
SLIDE 63

Justin Johnson September 23, 2020

Po Pooling Layers rs: Another way to downsample

Lecture 7 - 63

Hyperparameters: Kernel Size Stride Pooling function

64 x 224 x 224 64 x 112 x 112

slide-64
SLIDE 64

Justin Johnson September 23, 2020

Max Pooling

Lecture 7 - 64

1 1 2 4 5 6 7 8 3 2 1 1 2 3 4 Single depth slice x y

Max pooling with 2x2 kernel size and stride 2

6 8 3 4 Introduces invariance to small spatial shifts No learnable parameters!

64 x 224 x 224

slide-65
SLIDE 65

Justin Johnson September 23, 2020

Pooling Summary

Lecture 7 - 65

Input: C x H x W Hyperparameters:

  • Kernel size: K
  • Stride: S
  • Pooling function (max, avg)

Output: C x H’ x W’ where

  • H’ = (H – K) / S + 1
  • W’ = (W – K) / S + 1

Learnable parameters: None!

Common settings: max, K = 2, S = 2 max, K = 3, S = 2 (AlexNet)

slide-66
SLIDE 66

Justin Johnson September 23, 2020

Components of a Convolutional Network

Lecture 7 - 66

Convolution Layers Pooling Layers

x h s

Fully-Connected Layers Activation Function Normalization

! 𝑦!,# = 𝑦!,# βˆ’ 𝜈# 𝜏

# $ + 𝜁

slide-67
SLIDE 67

Justin Johnson September 23, 2020

Convolutional Networks

Lecture 7 - 67

Lecun et al, β€œGradient-based learning applied to document recognition”, 1998

Classic architecture: [Conv, ReLU, Pool] x N, flatten, [FC, ReLU] x N, FC Example: LeNet-5

slide-68
SLIDE 68

Justin Johnson September 23, 2020

Example: LeNet-5

Lecture 7 - 68

Layer Output Size Weight Size Input 1 x 28 x 28 Conv (Cout=20, K=5, P=2, S=1) 20 x 28 x 28 20 x 1 x 5 x 5 ReLU 20 x 28 x 28 MaxPool(K=2, S=2) 20 x 14 x 14 Conv (Cout=50, K=5, P=2, S=1) 50 x 14 x 14 50 x 20 x 5 x 5 ReLU 50 x 14 x 14 MaxPool(K=2, S=2) 50 x 7 x 7 Flatten 2450 Linear (2450 -> 500) 500 2450 x 500 ReLU 500 Linear (500 -> 10) 10 500 x 10

Lecun et al, β€œGradient-based learning applied to document recognition”, 1998

slide-69
SLIDE 69

Justin Johnson September 23, 2020

Example: LeNet-5

Lecture 7 - 69

Layer Output Size Weight Size Input 1 x 28 x 28 Conv (Cout=20, K=5, P=2, S=1) 20 x 28 x 28 20 x 1 x 5 x 5 ReLU 20 x 28 x 28 MaxPool(K=2, S=2) 20 x 14 x 14 Conv (Cout=50, K=5, P=2, S=1) 50 x 14 x 14 50 x 20 x 5 x 5 ReLU 50 x 14 x 14 MaxPool(K=2, S=2) 50 x 7 x 7 Flatten 2450 Linear (2450 -> 500) 500 2450 x 500 ReLU 500 Linear (500 -> 10) 10 500 x 10

Lecun et al, β€œGradient-based learning applied to document recognition”, 1998

slide-70
SLIDE 70

Justin Johnson September 23, 2020

Example: LeNet-5

Lecture 7 - 70

Layer Output Size Weight Size Input 1 x 28 x 28 Conv (Cout=20, K=5, P=2, S=1) 20 x 28 x 28 20 x 1 x 5 x 5 ReLU 20 x 28 x 28 MaxPool(K=2, S=2) 20 x 14 x 14 Conv (Cout=50, K=5, P=2, S=1) 50 x 14 x 14 50 x 20 x 5 x 5 ReLU 50 x 14 x 14 MaxPool(K=2, S=2) 50 x 7 x 7 Flatten 2450 Linear (2450 -> 500) 500 2450 x 500 ReLU 500 Linear (500 -> 10) 10 500 x 10

Lecun et al, β€œGradient-based learning applied to document recognition”, 1998

slide-71
SLIDE 71

Justin Johnson September 23, 2020

Example: LeNet-5

Lecture 7 - 71

Layer Output Size Weight Size Input 1 x 28 x 28 Conv (Cout=20, K=5, P=2, S=1) 20 x 28 x 28 20 x 1 x 5 x 5 ReLU 20 x 28 x 28 MaxPool(K=2, S=2) 20 x 14 x 14 Conv (Cout=50, K=5, P=2, S=1) 50 x 14 x 14 50 x 20 x 5 x 5 ReLU 50 x 14 x 14 MaxPool(K=2, S=2) 50 x 7 x 7 Flatten 2450 Linear (2450 -> 500) 500 2450 x 500 ReLU 500 Linear (500 -> 10) 10 500 x 10

Lecun et al, β€œGradient-based learning applied to document recognition”, 1998

slide-72
SLIDE 72

Justin Johnson September 23, 2020

Example: LeNet-5

Lecture 7 - 72

Layer Output Size Weight Size Input 1 x 28 x 28 Conv (Cout=20, K=5, P=2, S=1) 20 x 28 x 28 20 x 1 x 5 x 5 ReLU 20 x 28 x 28 MaxPool(K=2, S=2) 20 x 14 x 14 Conv (Cout=50, K=5, P=2, S=1) 50 x 14 x 14 50 x 20 x 5 x 5 ReLU 50 x 14 x 14 MaxPool(K=2, S=2) 50 x 7 x 7 Flatten 2450 Linear (2450 -> 500) 500 2450 x 500 ReLU 500 Linear (500 -> 10) 10 500 x 10

Lecun et al, β€œGradient-based learning applied to document recognition”, 1998

slide-73
SLIDE 73

Justin Johnson September 23, 2020

Example: LeNet-5

Lecture 7 - 73

Layer Output Size Weight Size Input 1 x 28 x 28 Conv (Cout=20, K=5, P=2, S=1) 20 x 28 x 28 20 x 1 x 5 x 5 ReLU 20 x 28 x 28 MaxPool(K=2, S=2) 20 x 14 x 14 Conv (Cout=50, K=5, P=2, S=1) 50 x 14 x 14 50 x 20 x 5 x 5 ReLU 50 x 14 x 14 MaxPool(K=2, S=2) 50 x 7 x 7 Flatten 2450 Linear (2450 -> 500) 500 2450 x 500 ReLU 500 Linear (500 -> 10) 10 500 x 10

Lecun et al, β€œGradient-based learning applied to document recognition”, 1998

slide-74
SLIDE 74

Justin Johnson September 23, 2020

Example: LeNet-5

Lecture 7 - 74

Layer Output Size Weight Size Input 1 x 28 x 28 Conv (Cout=20, K=5, P=2, S=1) 20 x 28 x 28 20 x 1 x 5 x 5 ReLU 20 x 28 x 28 MaxPool(K=2, S=2) 20 x 14 x 14 Conv (Cout=50, K=5, P=2, S=1) 50 x 14 x 14 50 x 20 x 5 x 5 ReLU 50 x 14 x 14 MaxPool(K=2, S=2) 50 x 7 x 7 Flatten 2450 Linear (2450 -> 500) 500 2450 x 500 ReLU 500 Linear (500 -> 10) 10 500 x 10

Lecun et al, β€œGradient-based learning applied to document recognition”, 1998

slide-75
SLIDE 75

Justin Johnson September 23, 2020

Example: LeNet-5

Lecture 7 - 75

Layer Output Size Weight Size Input 1 x 28 x 28 Conv (Cout=20, K=5, P=2, S=1) 20 x 28 x 28 20 x 1 x 5 x 5 ReLU 20 x 28 x 28 MaxPool(K=2, S=2) 20 x 14 x 14 Conv (Cout=50, K=5, P=2, S=1) 50 x 14 x 14 50 x 20 x 5 x 5 ReLU 50 x 14 x 14 MaxPool(K=2, S=2) 50 x 7 x 7 Flatten 2450 Linear (2450 -> 500) 500 2450 x 500 ReLU 500 Linear (500 -> 10) 10 500 x 10

Lecun et al, β€œGradient-based learning applied to document recognition”, 1998

slide-76
SLIDE 76

Justin Johnson September 23, 2020

Example: LeNet-5

Lecture 7 - 76

Layer Output Size Weight Size Input 1 x 28 x 28 Conv (Cout=20, K=5, P=2, S=1) 20 x 28 x 28 20 x 1 x 5 x 5 ReLU 20 x 28 x 28 MaxPool(K=2, S=2) 20 x 14 x 14 Conv (Cout=50, K=5, P=2, S=1) 50 x 14 x 14 50 x 20 x 5 x 5 ReLU 50 x 14 x 14 MaxPool(K=2, S=2) 50 x 7 x 7 Flatten 2450 Linear (2450 -> 500) 500 2450 x 500 ReLU 500 Linear (500 -> 10) 10 500 x 10

Lecun et al, β€œGradient-based learning applied to document recognition”, 1998

As we go through the network: Spatial size decreases (using pooling or strided conv) Number of channels increases (total β€œvolume” is preserved!)

slide-77
SLIDE 77

Justin Johnson September 23, 2020

Problem: Deep Networks very hard to train!

Lecture 7 - 77

slide-78
SLIDE 78

Justin Johnson September 23, 2020

Components of a Convolutional Network

Lecture 7 - 78

Convolution Layers Pooling Layers

x h s

Fully-Connected Layers Activation Function Normalization

! 𝑦!,# = 𝑦!,# βˆ’ 𝜈# 𝜏

# $ + 𝜁

slide-79
SLIDE 79

Justin Johnson September 23, 2020

Batch Normalization

Lecture 7 - 79

Ioffe and Szegedy, β€œBatch normalization: Accelerating deep network training by reducing internal covariate shift”, ICML 2015

Idea: β€œNormalize” the outputs of a layer so they have zero mean and unit variance Why? Helps reduce β€œinternal covariate shift”, improves optimization We can normalize a batch of activations like this: This is a differentiable function, so we can use it as an operator in our networks and backprop through it!

! 𝑦 = 𝑦 βˆ’ 𝐹 𝑦 π‘Šπ‘π‘  𝑦

slide-80
SLIDE 80

Justin Johnson September 23, 2020

Batch Normalization

Lecture 7 - 80

Ioffe and Szegedy, β€œBatch normalization: Accelerating deep network training by reducing internal covariate shift”, ICML 2015

Input:

Per-channel mean, shape is D Normalized x, Shape is N x D

X

N D

Per-channel std, shape is D

𝑦 ∈ ℝ!Γ—#

𝜈# = 1 𝑂 +

!%& '

𝑦!,# 𝜏

# $ = 1

𝑂 +

!%& '

𝑦!,# βˆ’ 𝜈#

$

! 𝑦!,# = 𝑦!,# βˆ’ 𝜈# 𝜏

# $ + 𝜁

slide-81
SLIDE 81

Justin Johnson September 23, 2020

Batch Normalization

Lecture 7 - 81

Ioffe and Szegedy, β€œBatch normalization: Accelerating deep network training by reducing internal covariate shift”, ICML 2015

Input:

X

N D

Problem: What if zero-mean, unit variance is too hard of a constraint?

Per-channel mean, shape is D Normalized x, Shape is N x D Per-channel std, shape is D

𝜈# = 1 𝑂 +

!%& '

𝑦!,# 𝜏

# $ = 1

𝑂 +

!%& '

𝑦!,# βˆ’ 𝜈#

$

! 𝑦!,# = 𝑦!,# βˆ’ 𝜈# 𝜏

# $ + 𝜁

𝑦 ∈ ℝ!Γ—#

slide-82
SLIDE 82

Justin Johnson September 23, 2020

Batch Normalization

Lecture 7 - 82

Learnable scale and shift parameters:

Output, Shape is N x D

Learning 𝛿 = 𝜏, 𝛾 = 𝜈 will recover the identity function (in expectation) Input: 𝑧!,# = 𝛿# ! 𝑦!,# + 𝛾#

Per-channel mean, shape is D Normalized x, Shape is N x D Per-channel std, shape is D

𝜈# = 1 𝑂 +

!%& '

𝑦!,# 𝜏

# $ = 1

𝑂 +

!%& '

𝑦!,# βˆ’ 𝜈#

$

! 𝑦!,# = 𝑦!,# βˆ’ 𝜈# 𝜏

# $ + 𝜁

𝑦 ∈ ℝ!Γ—# 𝛿, 𝛾 ∈ ℝ#

slide-83
SLIDE 83

Justin Johnson September 23, 2020

Batch Normalization

Lecture 7 - 83

Learnable scale and shift parameters:

Output, Shape is N x D

Learning 𝛿 = 𝜏, 𝛾 = 𝜈 will recover the identity function (in expectation) Input: 𝑧!,# = 𝛿# ! 𝑦!,# + 𝛾#

Per-channel mean, shape is D Normalized x, Shape is N x D Per-channel std, shape is D

𝜈# = 1 𝑂 +

!%& '

𝑦!,# 𝜏

# $ = 1

𝑂 +

!%& '

𝑦!,# βˆ’ 𝜈#

$

! 𝑦!,# = 𝑦!,# βˆ’ 𝜈# 𝜏

# $ + 𝜁

𝑦 ∈ ℝ!Γ—# 𝛿, 𝛾 ∈ ℝ#

Problem: Estimates depend on minibatch; can’t do this at test-time!

slide-84
SLIDE 84

Justin Johnson September 23, 2020

Batch Normalization: Test-Time

Lecture 7 - 84

Learnable scale and shift parameters: Input: Learning 𝛿 = 𝜏, 𝛾 = 𝜈 will recover the identity function (in expectation)

𝑦 ∈ ℝ!Γ—# 𝛿, 𝛾 ∈ ℝ#

Output, Shape is N x D

𝑧!,# = 𝛿# ! 𝑦!,# + 𝛾#

Per-channel mean, shape is D Normalized x, Shape is N x D Per-channel std, shape is D

𝜈# = 1 𝑂 +

!%& '

𝑦!,# 𝜏

# $ = 1

𝑂 +

!%& '

𝑦!,# βˆ’ 𝜈#

$

! 𝑦!,# = 𝑦!,# βˆ’ 𝜈# 𝜏

# $ + 𝜁

(Running) average of values seen during training

(Running) average of values seen during training

slide-85
SLIDE 85

Justin Johnson September 23, 2020

Batch Normalization: Test-Time

Lecture 7 - 85

Learnable scale and shift parameters: Input: 𝑦 ∈ ℝ!Γ—#

𝛿, 𝛾 ∈ ℝ#

Output, Shape is N x D

𝑧!,# = 𝛿# ! 𝑦!,# + 𝛾#

Per-channel mean, shape is D Normalized x, Shape is N x D Per-channel std, shape is D

𝜈# = 1 𝑂 +

!%& '

𝑦!,# 𝜏

# $ = 1

𝑂 +

!%& '

𝑦!,# βˆ’ 𝜈#

$

! 𝑦!,# = 𝑦!,# βˆ’ 𝜈# 𝜏

# $ + 𝜁

(Running) average of values seen during training

(Running) average of values seen during training

During testing batchnorm becomes a linear operator! Can be fused with the previous fully-connected or conv layer

slide-86
SLIDE 86

Justin Johnson September 23, 2020

𝑦 ∢ 𝑂 Γ— 𝐸 𝜈, 𝜏 ∢ 1 Γ— 𝐸 𝛿, 𝛾 ∢ 1 Γ— 𝐸 𝑧 = 𝑦 βˆ’ 𝜈 𝜏 𝛿 + 𝛾 Batch Normalization for ConvNets

Lecture 7 - 86

𝑦 ∢ 𝑂 Γ— 𝐷 Γ— 𝐼 Γ— 𝑋 𝜈, 𝜏 ∢ 1 Γ— 𝐷 Γ— 1 Γ— 1 𝛿, 𝛾 ∢ 1 Γ— 𝐷 Γ— 1 Γ— 1 𝑧 = 𝑦 βˆ’ 𝜈 𝜏 𝛿 + 𝛾

Normalize

Batch Normalization for fully-connected networks Batch Normalization for convolutional networks (Spatial Batchnorm, BatchNorm2D)

Normalize

slide-87
SLIDE 87

Justin Johnson September 23, 2020

Batch Normalization

Lecture 7 - 87

FC BN tanh FC BN tanh

Usually inserted after Fully Connected

  • r Convolutional layers, and before

nonlinearity.

Ioffe and Szegedy, β€œBatch normalization: Accelerating deep network training by reducing internal covariate shift”, ICML 2015

! 𝑦 = 𝑦 βˆ’ 𝐹 𝑦 π‘Šπ‘π‘  𝑦

slide-88
SLIDE 88

Justin Johnson September 23, 2020

Batch Normalization

Lecture 7 - 88

FC BN tanh FC BN tanh

  • Makes deep networks much easier to train!
  • Allows higher learning rates, faster convergence
  • Networks become more robust to initialization
  • Acts as regularization during training
  • Zero overhead at test-time: can be fused with conv!

Training iterations ImageNet accuracy

Ioffe and Szegedy, β€œBatch normalization: Accelerating deep network training by reducing internal covariate shift”, ICML 2015

slide-89
SLIDE 89

Justin Johnson September 23, 2020

Batch Normalization

Lecture 7 - 89

FC BN tanh FC BN tanh

  • Makes deep networks much easier to train!
  • Allows higher learning rates, faster convergence
  • Networks become more robust to initialization
  • Acts as regularization during training
  • Zero overhead at test-time: can be fused with conv!
  • Not well-understood theoretically (yet)
  • Behaves differently during training and testing: this

is a very common source of bugs!

Ioffe and Szegedy, β€œBatch normalization: Accelerating deep network training by reducing internal covariate shift”, ICML 2015

slide-90
SLIDE 90

Justin Johnson September 23, 2020

𝑦 ∢ 𝑂 Γ— 𝐸 𝜈, 𝜏 ∢ 1 Γ— 𝐸 𝛿, 𝛾 ∢ 1 Γ— 𝐸 𝑧 = 𝑦 βˆ’ 𝜈 𝜏 𝛿 + 𝛾 𝑦 ∢ 𝑂 Γ— 𝐸 𝜈, 𝜏 ∢ 𝑂 Γ— 1 𝛿, 𝛾 ∢ 1 Γ— 𝐸 𝑧 = 𝑦 βˆ’ 𝜈 𝜏 𝛿 + 𝛾 Layer Normalization

Lecture 7 - 90

Normalize

Batch Normalization for fully-connected networks

Normalize

Layer Normalization for fully- connected networks Same behavior at train and test! Used in RNNs, Transformers

slide-91
SLIDE 91

Justin Johnson September 23, 2020

Instance Normalization

Lecture 7 - 91

𝑦 ∢ 𝑂 Γ— 𝐷 Γ— 𝐼 Γ— 𝑋 𝜈, 𝜏 ∢ 1 Γ— 𝐷 Γ— 1 Γ— 1 𝛿, 𝛾 ∢ 1 Γ— 𝐷 Γ— 1 Γ— 1 𝑧 = 𝑦 βˆ’ 𝜈 𝜏 𝛿 + 𝛾

Batch Normalization for convolutional networks

Normalize

𝑦 ∢ 𝑂 Γ— 𝐷 Γ— 𝐼 Γ— 𝑋 𝜈, 𝜏 ∢ 𝑂 Γ— 𝐷 Γ— 1 Γ— 1 𝛿, 𝛾 ∢ 1 Γ— 𝐷 Γ— 1 Γ— 1 𝑧 = 𝑦 βˆ’ 𝜈 𝜏 𝛿 + 𝛾

Instance Normalization for convolutional networks

Normalize

slide-92
SLIDE 92

Justin Johnson September 23, 2020

Comparison of Normalization Layers

Lecture 7 - 92

Wu and He, β€œGroup Normalization”, ECCV 2018

slide-93
SLIDE 93

Justin Johnson September 23, 2020

Group Normalization

Lecture 7 - 93

Wu and He, β€œGroup Normalization”, ECCV 2018

slide-94
SLIDE 94

Justin Johnson September 23, 2020

Components of a Convolutional Network

Lecture 7 - 94

Convolution Layers Pooling Layers

x h s

Fully-Connected Layers Activation Function Normalization

! 𝑦!,# = 𝑦!,# βˆ’ 𝜈# 𝜏

# $ + 𝜁

slide-95
SLIDE 95

Justin Johnson September 23, 2020

Components of a Convolutional Network

Lecture 7 - 95

Convolution Layers Pooling Layers

x h s

Fully-Connected Layers Activation Function Normalization

Most computationally expensive!

! 𝑦!,# = 𝑦!,# βˆ’ 𝜈# 𝜏

# $ + 𝜁

slide-96
SLIDE 96

Justin Johnson September 23, 2020 Lecture 7 - 96

Summary: Components of a Convolutional Network

Convolution Layers Pooling Layers

x h s

Fully-Connected Layers Activation Function Normalization

! 𝑦!,# = 𝑦!,# βˆ’ 𝜈# 𝜏

# $ + 𝜁

slide-97
SLIDE 97

Justin Johnson September 23, 2020 Lecture 7 - 97

Summary: Components of a Convolutional Network

Problem: What is the right way to combine all these components?

slide-98
SLIDE 98

Justin Johnson September 23, 2020

Next time: CNN Architectures

Lecture 7 - 98