[PPT] - Neural Network Part 3: Convolutional Neural Networks CS PowerPoint Presentation

SLIDE 1

Neural Network Part 3: Convolutional Neural Networks

CS 760@UW-Madison

SLIDE 2

Goals for the lecture

you should understand the following concepts

convolutional neural networks (CNN)
convolution and its advantage
pooling and its advantage

2

SLIDE 3

Convolutional neural networks

Strong empirical application performance
Convolutional networks: neural networks that use convolution in

place of general matrix multiplication in at least one of their layers for a specific kind of weight matrix 𝑋 ℎ = 𝜏(𝑋𝑈𝑦 + 𝑐)

SLIDE 4

Convolution

SLIDE 5

Convolution: math formula

Given functions 𝑣(𝑢) and 𝑥(𝑢), their convolution is a function

𝑡 𝑢

Written as

𝑡 𝑢 = ∫ 𝑣 𝑏 𝑥 𝑢 − 𝑏 𝑒𝑏 𝑡 = 𝑣 ∗ 𝑥

r

𝑡 𝑢 = (𝑣 ∗ 𝑥)(𝑢)

SLIDE 6

Convolution: discrete version

Given array 𝑣𝑢 and 𝑥𝑢, their convolution is a function 𝑡𝑢
Written as
When 𝑣𝑢 or 𝑥𝑢 is not defined, assumed to be 0

𝑡𝑢 = ෍

𝑏=−∞ +∞

𝑣𝑏𝑥𝑢−𝑏 𝑡 = 𝑣 ∗ 𝑥

r

𝑡𝑢 = 𝑣 ∗ 𝑥 𝑢

SLIDE 7

Illustration 1

a b c d e f x y z xb+yc+zd 𝑥 = [z, y, x] 𝑣 = [a, b, c, d, e, f]

SLIDE 8

Illustration 1

a b c d e f x y z xc+yd+ze

SLIDE 9

Illustration 1

a b c d e f x y z xd+ye+zf

SLIDE 10

Illustration 1: boundary case

a b c d e f x y xe+yf

SLIDE 11

Illustration 1 as matrix multiplication

y z x y z x y z x y z x y z x y a b c d e f

SLIDE 12

Illustration 2: two dimensional case

a b c d e f g h i j k l w x y z wa + bx + ey + fz

SLIDE 13

Illustration 2

a b c d e f g h i j k l w x y z bw + cx + fy + gz wa + bx + ey + fz

SLIDE 14

Illustration 2

a b c d e f g h i j k l w x y z bw + cx + fy + gz wa + bx + ey + fz Kernel (or filter) Feature map Input

SLIDE 15

Illustration 2

All the units used the

same set of weights (kernel)

The units detect the

same “feature” but at different locations

[Figure from neuralnetworksanddeeplearning.com]

SLIDE 16

Advantage: sparse interaction

Figure from Deep Learning, by Goodfellow, Bengio, and Courville

Fully connected layer, 𝑛 × 𝑜 edges 𝑛 output nodes 𝑜 input nodes

SLIDE 17

Advantage: sparse interaction

Figure from Deep Learning, by Goodfellow, Bengio, and Courville

Convolutional layer, ≤ 𝑛 × 𝑙 edges 𝑛 output nodes 𝑜 input nodes 𝑙 kernel size

SLIDE 18

Advantage: sparse interaction

Figure from Deep Learning, by Goodfellow, Bengio, and Courville

Multiple convolutional layers: larger receptive field

SLIDE 19

Advantage: parameter sharing/weight tying

Figure from Deep Learning, by Goodfellow, Bengio, and Courville

The same kernel are used repeatedly. E.g., the black edge is the same weight in the kernel.

SLIDE 20

Advantage: equivariant representations

Equivariant: transforming the input = transforming the output
Example: input is an image, transformation is shifting
Convolution(shift(input)) = shift(Convolution(input))
Useful when care only about the existence of a pattern, rather

than the location

SLIDE 21

Pooling

SLIDE 22

Terminology

Figure from Deep Learning, by Goodfellow, Bengio, and Courville

SLIDE 23

Pooling

Summarizing the input (i.e., output the max of the input)

Figure from Deep Learning, by Goodfellow, Bengio, and Courville

SLIDE 24

Illustration

Each unit in a pooling

layer outputs a max, or similar function, of a subset of the units in the previous layer

[Figure from neuralnetworksanddeeplearning.com]

SLIDE 25

Advantage

Induce invariance

Figure from Deep Learning, by Goodfellow, Bengio, and Courville

SLIDE 26

Motivation from neuroscience

David Hubel and Torsten Wiesel studied early visual system in

human brain (V1 or primary visual cortex), and won Nobel prize for this

V1 properties
2D spatial arrangement
Simple cells: inspire convolution layers
Complex cells: inspire pooling layers

SLIDE 27

Example: LeNet

SLIDE 28

LeNet-5

Proposed in “Gradient-based learning applied to document

recognition” , by Yann LeCun, Leon Bottou, Yoshua Bengio and Patrick

Haffner, in Proceedings of the IEEE, 1998

Apply convolution on 2D images (MNIST) and use backpropagation
Structure: 2 convolutional layers (with pooling) + 3 fully connected layers
Input size: 32x32x1
Convolution kernel size: 5x5
Pooling: 2x2

SLIDE 29