[PPT] - Neural Network Part 3: Convolutional Neural Networks Yingyu Liang PowerPoint Presentation

SLIDE 1

Neural Network Part 3: Convolutional Neural Networks

Yingyu Liang Computer Sciences 760 Fall 2017

http://pages.cs.wisc.edu/~yliang/cs760/

Some of the slides in these lectures have been adapted/borrowed from materials developed by Mark Craven, David Page, Jude Shavlik, Tom Mitchell, Nina Balcan, Matt Gormley, Elad Hazan, Tom Dietterich, Pedro Domingos, and Kaiming He.

SLIDE 2

Goals for the lecture

you should understand the following concepts

convolutional neural networks (CNN)
convolution and its advantage
pooling and its advantage

2

SLIDE 3

Convolutional neural networks

Strong empirical application performance
Convolutional networks: neural networks that use convolution in

place of general matrix multiplication in at least one of their layers for a specific kind of weight matrix 𝑋 ℎ = 𝜏(𝑋𝑈𝑦 + 𝑐)

SLIDE 4

Convolution

SLIDE 5

Convolution: math formula

Given functions 𝑣(𝑢) and 𝑥(𝑢), their convolution is a function 𝑡 𝑢
Written as

𝑡 𝑢 = ∫ 𝑣 𝑏 𝑥 𝑢 − 𝑏 𝑒𝑏 𝑡 = 𝑣 ∗ 𝑥

r

𝑡 𝑢 = (𝑣 ∗ 𝑥)(𝑢)

SLIDE 6

Convolution: discrete version

Given array 𝑣𝑢 and 𝑥𝑢, their convolution is a function 𝑡𝑢
Written as
When 𝑣𝑢 or 𝑥𝑢 is not defined, assumed to be 0

𝑡𝑢 = ෍

𝑏=−∞ +∞

𝑣𝑏𝑥𝑢−𝑏 𝑡 = 𝑣 ∗ 𝑥

r

𝑡𝑢 = 𝑣 ∗ 𝑥 𝑢

SLIDE 7

Illustration 1

a b c d e f x y z xb+yc+zd 𝑥 = [z, y, x] 𝑣 = [a, b, c, d, e, f]

SLIDE 8

Illustration 1

a b c d e f x y z xc+yd+ze

SLIDE 9

Illustration 1

a b c d e f x y z xd+ye+zf

SLIDE 10

Illustration 1: boundary case

a b c d e f x y xe+yf

SLIDE 11

Illustration 1 as matrix multiplication

y z x y z x y z x y z x y z x y a b c d e f

SLIDE 12

Illustration 2: two dimensional case

a b c d e f g h i j k l w x y z wa + bx + ey + fz

SLIDE 13

Illustration 2

a b c d e f g h i j k l w x y z bw + cx + fy + gz wa + bx + ey + fz

SLIDE 14

Illustration 2

a b c d e f g h i j k l w x y z bw + cx + fy + gz wa + bx + ey + fz Kernel (or filter) Feature map Input

SLIDE 15

Advantage: sparse interaction

Figure from Deep Learning, by Goodfellow, Bengio, and Courville

Fully connected layer, 𝑛 × 𝑜 edges 𝑛 output nodes 𝑜 input nodes

SLIDE 16

Advantage: sparse interaction

Figure from Deep Learning, by Goodfellow, Bengio, and Courville

Convolutional layer, ≤ 𝑛 × 𝑙 edges 𝑛 output nodes 𝑜 input nodes 𝑙 kernel size

SLIDE 17

Advantage: sparse interaction

Figure from Deep Learning, by Goodfellow, Bengio, and Courville

Multiple convolutional layers: larger receptive field

SLIDE 18

Advantage: parameter sharing/weight tying

Figure from Deep Learning, by Goodfellow, Bengio, and Courville

The same kernel are used repeatedly. E.g., the black edge is the same weight in the kernel.

SLIDE 19

Advantage: equivariant representations

Equivariant: transforming the input = transforming the output
Example: input is an image, transformation is shifting
Convolution(shift(input)) = shift(Convolution(input))
Useful when care only about the existence of a pattern, rather than

the location

SLIDE 20

Pooling

SLIDE 21

Terminology

Figure from Deep Learning, by Goodfellow, Bengio, and Courville

SLIDE 22

Pooling

Summarizing the input (i.e., output the max of the input)

Figure from Deep Learning, by Goodfellow, Bengio, and Courville

SLIDE 23

Advantage

Induce invariance

Figure from Deep Learning, by Goodfellow, Bengio, and Courville

SLIDE 24

Motivation from neuroscience

David Hubel and Torsten Wiesel studied early visual system in human

brain (V1 or primary visual cortex), and won Nobel prize for this

V1 properties
2D spatial arrangement
Simple cells: inspire convolution layers
Complex cells: inspire pooling layers

SLIDE 25

Example: LeNet

SLIDE 26

LeNet-5

Proposed in “Gradient-based learning applied to document

recognition” , by Yann LeCun, Leon Bottou, Yoshua Bengio and Patrick Haffner,

in Proceedings of the IEEE, 1998

SLIDE 27

LeNet-5

Proposed in “Gradient-based learning applied to document

recognition” , by Yann LeCun, Leon Bottou, Yoshua Bengio and Patrick Haffner,

in Proceedings of the IEEE, 1998

Apply convolution on 2D images (MNIST) and use backpropagation

SLIDE 28

LeNet-5

Proposed in “Gradient-based learning applied to document

recognition” , by Yann LeCun, Leon Bottou, Yoshua Bengio and Patrick Haffner,

in Proceedings of the IEEE, 1998

Apply convolution on 2D images (MNIST) and use backpropagation
Structure: 2 convolutional layers (with pooling) + 3 fully connected layers
Input size: 32x32x1
Convolution kernel size: 5x5
Pooling: 2x2

SLIDE 29