CS 4803 / 7643: Deep Learning Topics: Specifying Layers Forward - - PowerPoint PPT Presentation

▶
cs 4803 7643 deep learning
SMART_READER_LITE
LIVE PREVIEW

CS 4803 / 7643: Deep Learning Topics: Specifying Layers Forward - - PowerPoint PPT Presentation

Jan 02, 2024 523 likes •1.04k views

CS 4803 / 7643: Deep Learning Topics: Specifying Layers Forward & Backward autodifferentiation (Beginning of) Convolutional neural networks Zsolt Kira Georgia Tech Projects We will release a set of project ideas for those

slide-1
SLIDE 1

CS 4803 / 7643: Deep Learning

Zsolt Kira Georgia Tech

Topics:

– Specifying Layers – Forward & Backward autodifferentiation – (Beginning of) Convolutional neural networks

slide-2
SLIDE 2

Projects

  • We will release a set of project ideas for those that

are having trouble coming up with them

– NEW: Facebook will develop a set of project ideas as well, with some of their researchers involved – Should be released early/mid Feb.

(C) Dhruv Batra & Zsolt Kira 2

slide-3
SLIDE 3

(C) Dhruv Batra & Zsolt Kira 3

slide-4
SLIDE 4

(C) Dhruv Batra & Zsolt Kira 4

slide-5
SLIDE 5

(C) Dhruv Batra & Zsolt Kira 5

slide-6
SLIDE 6

Types of Projects

  • Application

– Image to cooking recipe – Voice cloning – Etc. – Need to modify/implement something; not enough to just take code and run it

  • Rigorous empirical report

– Have a hypothesis or something you’re trying to test – Perform rigorous experiments with many (reasonable) conditions that would support/refute your hypothesis

  • Reproduction of a paper

– Not just running online code!

  • Theory
  • Novel new method

– (not necessary for the project, good if goal is publishing!)

(C) Dhruv Batra & Zsolt Kira 6

slide-7
SLIDE 7

Recap from last time

(C) Dhruv Batra & Zsolt Kira 7

slide-8
SLIDE 8

8

(x,y,z are scalars) x y z

* Modularized implementation: forward / backward API

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

slide-9
SLIDE 9

(C) Dhruv Batra & Zsolt Kira 9

+ sin( ) x1 x2 *

Example: Forward mode AD

https://www.cc.gatech.edu/classes/AY2020/cs7643_spring/slides/autodiff_forward_reverse.pdf

slide-10
SLIDE 10

(C) Dhruv Batra & Zsolt Kira 10

Example: Reverse mode AD

+ sin( ) x1 x2 *

https://www.cc.gatech.edu/classes/AY2020/cs7643_spring/slides/autodiff_forward_reverse.pdf

slide-11
SLIDE 11

11

Example: 200x200 image 40K hidden units

  • Spatial correlation is local
  • Waste of resources + we have not enough

training samples anyway..

Fully Connected Layer

Slide Credit: Marc'Aurelio Ranzato

~2B parameters!!!

slide-12
SLIDE 12

12

Example: 200x200 image 40K hidden units “Filter” size: 10x10 4M parameters Note: This parameterization is good when input image is registered (e.g., face recognition).

Locally Connected Layer

Slide Credit: Marc'Aurelio Ranzato

slide-13
SLIDE 13

13

STATIONARITY? Statistics similar at all locations

Locally Connected Layer

Slide Credit: Marc'Aurelio Ranzato

slide-14
SLIDE 14

14

Share the same parameters across different locations (assuming input is stationary): Convolutions with learned kernels

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

slide-15
SLIDE 15

How to implement this (and efficiently)?

slide-16
SLIDE 16

Convolutions for mathematicians

  • On operation on two functions

and to produce a third function

  • E.g. input

and kernel or weighting function

16 (C) Peter Anderson

slide-17
SLIDE 17

Convolutions!

  • Math vs. CS vs. programming viewpoints

(C) Dhruv Batra & Zsolt Kira 17

slide-18
SLIDE 18

Convolutions for mathematicians

  • On operation on two functions

and to produce a third function

  • E.g. input

and kernel or weighting function

18 (C) Peter Anderson

slide-19
SLIDE 19

Convolutions for mathematicians

  • One dimension

19

  • Two dimensions

1 2 1 2 1 2

(C) Peter Anderson

slide-20
SLIDE 20

Convolutions for programmers

  • Iterate over the kernel instead of the image

20

y

  • Later - implementation as matrix multiplication
  • Implement cross-correlation instead of convolution

(C) Peter Anderson

slide-21
SLIDE 21

Discrete convolution

  • Discrete Convolution!
  • Very similar to correlation but associative

2D Convolution 1D Convolution Filter

slide-22
SLIDE 22

A note on sizes

Filter m m Input N N Output N-m+1 N-m+1

MATLAB to the rescue!

  • conv2(x,w, ‘valid’)
slide-23
SLIDE 23

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 23

slide-24
SLIDE 24

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 24

slide-25
SLIDE 25

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 25

slide-26
SLIDE 26

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 26

slide-27
SLIDE 27

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 27

slide-28
SLIDE 28

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 28

slide-29
SLIDE 29

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 29

slide-30
SLIDE 30

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 30

slide-31
SLIDE 31

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 31

slide-32
SLIDE 32

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 32

slide-33
SLIDE 33

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 33

slide-34
SLIDE 34

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 34

slide-35
SLIDE 35

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 35

slide-36
SLIDE 36

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 36

slide-37
SLIDE 37

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 37

slide-38
SLIDE 38

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 38

slide-39
SLIDE 39

Convolution Explained

  • http://setosa.io/ev/image-kernels/
  • https://github.com/bruckner/deepViz

(C) Dhruv Batra & Zsolt Kira 40

slide-40
SLIDE 40

Learn multiple filters.

E.g.: 200x200 image 100 Filters Filter size: 10x10 10K parameters

Convolutional Layer

Slide Credit: Marc'Aurelio Ranzato

(C) Dhruv Batra & Zsolt Kira 41

slide-41
SLIDE 41

32 32 3

Convolution Layer

32x32x3 image -> preserve spatial structure

width height depth

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

slide-42
SLIDE 42

32 32 3

Convolution Layer

5x5x3 filter 32x32x3 image

Convolve the filter with the image i.e. “slide over the image spatially, computing dot products”

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

slide-43
SLIDE 43

32 32 3

Convolution Layer

5x5x3 filter 32x32x3 image

Convolve the filter with the image i.e. “slide over the image spatially, computing dot products” Filters always extend the full depth of the input volume

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

slide-44
SLIDE 44

32 32 3

32x32x3 image 5x5x3 filter

1 number: the result of taking a dot product between the filter and a small 5x5x3 chunk of the image (i.e. 5*5*3 = 75-dimensional dot product + bias)

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Convolution Layer

slide-45
SLIDE 45

32 32 3

32x32x3 image 5x5x3 filter

convolve (slide) over all spatial locations activation map 1 28 28

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Convolution Layer

slide-46
SLIDE 46

32 32 3

32x32x3 image 5x5x3 filter

convolve (slide) over all spatial locations activation maps 1 28 28

consider a second, green filter

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Convolution Layer

slide-47
SLIDE 47

32 32 3 Convolution Layer activation maps 6 28 28

For example, if we had 6 5x5 filters, we’ll get 6 separate activation maps: We stack these up to get a “new image” of size 28x28x6!

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

slide-48
SLIDE 48

Im2Col

(C) Dhruv Batra & Zsolt Kira 49

Figure Credit: https://petewarden.com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/

slide-49
SLIDE 49

General Matrix Multiply (GEMM)

(C) Dhruv Batra & Zsolt Kira 50

Figure Credit: https://petewarden.com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/

slide-50
SLIDE 50

Preview: ConvNet is a sequence of Convolution Layers, interspersed with activation functions 32 32 3 28 28 6 CONV, ReLU e.g. 6 5x5x3 filters

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n