lecture 7 convolutional networks
play

Lecture 7: Convolutional Networks Justin Johnson September 23, - PowerPoint PPT Presentation

Sep 10, 2022 •273 likes •1.26k views

Lecture 7: Convolutional Networks Justin Johnson September 23, 2020 Lecture 7 - 1 Reminder: A2 Due this Friday, 9/25/2020 Justin Johnson September 23, 2020 Lecture 7 - 2 Autograder Late Tokens - Our late policy is (from syllabus): - 3

  1. Lecture 7: Convolutional Networks Justin Johnson September 23, 2020 Lecture 7 - 1

  2. Reminder: A2 Due this Friday, 9/25/2020 Justin Johnson September 23, 2020 Lecture 7 - 2

  3. Autograder Late Tokens - Our late policy is (from syllabus): - 3 free late days - After that, late work gets 25% penalty per day - This was difficult to implement in autograder.io - We will keep track of free late days and penalties outside of autograder.io - We increased autograder.io late tokens to 1000 per student; this does not mean you can turn everything in a month late! Justin Johnson September 23, 2020 Lecture 7 - 3

  4. Last Time: Backpropagation During the backward pass, each node in the graph receives upstream gradients Represent complex expressions and multiplies them by local gradients to as computational graphs compute downstream gradients x s (scores) * hinge L + loss W R f Downstream gradients Local gradients Forward pass computes outputs Upstream gradient Backward pass computes gradients Justin Johnson September 23, 2020 Lecture 7 - 4

  5. Problem : So far our classifiers don’t f(x,W) = Wx respect the spatial structure of images! Stretch pixels into column 56 56 231 231 24 2 Input: x 24 h s W 1 W 2 3072 Input image Output: 10 2 Hidden layer: (2, 2) 100 (4,) Justin Johnson September 23, 2020 Lecture 7 - 5

  6. Problem : So far our classifiers don’t f(x,W) = Wx respect the spatial structure of images! Solution : Define new computational nodes that operate on images! Stretch pixels into column 56 56 231 231 24 2 Input: x 24 h s W 1 W 2 3072 Input image Output: 10 2 Hidden layer: (2, 2) 100 (4,) Justin Johnson September 23, 2020 Lecture 7 - 6

  7. Components of a Fully-Connected Network Fully-Connected Layers Activation Function x h s Justin Johnson September 23, 2020 Lecture 7 - 7

  8. Components of a Convolutional Network Fully-Connected Layers Activation Function x h s Pooling Layers Normalization Convolution Layers 𝑦 !,# = 𝑦 !,# − 𝜈 # ! $ + 𝜁 𝜏 # Justin Johnson September 23, 2020 Lecture 7 - 8

  9. Components of a Convolutional Network Fully-Connected Layers Activation Function x h s Pooling Layers Normalization Convolution Layers 𝑦 !,# = 𝑦 !,# − 𝜈 # ! $ + 𝜁 𝜏 # Justin Johnson September 23, 2020 Lecture 7 - 9

  10. Fully-Connected Layer 32x32x3 image -> stretch to 3072 x 1 Input Output 1 1 10 x 3072 3072 10 weights Justin Johnson September 23, 2020 Lecture 7 - 10

  11. Fully-Connected Layer 32x32x3 image -> stretch to 3072 x 1 Input Output 1 1 10 x 3072 3072 10 weights 1 number: the result of taking a dot product between a row of W and the input (a 3072- dimensional dot product) Justin Johnson September 23, 2020 Lecture 7 - 11

  12. Convolution Layer 3x32x32 image: preserve spatial structure 32 height 32 width depth / 3 channels Justin Johnson September 23, 2020 Lecture 7 - 12

  13. Convolution Layer 3x32x32 image 3x5x5 filter Convolve the filter with the image i.e. “slide over the image spatially, 32 height computing dot products” 32 width depth / 3 channels Justin Johnson September 23, 2020 Lecture 7 - 13

  14. Convolution Layer Filters always extend the full depth of the input volume 3x32x32 image 3x5x5 filter Convolve the filter with the image i.e. “slide over the image spatially, 32 height computing dot products” 32 width depth / 3 channels Justin Johnson September 23, 2020 Lecture 7 - 14

  15. Convolution Layer 3x32x32 image 3x5x5 filter 1 number: 32 the result of taking a dot product between the filter and a small 3x5x5 chunk of the image (i.e. 3*5*5 = 75-dimensional dot product + bias) 32 𝑥 ! 𝑦 + 𝑐 3 Justin Johnson September 23, 2020 Lecture 7 - 15

  16. Convolution Layer 1x28x28 activation map 3x32x32 image 3x5x5 filter 28 convolve (slide) over 32 all spatial locations 28 32 1 3 Justin Johnson September 23, 2020 Lecture 7 - 16

  17. Convolution Layer two 1x28x28 activation map Consider repeating with 3x32x32 image a second (green) filter: 3x5x5 filter 28 28 convolve (slide) over 32 all spatial locations 28 32 1 1 3 Justin Johnson September 23, 2020 Lecture 7 - 17

  18. Convolution Layer 6 activation maps, each 1x28x28 3x32x32 image Consider 6 filters, each 3x5x5 Convolution Layer 32 6x3x5x5 32 Stack activations to get a filters 3 6x28x28 output image! Justin Johnson September 23, 2020 Lecture 7 - 18

  19. Convolution Layer 6 activation maps, each 1x28x28 3x32x32 image Also 6-dim bias vector: Convolution Layer 32 6x3x5x5 32 Stack activations to get a filters 3 6x28x28 output image! Justin Johnson September 23, 2020 Lecture 7 - 19

  20. 28x28 grid, at each Convolution Layer point a 6-dim vector 3x32x32 image Also 6-dim bias vector: Convolution Layer 32 6x3x5x5 32 Stack activations to get a filters 3 6x28x28 output image! Justin Johnson September 23, 2020 Lecture 7 - 20

  21. Convolution Layer 2x6x28x28 Batch of outputs 2x3x32x32 Also 6-dim bias vector: Batch of images Convolution Layer 32 6x3x5x5 32 filters 3 Justin Johnson September 23, 2020 Lecture 7 - 21

  22. Convolution Layer N x C out x H’ x W’ Batch of outputs N x C in x H x W Also C out -dim bias vector: Batch of images Convolution Layer H W C out x C in x K w x K h C out filters C in Justin Johnson September 23, 2020 Lecture 7 - 22

  23. Stacking Convolutions 32 28 26 …. Conv Conv Conv W 1 : 6x3x5x5 W 2 : 10x6x3x3 W 3 : 12x10x3x3 b 1 : 6 b 2 : 10 b 3 : 12 32 28 26 3 6 10 Input: First hidden layer: Second hidden layer: N x 3 x 32 x 32 N x 6 x 28 x 28 N x 10 x 26 x 26 Justin Johnson September 23, 2020 Lecture 7 - 23

  24. Q : What happens if we stack Stacking Convolutions two convolution layers? 32 28 26 …. Conv Conv Conv W 1 : 6x3x5x5 W 2 : 10x6x3x3 W 3 : 12x10x3x3 b 1 : 6 b 2 : 10 b 3 : 12 32 28 26 3 6 10 Input: First hidden layer: Second hidden layer: N x 3 x 32 x 32 N x 6 x 28 x 28 N x 10 x 26 x 26 Justin Johnson September 23, 2020 Lecture 7 - 24

  25. (Recall y=W 2 W 1 x is Q : What happens if we stack Stacking Convolutions a linear classifier) two convolution layers? A : We get another convolution! 32 28 26 …. Conv ReLU Conv ReLU Conv ReLU W 1 : 6x3x5x5 W 2 : 10x6x3x3 W 3 : 12x10x3x3 b 1 : 6 b 2 : 10 b 3 : 12 32 28 26 3 6 10 Input: First hidden layer: Second hidden layer: N x 3 x 32 x 32 N x 6 x 28 x 28 N x 10 x 26 x 26 Justin Johnson September 23, 2020 Lecture 7 - 25

  26. What do convolutional filters learn? 32 28 26 …. Conv ReLU Conv ReLU Conv ReLU W 1 : 6x3x5x5 W 2 : 10x6x3x3 W 3 : 12x10x3x3 b 1 : 6 b 2 : 10 b 3 : 12 32 28 26 3 6 10 Input: First hidden layer: Second hidden layer: N x 3 x 32 x 32 N x 6 x 28 x 28 N x 10 x 26 x 26 Justin Johnson September 23, 2020 Lecture 7 - 26

  27. What do convolutional filters learn? 32 28 Linear classifier: One template per class Conv ReLU W 1 : 6x3x5x5 b 1 : 6 32 28 3 6 Input: First hidden layer: N x 3 x 32 x 32 N x 6 x 28 x 28 Justin Johnson September 23, 2020 Lecture 7 - 27

  28. What do convolutional filters learn? MLP: Bank of whole-image templates 32 28 Conv ReLU W 1 : 6x3x5x5 b 1 : 6 32 28 3 6 Input: First hidden layer: N x 3 x 32 x 32 N x 6 x 28 x 28 Justin Johnson September 23, 2020 Lecture 7 - 28

  29. What do convolutional filters learn? First-layer conv filters: local image templates (Often learns oriented edges, opposing colors) 32 28 Conv ReLU W 1 : 6x3x5x5 b 1 : 6 32 28 3 6 Input: First hidden layer: AlexNet: 64 filters, each 3x11x11 N x 3 x 32 x 32 N x 6 x 28 x 28 Justin Johnson September 23, 2020 Lecture 7 - 29

  30. A closer look at spatial dimensions 32 28 Conv ReLU W 1 : 6x3x5x5 b 1 : 6 32 28 3 6 Input: First hidden layer: N x 3 x 32 x 32 N x 6 x 28 x 28 Justin Johnson September 23, 2020 Lecture 7 - 30

  31. A closer look at spatial dimensions Input: 7x7 Filter: 3x3 7 7 Justin Johnson September 23, 2020 Lecture 7 - 31

  32. A closer look at spatial dimensions Input: 7x7 Filter: 3x3 7 7 Justin Johnson September 23, 2020 Lecture 7 - 32

  33. A closer look at spatial dimensions Input: 7x7 Filter: 3x3 7 7 Justin Johnson September 23, 2020 Lecture 7 - 33

  34. A closer look at spatial dimensions Input: 7x7 Filter: 3x3 7 7 Justin Johnson September 23, 2020 Lecture 7 - 34

  35. A closer look at spatial dimensions Input: 7x7 Filter: 3x3 Output: 5x5 7 7 Justin Johnson September 23, 2020 Lecture 7 - 35

  36. A closer look at spatial dimensions Input: 7x7 Filter: 3x3 Output: 5x5 In general: Problem: Feature 7 maps “shrink” Input: W with each layer! Filter: K Output: W – K + 1 7 Justin Johnson September 23, 2020 Lecture 7 - 36

  37. A closer look at spatial dimensions 0 0 0 0 0 0 0 0 0 Input: 7x7 0 0 Filter: 3x3 0 0 Output: 5x5 0 0 In general: Problem: Feature 0 0 maps “shrink” Input: W with each layer! 0 0 Filter: K Output: W – K + 1 0 0 0 0 Solution: padding Add zeros around the input 0 0 0 0 0 0 0 0 0 Justin Johnson September 23, 2020 Lecture 7 - 37

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12

Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12

Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12 Convolutional Networks Scale up neural networks to process very large images / video sequences Sparse connections Parameter sharing

552 views • 27 slides
Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi

Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi

Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi Convolutional Neural Networks (CNNs) Fully connected (dense) layers have no awareness of spatial information Key concept behind convolutional layers is

983 views • 41 slides
Introduction CSCE 970 CSCE 970 Lecture 4: Lecture 4: Convolutional Convolutional Neural

Introduction CSCE 970 CSCE 970 Lecture 4: Lecture 4: Convolutional Convolutional Neural

Introduction CSCE 970 CSCE 970 Lecture 4: Lecture 4: Convolutional Convolutional Neural Neural CSCE 970 Lecture 4: Networks Networks Good for data with a grid-like topology Stephen Scott Convolutional Neural Networks Stephen Scott

355 views • 3 slides
15-780 Graduate Artificial Intelligence: Convolutional and recurrent networks J. Zico Kolter

15-780 Graduate Artificial Intelligence: Convolutional and recurrent networks J. Zico Kolter

15-780 Graduate Artificial Intelligence: Convolutional and recurrent networks J. Zico Kolter (this lecture) and Ariel Procaccia Carnegie Mellon University Spring 2017 1 Outline Convolutional neural networks Applications of convolutional

1.32k views • 32 slides
CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute

1.07k views • 72 slides
Lecture 5: Convolution Princeton University COS 495 Instructor: Yingyu Liang Convolutional

Lecture 5: Convolution Princeton University COS 495 Instructor: Yingyu Liang Convolutional

Deep Learning Basics Lecture 5: Convolution Princeton University COS 495 Instructor: Yingyu Liang Convolutional neural networks Strong empirical application performance Convolutional networks: neural networks that use convolution in

997 views • 31 slides
Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural

Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural

Transfer Learning with Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural Networks A breakthough Convolutional Neural Networks VGG-16 example Layers of Convolutional filters Bottleneck

625 views • 23 slides
Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent

Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent

Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent Neural Networks and Bayesian Networks Reference: We will be referring to sections etc of Deep Learning by Yoshua Bengio, Ian J. Goodfellow

974 views • 54 slides
Lecture 7: Convolutional Networks Justin Johnson Lecture 7 - 1 September 24, 2019 Reminder: A2

Lecture 7: Convolutional Networks Justin Johnson Lecture 7 - 1 September 24, 2019 Reminder: A2

Lecture 7: Convolutional Networks Justin Johnson Lecture 7 - 1 September 24, 2019 Reminder: A2 Due Monday, September 30, 11:59pm (Even if you enrolled late!) Your submission must pass the validation script Justin Johnson Lecture 7 - 2

1.1k views • 98 slides
Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture you should understand the following concepts convolutional neural networks (CNN) convolution and its advantage pooling and its advantage 2

750 views • 50 slides
Lecture 11: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas, Mark

Lecture 11: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas, Mark

Lecture 11: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas, Mark Glickman and Chris Tanner 1 Outline Review from last lecture 1. Training CNNs 2. BackProp of MaxPooling layer 3. Layers Receptive Field 4.

2.21k views • 89 slides
Lecture 9: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas and Mark

Lecture 9: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas and Mark

Lecture 9: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas and Mark Glickman 1 Outline 1. Review from last lecture 2. BackProp of MaxPooling layer 3. A bit of history 4. Layers Receptive Field 5. Saliency maps 6.

1.15k views • 68 slides
Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN)

Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN)

Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN) A.k.a. CNN or ConvNet Adit Deshpande, A Beginner's Guide To Understanding Convolutional Neural Networks. Digital Images Input array: an images

1.42k views • 72 slides
Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

CSCI 5525 Machine Learning Fall 2019 Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1 Convolutional Neural Networks We will now study a special type of neural networks convolutional neural

662 views • 6 slides
CSC2515 Lecture 9: Convolutional Networks Marzyeh Ghassemi Material and slides developed by

CSC2515 Lecture 9: Convolutional Networks Marzyeh Ghassemi Material and slides developed by

CSC2515 Lecture 9: Convolutional Networks Marzyeh Ghassemi Material and slides developed by Roger Grosse, University of Toronto UofT CSC2515 Lec9 1 / 63 Neural Nets for Visual Object Recognition People are very good at recognizing shapes

944 views • 81 slides
Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing

Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing

Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing 2016 Prof. Luiz Velho Convolutional Neural Networks 1 Summary & References 08/11 ImageNet Classification with Deep Convolutional Neural Networks

648 views • 26 slides
or SoS Meets Program Obfuscation Ilan Komargodski Joint work with Boaz Barak (Harvard) Zvika

or SoS Meets Program Obfuscation Ilan Komargodski Joint work with Boaz Barak (Harvard) Zvika

Limits on Low-Degree PRGs or SoS Meets Program Obfuscation Ilan Komargodski Joint work with Boaz Barak (Harvard) Zvika Brakerski (Weizmann Institute) Pravesh K. Kothari (Princeton) Pseudorandom Generators (PRGs) : {0,1} {0,1}

679 views • 15 slides
March 22, Week 9 Today: Chapter 7, Elastic Potential Energy Homework Assignment #6 - Due Today

March 22, Week 9 Today: Chapter 7, Elastic Potential Energy Homework Assignment #6 - Due Today

March 22, Week 9 Today: Chapter 7, Elastic Potential Energy Homework Assignment #6 - Due Today Mastering Physics: 9 problems from chapters 5 and 6 Written Questions: 6.73 Homework Assignment #7 - Due March 29 Mastering Physics: 6 problems from

949 views • 67 slides
How to combine Debian and Yocto/Bitbake? Manuel Traut <manuel.traut@linutronix.de> ELCE

How to combine Debian and Yocto/Bitbake? Manuel Traut <manuel.traut@linutronix.de> ELCE

How to combine Debian and Yocto/Bitbake? Manuel Traut <manuel.traut@linutronix.de> ELCE 2017 - Prague 1. Why Yocto? What's next? 2. About Debian 3. Benefit of a combination 4. Existing solutions 5. Perfect combination 1) Why

301 views • 28 slides
The increasing_nvalue Constraint Nicolas Beldiceanu, Fabien Hermennier, Xavier Lorca and Thierry

The increasing_nvalue Constraint Nicolas Beldiceanu, Fabien Hermennier, Xavier Lorca and Thierry

The increasing_nvalue Constraint Nicolas Beldiceanu, Fabien Hermennier, Xavier Lorca and Thierry Petit LINA CNRS UMR 6241 CPAIOR 2010 Initial Motivation Def: nvalue( N , X) holds iff variable N is equal to the number of distinct values

659 views • 37 slides
Prometheus in Debian 4 th year in a row! Martina (Tina) Ferrari November 7th, 2019 Martina

Prometheus in Debian 4 th year in a row! Martina (Tina) Ferrari November 7th, 2019 Martina

Prometheus in Debian 4 th year in a row! Martina (Tina) Ferrari November 7th, 2019 Martina Ferrari (PromCon EU 2019) Prometheus in Debian November 7th, 2019 1 / 14 About Tina Debian contributor since 2005. (Debian Developer since 2008)

347 views • 14 slides
On the Concrete Security of Goldreichs PRG Yann Rotella Joint work with Geoffroy Couteau,

On the Concrete Security of Goldreichs PRG Yann Rotella Joint work with Geoffroy Couteau,

On the Concrete Security of Goldreichs PRG Yann Rotella Joint work with Geoffroy Couteau, Aurlien Dupin, Pierrick Maux and Mlissa Rossi January 31, 2019 Introduction A subexponential-time attack Algebraic cryptanalysis

1.22k views • 53 slides
Distributed computing of efficient routing schemes Nicolas Nisse 1 an Rapaport 2 Karol Suchan 3

Distributed computing of efficient routing schemes Nicolas Nisse 1 an Rapaport 2 Karol Suchan 3

Distributed computing of efficient routing schemes Nicolas Nisse 1 an Rapaport 2 Karol Suchan 3 Iv 1 MASCOTTE, INRIA, I3S, CNRS, UNS, Sophia Antipolis, France 2 DIM, Universidad de Chile, Santiago, Chile 3 Universidad Adolfo Ib a nez,

574 views • 34 slides
Motivation The operations and control structures of imperative languages are strongly influenced

Motivation The operations and control structures of imperative languages are strongly influenced

Motivation The operations and control structures of imperative languages are strongly influenced by the way most real computer hardware works. This makes imperative languages relatively easy to compile, but (arguably) less expressive; many

829 views • 44 slides

More recommend