lecture 9 convolutional neural networks 2
play

Lecture 9: Convolutional Neural Networks 2 CS109B Data Science 2 - PowerPoint PPT Presentation

Aug 09, 2023 •451 likes •1.15k views

Lecture 9: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas and Mark Glickman 1 Outline 1. Review from last lecture 2. BackProp of MaxPooling layer 3. A bit of history 4. Layers Receptive Field 5. Saliency maps 6.

  1. Lecture 9: Convolutional Neural Networks 2 CS109B Data Science 2 Pavlos Protopapas and Mark Glickman 1

  2. Outline 1. Review from last lecture 2. BackProp of MaxPooling layer 3. A bit of history 4. Layers Receptive Field 5. Saliency maps 6. Transfer Learning 7. CNN for text analysis (example) CS109B, P ROTOPAPAS , G LICKMAN 2

  3. Outline 1. Review from last lecture 2. BackProp of MaxPooling layer 3. A bit of history 4. Layers Receptive Field 5. Saliency maps 6. Transfer Learning 7. CNN for text analysis (example) CS109B, P ROTOPAPAS , G LICKMAN 3

  4. From last lecture + ReLU + ReLU CS109B, P ROTOPAPAS , G LICKMAN 4

  5. Examples I have a convolutional layer with 16 3x3 filters that takes an RGB • image as input. • What else can we define about this layer? • Activation function • Stride • Padding type • How many parameters does the layer have? 16 x 3 x 3 x 3 + 16 = 448 Number of Biases (one Number of Size of per filter) filters channels of Filters prev layer CS109B, P ROTOPAPAS , G LICKMAN 5

  6. Examples Let C be a CNN with the following disposition: • • Input: 32x32x3 images Conv1: 8 3x3 filters, stride 1, padding=same • • Conv2: 16 5x5 filters, stride 2, padding=same Flatten layer • • Dense1: 512 nodes Dense2: 4 nodes • • How many parameters does this network have? (8 x 3 x 3 x 3 + 8) + (16 x 5 x 5 x 8 + 16) + (16 x 16 x 16 x 512 + 512) + (512 x 4 + 4) Conv1 Conv2 Dense1 Dense2 CS109B, P ROTOPAPAS , G LICKMAN 6

  7. What do CNN layers learn? Each CNN layer learns filters of increasing complexity. • • The first layers learn basic feature detection filters: edges, corners, etc. • The middle layers learn filters that detect parts of objects. For faces, they might learn to respond to eyes, noses, etc. • The last layers have higher representations: they learn to recognize full objects, in different shapes and positions. CS109B, P ROTOPAPAS , G LICKMAN 7

  8. CS109B, P ROTOPAPAS , G LICKMAN 8

  9. 3D visualization of networks in action http://scs.ryerson.ca/~aharley/vis/conv/ https://www.youtube.com/watch?v=3JQ3hYko51Y CS109B, P ROTOPAPAS , G LICKMAN 9

  10. Outline 1. Review from last lecture 2. BackProp of MaxPooling layer 3. A bit of history 4. Layers Receptive Field 5. Saliency maps 6. Transfer Learning 7. CNN for text analysis (example) CS109B, P ROTOPAPAS , G LICKMAN 10

  11. Backward propagation of Maximum Pooling Layer Forward mode, 3x3 stride 1 2 4 8 3 6 9 3 4 2 5 5 4 6 3 1 2 3 1 3 4 2 7 4 5 7 CS109B, P ROTOPAPAS , G LICKMAN 11

  12. Backward propagation of Maximum Pooling Layer Forward mode, 3x3 stride 1 2 4 8 3 6 9 3 4 2 5 9 5 4 6 3 1 2 3 1 3 4 2 7 4 5 7 CS109B, P ROTOPAPAS , G LICKMAN 12

  13. Backward propagation of Maximum Pooling Layer Forward mode, 3x3 stride 1 2 4 8 3 6 9 3 4 2 5 9 8 5 4 6 3 1 2 3 1 3 4 2 7 4 5 7 CS109B, P ROTOPAPAS , G LICKMAN 13

  14. Backward propagation of Maximum Pooling Layer Forward mode, 3x3 stride 1 2 4 8 3 6 9 3 4 2 5 9 8 8 5 4 6 3 1 2 3 1 3 4 2 7 4 5 7 CS109B, P ROTOPAPAS , G LICKMAN 14

  15. Backward propagation of Maximum Pooling Layer Forward mode, 3x3 stride 1 2 4 8 3 6 9 3 4 2 5 9 8 8 5 4 6 3 1 9 6 6 7 7 7 2 3 1 3 4 2 7 4 5 7 CS109B, P ROTOPAPAS , G LICKMAN 15

  16. Backward propagation of Maximum Pooling Layer Backward mode. Large fonts represents the values of the derivatives of the current layer (max-pool) and small font the corresponding value of the previous layer. 2 4 8 3 6 9 3 4 2 5 1 9 3 8 1 8 5 4 6 3 1 1 9 4 6 2 6 6 7 2 7 1 7 2 3 1 3 4 2 7 4 5 7 CS109B, P ROTOPAPAS , G LICKMAN 16

  17. Backward propagation of Maximum Pooling Layer Backward mode. Large fonts represents the values of the derivatives of the current layer (max-pool) and small font the corresponding value of the previous layer. 2 4 8 3 6 9 3 4 2 5 1 9 3 8 1 8 5 4 6 3 1 1 9 4 6 2 6 6 7 2 7 1 7 2 3 1 3 4 2 7 4 5 7 CS109B, P ROTOPAPAS , G LICKMAN 17

  18. Backward propagation of Maximum Pooling Layer Backward mode. Large fonts represents the values of the derivatives of the current layer (max-pool) and small font the corresponding value of the previous layer. 2 4 8 3 6 9 3 4 2 5 1 9 3 8 1 8 5 4 6 3 1 1 9 4 6 2 6 6 7 2 7 1 7 2 3 1 3 4 2 7 4 5 7 CS109B, P ROTOPAPAS , G LICKMAN 18

  19. Backward propagation of Maximum Pooling Layer Backward mode. Large fonts represents the values of the derivatives of the current layer (max-pool) and small font the corresponding value of the previous layer. 2 4 8 3 6 +1 9 3 4 2 5 1 9 3 8 1 8 5 4 6 3 1 1 9 4 6 2 6 6 7 2 7 1 7 2 3 1 3 4 2 7 4 5 7 CS109B, P ROTOPAPAS , G LICKMAN 19

  20. Backward propagation of Maximum Pooling Layer Backward mode. Large fonts represents the values of the derivatives of the current layer (max-pool) and small font the corresponding value of the previous layer. 2 4 8 3 6 +1 9 3 4 2 5 1 9 3 8 1 8 5 4 6 3 1 1 9 4 6 2 6 6 7 2 7 1 7 2 3 1 3 4 2 7 4 5 7 CS109B, P ROTOPAPAS , G LICKMAN 20

  21. Backward propagation of Maximum Pooling Layer Backward mode. Large fonts represents the values of the derivatives of the current layer (max-pool) and small font the corresponding value of the previous layer. +3 2 4 8 3 6 +1 9 3 4 2 5 1 9 3 8 1 8 5 4 6 3 1 1 9 4 6 2 6 6 7 2 7 1 7 2 3 1 3 4 2 7 4 5 7 CS109B, P ROTOPAPAS , G LICKMAN 21

  22. Backward propagation of Maximum Pooling Layer Backward mode. Large fonts represents the values of the derivatives of the current layer (max-pool) and small font the corresponding value of the previous layer. +3 2 4 8 3 6 +1 9 3 4 2 5 1 9 3 8 1 8 5 4 6 3 1 1 9 4 6 2 6 6 7 2 7 1 7 2 3 1 3 4 2 7 4 5 7 CS109B, P ROTOPAPAS , G LICKMAN 22

  23. Backward propagation of Maximum Pooling Layer Backward mode. Large fonts represents the values of the derivatives of the current layer (max-pool) and small font the corresponding value of the previous layer. +3 2 4 8 3 6 +1 9 3 4 2 5 1 9 3 8 1 8 5 4 6 3 1 1 9 4 6 2 6 6 7 2 7 1 7 2 3 1 3 4 2 7 4 5 7 CS109B, P ROTOPAPAS , G LICKMAN 23

  24. Backward propagation of Maximum Pooling Layer Backward mode. Large fonts represents the values of the derivatives of the current layer (max-pool) and small font the corresponding value of the previous layer. +4 2 4 8 3 6 +1 9 3 4 2 5 1 9 3 8 1 8 5 4 6 3 1 1 9 4 6 2 6 6 7 2 7 1 7 2 3 1 3 4 2 7 4 5 7 CS109B, P ROTOPAPAS , G LICKMAN 24

  25. Outline 1. Review from last lecture 2. BackProp of MaxPooling layer 3. A bit of history 4. Layers Receptive Field 5. Saliency maps 6. Transfer Learning 7. CNN for text analysis (example) CS109B, P ROTOPAPAS , G LICKMAN 25

  26. Initial ideas • The first piece of research proposing something similar to a Convolutional Neural Network was authored by Kunihiko Fukushima in 1980, and was called the NeoCognitron 1 . • Inspired by discoveries on visual cortex of mammals. Fukushima applied the NeoCognitron to hand-written character • recognition. • End of the 80’s: several papers advanced the field Backpropagation published in French by Yann LeCun in 1985 (independently • discovered by other researchers as well) • TDNN by Waiber et al., 1989 - Convolutional-like network trained with backprop. • Backpropagation applied to handwritten zip code recognition by LeCun et al., 1989 1 K. Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4): 93-202, 1980. CS109B, P ROTOPAPAS , G LICKMAN 26

  27. LeNet November 1998: LeCun publishes one of his most recognized papers • describing a “ modern ” CNN architecture for document recognition, called LeNet 1 . Not his first iteration, this was in fact LeNet-5, but this paper is the • commonly cited publication when talking about LeNet. 1 LeCun, Yann, et al. "Gradient-based learning applied to document recognition." Proceedings of the IEEE 86.11 (1998): 2278-2324. CS109B, P ROTOPAPAS , G LICKMAN 27

  28. AlexNet • Developed by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton at Utoronto in 2012. More than 25000 citations. • Destroyed the competition in the 2012 ImageNet Large Scale Visual Recognition Challenge. Showed benefits of CNNs and kickstarted AI revolution. AlexNet • top-5 error of 15.3%, more than 10.8 percentage points lower than runner-up. Main contributions: • Trained on ImageNet with data • augmentation Increased depth of model, GPU • training ( five to six days ) Smart optimizer and Dropout layers • ReLU activation! • CS109B, P ROTOPAPAS , G LICKMAN 28

  29. ZFNet Introduced by Matthew Zeiler and Rob Fergus from NYU, won ILSVRC • 2013 with 11.2% error rate. Decreased sizes of filters. • Trained for 12 days. Paper presented a visualization technique named Deconvolutional • Network , which helps to examine different feature activations and their relation to the input space. CS109B, P ROTOPAPAS , G LICKMAN 29

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi

Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi

Convolutional Neural Networks in Speech Lecture 20 CS 753 Instructor: Preethi Jyothi Convolutional Neural Networks (CNNs) Fully connected (dense) layers have no awareness of spatial information Key concept behind convolutional layers is

983 views • 41 slides
Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural

Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural

Transfer Learning with Convolutional Neural Networks ---- Off the shelf top notch performances Convolutional Neural Networks A breakthough Convolutional Neural Networks VGG-16 example Layers of Convolutional filters Bottleneck

625 views • 23 slides
Introduction CSCE 970 CSCE 970 Lecture 4: Lecture 4: Convolutional Convolutional Neural

Introduction CSCE 970 CSCE 970 Lecture 4: Lecture 4: Convolutional Convolutional Neural

Introduction CSCE 970 CSCE 970 Lecture 4: Lecture 4: Convolutional Convolutional Neural Neural CSCE 970 Lecture 4: Networks Networks Good for data with a grid-like topology Stephen Scott Convolutional Neural Networks Stephen Scott

355 views • 3 slides
Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN)

Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN)

Convolutional Kuan-Ting Lai 2020/3/31 Neural Network Convolutional Neural Networks (CNN) A.k.a. CNN or ConvNet Adit Deshpande, A Beginner's Guide To Understanding Convolutional Neural Networks. Digital Images Input array: an images

1.42k views • 72 slides
CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute

1.07k views • 72 slides
Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture you should understand the following concepts convolutional neural networks (CNN) convolution and its advantage pooling and its advantage 2

750 views • 50 slides
Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly

Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly

Convolutional Neural Nets 4-25-16 Reading Quiz Convolutional neural networks are most commonly used for what sort of task? a) Recognizing images b) Transcribing speech c) Translating documents d) Playing games Neural network review

431 views • 13 slides
Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

CSCI 5525 Machine Learning Fall 2019 Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1 Convolutional Neural Networks We will now study a special type of neural networks convolutional neural

662 views • 6 slides
Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent

Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent

Lecture 18: Concluding Convolutional Neural Networks, Graphical Models as Foundation for Recurrent Neural Networks and Bayesian Networks Reference: We will be referring to sections etc of Deep Learning by Yoshua Bengio, Ian J. Goodfellow

974 views • 54 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing

Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing

Convolutional Neural Networks 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing 2016 Prof. Luiz Velho Convolutional Neural Networks 1 Summary & References 08/11 ImageNet Classification with Deep Convolutional Neural Networks

648 views • 26 slides
Lecture 5: Convolution Princeton University COS 495 Instructor: Yingyu Liang Convolutional

Lecture 5: Convolution Princeton University COS 495 Instructor: Yingyu Liang Convolutional

Deep Learning Basics Lecture 5: Convolution Princeton University COS 495 Instructor: Yingyu Liang Convolutional neural networks Strong empirical application performance Convolutional networks: neural networks that use convolution in

997 views • 31 slides
MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks

MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks

Run 3493 Event 41075, Oct. 23rd, 2015 CONVOLUTIONAL NEURAL NETWORKS IN MICROBOONE Taritree Wongjirad DPF 2017 Tufts/MIT Outline Convolutional neural networks (CNNs) are a type of deep, feed-forward neural networks that have been

831 views • 43 slides
Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image

Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image

Convolutional Neural Networks (Part III) 08, 10 & 17 Nov, 2016 J. Ezequiel Soto S. Image Processing 2016 Prof. Luiz Velho Convolutional Neural Networks 1 Summary & References 08/11 ImageNet Classification with Deep Convolutional

444 views • 27 slides
Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12

Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12

Convolutional Networks Lecture slides for Chapter 9 of Deep Learning Ian Goodfellow 2016-09-12 Convolutional Networks Scale up neural networks to process very large images / video sequences Sparse connections Parameter sharing

552 views • 27 slides
Convolutional Neural Networks Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline GPUs

Convolutional Neural Networks Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline GPUs

Convolutional Neural Networks Rachel Hu and Zhi Zhang Amazon AI d2l.ai Outline GPUs Convolutions Pooling, Padding and Stride Convolutional Neural Networks (LeNet) Deep ConvNets (AlexNet) Networks using Blocks (VGG)

1.91k views • 89 slides
Fuzzy Self-Organizing Map based on Regularized Fuzzy c-means Clustering Sndor Migly, Jnos

Fuzzy Self-Organizing Map based on Regularized Fuzzy c-means Clustering Sndor Migly, Jnos

Fuzzy Self-Organizing Map based on Regularized Fuzzy c-means Clustering Sndor Migly, Jnos Abonyi and Ferenc Szeifert University of Veszprm, Department of Process Engineering www.fmt.vein.hu/ softcomp 2/10 Overview Steps and tasks

620 views • 12 slides
to accelerate similarity search Fanny Bonachera, Gilles Marcou, Natalia Kireeva, Alexandre Varnek,

to accelerate similarity search Fanny Bonachera, Gilles Marcou, Natalia Kireeva, Alexandre Varnek,

YOUR LOGO Using Self-Organizing maps to accelerate similarity search Fanny Bonachera, Gilles Marcou, Natalia Kireeva, Alexandre Varnek, Dragos Horvath Laboratoire d Infochimie, UMR 7177. 1, rue Blaise Pascal, 67000 Strasbourg YOUR LOGO The

426 views • 18 slides
August 14, 2012 Sussex County August 15, 2012 Kent County August 17, 2012 New Castle County

August 14, 2012 Sussex County August 15, 2012 Kent County August 17, 2012 New Castle County

August 14, 2012 Sussex County August 15, 2012 Kent County August 17, 2012 New Castle County www.dnrec.delaware.gov/swc/Pages/FloodplainandDrainageCodeWorkGroupCommittee.aspx AGENDA Introductions. Overview of Senate Bill 64 and purpose of

1.43k views • 99 slides
Article 10 of the Public Service Law What You Should Know About the Siting of Major Electric

Article 10 of the Public Service Law What You Should Know About the Siting of Major Electric

Article 10 of the Public Service Law What You Should Know About the Siting of Major Electric Generating Facilities May 8, 2018 May 8, 2018 2 Overview Article 10 provides a unified review and approval process for major electric generating

522 views • 27 slides
Ongoing Emergence: A Core Concept in Epigenetic Robotics Christopher G. Prince University of

Ongoing Emergence: A Core Concept in Epigenetic Robotics Christopher G. Prince University of

Ongoing Emergence: A Core Concept in Epigenetic Robotics Christopher G. Prince University of Minnesota Duluth, Dept. of Computer Science Nathan A. Helder George J. Hollich Purdue University, Dept. of Psychological Sciences

593 views • 38 slides
FO FOSS4G SS4G, , Seo Seoul ul Septemb Sep tember er 17 17, , 2015 2015 Priska Pri

FO FOSS4G SS4G, , Seo Seoul ul Septemb Sep tember er 17 17, , 2015 2015 Priska Pri

Canton of Zrich, Switzerland Building Department, Office for Spatial Development Department of Geoinformation Use Case of a Dual Open Strategy in the Canton of Zrich, Switzerland FO FOSS4G SS4G, , Seo Seoul ul Septemb Sep tember

709 views • 45 slides
Enter the Agile Dojo Presented by: Christina Ambers

Enter the Agile Dojo Presented by: Christina Ambers

AW18 Agile Practices Wednesday, November 6th, 2019 3:00 PM Enter the Agile Dojo Presented by: Christina Ambers Thomson

468 views • 21 slides
Realistic Scenarios for System-Level Simulations of LTE Networks with SON Features 7 th COST

Realistic Scenarios for System-Level Simulations of LTE Networks with SON Features 7 th COST

FP7 ICT-SOCRATES Realistic Scenarios for System-Level Simulations of LTE Networks with SON Features 7 th COST 2100 Committee Meeting February 16 th - 18 th 2009 Braunschweig, Germany Authors: Andreas Eisenbltter, atesio, Berlin Thomas

209 views • 16 slides

More recommend