cs 4803 7643 deep learning
play

CS 4803 / 7643: Deep Learning Topics: Forward and backward though - PowerPoint PPT Presentation

Sep 08, 2022 •344 likes •928 views

CS 4803 / 7643: Deep Learning Topics: Forward and backward though conv (Beginning) of convolutional neural network (CNN) architectures Zsolt Kira Georgia Tech Administrative PS1/HW1 Due Feb 11 th ! (C) Dhruv Batra & Zsolt

  1. CS 4803 / 7643: Deep Learning Topics: – Forward and backward though conv – (Beginning) of convolutional neural network (CNN) architectures Zsolt Kira Georgia Tech

  2. Administrative • PS1/HW1 Due Feb 11 th ! (C) Dhruv Batra & Zsolt Kira 2

  3. Example: Reverse mode AD + sin( ) * x 1 x 2 (C) Dhruv Batra 3

  4. Duality in Fprop and Bprop FPROP BPROP SUM + COPY + (C) Dhruv Batra and Zsolt Kira 4

  5. Convolutions for programmers y • Iterate over the kernel instead of the image • Implement cross-correlation instead of convolution • Later - implementation as matrix multiplication (C) Peter Anderson 5

  6. Discrete convolution • Discrete Convolution! • Very similar to correlation but associative 1D Convolution 2D Convolution Filter

  7. Convolutional Layer (C) Dhruv Batra & Zsolt Kira 7 Slide Credit: Marc'Aurelio Ranzato

  8. Convolutional Layer (C) Dhruv Batra & Zsolt Kira 8 Slide Credit: Marc'Aurelio Ranzato

  9. Convolutional Layer (C) Dhruv Batra & Zsolt Kira 9 Slide Credit: Marc'Aurelio Ranzato

  10. Convolutional Layer (C) Dhruv Batra & Zsolt Kira 10 Slide Credit: Marc'Aurelio Ranzato

  11. Convolutional Layer (C) Dhruv Batra & Zsolt Kira 11 Slide Credit: Marc'Aurelio Ranzato

  12. Convolutional Layer (C) Dhruv Batra & Zsolt Kira 12 Slide Credit: Marc'Aurelio Ranzato

  13. Convolutional Layer (C) Dhruv Batra & Zsolt Kira 13 Slide Credit: Marc'Aurelio Ranzato

  14. Convolutional Layer (C) Dhruv Batra & Zsolt Kira 14 Slide Credit: Marc'Aurelio Ranzato

  15. Convolutional Layer (C) Dhruv Batra & Zsolt Kira 15 Slide Credit: Marc'Aurelio Ranzato

  16. Convolutional Layer (C) Dhruv Batra & Zsolt Kira 16 Slide Credit: Marc'Aurelio Ranzato

  17. Convolutional Layer (C) Dhruv Batra & Zsolt Kira 17 Slide Credit: Marc'Aurelio Ranzato

  18. Convolutional Layer (C) Dhruv Batra & Zsolt Kira 18 Slide Credit: Marc'Aurelio Ranzato

  19. Convolutional Layer (C) Dhruv Batra & Zsolt Kira 19 Slide Credit: Marc'Aurelio Ranzato

  20. Convolutional Layer (C) Dhruv Batra & Zsolt Kira 20 Slide Credit: Marc'Aurelio Ranzato

  21. Convolutional Layer (C) Dhruv Batra & Zsolt Kira 21 Slide Credit: Marc'Aurelio Ranzato

  22. Convolutional Layer (C) Dhruv Batra & Zsolt Kira 22 Slide Credit: Marc'Aurelio Ranzato

  23. Convolution Layer Filters always extend the full depth of the input volume 32x32x3 image 5x5x3 filter 32 Convolve the filter with the image i.e. “slide over the image spatially, computing dot products” 32 3 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  24. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter 7 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  25. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter 7 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  26. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter 7 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  27. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter 7 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  28. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter => 5x5 output 7 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  29. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter applied with stride 2 7 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  30. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter applied with stride 2 7 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  31. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter applied with stride 2 => 3x3 output! 7 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  32. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter applied with stride 3? 7 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  33. A closer look at spatial dimensions: 7 7x7 input (spatially) assume 3x3 filter applied with stride 3? doesn’t fit! 7 cannot apply 3x3 filter on 7x7 input with stride 3. Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  34. N Output size: (N - F) / stride + 1 F e.g. N = 7, F = 3: N F stride 1 => (7 - 3)/1 + 1 = 5 stride 2 => (7 - 3)/2 + 1 = 3 stride 3 => (7 - 3)/3 + 1 = 2.33 :\ Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  35. In practice: Common to zero pad the border 0 0 0 0 0 0 e.g. input 7x7 3x3 filter, applied with stride 1 0 pad with 1 pixel border => what is the output? 0 0 0 (recall:) (N - F) / stride + 1 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  36. In practice: Common to zero pad the border 0 0 0 0 0 0 e.g. input 7x7 3x3 filter, applied with stride 1 0 pad with 1 pixel border => what is the output? 0 7x7 output! 0 0 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  37. In practice: Common to zero pad the border 0 0 0 0 0 0 e.g. input 7x7 3x3 filter, applied with stride 1 0 pad with 1 pixel border => what is the output? 0 7x7 output! 0 in general, common to see CONV layers with 0 stride 1, filters of size FxF, and zero-padding with (F-1)/2. (will preserve size spatially) e.g. F = 3 => zero pad with 1 F = 5 => zero pad with 2 F = 7 => zero pad with 3 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  38. Convolutional Layer Learn multiple filters. E.g.: 200x200 image 100 Filters Filter size: 10x10 10K parameters (C) Dhruv Batra & Zsolt Kira 38 Slide Credit: Marc'Aurelio Ranzato

  39. For example, if we had 6 5x5 filters, we’ll get 6 separate activation maps: activation maps 32 28 Convolution Layer 28 32 3 6 We stack these up to get a “new image” of size 28x28x6! Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  40. General Matrix Multiply (GEMM) (C) Dhruv Batra & Zsolt Kira 40 Figure Credit: https://petewarden.com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/

  41. Examples time: Input volume: 32x32x3 10 5x5 filters with stride 1, pad 2 Output volume size: ? Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  42. Examples time: Input volume: 32x32x3 10 5x5 filters with stride 1, pad 2 Output volume size: (32+2*2-5)/1+1 = 32 spatially, so 32x32x10 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  43. Examples time: Input volume: 32x32x3 10 5x5 filters with stride 1, pad 2 Number of parameters in this layer? Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  44. Examples time: Input volume: 32x32x3 10 5x5 filters with stride 1, pad 2 Number of parameters in this layer? each filter has 5*5*3 + 1 = 76 params (+1 for bias) => 76*10 = 760 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  45. Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  46. Common settings: K = (powers of 2, e.g. 32, 64, 128, 512) - F = 3, S = 1, P = 1 - F = 5, S = 1, P = 2 - F = 5, S = 2, P = ? (whatever fits) - F = 1, S = 1, P = 0 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  47. Example: CONV layer in Torch Torch is licensed under BSD 3-clause. Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  48. Preview: ConvNet is a sequence of Convolution Layers, interspersed with activation functions 32 28 CONV, ReLU e.g. 6 5x5x3 32 28 filters 3 6 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  49. Backprop through Conv (C) Dhruv Batra Image Credit: Yann LeCun, Kevin Murphy 49

  50. Preview: ConvNet is a sequence of Convolutional Layers, interspersed with activation functions 32 28 24 …. CONV, CONV, CONV, ReLU ReLU ReLU e.g. 6 e.g. 10 5x5x3 5x5x 6 32 28 24 filters filters 3 6 10 Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

  51. Convolutional Neural Networks (C) Dhruv Batra Image Credit: Yann LeCun, Kevin Murphy 51

  52. The architecture of LeNet5

  53. Handwriting Recognition Example

  54. Translation Invariance

  55. Some Rotation Invariance

  56. Some Scale Invariance

  57. Case Studies • There are several generations of ConvNets – 2012 – 2014: AlexNet, ZNet, VGGNet • Conv-Relu, Pooling, Fully connected, Softmax • Deeper ones (VGGNet) tend to do better – 2014 • Fully-convolutional networks for semantic segmentation • Matrix outputs rather than just one probability distribution – 2014-2016 • Fully-convolutional networks for classification • Less parameters, faster than comparable Gen1 networks • GoogleNet, ResNet – 2014-2016 • Detection layers (proposals) • Caption generation (combine with RNNs for language)

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

CS 4803 / 7643: Deep Learning Topics: Image Classification Supervised Learning view

CS 4803 / 7643: Deep Learning Topics: Image Classification Supervised Learning view

CS 4803 / 7643: Deep Learning Topics: Image Classification Supervised Learning view K-NN Linear Classifier Zsolt Kira Georgia Tech Last Time High-level intro to what deep learning is Fast brief of logistics

1.24k views • 100 slides
CS 4803 / 7643: Deep Learning Topics: Dynamic Programming (Q-Value Iteration)

CS 4803 / 7643: Deep Learning Topics: Dynamic Programming (Q-Value Iteration)

CS 4803 / 7643: Deep Learning Topics: Dynamic Programming (Q-Value Iteration) Reinforcement Learning (Intro, Q-Learning, DQNs) Nirbhay Modhe Georgia Tech Topics well cover Overview of RL RL vs other forms of learning

1.01k views • 65 slides
CS 4803 / 7643: Deep Learning Topics: Specifying Layers Forward & Backward

CS 4803 / 7643: Deep Learning Topics: Specifying Layers Forward & Backward

CS 4803 / 7643: Deep Learning Topics: Specifying Layers Forward & Backward autodifferentiation (Beginning of) Convolutional neural networks Zsolt Kira Georgia Tech Projects We will release a set of project ideas for those

1.04k views • 50 slides
CS 4803 / 7643: Deep Learning Topics: Specifying Layers Forward & Backward

CS 4803 / 7643: Deep Learning Topics: Specifying Layers Forward & Backward

CS 4803 / 7643: Deep Learning Topics: Specifying Layers Forward & Backward autodifferentiation (Beginning of) Convolutional neural networks Zsolt Kira Georgia Tech Administrivia PS0 released mean of 20.7

1.39k views • 92 slides
CS 4803 / 7643: Deep Learning Topic: Reinforcement Learning (RL) Overview Markov

CS 4803 / 7643: Deep Learning Topic: Reinforcement Learning (RL) Overview Markov

CS 4803 / 7643: Deep Learning Topic: Reinforcement Learning (RL) Overview Markov Decision Processes Zsolt Kira Georgia Tech Administrative PS3/HW3 due March 15 th ! Projects 2 new FB projects up

1.06k views • 85 slides
CS 4803 / 7643: Deep Learning Topics: Variational Auto-Encoders (VAEs)

CS 4803 / 7643: Deep Learning Topics: Variational Auto-Encoders (VAEs)

CS 4803 / 7643: Deep Learning Topics: Variational Auto-Encoders (VAEs) Reparameterization trick Dhruv Batra Georgia Tech Administrativia HW4 Grades Released Regrade requests close: 12/03, 11:55pm Please check solutions

627 views • 61 slides
CS 4803 / 7643: Deep Learning Topics: Regularization Neural Networks Optimization

CS 4803 / 7643: Deep Learning Topics: Regularization Neural Networks Optimization

CS 4803 / 7643: Deep Learning Topics: Regularization Neural Networks Optimization Computing Gradients Zsolt Kira Georgia Tech Administrativia HW0 Reminder Due: 01/18, 11:55pm Plagiarism No Tolerance

987 views • 84 slides
CS 4803 / 7643: Deep Learning Topics: Moving beyond supervised learning Zsolt Kira Georgia

CS 4803 / 7643: Deep Learning Topics: Moving beyond supervised learning Zsolt Kira Georgia

CS 4803 / 7643: Deep Learning Topics: Moving beyond supervised learning Zsolt Kira Georgia Tech Administrativia Projects! Due April 30 th Template online Can use MS Word but follow the organization/rubric! No

861 views • 82 slides
CS 4803 / 7643: Deep Learning Website: http://www.cc.gatech.edu/classes/AY2020/cs7643_spring/

CS 4803 / 7643: Deep Learning Website: http://www.cc.gatech.edu/classes/AY2020/cs7643_spring/

CS 4803 / 7643: Deep Learning Website: http://www.cc.gatech.edu/classes/AY2020/cs7643_spring/ Piazza: https://piazza.com/gatech/spring2020/cs4803dl7643a/ Staff mailing list (personal questions): cs4803-7643-staff@lists.gatech.edu Gradescope:

1.12k views • 110 slides
CS 4803 / 7643: Deep Learning Topics: Backpropagation Vector/Matrix/Tensor math

CS 4803 / 7643: Deep Learning Topics: Backpropagation Vector/Matrix/Tensor math

CS 4803 / 7643: Deep Learning Topics: Backpropagation Vector/Matrix/Tensor math Deriving vectorized gradients for ReLU Zsolt Kira Georgia Tech Administrivia PS1/HW1 out Start thinking about project topics/teams (C)

482 views • 21 slides
CS 4803 / 7643: Deep Learning Topics: (Finish) Computing Gradients Backprop in Conv

CS 4803 / 7643: Deep Learning Topics: (Finish) Computing Gradients Backprop in Conv

CS 4803 / 7643: Deep Learning Topics: (Finish) Computing Gradients Backprop in Conv Layers Forward mode vs Reverse mode AD Modern CNN Architectures Zsolt Kira Georgia Tech The architecture of LeNet5 Handwriting Recognition

1.16k views • 84 slides
CS 4803 / 7643: Deep Learning Topics: Linear Classifiers Loss Functions Dhruv Batra

CS 4803 / 7643: Deep Learning Topics: Linear Classifiers Loss Functions Dhruv Batra

CS 4803 / 7643: Deep Learning Topics: Linear Classifiers Loss Functions Dhruv Batra Georgia Tech Administrativia Notes and readings on class webpage https://www.cc.gatech.edu/classes/AY2020/cs7643_fall/ HW0 solutions and

969 views • 76 slides
CS 4803 / 7643: Deep Learning Topics: (Finish) Computing Gradients Backprop in Conv

CS 4803 / 7643: Deep Learning Topics: (Finish) Computing Gradients Backprop in Conv

CS 4803 / 7643: Deep Learning Topics: (Finish) Computing Gradients Backprop in Conv Layers Forward mode vs Reverse mode AD Modern CNN Architectures Zsolt Kira Georgia Tech Administrivia HW1 due date moved! Due:

1.03k views • 44 slides
CS 4803 / 7643: Deep Learning Topics: (Continue) Low-label ML Formulations Zsolt Kira

CS 4803 / 7643: Deep Learning Topics: (Continue) Low-label ML Formulations Zsolt Kira

CS 4803 / 7643: Deep Learning Topics: (Continue) Low-label ML Formulations Zsolt Kira Georgia Tech Administrative Projects! Poster details out on piazza Note: No late days for anything project related! Also note: Keep

864 views • 34 slides
CS 4803 / 7643: Deep Learning Topics: Linear Classifiers Loss Functions Zsolt Kira

CS 4803 / 7643: Deep Learning Topics: Linear Classifiers Loss Functions Zsolt Kira

CS 4803 / 7643: Deep Learning Topics: Linear Classifiers Loss Functions Zsolt Kira Georgia Tech Administrativia Office hours started this week For now CCB commons area for Tas CCB 222 for instructor Any changes will

815 views • 69 slides
CS 4803 / 7643: Deep Learning Topics: Policy Gradients Actor Critic Ashwin Kalyan

CS 4803 / 7643: Deep Learning Topics: Policy Gradients Actor Critic Ashwin Kalyan

CS 4803 / 7643: Deep Learning Topics: Policy Gradients Actor Critic Ashwin Kalyan Georgia Tech Topics well cover Overview of RL RL vs other forms of learning RL API Applications Framework: Markov

939 views • 51 slides
The Work Up of PAD in the Young Be Careful (Likely Not PAD) David Rigberg, MD Professor of

The Work Up of PAD in the Young Be Careful (Likely Not PAD) David Rigberg, MD Professor of

4/15/2016 No relevant disclosures The Work Up of PAD in the Young Be Careful (Likely Not PAD) David Rigberg, MD Professor of Surgery UCLA/DGSOM Vascular Disease in Young Adults Causes Accelerated Atherosclerosis Not common,

269 views • 11 slides
Medical Treatment for Claudication: What Works and What is on the Horizon? Ehrin J. Armstrong,

Medical Treatment for Claudication: What Works and What is on the Horizon? Ehrin J. Armstrong,

Medical Treatment for Claudication: What Works and What is on the Horizon? Ehrin J. Armstrong, MD MSc MAS Director, Interventional Cardiology Director, Vascular Laboratory VA Eastern Colorado Healthcare System Associate Professor Of Medicine

497 views • 32 slides
PTFE Pipe Slide Assemblies Overview Application Anvil PTFE pipe slide assemblies are designed to

PTFE Pipe Slide Assemblies Overview Application Anvil PTFE pipe slide assemblies are designed to

GUIDES & SLIDES PTFE Pipe Slide Assemblies Overview Application Anvil PTFE pipe slide assemblies are designed to support the pipe and provide for lateral and axial movement due to thermal expansion and contraction of the piping system.

517 views • 7 slides
Example: Car 3 Mission: Reaching the destination safely. Controlled System: Car.

Example: Car 3 Mission: Reaching the destination safely. Controlled System: Car.

Real-Time Systems 1 Basic Concepts Typical RTS 2 Example: Car 3 Mission: Reaching the destination safely. Controlled System: Car. Operating environment: Road conditions and

504 views • 39 slides
Lecture 7: Convolutional Neural Networks Fei-Fei Li & Andrej Karpathy & Justin Johnson

Lecture 7: Convolutional Neural Networks Fei-Fei Li & Andrej Karpathy & Justin Johnson

Lecture 7: Convolutional Neural Networks Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - Lecture 7 - 27 Jan 2016 27 Jan 2016 1 Administrative A2 is due Feb 5 (next

1.29k views • 89 slides
Segmentation Segmentation Segmentation Define the accurate boundaries of all objects in an image

Segmentation Segmentation Segmentation Define the accurate boundaries of all objects in an image

Day 4 Lecture 2 Segmentation Segmentation Segmentation Define the accurate boundaries of all objects in an image Segmentation: Datasets Pascal Visual Object Classes Microsoft COCO 20 Classes 80 Classes ~ 5.000 images ~ 300.000 images

995 views • 26 slides
ASIC Physical Design Top-Level Chip Layout References: M. Smith, Application Specific Integrated

ASIC Physical Design Top-Level Chip Layout References: M. Smith, Application Specific Integrated

ASIC Physical Design Top-Level Chip Layout References: M. Smith, Application Specific Integrated Circuits , Chap. 16 Cadence Virtuoso User Manual Top-level IC design process Typically done before individual circuit block layouts

478 views • 34 slides
FEDERAL ENERGY REGULATORY COMMISSION Multi-Stakeholder ILP Effectiveness Technical Conference

FEDERAL ENERGY REGULATORY COMMISSION Multi-Stakeholder ILP Effectiveness Technical Conference

FEDERAL ENERGY REGULATORY COMMISSION Multi-Stakeholder ILP Effectiveness Technical Conference Thursday, June 23, 2005 11:00 a.m. (EDT) 3:00 p.m. Objectives Share feedback from ILP Effectiveness Evaluation 62 Telephone Interviews

617 views • 32 slides

More recommend