lecture 4 backpropagation and neural networks part 1
play

Lecture 4: Backpropagation and Neural Networks part 1 Fei-Fei Li - PowerPoint PPT Presentation

Nov 04, 2023 •398 likes •1.26k views

Lecture 4: Backpropagation and Neural Networks part 1 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 1 Administrative A1 is due

  1. Lecture 4: Backpropagation and Neural Networks part 1 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 1

  2. Administrative A1 is due Jan 20 (Wednesday). ~150 hours left Warning: Jan 18 (Monday) is Holiday (no class/office hours) Also note: Lectures are non-exhaustive. Read course notes for completeness. I’ll hold make up office hours on Wed Jan20, 5pm @ Gates 259 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 2

  3. Where we are... scores function SVM loss data loss + regularization want Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 3

  4. Optimization (image credits to Alec Radford) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 4

  5. Gradient Descent Numerical gradient : slow :(, approximate :(, easy to write :) Analytic gradient : fast :), exact :), error-prone :( In practice: Derive analytic gradient, check your implementation with numerical gradient Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 5

  6. Computational Graph x s (scores) * hinge L + loss W R Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 6

  7. Convolutional Network (AlexNet) input image weights loss Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 7

  8. Neural Turing Machine input tape loss Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 8

  9. Neural Turing Machine Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 9

  10. e.g. x = -2, y = 5, z = -4 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 10

  11. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 11

  12. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 12

  13. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 13

  14. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 14

  15. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 15

  16. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 16

  17. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 17

  18. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 18

  19. e.g. x = -2, y = 5, z = -4 Chain rule: Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 19

  20. e.g. x = -2, y = 5, z = -4 Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 20

  21. e.g. x = -2, y = 5, z = -4 Chain rule: Want: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 21

  22. activations f Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 22

  23. activations “local gradient” f Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 23

  24. activations “local gradient” f gradients Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 24

  25. activations “local gradient” f gradients Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 25

  26. activations “local gradient” f gradients Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 26

  27. activations “local gradient” f gradients Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 27

  28. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 28

  29. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 29

  30. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 30

  31. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 31

  32. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 32

  33. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 33

  34. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 34

  35. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 35

  36. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 36

  37. Another example: (-1) * (-0.20) = 0.20 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 37

  38. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 38

  39. Another example: [local gradient] x [its gradient] [1] x [0.2] = 0.2 [1] x [0.2] = 0.2 (both inputs!) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 39

  40. Another example: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 40

  41. Another example: [local gradient] x [its gradient] x0: [2] x [0.2] = 0.4 w0: [-1] x [0.2] = -0.2 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 41

  42. sigmoid function sigmoid gate Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 42

  43. sigmoid function sigmoid gate (0.73) * (1 - 0.73) = 0.2 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 4 - Lecture 4 - 13 Jan 2016 13 Jan 2016 43

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Neural Networks for Machine Learning Lecture 13a The ups and downs of backpropagation Geoffrey

Neural Networks for Machine Learning Lecture 13a The ups and downs of backpropagation Geoffrey

Neural Networks for Machine Learning Lecture 13a The ups and downs of backpropagation Geoffrey Hinton Nitish Srivastava, Kevin Swersky Tijmen Tieleman Abdel-rahman Mohamed A brief history of backpropagation The backpropagation algorithm

475 views • 26 slides
Learning From Data Lecture 21 Neural Networks: Backpropagation Forward propagation: algorithmic

Learning From Data Lecture 21 Neural Networks: Backpropagation Forward propagation: algorithmic

Learning From Data Lecture 21 Neural Networks: Backpropagation Forward propagation: algorithmic computation h ( x ) e ( x ) Backpropagation: algorithmic computation of weights M. Magdon-Ismail CSCI 4100/6100 recap: The Neural Network

294 views • 14 slides
Natural Language Understanding Lecture 2: Revision of neural networks and backpropagation Adam

Natural Language Understanding Lecture 2: Revision of neural networks and backpropagation Adam

Natural Language Understanding Lecture 2: Revision of neural networks and backpropagation Adam Lopez Credits: Mirella Lapata and Frank Keller 19 January 2018 School of Informatics University of Edinburgh alopez@inf.ed.ac.uk 1 Biological

1.61k views • 96 slides
CSC321 Lecture 6: Backpropagation Roger Grosse Roger Grosse CSC321 Lecture 6: Backpropagation 1

CSC321 Lecture 6: Backpropagation Roger Grosse Roger Grosse CSC321 Lecture 6: Backpropagation 1

CSC321 Lecture 6: Backpropagation Roger Grosse Roger Grosse CSC321 Lecture 6: Backpropagation 1 / 21 Overview Weve seen that multilayer neural networks are powerful. But how can we actually learn them? Backpropagation is the central

577 views • 21 slides
Backpropagation Why backpropagation Neural networks are sequences of parametrized functions

Backpropagation Why backpropagation Neural networks are sequences of parametrized functions

Backpropagation Why backpropagation Neural networks are sequences of parametrized functions x ($; !) linear conv subsample conv subsample filters filters weights Parameters ! Why backpropagation Neural networks are

767 views • 45 slides
Lecture 10: Neural Networks (Part 2) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

Lecture 10: Neural Networks (Part 2) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

CSCI 5525 Machine Learning Fall 2019 Lecture 10: Neural Networks (Part 2) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1 Backpropagation Now we consider ERM problem of minimizing the following empirical risk function over : n R (

421 views • 3 slides
Neural Networks: Backpropagation Machine Learning Based on slides and material from Geoffrey

Neural Networks: Backpropagation Machine Learning Based on slides and material from Geoffrey

Neural Networks: Backpropagation Machine Learning Based on slides and material from Geoffrey Hinton, Richard Socher, Dan Roth, 1 Yoav Goldberg, Shai Shalev-Shwartz and Shai Ben-David, and others This lecture What is a neural network?

1.1k views • 55 slides
Neural Networks + Backpropagation Last Class Softmax Classifier Generalization /

Neural Networks + Backpropagation Last Class Softmax Classifier Generalization /

CS4501: Introduction to Computer Vision Neural Networks + Backpropagation Last Class Softmax Classifier Generalization / Overfitting Pytorch Todays Class Global Features The perceptron model Neural Networks

838 views • 34 slides
Artificial Neural Networks (Part 2) Gradient Descent Learning and Backpropagation Christian Jacob

Artificial Neural Networks (Part 2) Gradient Descent Learning and Backpropagation Christian Jacob

Artificial Neural Networks (Part 2) Gradient Descent Learning and Backpropagation Christian Jacob CPSC 533 Winter 2001 Learning by Gradient Descent Definition of the Learning Problem Let us start with the simple case of linear cells, which

558 views • 22 slides
An Introduction to Neural Networks - Feedforward NN Backpropagation Agathe Merceron Beuth

An Introduction to Neural Networks - Feedforward NN Backpropagation Agathe Merceron Beuth

An Introduction to Neural Networks - Feedforward NN Backpropagation Agathe Merceron Beuth University of Applied Sciences Berlin, Germany 1 Agenda Artificial neuron Activation function Feedforward neural networks Forward

758 views • 39 slides
CS7015 (Deep Learning): Lecture 4 Feedforward Neural Networks, Backpropagation Mitesh M. Khapra

CS7015 (Deep Learning): Lecture 4 Feedforward Neural Networks, Backpropagation Mitesh M. Khapra

CS7015 (Deep Learning): Lecture 4 Feedforward Neural Networks, Backpropagation Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute of Technology Madras 1/9 Mitesh M. Khapra CS7015 (Deep Learning): Lecture 4

1.85k views • 59 slides
Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Feed-forward Networks Network Training Error Backpropagation Deep Learning Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural networks arise from attempts to model Neural Networks

380 views • 9 slides
Neural Networks and Backpropagation Neural Net Readings: Matt Gormley Murphy

Neural Networks and Backpropagation Neural Net Readings: Matt Gormley Murphy

10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Neural Networks and Backpropagation Neural Net Readings: Matt Gormley Murphy

1.2k views • 77 slides
Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

CSCI 5525 Machine Learning Fall 2019 Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1 Convolutional Neural Networks We will now study a special type of neural networks convolutional neural

662 views • 6 slides
CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided

CS7015 (Deep Learning) : Lecture 13 Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute

1.07k views • 72 slides
CS489/698 Lecture 9: Feb 1, 2017 Multi-layer Neural Networks, Error Backpropagation [D] Chapt.

CS489/698 Lecture 9: Feb 1, 2017 Multi-layer Neural Networks, Error Backpropagation [D] Chapt.

CS489/698 Lecture 9: Feb 1, 2017 Multi-layer Neural Networks, Error Backpropagation [D] Chapt. 10, [HTF] Chapt. 11, [B] Sec. 5.2, 5.3, [M] Sec. 16.5, [RN] Sec. 18.7 CS489/698 (c) 2017 P. Poupart 1 Quick Recap: Linear Models Linear

604 views • 18 slides
The Embedded Graphs of a Knot and the Partial Duals of a Plane Graph Iain Moffatt University of

The Embedded Graphs of a Knot and the Partial Duals of a Plane Graph Iain Moffatt University of

The Embedded Graphs of a Knot and the Partial Duals of a Plane Graph Iain Moffatt University of South Alabama SIAM Conference on Discrete Mathematics, 14 th June 2010 I. Moffatt (South Alabama) Partial Duals of a Plane Graph SIAM DM 10 1

542 views • 22 slides
Quick Changeover Examples (SMED) AMF Process Improvement Group 16 February 2017 Proprietary

Quick Changeover Examples (SMED) AMF Process Improvement Group 16 February 2017 Proprietary

Quick Changeover Examples (SMED) AMF Process Improvement Group 16 February 2017 Proprietary information of New Potential www.newpotentialcbs.com Remember the Basics OBSERVE the process IDENTIFY steps that NEED to be done whilst

389 views • 13 slides
Announcements Wednesday, August 22 Everything youll need to know is on the master website:

Announcements Wednesday, August 22 Everything youll need to know is on the master website:

Announcements Wednesday, August 22 Everything youll need to know is on the master website: http://people.math.gatech.edu/~cjankowski3/18f/m1553/webpage/ or on the website for this section:

571 views • 18 slides
3D Viewing: the Synthetic Camera Programmers reference model for specifying 3D view

3D Viewing: the Synthetic Camera Programmers reference model for specifying 3D view

I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S I N T R O D U C T I O N T O C O M P U T E R G R A P H I C S 3D Viewing: the Synthetic Camera Programmers reference model for specifying 3D view

324 views • 6 slides
Backpropagation I2DL: Prof. Niessner, Prof. Leal-Taix 1 Lecture 3 Recap I2DL: Prof. Niessner,

Backpropagation I2DL: Prof. Niessner, Prof. Leal-Taix 1 Lecture 3 Recap I2DL: Prof. Niessner,

Optimization and Backpropagation I2DL: Prof. Niessner, Prof. Leal-Taix 1 Lecture 3 Recap I2DL: Prof. Niessner, Prof. Leal-Taix 2 Neural Network Linear score function = On CIFAR-10 Credit: Li/Karpathy/Johnson On

1.75k views • 75 slides
Multi-layer Perceptrons & the Back-propagation Algorithm Instructor: Sham Kakade Please email

Multi-layer Perceptrons & the Back-propagation Algorithm Instructor: Sham Kakade Please email

CSE 446: Machine Learning Lecture Multi-layer Perceptrons & the Back-propagation Algorithm Instructor: Sham Kakade Please email the staff mailing list should you find typos/errors in the notes. Thank you! The treatment in Bishop is good as

288 views • 4 slides
Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr.

Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr.

Lecture 16: Introduction to Neural Networks, Feed-forward Networks and Back-propagation Dr. Chengjiang Long Computer Vision Researcher at Kitware Inc. Adjunct Professor at RPI. Email: longc3@rpi.edu Recap Previous Lecture 2 C. Long Lecture

1.12k views • 68 slides
Neural Network Backpropagation 3-2-16 Recall from Monday... Perceptrons can only classify

Neural Network Backpropagation 3-2-16 Recall from Monday... Perceptrons can only classify

Neural Network Backpropagation 3-2-16 Recall from Monday... Perceptrons can only classify linearly separable data. Multi-layer networks Can represent any boolean function. We dont want to build them by hand, so we need a way to

570 views • 11 slides

More recommend