lecture 6 training neural networks part 2
play

Lecture 6: Training Neural Networks, Part 2 Fei-Fei Li & - PowerPoint PPT Presentation

Dec 13, 2023 •408 likes •1.28k views

Lecture 6: Training Neural Networks, Part 2 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 1 Administrative A2 is out. Its

  1. Lecture 6: Training Neural Networks, Part 2 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 1

  2. Administrative A2 is out. It’s meaty. It’s due Feb 5 (next Friday) You’ll implement: Neural Nets (with Layer Forward/Backward API) Batch Norm Dropout ConvNets Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 2

  3. Mini-batch SGD Loop: 1. Sample a batch of data 2. Forward prop it through the graph, get loss 3. Backprop to calculate the gradients 4. Update the parameters using the gradient Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 3

  4. Leaky ReLU Activation Functions max(0.1x, x) Sigmoid Maxout tanh tanh(x) ELU ReLU max(0,x) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 4

  5. Data Preprocessing Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 5

  6. “Xavier initialization” [Glorot et al., 2010] Reasonable initialization. (Mathematical derivation assumes linear activations) Weight Initialization Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 6

  7. [Ioffe and Szegedy, 2015] Batch Normalization - Improves gradient flow Normalize: through the network - Allows higher learning rates - Reduces the strong dependence on initialization And then allow the network to squash - Acts as a form of the range if it wants to: regularization in a funny way, and slightly reduces the need for dropout, maybe Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 7

  8. Babysitting the Cross-validation learning process Loss barely changing: Learning rate is probably too low Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 8

  9. TODO - Parameter update schemes - Learning rate schedules - Dropout - Gradient checking - Model ensembles Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 9

  10. Parameter Updates Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 10

  11. Training a neural network, main loop: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 11

  12. Training a neural network, main loop: simple gradient descent update now: complicate. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 12

  13. Image credits: Alec Radford Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 13

  14. Suppose loss function is steep vertically but shallow horizontally: Q: What is the trajectory along which we converge towards the minimum with SGD? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 14

  15. Suppose loss function is steep vertically but shallow horizontally: Q: What is the trajectory along which we converge towards the minimum with SGD? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 15

  16. Suppose loss function is steep vertically but shallow horizontally: Q: What is the trajectory along which we converge towards the minimum with SGD? very slow progress along flat direction, jitter along steep one Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 16

  17. Momentum update - Physical interpretation as ball rolling down the loss function + friction (mu coefficient). - mu = usually ~0.5, 0.9, or 0.99 (Sometimes annealed over time, e.g. from 0.5 -> 0.99) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 17

  18. Momentum update - Allows a velocity to “build up” along shallow directions - Velocity becomes damped in steep direction due to quickly changing sign Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 18

  19. SGD vs Momentum notice momentum overshooting the target, but overall getting to the minimum much faster. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 19

  20. Nesterov Momentum update Ordinary momentum update: momentum step actual step gradient step Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 20

  21. Nesterov Momentum update Nesterov momentum update Momentum update “lookahead” gradient step (bit different than momentum momentum original) step step actual step actual step gradient step Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 21

  22. Nesterov Momentum update Nesterov momentum update Momentum update “lookahead” gradient step (bit different than momentum momentum original) step step actual step actual step gradient Nesterov: the only difference... step Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 22

  23. Nesterov Momentum update Slightly inconvenient… usually we have : Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 23

  24. Nesterov Momentum update Slightly inconvenient… usually we have : Variable transform and rearranging saves the day: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 24

  25. Nesterov Momentum update Slightly inconvenient… usually we have : Variable transform and rearranging saves the day: Replace all thetas with phis, rearrange and obtain: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 25

  26. nag = Nesterov Accelerated Gradient Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 26

  27. [Duchi et al., 2011] AdaGrad update Added element-wise scaling of the gradient based on the historical sum of squares in each dimension Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 27

  28. AdaGrad update Q: What happens with AdaGrad? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 28

  29. AdaGrad update Q2: What happens to the step size over long time? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 29

  30. [Tieleman and Hinton, 2012] RMSProp update Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 30

  31. Introduced in a slide in Geoff Hinton’s Coursera class, lecture 6 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 31

  32. Introduced in a slide in Geoff Hinton’s Coursera class, lecture 6 Cited by several papers as: Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 32

  33. adagrad rmsprop Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 33

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Lecture 10: Training Neural Networks (Part 1) Justin Johnson Lecture 1 - 1 October 7, 2019

Lecture 10: Training Neural Networks (Part 1) Justin Johnson Lecture 1 - 1 October 7, 2019

Lecture 10: Training Neural Networks (Part 1) Justin Johnson Lecture 1 - 1 October 7, 2019 Reminder: A3 Due Monday, October 14 (1 week from today!) Remember to run the validation script! Justin Johnson Lecture 10 - 2 October 7, 2019

1.45k views • 101 slides
Lecture 10: Training Neural Networks (Part 1) Justin Johnson October 5, 2020 Lecture 1 - 1

Lecture 10: Training Neural Networks (Part 1) Justin Johnson October 5, 2020 Lecture 1 - 1

Lecture 10: Training Neural Networks (Part 1) Justin Johnson October 5, 2020 Lecture 1 - 1 Reminder: A3 Due Friday, October 9 Justin Johnson October 5, 2020 Lecture 10 - 2 Midterm Exam We are still working out details! Will share more

1.31k views • 102 slides
Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training

Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training

Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training the network Given a training set of input-output pairs Minimize the following function w.r.t This is problem of function minimization

1.79k views • 165 slides
Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

CSCI 5525 Machine Learning Fall 2019 Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1 Convolutional Neural Networks We will now study a special type of neural networks convolutional neural

662 views • 6 slides
Lecture 4: Recurrent neural networks for natural language processing Plan of the lecture Part

Lecture 4: Recurrent neural networks for natural language processing Plan of the lecture Part

Neural Natural Language Processing Lecture 4: Recurrent neural networks for natural language processing Plan of the lecture Part 1 : Language modeling. Part 2 : Recurrent neural networks. Part 3 : Long-Short Term Memory (LSTM).

1.07k views • 78 slides
TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja,

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja,

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja, Alena Shilova Also: Olivier Beaumont Jan Huckelheim Nicola Ferrier Paul Hovland Gerard Gorman BACKGROUND BACKGROUND Typical data ow pattern

779 views • 35 slides
Lecture 5: Training Neural Networks, Part I Fei-Fei Li & Andrej Karpathy & Justin

Lecture 5: Training Neural Networks, Part I Fei-Fei Li & Andrej Karpathy & Justin

Lecture 5: Training Neural Networks, Part I Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 1 Administrative A1 is due today

1.36k views • 102 slides
Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture you should understand the following concepts convolutional neural networks (CNN) convolution and its advantage pooling and its advantage 2

750 views • 50 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
343H: Honors AI Lecture 25: Neural networks Applications, part 1 4/24/2014 Kristen Grauman UT

343H: Honors AI Lecture 25: Neural networks Applications, part 1 4/24/2014 Kristen Grauman UT

343H: Honors AI Lecture 25: Neural networks Applications, part 1 4/24/2014 Kristen Grauman UT Austin Today Neural networks Supervised learning in visual recognition What does recognition involve? Verification: is that a lamp?

774 views • 49 slides
Training neural networks Today's lecture Learning from small data Curriculum: Active

Training neural networks Today's lecture Learning from small data Curriculum: Active

Training neural networks Today's lecture Learning from small data Curriculum: Active learning - How transferable are features in deep neural When you are not learning networks? Surrogat losses

1k views • 69 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network architecture Geoffrey Hinton with Nitish Srivastava Kevin Swersky Feed-forward neural networks These are the commonest type of neural

534 views • 32 slides
Lecture 9: Neural Networks (Part 1) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu We have

Lecture 9: Neural Networks (Part 1) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu We have

CSCI 5525 Machine Learning Fall 2019 Lecture 9: Neural Networks (Part 1) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu We have just learned about kernel functions that allow us to implicitly lift the raw feature vector x to expanded

371 views • 5 slides
Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training

Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training

Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training Neural Networks 2.Leveraging heterogeneous hardware Source: wikipedia Modern Neural Networks Applications: Image classification Modern Neural

513 views • 20 slides
Lecture 10: Neural Networks (Part 2) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

Lecture 10: Neural Networks (Part 2) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

CSCI 5525 Machine Learning Fall 2019 Lecture 10: Neural Networks (Part 2) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1 Backpropagation Now we consider ERM problem of minimizing the following empirical risk function over : n R (

421 views • 3 slides
The Application of Airborne Remote Sensing for OSI Aled Rowlands and Rainier Arndt OSI/Equipment

The Application of Airborne Remote Sensing for OSI Aled Rowlands and Rainier Arndt OSI/Equipment

The Application of Airborne Remote Sensing for OSI Aled Rowlands and Rainier Arndt OSI/Equipment Section Science and Technology Conference, June, 2013 Page 1 Structure . Context Expert Meetings Field Tests Sensors Flight Operations

763 views • 22 slides
Decompositions of log-correlated fields with applications Eero Saksman (University of Helsinki)

Decompositions of log-correlated fields with applications Eero Saksman (University of Helsinki)

Decompositions of log-correlated fields with applications Eero Saksman (University of Helsinki) Based on joint work with Janne Junnila (EPFL Lausanne) and Christian Webb (Aalto University) CONFERENCE IN HONOR OF JOHN AND DON AND JOHN Seattle

1.01k views • 75 slides
Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment Ruofei Du,

Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment Ruofei Du,

Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment Ruofei Du, Sujal Bista, Amitabh Varshney The Augmentarium| UMIACS | University of Maryland, College Park {ruofei, sujal, varshney} @ cs.umd.edu

980 views • 83 slides
August 15, 2012: Riot Grrrl, Interrupted Photo: William Webster/Reuters WEEK 2 IAT100

August 15, 2012: Riot Grrrl, Interrupted Photo: William Webster/Reuters WEEK 2 IAT100

August 15, 2012: Riot Grrrl, Interrupted Photo: William Webster/Reuters WEEK 2 IAT100 Digital Image Design Fall 2013 Composi'on: Principles & Elements of Design Before

580 views • 45 slides
ASSISTANT SUPERINTENDENTS MEETING DECEMBER 2, 2016 Assistant Superintendents Meeting

ASSISTANT SUPERINTENDENTS MEETING DECEMBER 2, 2016 Assistant Superintendents Meeting

ASSISTANT SUPERINTENDENTS MEETING DECEMBER 2, 2016 Assistant Superintendents Meeting December 2, 2016 2 State Board of Education Update CISC GENERAL MEMBERSHIP MEETING Cindy Kazanis, Director Analysis, Measurement, and

628 views • 38 slides
Regeneration Kattie Sepehri Every species is capable of regeneration but some species can

Regeneration Kattie Sepehri Every species is capable of regeneration but some species can

Regeneration Kattie Sepehri Every species is capable of regeneration but some species can do it better than others Regeneration The process of renewal wal, restorati tion, and tissue growth that makes genomes, cells, organisms,

183 views • 15 slides
Recruitment & Support of the Vital Few Created By the Members of the Ohio Family Care

Recruitment & Support of the Vital Few Created By the Members of the Ohio Family Care

Recruitment & Support of the Vital Few Created By the Members of the Ohio Family Care Association Leadership Team Bobbi Pedersen, Georgetta Lake, Dot Erickson-Anderson 9/2020 Agencies with a foster care service: 260 173 FFH

491 views • 20 slides
Presented by: Laurie Darst Mayo Clinic Mary Lynn Bushman NGS February 16, 2016

Presented by: Laurie Darst Mayo Clinic Mary Lynn Bushman NGS February 16, 2016

Presented by: Laurie Darst Mayo Clinic Mary Lynn Bushman NGS February 16, 2016 Attachment Pilot Testimony Overview of Attachment Project Mayo Clinic comments: Results from the Claim Attachment Project(s) NGS comments:

318 views • 20 slides

More recommend