lecture 5 training neural networks part i
play

Lecture 5: Training Neural Networks, Part I Fei-Fei Li & - PowerPoint PPT Presentation

Oct 27, 2023 •327 likes •1.36k views

Lecture 5: Training Neural Networks, Part I Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 1 Administrative A1 is due today

  1. Activation Functions - Squashes numbers to range [0,1] - Historically popular since they have nice interpretation as a saturating “firing rate” of a neuron 3 problems: 1. Saturated neurons “kill” the Sigmoid gradients 2. Sigmoid outputs are not zero- centered Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 33

  2. Consider what happens when the input to a neuron (x) is always positive: What can we say about the gradients on w ? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 34

  3. Consider what happens when the input to a neuron is always positive... allowed gradient update directions zig zag path allowed gradient update directions hypothetical What can we say about the gradients on w ? optimal w vector Always all positive or all negative :( (this is also why you want zero-mean data!) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 35

  4. Activation Functions - Squashes numbers to range [0,1] - Historically popular since they have nice interpretation as a saturating “firing rate” of a neuron 3 problems: 1. Saturated neurons “kill” the Sigmoid gradients 2. Sigmoid outputs are not zero- centered 3. exp() is a bit compute expensive Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 36

  5. Activation Functions - Squashes numbers to range [-1,1] - zero centered (nice) - still kills gradients when saturated :( tanh(x) [LeCun et al., 1991] Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 37

  6. - Computes f(x) = max(0,x) Activation Functions - Does not saturate (in +region) - Very computationally efficient - Converges much faster than sigmoid/tanh in practice (e.g. 6x) ReLU (Rectified Linear Unit) [Krizhevsky et al., 2012] Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 38

  7. - Computes f(x) = max(0,x) Activation Functions - Does not saturate (in +region) - Very computationally efficient - Converges much faster than sigmoid/tanh in practice (e.g. 6x) - Not zero-centered output - An annoyance: ReLU (Rectified Linear Unit) hint: what is the gradient when x < 0? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 39

  8. x ReLU gate What happens when x = -10? What happens when x = 0? What happens when x = 10? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 40

  9. active ReLU DATA CLOUD dead ReLU will never activate => never update Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 41

  10. active ReLU DATA CLOUD => people like to initialize dead ReLU ReLU neurons with slightly will never activate positive biases (e.g. 0.01) => never update Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 42

  11. [Mass et al., 2013] Activation Functions [He et al., 2015] - Does not saturate - Computationally efficient - Converges much faster than sigmoid/tanh in practice! (e.g. 6x) - will not “die”. Leaky ReLU Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 43

  12. [Mass et al., 2013] Activation Functions [He et al., 2015] - Does not saturate - Computationally efficient - Converges much faster than sigmoid/tanh in practice! (e.g. 6x) - will not “die”. Parametric Rectifier (PReLU) Leaky ReLU backprop into \alpha (parameter) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 44

  13. [Clevert et al., 2015] Activation Functions Exponential Linear Units (ELU) - All benefits of ReLU - Does not die - Closer to zero mean outputs - Computation requires exp() Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 45

  14. [Goodfellow et al., 2013] Maxout “Neuron” - Does not have the basic form of dot product -> nonlinearity - Generalizes ReLU and Leaky ReLU - Linear Regime! Does not saturate! Does not die! Problem: doubles the number of parameters/neuron :( Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 46

  15. TLDR: In practice: - Use ReLU. Be careful with your learning rates - Try out Leaky ReLU / Maxout / ELU - Try out tanh but don’t expect much - Don’t use sigmoid Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 47

  16. Data Preprocessing Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 48

  17. Step 1: Preprocess the data (Assume X [NxD] is data matrix, each example in a row) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 49

  18. Step 1: Preprocess the data In practice, you may also see PCA and Whitening of the data (data has diagonal (covariance matrix is the covariance matrix) identity matrix) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 50

  19. TLDR: In practice for Images: center only e.g. consider CIFAR-10 example with [32,32,3] images - Subtract the mean image (e.g. AlexNet) (mean image = [32,32,3] array) - Subtract per-channel mean (e.g. VGGNet) (mean along each channel = 3 numbers) Not common to normalize variance, to do PCA or whitening Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 51

  20. Weight Initialization Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 52

  21. - Q: what happens when W=0 init is used? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 53

  22. - First idea: Small random numbers (gaussian with zero mean and 1e-2 standard deviation) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 54

  23. - First idea: Small random numbers (gaussian with zero mean and 1e-2 standard deviation) Works ~okay for small networks, but can lead to non-homogeneous distributions of activations across the layers of a network. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 55

  24. Lets look at some activation statistics E.g. 10-layer net with 500 neurons on each layer, using tanh non- linearities, and initializing as described in last slide. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 56

  25. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 57

  26. All activations become zero! Q: think about the backward pass. What do the gradients look like? Hint: think about backward pass for a W*X gate. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 58

  27. Almost all neurons completely *1.0 instead of *0.01 saturated, either -1 and 1. Gradients will be all zero. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 59

  28. “Xavier initialization” [Glorot et al., 2010] Reasonable initialization. (Mathematical derivation assumes linear activations) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 60

  29. but when using the ReLU nonlinearity it breaks. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 61

  30. He et al., 2015 (note additional /2) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 62

  31. He et al., 2015 (note additional /2) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 63

  32. Proper initialization is an active area of research… Understanding the difficulty of training deep feedforward neural networks by Glorot and Bengio, 2010 Exact solutions to the nonlinear dynamics of learning in deep linear neural networks by Saxe et al, 2013 Random walk initialization for training very deep feedforward networks by Sussillo and Abbott, 2014 Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification by He et al., 2015 Data-dependent Initializations of Convolutional Neural Networks by Krähenbühl et al., 2015 All you need is a good init , Mishkin and Matas, 2015 … Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 64

  33. [Ioffe and Szegedy, 2015] Batch Normalization “you want unit gaussian activations? just make them so.” consider a batch of activations at some layer. To make each dimension unit gaussian, apply: this is a vanilla differentiable function... Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 65

  34. [Ioffe and Szegedy, 2015] Batch Normalization “you want unit gaussian activations? just make them so.” 1. compute the empirical mean and variance independently for each dimension. N X 2. Normalize D Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 66

  35. [Ioffe and Szegedy, 2015] Batch Normalization Usually inserted after Fully FC Connected / (or Convolutional, as BN we’ll see soon) layers, and before nonlinearity. tanh FC Problem: do we BN necessarily want a unit gaussian input to a tanh tanh layer? ... Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 67

  36. [Ioffe and Szegedy, 2015] Batch Normalization Normalize: Note, the network can learn: And then allow the network to squash the range if it wants to: to recover the identity mapping. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 68

  37. [Ioffe and Szegedy, 2015] Batch Normalization - Improves gradient flow through the network - Allows higher learning rates - Reduces the strong dependence on initialization - Acts as a form of regularization in a funny way, and slightly reduces the need for dropout, maybe Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 69

  38. [Ioffe and Szegedy, 2015] Batch Normalization Note: at test time BatchNorm layer functions differently: The mean/std are not computed based on the batch. Instead, a single fixed empirical mean of activations during training is used. (e.g. can be estimated during training with running averages) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 70

  39. Babysitting the Learning Process Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 71

  40. Step 1: Preprocess the data (Assume X [NxD] is data matrix, each example in a row) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 72

  41. Step 2: Choose the architecture: say we start with one hidden layer of 50 neurons: 50 hidden neurons 10 output output layer neurons, one input CIFAR-10 per class hidden layer layer images, 3072 numbers Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 73

  42. Double check that the loss is reasonable: disable regularization loss ~2.3. returns the loss and the “correct “ for gradient for all parameters 10 classes Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 74

  43. Double check that the loss is reasonable: crank up regularization loss went up, good. (sanity check) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 75

  44. Lets try to train now… Tip : Make sure that you can overfit very small portion of the The above code: training data - take the first 20 examples from CIFAR-10 - turn off regularization (reg = 0.0) - use simple vanilla ‘sgd’ Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 76

  45. Lets try to train now… Tip : Make sure that you can overfit very small portion of the training data Very small loss, train accuracy 1.00, nice! Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 77

  46. Lets try to train now… I like to start with small regularization and find learning rate that makes the loss go down. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 78

  47. Lets try to train now… I like to start with small regularization and find learning rate that makes the loss go down. Loss barely changing Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 79

  48. Lets try to train now… I like to start with small regularization and find learning rate that makes the loss go down. Loss barely changing: Learning rate is loss not going down: probably too low learning rate too low Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 80

  49. Lets try to train now… I like to start with small regularization and find learning rate that makes the loss go down. Loss barely changing: Learning rate is loss not going down: probably too low learning rate too low Notice train/val accuracy goes to 20% though, what’s up with that? (remember this is softmax) Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 81

  50. Lets try to train now… I like to start with small regularization and find learning rate that Okay now lets try learning rate 1e6. What could makes the loss go possibly go wrong? down. loss not going down: learning rate too low Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 82

  51. Lets try to train now… I like to start with small regularization and find learning rate that makes the loss go down. cost: NaN almost loss not going down: always means high learning rate too low learning rate... loss exploding: learning rate too high Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 83

  52. Lets try to train now… I like to start with small regularization and find learning rate that makes the loss go down. 3e-3 is still too high. Cost explodes…. loss not going down: => Rough range for learning rate we learning rate too low should be cross-validating is loss exploding: somewhere [1e-3 … 1e-5] learning rate too high Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 84

  53. Hyperparameter Optimization Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 85

  54. Cross-validation strategy I like to do coarse -> fine cross-validation in stages First stage : only a few epochs to get rough idea of what params work Second stage : longer running time, finer search … (repeat as necessary) Tip for detecting explosions in the solver: If the cost is ever > 3 * original cost, break out early Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 86

  55. For example: run coarse search for 5 epochs note it’s best to optimize in log space! nice Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 87

  56. Now run finer search... adjust range 53% - relatively good for a 2-layer neural net with 50 hidden neurons. Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 88

  57. Now run finer search... adjust range 53% - relatively good for a 2-layer neural net with 50 hidden neurons. But this best cross- validation result is worrying. Why? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 89

  58. Random Search vs. Grid Search Random Search for Hyper-Parameter Optimization Bergstra and Bengio, 2012 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 90

  59. Hyperparameters to play with: - network architecture - learning rate, its decay schedule, update type - regularization (L2/Dropout strength) neural networks practitioner music = loss function Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 91

  60. My cross-validation “command center” Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 92

  61. Monitor and visualize the loss curve Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 93

  62. Loss time Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 94

  63. Loss Bad initialization a prime suspect time Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 95

  64. lossfunctions.tumblr.com Loss function specimen Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 96

  65. lossfunctions.tumblr.com Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 97

  66. lossfunctions.tumblr.com Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 98

  67. Monitor and visualize the accuracy: big gap = overfitting => increase regularization strength? no gap => increase model capacity? Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 99

  68. Track the ratio of weight updates / weight magnitudes: ratio between the values and updates: ~ 0.0002 / 0.02 = 0.01 (about okay) want this to be somewhere around 0.001 or so 10 Fei-Fei Li & Andrej Karpathy & Justin Johnson Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - Lecture 5 - 20 Jan 2016 20 Jan 2016 0

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Lecture 10: Training Neural Networks (Part 1) Justin Johnson Lecture 1 - 1 October 7, 2019

Lecture 10: Training Neural Networks (Part 1) Justin Johnson Lecture 1 - 1 October 7, 2019

Lecture 10: Training Neural Networks (Part 1) Justin Johnson Lecture 1 - 1 October 7, 2019 Reminder: A3 Due Monday, October 14 (1 week from today!) Remember to run the validation script! Justin Johnson Lecture 10 - 2 October 7, 2019

1.45k views • 101 slides
Lecture 10: Training Neural Networks (Part 1) Justin Johnson October 5, 2020 Lecture 1 - 1

Lecture 10: Training Neural Networks (Part 1) Justin Johnson October 5, 2020 Lecture 1 - 1

Lecture 10: Training Neural Networks (Part 1) Justin Johnson October 5, 2020 Lecture 1 - 1 Reminder: A3 Due Friday, October 9 Justin Johnson October 5, 2020 Lecture 10 - 2 Midterm Exam We are still working out details! Will share more

1.31k views • 102 slides
Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training

Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training

Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training the network Given a training set of input-output pairs Minimize the following function w.r.t This is problem of function minimization

1.79k views • 165 slides
Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

CSCI 5525 Machine Learning Fall 2019 Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1 Convolutional Neural Networks We will now study a special type of neural networks convolutional neural

662 views • 6 slides
Lecture 4: Recurrent neural networks for natural language processing Plan of the lecture Part

Lecture 4: Recurrent neural networks for natural language processing Plan of the lecture Part

Neural Natural Language Processing Lecture 4: Recurrent neural networks for natural language processing Plan of the lecture Part 1 : Language modeling. Part 2 : Recurrent neural networks. Part 3 : Long-Short Term Memory (LSTM).

1.07k views • 78 slides
TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja,

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja,

TRAINING NEURAL TRAINING NEURAL NETWORKS ON THE NETWORKS ON THE EDGE EDGE Navjot Kukreja, Alena Shilova Also: Olivier Beaumont Jan Huckelheim Nicola Ferrier Paul Hovland Gerard Gorman BACKGROUND BACKGROUND Typical data ow pattern

779 views • 35 slides
Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture you should understand the following concepts convolutional neural networks (CNN) convolution and its advantage pooling and its advantage 2

750 views • 50 slides
Lecture 6: Training Neural Networks, Part 2 Fei-Fei Li &amp; Andrej Karpathy &amp; Justin

Lecture 6: Training Neural Networks, Part 2 Fei-Fei Li &amp; Andrej Karpathy &amp; Justin

Lecture 6: Training Neural Networks, Part 2 Fei-Fei Li &amp; Andrej Karpathy &amp; Justin Johnson Fei-Fei Li &amp; Andrej Karpathy &amp; Justin Johnson Lecture 6 - Lecture 6 - 25 Jan 2016 25 Jan 2016 1 Administrative A2 is out. Its

1.28k views • 86 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
343H: Honors AI Lecture 25: Neural networks Applications, part 1 4/24/2014 Kristen Grauman UT

343H: Honors AI Lecture 25: Neural networks Applications, part 1 4/24/2014 Kristen Grauman UT

343H: Honors AI Lecture 25: Neural networks Applications, part 1 4/24/2014 Kristen Grauman UT Austin Today Neural networks Supervised learning in visual recognition What does recognition involve? Verification: is that a lamp?

774 views • 49 slides
Training neural networks Today's lecture Learning from small data Curriculum: Active

Training neural networks Today's lecture Learning from small data Curriculum: Active

Training neural networks Today's lecture Learning from small data Curriculum: Active learning - How transferable are features in deep neural When you are not learning networks? Surrogat losses

1k views • 69 slides
CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks

CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS Neural networks Fully connected networks Neuron Non-linearity Softmax layer DNN training Loss function and regularization SGD and backprop Learning rate Overfitting

1.8k views • 86 slides
Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network architecture Geoffrey Hinton with Nitish Srivastava Kevin Swersky Feed-forward neural networks These are the commonest type of neural

534 views • 32 slides
Lecture 9: Neural Networks (Part 1) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu We have

Lecture 9: Neural Networks (Part 1) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu We have

CSCI 5525 Machine Learning Fall 2019 Lecture 9: Neural Networks (Part 1) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu We have just learned about kernel functions that allow us to implicitly lift the raw feature vector x to expanded

371 views • 5 slides
Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training

Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training

Modern Systems for Neural Networks Valentin Dalibard This talk 1.Practicalities of training Neural Networks 2.Leveraging heterogeneous hardware Source: wikipedia Modern Neural Networks Applications: Image classification Modern Neural

513 views • 20 slides
Lecture 10: Neural Networks (Part 2) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

Lecture 10: Neural Networks (Part 2) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

CSCI 5525 Machine Learning Fall 2019 Lecture 10: Neural Networks (Part 2) Feb 25th, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1 Backpropagation Now we consider ERM problem of minimizing the following empirical risk function over : n R (

421 views • 3 slides
Different aspects in correlation products pricing Pascal DELANOE, Structured Equity Derivatives

Different aspects in correlation products pricing Pascal DELANOE, Structured Equity Derivatives

Different aspects in correlation products pricing Pascal DELANOE, Structured Equity Derivatives HSBC 2nd March 2012 Structured Equity Research (HSBC) Different aspects of Correlation 2nd March 2012 1 / 46 First part : Local correlation

827 views • 46 slides
CLIMATE FINANCE? Sren E. Ltken UNEP Risoe UNU, Helsinki September 28 th 2012 UNEP RIS C

CLIMATE FINANCE? Sren E. Ltken UNEP Risoe UNU, Helsinki September 28 th 2012 UNEP RIS C

is there such a thing as CLIMATE FINANCE? Sren E. Ltken UNEP Risoe UNU, Helsinki September 28 th 2012 UNEP RIS C E N T R E ENERGY, CLIMATE &amp; SUSTAINABLE DEVELOPMENT Agenda Penny Wise, Pound Foolish? Can we fix it?

545 views • 15 slides
BIG DATA CONFERENCE How to transform data into money using Big Data technologies INTRO THE

BIG DATA CONFERENCE How to transform data into money using Big Data technologies INTRO THE

APACHE BIG DATA CONFERENCE How to transform data into money using Big Data technologies INTRO THE FIRST SPARK-BASED BIG DATA PLATFORM RELEASED After almost a decade developing Big Data projects in Paradigma, through its R+D department we

640 views • 34 slides
Welcome! The Basics of Show Structure Agenda Housekeeping Catching Up... The

Welcome! The Basics of Show Structure Agenda Housekeeping Catching Up... The

Welcome! The Basics of Show Structure Agenda Housekeeping Catching Up... The Basics of Show Structure What to Consider Putting it All Together Q&amp;A Future Planning Thank you, Venture X!

499 views • 16 slides
St StreamBo eamBox-HB HBM Stream Analytics on High Bandwidth Hybrid Memory Hongyu Miao, Purdue

St StreamBo eamBox-HB HBM Stream Analytics on High Bandwidth Hybrid Memory Hongyu Miao, Purdue

St StreamBo eamBox-HB HBM Stream Analytics on High Bandwidth Hybrid Memory Hongyu Miao, Purdue ECE; Myeongjae Jeon, UNIST ; Gennady Pekhimenko, UToronto; Kathryn S. McKinley, Google; Felix Xiaozhu Lin, Purdue ECE http://xsel.rocks/p/streambox

647 views • 35 slides
Course summary SO 2020_2021_Q1 1.1 Outline Goals Competences Methodology

Course summary SO 2020_2021_Q1 1.1 Outline Goals Competences Methodology

9/21/20 Course summary SO 2020_2021_Q1 1.1 Outline Goals Competences Methodology Course scheduling Assessment Communication Channels http://docencia.ac.upc.edu/FIB/grau/SO/index.html 1.2 1 9/21/20

341 views • 7 slides
Thyroid Cases Case Based Discussion Chienying Liu: no disclosures Jennifer Park-Sigal: no

Thyroid Cases Case Based Discussion Chienying Liu: no disclosures Jennifer Park-Sigal: no

4/13/2018 Thyroid Cases Case Based Discussion Chienying Liu: no disclosures Jennifer Park-Sigal: no disclosures 1 4/13/2018 CASE 1 69 yo healthy active man with abnormal thyroid tests PMH BPH, anxiety, mild hypertension, GERD

576 views • 32 slides
r r sts t

r r sts t

r r sts t rtrs ts rr rst

206 views • 19 slides

More recommend