neural networks learning the network part 3
play

Neural Networks Learning the network: Part 3 11-785, Fall 2020 - PowerPoint PPT Presentation

Jun 25, 2023 •134 likes •1.79k views

Neural Networks Learning the network: Part 3 11-785, Fall 2020 Lecture 5 1 Recap : Training the network Given a training set of input-output pairs Minimize the following function w.r.t This is problem of function minimization

  1. Calculus Refresher: Basic rules of calculus For any differentiable function with derivative �� �� the following must hold for sufficiently small For any differentiable function � � � with partial derivatives Both by the �� �� �� definition �� � �� � �� � the following must hold for sufficiently small � � � � 25

  2. Calculus Refresher: Chain rule For any nested function Check – we can confirm that : 26

  3. Calculus Refresher: Distributed Chain rule Check: Let � � � � � � � � � 27 � � �

  4. Calculus Refresher: Distributed Chain rule Check: � � � � � � � � � 28 � � �

  5. Distributed Chain Rule: Influence Diagram � � � � � � • affects through each of 29

  6. Distributed Chain Rule: Influence Diagram � � � � � � � � � � • Small perturbations in cause small perturbations in each of each of which individually additively perturbs 30

  7. Returning to our problem • How to compute 31

  8. A first closer look at the network • Showing a tiny 2-input network for illustration – Actual network would have many more neurons and inputs 32

  9. A first closer look at the network + + 𝑔(. ) 𝑔(. ) + 𝑔(. ) + + 𝑔(. ) 𝑔(. ) • Showing a tiny 2-input network for illustration – Actual network would have many more neurons and inputs • Explicitly separating the weighted sum of inputs from the activation 33

  10. A first closer look at the network (�) (�) �,� + �,� + (�) (�) (�) �,� �,� �,� + (�) (�) (�) �,� �,� �,� (�) (�) + + �,� �,� (�) (�) (�) (�) (�) �,� �,� �,� �,� �,� • Showing a tiny 2-input network for illustration – Actual network would have many more neurons and inputs • Expanded with all weights shown • Lets label the other variables too… 34

  11. Computing the derivative for a single input (�) (�) �,� �,� (�) (�) (�) (�) � � + + � � 1 2 (�) (�) (�) �,� �,� �,� (�) + � Div 3 (�) (�) �,� (�) �,� �,� (�) (�) (�) (�) (�) �,� + + � � � �,� 1 2 (�) � (�) �,� (�) (�) (�) (�) �,� �,� �,� �,� 35

  12. Computing the derivative for a single input 𝒆𝑬𝒋𝒘(𝒁,𝒆) What is: (�) (�) (�) 𝒆� �,� �,� �,� (�) (�) (�) (�) � � + + � � 1 2 (�) (�) (�) �,� �,� �,� (�) + � Div 3 (�) (�) �,� (�) �,� �,� (�) (�) (�) (�) (�) �,� + + � � � �,� 1 2 (�) � (�) �,� (�) (�) (�) (�) �,� �,� �,� �,� 36

  13. Computing the gradient • Note: computation of the derivative requires (�) �,� intermediate and final output values of the network in response to the input 37

  14. The “forward pass” y (0) z (1) y (1) z (2) y (2) z (3) y (3) z (N-1) y (N-1) ��� � � � z (N) y (N) f N � � � ��� f N ��� � � � ��� � � � 1 1 1 1 We will refer to the process of computing the output from an input as the forward pass We will illustrate the forward pass in the following slides 38

  15. The “forward pass” y (0) z (1) y (1) z (2) y (2) z (3) y (3) z (N-1) y (N-1) ��� � � � z (N) y (N) f N � � � ��� f N ��� � � � ��� � � � 1 1 1 1 Setting � (�) � for notational convenience (�) and � Assuming (�) (�) -- assuming the bias is a weight and extending �� � the output of every layer by a constant 1, to account for the biases 39

  16. The “forward pass” y (0) z (1) y (1) z (2) y (2) z (3) y (3) z (N-1) y (N-1) ��� � � � z (N) y (N) f N � � � ��� f N ��� � � � ��� � � � 1 1 1 1 40

  17. The “forward pass” y (0) z (1) y (1) z (2) y (2) z (3) y (3) z (N-1) y (N-1) ��� � � � z (N) y (N) f N � � � ��� f N ��� � � � ��� � � � 1 1 1 1 (�) (�) (�) � �� � � 41

  18. y (0) z (1) y (1) z (2) y (2) z (3) y (3) z (N-1) y (N-1) ��� � � � z (N) y (N) f N � � � ��� f N ��� � � � ��� � � � 1 1 1 1 (�) (�) (�) (�) (�) � � � � �� � � 42

  19. y (0) z (1) y (1) z (2) y (2) z (3) y (3) z (N-1) y (N-1) ��� � � � z (N) y (N) f N � � � ��� f N ��� � � � ��� � � � 1 1 1 1 (�) (�) (�) (�) (�) (�) (�) (�) � � � � �� � � �� � � � 43

  20. y (0) z (1) y (1) z (2) y (2) z (3) y (3) z (N-1) y (N-1) ��� � � � z (N) y (N) f N � � � ��� f N ��� � � � ��� � � � 1 1 1 1 (�) (�) (�) (�) (�) (�) (�) (�) (�) (�) � � � � � � � �� � � �� � � � 44

  21. y (0) z (3) z (1) y (1) z (2) y (2) y (3) z (N-1) y (N-1) ��� � � � z (N) y (N) f N � � � ��� f N ��� � � � ��� � � � 1 1 1 1 (�) (�) (�) (�) (�) (�) (�) (�) (�) (�) � � � � � � � �� � � �� � � � (�) (�) (�) � �� � 45 �

  22. y (0) z (3) z (1) y (3) y (1) z (2) y (2) z (N-1) y (N-1) ��� � � � z (N) y (N) f N � � � ��� f N ��� � � � ��� � � � 1 1 1 1 (�) (�) (�) (�) (�) (�) (�) (�) (�) (�) � � � � � � � �� � � �� � � � (�) (�) (�) (�) (�) � �� � � � � 46 �

  23. y (0) z (3) y (N-1) z (1) y (3) y (1) z (2) y (2) z (N-1) ��� � � � z (N) y (N) f N � � � ��� f N ��� � � � ��� � � � 1 1 1 1 (�) (�) (���) (���) (���) (�) (�) � �� � ��� � � � � 47

  24. Forward Computation y (0) z (3) z (1) y (1) z (2) y (3) y (2) z (N-1) y (N-1) ��� � � � z (N) y (N) f N � � � ��� f N ��� � � � ��� � � � 1 1 1 1 ITERATE FOR k = 1:N for j = 1:layer-width 48

  25. Forward “Pass” • Input: dimensional vector • Set: , is the width of the 0 th (input) layer – – ; • For layer – For D k is the size of the kth layer (�) � ��� (�) (���) • � ��� �,� � (�) (�) • � � � • Output: – 49

  26. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 We have computed all these intermediate values in the forward computation We must remember them – we will need them to compute the derivatives 50

  27. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 First, we compute the divergence between the output of the net y = y (N) and the desired output 51

  28. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 We then compute � (�) the derivative of the divergence w.r.t. the final output of the network y (N) 52

  29. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 We then compute � (�) the derivative of the divergence w.r.t. the final output of the network y (N) We then compute � (�) the derivative of the divergence w.r.t. the pre-activation affine combination z (N) using the chain rule 53

  30. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 Continuing on, we will compute � (�) the derivative of the divergence with respect to the weights of the connections to the output layer 54

  31. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 Continuing on, we will compute � (�) the derivative of the divergence with respect to the weights of the connections to the output layer Then continue with the chain rule to compute � (���) the derivative of the divergence w.r.t. the output of the N-1th layer 55

  32. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 We continue our way backwards in the order shown 56

  33. y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 We continue our way backwards in the order shown 57

  34. y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 We continue our way backwards in the order shown 58

  35. y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 We continue our way backwards in the order shown 59

  36. y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 We continue our way backwards in the order shown 60

  37. y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 We continue our way backwards in the order shown 61

  38. y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 We continue our way backwards in the order shown 62

  39. Backward Gradient Computation • Lets actually see the math.. 63

  40. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 64

  41. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 The derivative w.r.t the actual output of the final layer of the network is simply the derivative w.r.t to the output of the network 65

  42. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 66

  43. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 Already computed 67

  44. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � (�) � 1 1 1 � � Derivative of activation function 68

  45. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � (�) � 1 1 1 � � Derivative of activation function Computed in forward pass 69

  46. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 70

  47. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 71

  48. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 (�) � (�) (�) (�) �� �� � 72

  49. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 (�) � (�) (�) (�) Just computed �� �� � 73

  50. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 Because (���) (�) (�) (�) (���) � � �� � � (�) (�) (�) �� �� � 74

  51. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 Because (���) (�) (�) (�) (���) � � �� � � (�) (�) (�) �� �� � Computed in forward pass 75

  52. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 (���) � (�) (�) �� � 76

  53. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 (���) For the bias term � (���) � (�) (�) �� � 77

  54. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 (�) � (���) (���) (�) � � � � 78

  55. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 (�) � Already computed (���) (���) (�) � � � � 79

  56. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 Because (�) (�) (�) (�) (���) � �� � �� � (���) (���) (�) � � � � 80

  57. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 (�) �� (���) (�) � � � 81

  58. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 (�) �� (���) (�) � � � 82

  59. Computing derivatives y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 We continue our way backwards in the order shown (���) � ��� � (���) (���) � � 83

  60. y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 We continue our way backwards in the order shown (���) For the bias term � (���) � (���) (���) �� � 84

  61. y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 We continue our way backwards in the order shown (���) �� (���) (���) � � � 85

  62. y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 We continue our way backwards in the order shown (���) � ��� � (���) (���) � � 86

  63. y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 We continue our way backwards in the order shown (�) �� (�) (�) � � � 87

  64. y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 We continue our way backwards in the order shown (�) � � � (�) (�) � � 88

  65. y (0) y (N-2) y (N-1) z (1) y (1) z (N-2) z (N-1) ��� ��� � z (N) y (N) f N Div(Y,d) � ��� ��� f N ��� ��� � ��� ��� � 1 1 1 (�) � (�) (�) We continue our way backwards in the order shown �� � 89

  66. Gradients: Backward Computation z (k-1) y (k-1) z (k) y (k) z (N-1) y (N-1) z (N) y (N) f N Div(Y,d) Div(Y,d) f N Figure assumes, but does not show the “1” bias nodes Initialize: Gradient w.r.t network output (���) (�) � � �� � (�) (���) (�) (�) (�) � � � � � � � (�) (�) � � (���) (���) � � (�) (�) 90 �� � � �

  67. Backward Pass • Output layer (N) : – For � ���� (�) = ����(�,�) • �� � �� � ���� (�) = ���� � 𝑨 � (�) • (�) 𝑔 � �� � �� � • For layer – For � ���� ���� (���) • (�) = ∑ 𝑥 �� � (���) �� � �� � ���� (�) = ���� � 𝑨 � (�) • (�) 𝑔 � �� � �� � ���� ���� (�) • (���) = 𝑧 � (���) for 𝑘 = 1 … 𝐸 � �� �� �� � ���� (�) ���� – (�) for � (�) � �� �� �� � 91

  68. Backward Pass Called “Backpropagation” because • Output layer (N) : the derivative of the loss is – For � propagated “backwards” through the network ���� (�) = ����(�,�) • �� � �� � ���� (�) = ���� � 𝑨 � (�) • (�) 𝑔 � �� � �� � • Very analogous to the forward pass: For layer – For � Backward weighted combination of next layer ���� ���� (���) • (�) = ∑ 𝑥 �� � (���) �� � �� � Backward equivalent of activation ���� (�) = ���� � 𝑨 � (�) • (�) 𝑔 � �� � �� � ���� ���� (�) • (���) = 𝑧 � (���) for 𝑘 = 1 … 𝐸 � �� �� �� � ���� (�) ���� – (�) for � (�) � �� �� �� � 92

  69. Backward Pass ����(�,�) Using notation etc (overdot represents derivative of w.r.t variable) �� Called “Backpropagation” because • Output layer (N) : the derivative of the loss is – For propagated “backwards” through � the network ���� (�) • � �� � (�) (�) (�) � • � � � � • For layer Very analogous to the forward pass: – For � Backward weighted combination (�) (���) (���) of next layer • � � �� � Backward equivalent of activation (�) (�) (�) � • � � � � ���� (�) (���) for • � (���) � � �� �� ���� (�) (�) for – � (�) � � �� �� 93

  70. For comparison: the forward pass again • Input: dimensional vector • Set: , is the width of the 0 th (input) layer – – ; • For layer – For (�) � � (�) (���) • � ��� �,� � (�) (�) • � � � • Output: – 94

  71. Special cases • Have assumed so far that 1. The computation of the output of one neuron does not directly affect computation of other neurons in the same (or previous) layers 2. Inputs to neurons only combine through weighted addition 3. Activations are actually differentiable – All of these conditions are frequently not applicable • Will not discuss all of these in class, but explained in slides – Will appear in quiz. Please read the slides 95

  72. Special Case 1. Vector activations y (k-1) z (k) y (k) y (k-1) z (k) y (k) • Vector activations: all outputs are functions of all inputs 96

  73. Special Case 1. Vector activations y (k-1) y (k-1) z (k) z (k) y (k) y (k) Scalar activation: Modifying a Vector activation: Modifying a only changes corresponding potentially changes all, 97

  74. “Influence” diagram y (k-1) y (k-1) z (k) y (k) z (k) y (k) Scalar activation: Each Vector activation: Each influences one influences all, 98

  75. The number of outputs y (k-1) y (k-1) z (k) y (k) z (k) y (k) • Note: The number of outputs (y (k) ) need not be the same as the number of inputs (z (k) ) • May be more or fewer 99

  76. Scalar Activation: Derivative rule y (k-1) z (k) y (k) • In the case of scalar activation functions, the derivative of the error w.r.t to the input to the unit is a simple product of derivatives 100

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Neural Networks Learning the network: Part 1 11-785, Fall 2020 Lecture 3 1 Topics for the day

Neural Networks Learning the network: Part 1 11-785, Fall 2020 Lecture 3 1 Topics for the day

Neural Networks Learning the network: Part 1 11-785, Fall 2020 Lecture 3 1 Topics for the day The problem of learning The perceptron rule for perceptrons And its inapplicability to multi-layer perceptrons Greedy solutions for

1.96k views • 160 slides
Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network

Neural Networks for Machine Learning Lecture 2a An overview of the main types of neural network architecture Geoffrey Hinton with Nitish Srivastava Kevin Swersky Feed-forward neural networks These are the commonest type of neural

534 views • 32 slides
(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning

(Very) Brief Introduction to Neural Networks IITP-03 Algorithms for NLP 1 / 31 Learning objectives What are neural networks? What are deep neural networks? How do we train neural networks? What variants of neural network

331 views • 31 slides
Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training

Neural Networks and their Application to Go A. Bausch Neural Networks and their Application to Go Neural Networks Learning Blackjack Theory Training neural networks Problems AlphaGo Anne-Marie Bausch The Game of Go Policy Network

280 views • 24 slides
Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics

Deep Learning Primer Nishith Khandwala Neural Networks Overview Neural Network Basics Activation Functions Stochastic Gradient Descent (SGD) Regularization (Dropout) Training Tips and Tricks Neural Network (NN)

828 views • 45 slides
Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Propagating Error Backward Hyperparameters for Neural Networks } Multi-layer (deep) neural

Learning in Neural Networks w 1,3 w 3,5 1 3 5 w w 1,4 3,6 w w 2,3 4,5 2 4 6 w w 2,4 4,6 Class #18: Back-Propagation; } A neural network can learn a classification function by Tuning Hyper-Parameters adjusting its weights to

259 views • 4 slides
Neural Network Part 3: Convolutional Neural Networks Yingyu Liang Computer Sciences 760 Fall

Neural Network Part 3: Convolutional Neural Networks Yingyu Liang Computer Sciences 760 Fall

Neural Network Part 3: Convolutional Neural Networks Yingyu Liang Computer Sciences 760 Fall 2017 http://pages.cs.wisc.edu/~yliang/cs760/ Some of the slides in these lectures have been adapted/borrowed from materials developed by Mark Craven,

818 views • 48 slides
Neural Networks Learning the network: Part 1 11-785, Spring 2018 Lecture 3 1 Designing a net..

Neural Networks Learning the network: Part 1 11-785, Spring 2018 Lecture 3 1 Designing a net..

Neural Networks Learning the network: Part 1 11-785, Spring 2018 Lecture 3 1 Designing a net.. Input: An N-D real vector Output: A class (binary classification) Input units? Output units? Architecture? Output

1.98k views • 156 slides
Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 Story so far Neural

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 Story so far Neural

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 Story so far Neural networks are universal approximators Can model any odd thing Provided they have the right architecture We must train them to approximate

1.37k views • 134 slides
Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Neural Networks Neural networks arise from attempts to model Neural Networks human/animal

Feed-forward Networks Network Training Error Backpropagation Deep Learning Feed-forward Networks Network Training Error Backpropagation Deep Learning Neural Networks Neural networks arise from attempts to model Neural Networks

380 views • 9 slides
Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture

Neural Network Part 3: Convolutional Neural Networks CS 760@UW-Madison Goals for the lecture you should understand the following concepts convolutional neural networks (CNN) convolution and its advantage pooling and its advantage 2

750 views • 50 slides
Neural Network Part 4: Recurrent Neural Networks Yingyu Liang Computer Sciences 760 Fall 2017

Neural Network Part 4: Recurrent Neural Networks Yingyu Liang Computer Sciences 760 Fall 2017

Neural Network Part 4: Recurrent Neural Networks Yingyu Liang Computer Sciences 760 Fall 2017 http://pages.cs.wisc.edu/~yliang/cs760/ Some of the slides in these lectures have been adapted/borrowed from materials developed by Mark Craven,

384 views • 37 slides
Artificial Neural Networks By: Kodi Neumiller Overview What is an artificial neural network

Artificial Neural Networks By: Kodi Neumiller Overview What is an artificial neural network

Artificial Neural Networks By: Kodi Neumiller Overview What is an artificial neural network Perceptron Feed forward Backpropagation Learning rate/momentum What is a neural network? Artificial Human Neurons Each

888 views • 12 slides
Neural Networks: What can a network represent Deep Learning, Fall 2020 1 Recap : Neural

Neural Networks: What can a network represent Deep Learning, Fall 2020 1 Recap : Neural

Neural Networks: What can a network represent Deep Learning, Fall 2020 1 Recap : Neural networks have taken over AI Tasks that are made possible by NNs, aka deep learning Tasks that were once assumed to be purely in the human domain

1.92k views • 150 slides
Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1

CSCI 5525 Machine Learning Fall 2019 Lecture 11: Neural Networks (Part 3) March 2nd, 2020 Lecturer: Steven Wu Scribe: Steven Wu 1 Convolutional Neural Networks We will now study a special type of neural networks convolutional neural

662 views • 6 slides
Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 1 Story so far

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 1 Story so far

Neural Networks: Optimization Part 1 Intro to Deep Learning, Fall 2017 1 Story so far Neural networks are universal approximators Can model any odd thing Provided they have the right architecture We must train them to approximate

1.58k views • 134 slides
Pair Programming Review and Practice Lots of time to work on Homeworks 3 and 4 Session 4 CSSE

Pair Programming Review and Practice Lots of time to work on Homeworks 3 and 4 Session 4 CSSE

As you arrive: 1. Start up your computer and plug it in 2. Log into Angel and go to CSSE 120 3. Do the Attendance Widget the PIN is on the board 4. Go to the course Schedule Page From your bookmark , or from the Lessons tab in Angel 5.

118 views • 10 slides
texdoc 2.0 An update on creating LaTeX documents from within Stata Ben Jann University of Bern,

texdoc 2.0 An update on creating LaTeX documents from within Stata Ben Jann University of Bern,

texdoc 2.0 An update on creating LaTeX documents from within Stata Ben Jann University of Bern, ben.jann@soz.unibe.ch 2016 German Stata Users Group Meeting GESIS, Cologne, June 10, 2016 Ben Jann (University of Bern) texdoc 2.0 Cologne,

716 views • 17 slides
Creating LaTeX and HTML documents from within Stata using textdoc and webdoc Ben Jann University

Creating LaTeX and HTML documents from within Stata using textdoc and webdoc Ben Jann University

Creating LaTeX and HTML documents from within Stata using textdoc and webdoc Ben Jann University of Bern, ben.jann@soz.unibe.ch Swiss Stata Users Group meeting Bern, November 17, 2016 Ben Jann (University of Bern) texdoc/webdoc Bern,

254 views • 23 slides
CS7015 (Deep Learning) : Lecture 11 Convolutional Neural Networks, LeNet, AlexNet, ZF-Net, VGGNet,

CS7015 (Deep Learning) : Lecture 11 Convolutional Neural Networks, LeNet, AlexNet, ZF-Net, VGGNet,

CS7015 (Deep Learning) : Lecture 11 Convolutional Neural Networks, LeNet, AlexNet, ZF-Net, VGGNet, GoogLeNet and ResNet Mitesh M. Khapra Department of Computer Science and Engineering Indian Institute of Technology Madras 1/1 Mitesh M. Khapra

845 views • 72 slides
Recurrent Networks : 1 Spring 2020 Instructor: Bhiksha Raj 1 Which open source project? 2

Recurrent Networks : 1 Spring 2020 Instructor: Bhiksha Raj 1 Which open source project? 2

Deep Learning Recurrent Networks : 1 Spring 2020 Instructor: Bhiksha Raj 1 Which open source project? 2 Related math. What is it talking about? 3 And a Wikipedia page explaining it all 4 The unreasonable effectiveness of recurrent

1.41k views • 117 slides
Multiple Input and Output Channels Multiple Input and Output Channels Multiple Input Channels In

Multiple Input and Output Channels Multiple Input and Output Channels Multiple Input Channels In

2/23/2019 channels slides Multiple Input and Output Channels Multiple Input and Output Channels Multiple Input Channels In [1]: import d2l from mxnet import nd def corr2d_multi_in(X, K): # First, traverse along the 0th dimension (channel

457 views • 6 slides
Introduction to Deep Learning Georgia Tech CS 4650/7650 Fall 2020 Outline Deep Learning

Introduction to Deep Learning Georgia Tech CS 4650/7650 Fall 2020 Outline Deep Learning

Introduction to Deep Learning Georgia Tech CS 4650/7650 Fall 2020 Outline Deep Learning CNN RNN Attention Transformer Pytorch Introduction Basics Examples CNNs Some slides borrowed from

575 views • 56 slides
T HE Z ERO L OWER B OUND , THE D UAL M ANDATE , AND U NCONVENTIONAL D YNAMICS William T. Gavin

T HE Z ERO L OWER B OUND , THE D UAL M ANDATE , AND U NCONVENTIONAL D YNAMICS William T. Gavin

I NTRODUCTION M ODELS M ODEL 1 R ESULTS M ODEL 2 R ESULTS M ODEL C OMPARISON C ONCLUSION T HE Z ERO L OWER B OUND , THE D UAL M ANDATE , AND U NCONVENTIONAL D YNAMICS William T. Gavin Federal Reserve Bank of St. Louis Benjamin D. Keen University

652 views • 29 slides

More recommend