Multi-Layer Networks & Back-Propagation M. Soleymani Deep - PowerPoint PPT Presentation

Choosing cost function: Examples } Regression problem + 𝐹 = 1 𝐹 5 56$ – SSE 𝑦 $ 𝐹 5 = 1 ) 2 𝑝 5 − 𝑧 5 One dimensional output 𝑧 $ 𝐹 5 = 1 > ) ) 𝒑 5 − 𝒛 5 5 − 𝑧 ; 5 𝑧 > = 1 𝑝 ; Multi-dimensional output 2 𝑦 9 ;6$ } Classification problem – Cross-entropy • Binary classification 𝑚𝑝𝑡𝑡 5 = −𝑧 5 log 𝑝 5 − (1 − 𝑧 5 ) log(1 − 𝑝 5 ) Output layer uses sigmoid activation function 31

Multi-class output: One-hot representations • Consider a network that must distinguish if an input is a cat, a dog, a camel, a hat, or a flower • For inputs of each of the five classes the desired output is: Cat : [1 0 0 0 0 ] T dog : [0 1 0 0 0 ] T camel : [0 0 1 0 0 ] T hat : [0 0 0 1 0 ] T flower : [0 0 0 0 1 ] T • For input of any class, we will have a five-dimensional vector output with four zeros and a single 1 at the position of the class • This is a one hot vector 32

Multi-class networks Input Hidden Layers L ayer Output Layer • For a multi-class classifier with N classes, the one-hot representation will have N binary outputs – An N-dimensional binary vector • The neural network’s output too must ideally be binary (N-1 zeros and a single 1 in the right place) • More realistically, it will be aprobability vector – N probability values that sum to 1. 33

Vector activation example: Softmax • Example: Softmax vector activation Parameters are weights and bias 𝑝 U 34

Vector Activations Input Hidden Layers L ayer Output Layer • We can also have neuron that have multiple couple outputs 𝑧 $ , 𝑧 ) , … , 𝑧 V = 𝑔(𝑦 $ , 𝑦 ) , … , 𝑦 ; ; 𝑿) – Function 𝑔(. ) operates on set of inputs to produce set of outputs – Modifying a single parameter in 𝑿 will affect all outputs 35

� � Multi-class classification: Output Input Hidden Layers L ayer Output Layer s o f t m a x • Softmax vector activation is often used at the output of multi-class classifier nets (V) 𝑏 Y (5[$) 𝑨 U = 1 𝑥 YU Y 𝑝 U = exp (𝑨 U ) ∑ exp (𝑨 Y ) Y • This can be viewed as the probability 𝑝 U = 𝑄 𝑑𝑚𝑏𝑡𝑡 = 𝑗 𝒚 36

� For multi-class classification y 1 y 2 y 3 y 4 K L Div() E • Desired output 𝑧 is one hot vector 0 0 … 1 … 0 0 0 wit the 1 in the 𝑑 -th position(for class c) • Actual output will be probability distribution [𝑝 $ , 𝑝 ) , … , 𝑝 V ] • The cross-entropy between the desired one-hot output and actual output 𝑀 𝒛, 𝒑 = − 1 𝑧 U 𝑚𝑝𝑕𝑝 U = −𝑚𝑝𝑕𝑝 a U • Derivative = b− 1 𝑒𝑀(𝒛, 𝒑) 𝑔𝑝𝑠 𝑢ℎ𝑓 𝑑 𝑢ℎ 𝑑𝑝𝑛𝑞𝑝𝑜𝑓𝑜𝑢 The slopeis negative w.r.t. 𝑝 a 𝑝 a 𝑒𝑝 U Indicates increasing 𝑝 a will reduce divergence 0 𝑔𝑝𝑠 𝑠𝑓𝑛𝑏𝑗𝑜𝑗𝑜𝑕 𝑑𝑝𝑛𝑞𝑝𝑜𝑓𝑜𝑢 𝒑 𝑀(𝒛, 𝒑) = [0 0 … −1 𝛼 … 0 0 ] 𝑝 a 37

� For multi-class classification y 1 y 2 y 3 y 4 K L Div() E • Desired output 𝑧 is one hot vector 0 0 … 1 … 0 0 0 wit the 1 in the 𝑑 -th position(for class c) • Actual output will be probability distribution [𝑝 $ , 𝑝 ) , … , 𝑝 V ] • The cross-entropy between the desired one-hot output and actual output 𝑀 𝒛, 𝒑 = − 1 𝑧 U 𝑚𝑝𝑕𝑝 U = −𝑚𝑝𝑕𝑝 a U • Derivative The slopeis negative w.r.t. 𝑝 a = b− 1 Indicates increasing 𝑝 a will reduce divergence 𝑒𝑀(𝒛, 𝒑) 𝑔𝑝𝑠 𝑢ℎ𝑓 𝑑 𝑢ℎ 𝑑𝑝𝑛𝑞𝑝𝑜𝑓𝑜𝑢 𝑝 a 𝑒𝑝 U Note: when 𝒛 = 𝒑 the 0 𝑔𝑝𝑠 𝑠𝑓𝑛𝑏𝑗𝑜𝑗𝑜𝑕 𝑑𝑝𝑛𝑞𝑝𝑜𝑓𝑜𝑢 derivative is not 0 𝒑 𝑀(𝒛, 𝒑) = [0 0 … −1 𝛼 … 0 0 ] 𝑝 a 38

Choosing cost function: Examples } Regression problem + 𝐹 = 1 𝐹 5 56$ – SSE 𝑦 $ 𝐹 5 = 1 ) 2 𝑝 5 − 𝑧 5 One dimensional output 𝑧 $ 𝐹 5 = 1 > ) ) 𝒑 5 − 𝒛 5 5 − 𝑧 ; 5 𝑧 > = 1 𝑝 ; Multi-dimensional output 2 𝑦 9 ;6$ } Classification problem – Cross-entropy 1 𝑚𝑝𝑡𝑡 5 = −𝑧 5 log 𝑝 5 − (1 − 𝑧 5 ) log(1 − 𝑝 5 ) 𝑝 = • Binary classification 1 + 𝑓 s • Multi-class classification Output layer uses sigmoid activation function 𝑚𝑝𝑡𝑡 5 = −log 𝑝 i (j) k lm Output is found by a softmax layer 𝑝 U = k ln o ∑ npq 40

Problem setup • Given: the architecture of thenetwork • Training data: A set of input-output pairs 𝒚 ($) , 𝒛 ($) , 𝒚 ()) , 𝒛 ()) , … , (𝒚 (+) , 𝒛 (+) ) • We need a loss function to show how penalizes the obtained output 𝑝 = 𝑔(𝒚; 𝑿) when the desired output is 𝒛 + + = 1 𝐹(𝑿) = 1 𝑚𝑝𝑡𝑡 𝒑 (5) , 𝒛 (5) 𝑂 1 𝑚𝑝𝑡𝑡 𝑔 𝒚 (5) ; 𝑿 , 𝒛 (5) 56$ 56$ ; , 𝑐 [;] • Minimize 𝐹 w.r.t. 𝑿 that containts 𝑥 U,Y Y 41

How to adjust weights for multi layer networks? • We need multiple layers of adaptive, non-linear hidden units. But how can we train such nets? – We need an efficient way of adapting all the weights, not just the last layer. – Learning the weights going into hidden units is equivalent to learning features. – This is difficult because nobody is telling us directly what the hidden units should do. 42

Find the weights by optimizing the cost • Start from random weights and then adjust them iteratively to get lower cost. • Update the weights according to the gradient of the cost function Source: http://3b1b.co 43

How does the network learn? • Which changes to the weights do improve the most? 𝛼𝐹 𝛼𝐹 • The magnitude of each element shows how sensitive the cost is to that weight or bias. Source: http://3b1b.co 44

Training multi-layer networks • Back-propagation – Training algorithm that is used to adjust weights in multi-layer networks (based on the training data) – The back-propagation algorithm is based on gradient descent – Use chain rule and dynamic programming to efficiently compute gradients 45

Training Neural Nets through Gradient Descent Total training error: + 𝐹 = 1 𝑚𝑝𝑡𝑡 𝒑 (5) , 𝒛 (5) 56$ • Gradient descent algorithm Assuming the bias is also [;] • Initialize all weights and biases 𝑥 UY represented as a weight – Using the extended notation : the bias is also weight • Do : – For every layer 𝑙 for all 𝑗, 𝑘 update: [;] = 𝑥 U,Y [;] − 𝜃 9w • 𝑥 U,Y [y] 9x m,n • Until 𝐹 has converged 46

The derivative Total training error: + 𝐹 = 1 𝑚𝑝𝑡𝑡 𝒑 (5) , 𝒛 (5) 56$ • Computing the derivative Total derivative: + [;] = 1 𝑚𝑝𝑡𝑡 𝒑 (5) , 𝒛 (5) 𝑒𝐹 [;] 𝑒𝑥 U,Y 𝑒𝑥 U,Y 56$ 47

Training by gradient descent [;] • Initialize all weights 𝑥 UY • Do : 9w – For all 𝑗 , 𝑘 , 𝑙, initialize [y] = 0 9x m,n – For all 𝑜 = 1: 𝑂 • For every layer 𝑙 for all 𝑗, 𝑘 : 9 V{|| { j ,i j • Compute [y] 9x m,n 9 V{|| { j ,i j 9w [y] += • [y] 9x m,n 9x m,n – For every layer 𝑙 for all 𝑗, 𝑘: [;] = 𝑥 U,Y [;] − } 9w 𝑥 U,Y [y] T 9x m,n 48

The derivative Total training error: + 𝐹 = 1 𝑚𝑝𝑡𝑡 𝒑 (5) , 𝒛 (5) 56$ Total derivative: + [;] = 1 𝑒 𝑚𝑝𝑡𝑡 𝒑 (5) , 𝒛 (5) 𝑒𝐹 [;] 𝑒𝑥 U,Y 𝑒𝑥 U,Y 56$ • So we must first figure out how to compute the derivative of divergences of individual training inputs 49

Calculus Refresher: Basic rules of calculus 𝑒𝑧 𝑒𝑦 ≈ Δ𝑧 with derivative dy Δ𝑦 dx the following must hold for sufficiently small For any differentiable function 1 2 M with partial derivatives ∂ y ∂ y ∂ y ∂ x 1 ∂ x 2 ∂ x M the following must hold for sufficiently small 1 2 M 50

Calculus Refresher: Chainrule For any nested function Check –we can confirm that : 51

Calculus Refresher: Distributed Chain rule Check: 1 2 M 1 2 M 1 2 M 1 2 M 52

Distributed Chain Rule: Influence Diagram 1 1 2 2 M M • 𝑦 affects 𝑧 through each 𝑕 $ , … , 𝑕 € 53

Distributed Chain Rule: Influence Diagram 1 1 1 2 M M M • Small perturbations in cause small perturbations in each o each of which individually additively perturbs 54

Simple chain rule • 𝑨 = 𝑔 𝑕 𝑦 • 𝑧 = 𝑕(𝑦) 55

Multiple paths chain rule 56

Backpropagation: a simple example 57

Backpropagation: a simple example 58

Backpropagation: a simple example 59

Backpropagation: a simple example 60

Backpropagation: a simple example 61

Backpropagation: a simple example 62

How to propagate the gradients backward 𝑨 = 𝑔(𝑦, 𝑧) 63

How to propagate the gradients backward 64

Returning to ourproblem 𝑒 𝑚𝑝𝑡𝑡 𝒑, 𝒛 • How to compute [;] 𝑒𝑥 U,Y 65

A first closer look at the network + + 𝑔 (.) 𝑔 (.) + 𝑝 𝑔 (.) + + 𝑔 (.) 𝑔 (.) • Showing a tiny 2-input network forillustration – Actual network would have many more neurons and inputs • Explicitly separating the weighted sum of inputs from the activation 66

A first closer look at the network (1) (2) + + 1,1 1,1 (3) (1) (2) 1,1 1,2 1,2 + 𝑝 (1) (2) (3) 2,1 2,1 2,1 (1) (2) + + 2,2 2,2 (3) (1) (1) (2) (2) 3,1 3,1 3,2 3,1 3,2 • Showing a tiny 2-input network forillustration – Actual network would have many more neurons and inputs • Expanded with all weights andactivations shown • The overall function is differentiable w.r.t every weight,bias and input 67

Backpropagation: Notation • 𝒃 [‚] ← 𝐽𝑜𝑞𝑣𝑢 • 𝑝𝑣𝑢𝑞𝑣𝑢 ← 𝒃 [†] 𝑔(. ) 𝑔(. ) 𝒃 [V[$] 𝒃 [V] 𝒜 [V] 𝑔(. ) 𝑔(. ) 𝑔(. ) 68

Backpropagation: Last layer gradient For squared error loss: 𝜖𝑚𝑝𝑡𝑡 [†] = 𝑔 𝑨 U † ) [†] 𝑚𝑝𝑡𝑡 = 1 𝑏 U ) † = (𝑧 Y − 𝑏 Y 2 1 𝑝 Y − 𝑧 Y € 𝜖𝑏 Y [†] = 1 𝑥 UY Y [†] 𝑏 U [†[$] 𝑨 Y † 𝑝 Y = 𝑏 Y U6‚ † 𝜖𝑏 Y 𝜖𝑚𝑝𝑡𝑡 [†] = 𝜖𝑚𝑝𝑡𝑡 Output j 𝜖𝑚𝑝𝑡𝑡 [†] [†] † 𝑏 Y † 𝜖𝑥 UY 𝜖𝑏 Y 𝜖𝑥 UY 𝜖𝑏 Y 𝑔 [†] [†] 𝑨 𝜖𝑨 𝜖𝑏 [†] Y [†] 𝑏 U [†] = 𝑔 ‰ 𝑨 Y [†] = 𝑔 ‰ 𝑨 [†] [†[$] 𝜖𝑚𝑝𝑡𝑡 Y Y [†] 𝜖𝑥 UY 𝜖𝑥 UY 𝑥 UY [†] 𝜖𝑥 UY [V[$] 𝑏 U 𝜖𝑚𝑝𝑡𝑡 [†] = 𝜖𝑚𝑝𝑡𝑡 [†] 𝑏 U † 𝑔 ‰ 𝑨 [†[$] Y 𝜖𝑥 UY 𝜖𝑏 Y 69

Activations and theirderivatives 2 [*] • Some popular activation functions andtheir derivatives 70

Previous layers gradients 𝜖 𝑚𝑝𝑡𝑡 [V] 𝑏 Y V [V] = 𝑔 𝑨 U 𝑔 𝜖𝑏 Y [V] 𝑏 U [V] 𝑨 V Y 𝜖𝑏 Y € 𝜖 𝑚𝑝𝑡𝑡 [V] = 𝜖 𝑚𝑝𝑡𝑡 [V] = 1 𝑥 UY [V] 𝑏 U [V[$] 𝑨 Y 𝜖 𝑚𝑝𝑡𝑡 [V] V 𝜖𝑥 UY 𝜖𝑏 Y 𝜖𝑥 UY [V] 𝑥 UY [V] U6‚ 𝜖𝑥 UY [V[$] [V] [V] 𝑏 U 𝜖𝑏 [V] 𝜖𝑏 Y 𝜖𝑨 [V] 𝑏 U [V] = 𝑔 ‰ 𝑨 Y [V[$] Y [V] = [V] × 𝜖𝑥 UY 𝜖𝑨 𝜖𝑥 UY Y 𝜖 𝑚𝑝𝑡𝑡 𝜖 𝑚𝑝𝑡𝑡 [V] [V] 9 [‹] 𝜖𝑏 $ [V] [V] 𝜖𝑏 9 [‹] [V] 𝜖𝑏 Y 𝜖𝑨 Y 𝑏 Y 𝜖 𝑚𝑝𝑡𝑡 [V[$] = 1 𝜖 𝑚𝑝𝑡𝑡 [V] × [V] × [V[$] 𝜖𝑏 U 𝜖𝑏 Y 𝜖𝑨 Y 𝜖𝑏 U [V] 𝑨 Y6$ Y 9 [‹] = 1 𝜖 𝑚𝑝𝑡𝑡 [V] ×𝑥 UY [V] ×𝑔 ‰ 𝑨 Y [V] [V] 𝜖𝑏 Y 𝑥 UY Y6$ 𝜖 𝑚𝑝𝑡𝑡 [V[$] 𝑏 U [V[$] 𝜖𝑏 U 71

Backpropagation: [V] 𝑏 Y [V] 𝜖𝑏 Y 𝜖 𝑚𝑝𝑡𝑡 [V] = 𝜖 𝑚𝑝𝑡𝑡 [V] = 𝑔 𝑨 U 𝑔 [V] 𝑏 U [V] × [V] € [V] 𝑨 Y 𝜖𝑥 UY 𝜖𝑏 Y 𝜖𝑥 UY [V] = 1 𝑥 UY [V] 𝑏 U [V[$] 𝑨 Y U6‚ [V] [V[$] ×𝑔 ‰ 𝑨 Y [V] ×𝑏 U [V] 𝑥 UY = 𝜀 Y [V[$] 𝑏 U [V] = • V{|| [V] } 𝜀 [‹] is the sensitivity of the output to 𝑏 Y Y •Ž n } Sensitivity vectors can be obtained by running a backward process in the network architecture (hence the name backpropagation.) We will compute 𝜺 [V[$] from 𝜺 [V] : 9 [‹] [V[$] = 1 𝜀 [V] ×𝑥 UY [V] ×𝑔 ‰ 𝑨 [V] 𝜀 U Y Y Y6$ 72

Find and save 𝜺 [†] • Called error, computed recursively in backward manner • For the final layer 𝑚 = 𝑀 : [†] = 𝜖 𝑚𝑝𝑡𝑡 𝜀 Y [†] 𝜖𝑏 Y 73

Compute 𝜺 [V[$] from 𝜺 [V] [V] = • V{|| [V] } 𝜀 [‹] is the sensitivity of the output to 𝑏 Y Y •Ž n [V] = 𝑔 𝑨 U [V] 𝑏 U € } Sensitivity vectors can be obtained by running a backward process in [V] = 1𝑥 UY [V] 𝑏 U [V[$] 𝑨 Y the network architecture (hence the name backpropagation.) Y6‚ [V] 𝑏 Y 9 [‹] 𝑔 [V[$] = 1 𝜀 [V] ×𝑥 UY [V] ×𝑔 ‰ 𝑨 [V] 𝜀 U [V] Y Y 𝑨 Y Y6$ [V] 𝑥 UY [V[$] 𝑏 U 74

Backpropagation Algorithm • Initialize all weights to small random numbers. • While not satisfied • For each training example do : 1. Feed forward the training example to the network and compute the outputs of all units in forward step (z and a) and the loss 2. For each unit find its 𝜀 in the backward step [V] as 𝑥 UY [V] ← 𝑥 UY [V] − 𝜃 • V{|| • V{|| [V] ×𝑏 U [V[$] × 3. Update each network weight 𝑥 UY [‹] where [‹] = 𝜀 Y •x mn •x mn 𝑔 ‰ 𝑨 [V] Y 75

Another example 76

Another example 77

Another example 78

Another example 79

Another example 80

Another example 81

Another example 82

Another example 83

Another example 84

Another example 85

Another example 86

Another example 87

Another example 88

Another example [local gradient] x [upstream gradient] x0: [2] x [0.2] = 0.4 w0: [-1] x [0.2] = -0.2 89

Derivative of sigmoid function 90

Derivative of sigmoid function 91

Patterns in backward flow • add gate: gradient distributor • max gate: gradient router 92

Modularized implementation: forward / backward API 93

Modularized implementation: forward / backward API 94

Modularized implementation: forward / backward API 95

Ve Vector formulation • For layered networks it is generally simpler to think of the process in terms of vector operations – Simpler arithmetic – Fast matrix libraries make operations much faster • We can restate the entire process in vector terms – This is what is actually used in any real system 97

The Ja Th Jacobi bian • The derivative of a vector function w.r.t. vector input is called a Jacobian • It is the matrix of partial derivatives given below 𝑧 $ 𝑦 $ 𝜖𝑧 $ 𝜖𝑧 $ 𝜖𝑧 $ ⋯ 𝑧 ) 𝑦 ) 𝜖𝑦 $ 𝜖𝑦 ) 𝜖𝑦 9 = 𝑔 ⋮ ⋮ 𝜖𝑧 ) 𝜖𝑧 ) 𝜖𝑧 ) 𝜖𝒛 ⋯ 𝑧 ‘ 𝑦 9 𝜖𝒚 = 𝜖𝑦 $ 𝜖𝑦 ) 𝜖𝑦 9 ⋯ ⋯ ⋱ ⋯ Using vector notation 𝜖𝑧 ‘ 𝜖𝑧 ‘ 𝜖𝑧 ‘ 𝐳 = 𝑔 𝐲 ⋯ 𝜖𝑦 $ 𝜖𝑦 ) 𝜖𝑦 9 ∆𝐳 = 𝜖𝒛 Check: 𝜖𝒚 ∆𝐲 98

Matrix calculus 𝜖𝑧 𝜖𝑧 𝜖𝑧 • Scalar-by-Vector 𝜖𝒚 = … 𝜖𝑦 $ 𝜖𝑦 5 𝜖𝑧 $ 𝜖𝑧 $ … 𝜖𝑦 $ 𝜖𝑦 5 𝜖𝒛 • Vector-by-Vector 𝜖𝒚 = ⋮ ⋱ ⋮ 𝜖𝑧 ‘ 𝜖𝑧 ‘ … 𝜖𝑦 $ 𝜖𝑦 5 𝜖𝑧 𝜖𝑧 … 𝜖𝐵 $$ 𝜖𝐵 $5 𝜖𝑧 • Scalar-by-Matrix 𝜖𝑩 = ⋮ ⋱ ⋮ 𝜖𝑧 𝜖𝑧 … 𝜖𝐵 ‘$ 𝜖𝐵 ‘5 • Vector-by-Matrix 𝜖𝑧 = 𝜖𝑧 𝜖𝒜 𝜖𝐵 UY 𝜖𝒜 𝜖𝐵 UY 99

Vector-by-matrix gradients 100

Examples 101

Ja Jacob obians c can d des escribe t e the d e der erivatives es of n of neu eural a activation ons w. w.r.t their input z a 𝑒𝑏 $ 0 ⋯ 0 𝑒𝑨 $ 𝑒𝑏 ) 𝜖𝒃 0 ⋯ 0 𝜖𝒜 = 𝑒𝑨 ) ⋯ ⋯ ⋱ ⋯ 𝑒𝑏 ‘ 0 0 ⋯ 𝑒𝑨 ‘ • For Scalar activations – Number of outputs is identical to the number of inputs • Jacobian is a diagonal matrix – Diagonal entries are individual derivatives of outputs w.r.t inputs – Not showing the superscript “[k]” in equations for brevity 102

Ja Jacob obians c can d des escribe t e the d e der erivatives es of n of neu eural ac activ tivatio tions ns w.r.t t the their ir input input z a 𝑏 U = 𝑔 𝑨 U 𝑔′ 𝑨 $ 0 ⋯ 0 𝜖𝒃 0 𝑔′ 𝑨 ) ⋯ 0 𝜖𝒜 = ⋯ ⋯ ⋱ ⋯ 0 0 ⋯ 𝑔′ 𝑨 ‘ • For scalar activations (shorthand notation): – Jacobian is a diagonal matrix – Diagonal entries are individual derivatives of outputs w.r.t inputs 103