Neural Net Backpropagation 3/20/17 Recall: Limitations of - - PowerPoint PPT Presentation

Neural Net Backpropagation

3/20/17

Recall: Limitations of Perceptrons

• vs.

AND and OR are linearly separable. XOR isn’t

What is the output of the network?

f(x) = ( x < 0 1 x ≥ 0 f(x) = 1 1 + e−x f(x) = ( x < 0 x x ≥ 0

How can we train these networks?

Two reasons the perceptron algorithm won’t work:

• 1. Non-threshold activation functions.
• 2. Multiple layers (what’s the correction for hidden

nodes?). Key idea: stochastic gradient descent (SGD).

• Compute the error on a random training example.
• Compute the derivative of the error with respect to

each weight.

• Update weights in the direction that reduces error.

Problem: SGD on threshold functions

• The derivative of this function is always 0.
• We can’t “move in the direction of the gradient”.

Better Activation Functions

sigmoid tanh RELU

RELU(x) = ( x < 0 x x ≥ 0 σ(x) = 1 1 + e−x tanh(x) = 1 + e−2x 1 − e−2x

Derivatives of Activation Functions

σ(x) = 1 1 + e−x dσ(x) dx = σ(x)(1 − σ(x)) d tanh(x) dx = 1 − tanh2(x) tanh(x) = 1 + e−2x 1 − e−2x RELU(x) = ( x < 0 x x ≥ 0

sigmoid tanh RELU

dRELU(x) dx = ( x ≤ 0 1 x > 0

Error Gradient

• Define training error as squared difference

between a node’s output and the target:

• Compute gradient of error with respect to weights:

… … … algebra ensues … … …

E(~ w, ~ x) = (t − o)2 ∂E ∂wi

• =

1 1 + e− ~

w·~ x

~ w · ~ x = X

i

wixi

sigmoid

∂E ∂wi = −o(1 − o)(t − o)xi

Output Node Gradient Descent Step

wi+ = −α ∂E ∂wi wi+ = α(o)(1 − o)(t − o)xi

sigmoid

w0 += .5 · .7(1 − .7)(.9 − .7)2 → wi = 1.04 w1 += .5 · .7(1 − .7)(.9 − .7)1.2 → wi = −.97 α = .5

What about hidden layers?

• Use the chain rule to compute error derivatives for

previous layers.

• This turns out to be much easier than it sounds.

Let 𝜀k be the error we computed for output-node k. The error for hidden node h comes from the sum of its contribution to the errors for each output node.

sigmoid

δk = ok(1 − ok)(tk − ok) X

k∈output

whkδk

Hidden Node Gradient Descent Step

• Compute the contribution to next-layer errors:
• Update incoming weights using 𝜀h as the error:

δh = oh(1 − oh) X

k∈next layer

whkδk wi+ = αδhxi

Backpropagation Algorithm

for 1:training runs for example in shuffled training data: run example through network compute error for each output node for each layer (starting from output): for each node in layer: gradient descent update on incoming weights

Example Backpropagation Update

wi+ = α(o)(1 − o)(t − o)xi δh = oh(1 − oh) X

k∈next layer

whkδk

σ(x) = 1 1 + e−x