Recurrent Neural Networks: Stability analysis and LSTMs M. - - PowerPoint PPT Presentation
Recurrent Neural Networks: Stability analysis and LSTMs M. Soleymani Sharif University of Technology Spring 2020 Most slides have been adopted from Bhiksha Raj, 11-785, CMU 2019 and some from Fei Fei Li and colleagues lectures, cs231n,
Sharif University of Technology Spring 2020 Most slides have been adopted from Bhiksha Raj, 11-785, CMU 2019 and some from Fei Fei Li and colleagues lectures, cs231n, Stanford 2017.
– These are “Time delay Neural Nets” (TDNNs), AKA convnets
Stock vector X(t) X(t+1) X(t+2) X(t+3) X(t+4) X(t+5) X(t+6) X(t+7) Y(t+6)
2
– These are recurrent neural networks
Time X(t) Y(t) t=0 h-1
3
number
– Input is binary – Will require large number of training instances
– Network trained for N-bit numbers will not work for N+1 bit numbers
1 0 0 0 1 1 0 0 1 0 1 1 0 0 1 0 1 1 0 0 MLP 1 0 1 0 1 0 1 1 1 1 0
4
1 1 RNN unit Previous carry Carry
5
– XOR network, quite complex – Fixed input size
1 0 0 0 1 1 0 0 1 0 MLP 1
6
1 1 RNN unit Previous
7
– Through gradient descent and backpropagation
Time X(t) Y(t) t=0 h-1 Loss Ydesired(t)
8
h0
𝑌(1) 𝑌(2) 𝑌(𝑈 − 2) 𝑌(𝑈 − 1) 𝑌(𝑈) 𝑍(1) 𝑍(2) 𝑍(𝑈 − 2) 𝑍(𝑈 − 1) 𝑍(𝑈) 𝑍)*+,-)(1. . 𝑈) 𝑀𝑝𝑡𝑡
by the network and the desired sequence of outputs
§ Unless we explicitly define it that way
9
divergence at individual instants
𝑀𝑝𝑡𝑡 𝑍
)*+,-) 1 … 𝑈 , 𝑍(1 … 𝑈) = 5 𝑀𝑝𝑡𝑡 𝑍 )*+,-) 𝑢 , 𝑍(𝑢)
𝛼9())𝑀𝑝𝑡𝑡 𝑍
)*+,-) 1 … 𝑈 , 𝑍(1 … 𝑈) = 𝛼9())𝑀𝑝𝑡𝑡 𝑍 )*+,-) 𝑢 , 𝑍(𝑢)
Time X(t) Y(t) t=1 h0 Y(t) Loss
Ytarget(t)
10
– If the largest Eigen value is greater than 1, the system will “blow up” – If it is lesser than 1, the response will “vanish” very quickly
11
– This is a highly desirable characteristic
12
– The function 𝑔() has bounded output for bounded input
– 𝑌(𝑢) is bounded
X(t+1) X(t+2) X(t+3) X(t+4) X(t+5) X(t+6) X(t+7) Y(t+5)
13
𝑍 𝑢 = 𝑔(𝑌 𝑢 − 𝑗 , 𝑗 = 1. . 𝐿)
Time X(t) Y(t) t=0 h-1
14
– Guaranteed if output and hidden activations are bounded
– What if the activations are linear?
Time X(t) Y(t) t=0 h-1
15
– Will assume only a single hidden layer for simplicity
Time X(t) Y(t) t=0 h-1
16
17
– Will attempt to extrapolate to non-linear systems subsequently
– 𝑨? = 𝑋
Bℎ?CD + 𝑋 F𝑦?, ℎ?= 𝑨?
Time X(t) Y(t) t=0 h-1
18
Bℎ?CD + 𝑋 F𝑦?
– ℎ?CD = 𝑋
Bℎ?CH + 𝑋 F𝑦?CD
B Hℎ?CH + 𝑋 B𝑋 F𝑦?CD + 𝑋 F𝑦?
19
Bℎ?CD + 𝑋 F𝑦?
– ℎ?CD = 𝑋
Bℎ?CH + 𝑋 F𝑦?CD
B Hℎ?CH + 𝑋 B𝑋 F𝑦?CD + 𝑋 F𝑦?
B ?IDℎCD + 𝑋 B ?𝑋 F𝑦J + 𝑋 B ?CD𝑋 F𝑦D + ⋯
20
Bℎ?CD + 𝑋 F𝑦?
– ℎ?CD = 𝑋
Bℎ?CH + 𝑋 F𝑦?CD
B Hℎ?CH + 𝑋 B𝑋 F𝑦?CD + 𝑋 F𝑦?
B ?IDℎCD + 𝑋 B ?𝑋 F𝑦J + 𝑋 B ?CD𝑋 F𝑦D + ⋯
= ℎCD𝐼?(1CD) + 𝑦J𝐼?(1J) + 𝑦D𝐼?(1D) + 𝑦H𝐼?(1H) + ⋯
[0 0 0 … 1 0 . . 0] (where the 1 occurs in the t-th position)
21
– Principle of superposition in linear systems:
ℎ? = ℎCD𝐼?(1CD) + 𝑦J𝐼?(1J) + 𝑦D𝐼?(1D) + 𝑦H𝐼?(1H) + ⋯
Time X(t) Y(t) t=0 h-1
22
– ℎ 𝑢 = 𝑥Bℎ 𝑢 − 1 + 𝑥F𝑦(𝑢) – ℎ D 𝑢 = 𝑥B
)𝑥F𝑦 1
23
ℎ D 𝑙
– ℎ 𝑢 = 𝑋
Bℎ 𝑢 − 1 + 𝑋 F𝑦(𝑢)
– ℎ D 𝑢 = 𝑋
B )𝑋 F𝑦 1
B = 𝑉Λ𝑉CD
– 𝑋
B𝑣X = 𝜇X𝑣X
– For any vector 𝑤 we can write
B𝑤 = 𝑏D𝜇D𝑣D + 𝑏H𝜇H𝑣H + ⋯ + 𝑏\𝜇\𝑣\
B )𝑤 = 𝑏D𝜇D )𝑣D + 𝑏H𝜇H ) 𝑣H + ⋯ + 𝑏\𝜇\ ) 𝑣\
– lim
)→a 𝑋 B )𝑤 = 𝑏b𝜇b ) 𝑣b where 𝑛 = argmax h
𝜇h
24
– ℎ 𝑢 = 𝑋
Bℎ 𝑢 − 1 + 𝑋 F𝑦(𝑢)
– ℎ D 𝑢 = 𝑋
B )𝑋 F𝑦 1
B = 𝑉Λ𝑉CD
– 𝑋
B𝑣X = 𝜇X𝑣X
– For any vector 𝑤 we can write
B𝑤 = 𝑏D𝜇D𝑣D + 𝑏H𝜇H𝑣H + ⋯ + 𝑏\𝜇\𝑣\
B )𝑤 = 𝑏D𝜇D )𝑣D + 𝑏H𝜇H ) 𝑣H + ⋯ + 𝑏\𝜇\ ) 𝑣\
– lim
)→a 𝑋 B )𝑤 = 𝑏b𝜇b ) 𝑣b where 𝑛 = argmax h
𝜇h For any input, for large 𝑢 the length of the hidden vector will expand or contract according to the 𝑢 th power of the largest eigen value of the hidden-layer weight matrix
25
– ℎ 𝑢 = 𝑋
Bℎ 𝑢 − 1 + 𝑋 F𝑦(𝑢)
– ℎ D 𝑢 = 𝑋
B )𝑋 F𝑦 1
B = 𝑉Λ𝑉CD
– 𝑋
B𝑣X = 𝜇X𝑣X
– For any vector 𝑤 we can write
B𝑤 = 𝑏D𝜇D𝑣D + 𝑏H𝜇H𝑣H + ⋯ + 𝑏\𝜇\𝑣\
B )𝑤 = 𝑏D𝜇D )𝑣D + 𝑏H𝜇H ) 𝑣H + ⋯ + 𝑏\𝜇\ ) 𝑣\
– lim
)→a 𝑋 B )𝑤 = 𝑏b𝜇b ) 𝑣b where 𝑛 = argmax h
𝜇h
For any input, for large 𝑢, the length of the hidden vector will expand or contract according to the 𝑢 th power of the largest eigen value of the hidden-layer weight matrix
Unless it has no component along the eigen vector corresponding to the largest eigen value. In that case it will grow according to the second largest Eigen value.. And so on..
26
– ℎ 𝑢 = 𝑋
Bℎ 𝑢 − 1 + 𝑋 F𝑦(𝑢)
– ℎ D 𝑢 = 𝑋
B )𝑋 F𝑦 1
B = 𝑉Λ𝑉CD
– 𝑋
B𝑣X = 𝜇X𝑣X
– For any vector 𝑤 we can write
B𝑤 = 𝑏D𝜇D𝑣D + 𝑏H𝜇H𝑣H + ⋯ + 𝑏\𝜇\𝑣\
B )𝑤 = 𝑏D𝜇D )𝑣D + 𝑏H𝜇H ) 𝑣H + ⋯ + 𝑏\𝜇\ ) 𝑣\
– lim
)→a 𝑋 B )𝑤 = 𝑏b𝜇b ) 𝑣b where 𝑛 = argmax h
𝜇h
If 𝜇b*F > 1 it will blow up, otherwise it will contract and shrink to 0 rapidly
27
For any input, for large 𝑢 the length of the hidden vector will expand or contract according to the 𝑢 th power of the largest eigen value of the hidden-layer weight matrix
Unless it has no component along the eigen vector corresponding to the largest eigen value. In that case it will grow according to the second largest Eigen value.. And so on..
– ℎ 𝑢 = 𝑋
Bℎ 𝑢 − 1 + 𝑋 F𝑦(𝑢)
– ℎ D 𝑢 = 𝑋
B )𝑋 F𝑦 1
B = 𝑉Λ𝑉CD
– 𝑋
B𝑣X = 𝜇X𝑣X
– For any vector 𝑤 we can write
B𝑤 = 𝑏D𝜇D𝑣D + 𝑏H𝜇H𝑣H + ⋯ + 𝑏\𝜇\𝑣\
B )𝑤 = 𝑏D𝜇D )𝑣D + 𝑏H𝜇H ) 𝑣H + ⋯ + 𝑏\𝜇\ ) 𝑣\
– lim
)→a 𝑋 B )𝑤 = 𝑏b𝜇b ) 𝑣b where 𝑛 = argmax h
𝜇h
If 𝜇b*F > 1 it will blow up, otherwise it will contract and shrink to 0 rapidly What about at middling values of 𝑢? It will depend on the
28
For any input, for large 𝑢 the length of the hidden vector will expand or contract according to the 𝑢 th power of the largest eigen value of the hidden-layer weight matrix
Unless it has no component along the eigen vector corresponding to the largest eigen value. In that case it will grow according to the second largest Eigen value.. And so on..
– ℎ 𝑢 = 𝑋
Bℎ 𝑢 − 1 + 𝑋 F𝑦(𝑢)
– ℎ D 𝑢 = 𝑋
B )𝑋 F𝑦 1
𝜇b*F = 0.9 𝜇b*F = 1 𝜇b*F = 1.1 𝜇b*F = 1 𝜇b*F = 1.1 Complex Eigenvalues 𝜇H\k = 0.5 𝜇H\k = 0.1
29
– If the largest Eigen value is greater than 1, the system will “blow up” – If it is lesser than 1, the response will “vanish” very quickly – Complex Eigen values cause oscillatory response
30
– Sigmoid: Saturates in a limited number of steps, regardless of 𝑥B – Tanh: Sensitive to 𝑥B, but eventually saturates
– Relu: Sensitive to 𝑥B, can blow up
31
– Sigmoid: Saturates in a limited number of steps, regardless of 𝑥B – Tanh: Sensitive to 𝑥B, but eventually saturates – Relu: For negative starts, has no response
32
– 1,1,1, … / 𝑂
– Interestingly, RELU is more prone to blowing up (why?)
sigmoid tanh relu
33
– −1, −1, −1, … / 𝑂
– Interestingly, RELU is more prone to blowing up (why?)
sigmoid tanh relu
34
– Alternately, Routh’s criterion and/or pole-zero analysis – Positive definite functions evaluated at ℎ – Conclusions are similar: only the tanh activation gives us any reasonable behavior
– Bipolar activations (e.g. tanh) have the best memory behavior – Still sensitive to Eigenvalues of 𝑋 – Best case memory is short – Exponential memory behavior
35
– If the largest Eigen value of the recurrent weights matrix is greater than 1, the network response may blow up – If its less than one, the response dies down very quickly
units
– Sigmoid activations saturate and the network becomes unable to retain new information – RELUs blow up – Tanh activations are the most effective at storing memory
36
– Time-series prediction – Time-series classification – Sequence prediction.. – They can even simplify problems that are difficult for MLPs
– Also..
37
– (Any deep network, not just recurrent nets) – The gradient of the error with respect to weights is unstable..
38
𝑃𝑣𝑢𝑞𝑣𝑢 = 𝑏[u] = 𝑔 𝑨[u] = 𝑔 𝑋[u]𝑏[uCD] = 𝑔 𝑋[u]𝑔(𝑋[uCD]𝑏[uCH] = 𝑔 𝑋[u]𝑔 𝑋[uCD] … 𝑔 𝑋[H]𝑔 𝑋[D]𝑦
For convenience, we use the same activation functions for all layers. However, output layer neurons most commonly do not need activation function (they show class scores or real-valued targets.)
𝑋[D] 𝑦 × 𝑔 𝑋[H] × 𝑔 𝑋[u] × 𝑔 𝑨[D] 𝑏[D] 𝑨[H] 𝑏[uCD] 𝑨[u] 𝑏[u] 𝑏[u] = 𝑝𝑣𝑢𝑞𝑣𝑢
39
𝑀𝑝𝑡𝑡(𝑦) = 𝐹 𝑔[u] 𝑋[u]𝑔[uCD] 𝑋[uCD]𝑔[uCH] … 𝑋[D]𝑦
𝛼
x[y]𝑀𝑝𝑡𝑡 = 𝛼 x[z]𝑀𝑝𝑡𝑡. 𝛼𝑔[u]. 𝑋[u]. 𝛼𝑔[uCD]. 𝑋[uCD] … 𝛼𝑔[{ID]. 𝑋[{ID]
– 𝛼
x[y]𝑀𝑝𝑡𝑡 is the gradient of the error w.r.t the output of the l-th layer of the network
– 𝛼𝑔[{] is jacobian of 𝑔[{] w.r.t. to its current input – All blue terms are matrices
40
– Resulting in insignificant or unstable gradient descent updates – Problem gets worse as network depth increases
𝛼
x[y]𝑀𝑝𝑡𝑡 = 𝛼 x[z]𝑀𝑝𝑡𝑡. 𝛼𝑔[u]. 𝑋[u]. 𝛼𝑔[uCD]. 𝑋[uCD] … 𝛼𝑔[{ID]. 𝑋[{ID]
41
– After a few layers the derivative of the loss at any time is totally “forgotten”
𝛼
x[y]𝑀𝑝𝑡𝑡 = 𝛼 x[z]𝑀𝑝𝑡𝑡. 𝛼𝑔[u]. 𝑋[u]. 𝛼𝑔[uCD]. 𝑋[uCD] … 𝛼𝑔[{ID]. 𝑋[{ID]
42
x|𝑀𝑝𝑡𝑡 = 𝛼𝑀𝑝𝑡𝑡. 𝛼𝑔 }. 𝑋 }CD. 𝛼𝑔 }CD. 𝑋 }CH … 𝛼𝑔 ?ID𝑋 ?
x|𝑀𝑝𝑡𝑡 will
– Expand 𝛼𝑀𝑝𝑡𝑡 in directions where each stage has singular values greater than 1 – Shrink 𝛼𝑀𝑝𝑡𝑡 in directions where each stage has singular values less than 1
43
– Your results may vary
– In reality a tiny number will actually blow up.
44
– For vector activations: A full matrix – For scalar activations: A matrix where the diagonal entries are the derivatives of the activation of the recurrent hidden layer
ℎX(𝑢) = 𝑔 𝑨X 𝑢 𝑌 ℎ 𝑍 𝛼𝑔 𝑨(𝑢) = 𝑔~(𝑨),D) ⋯ 𝑔~(𝑨),H) ⋯ ⋮ ⋮ ⋱ ⋮ ⋯ 𝑔~(𝑨),})
45
– The diagonals (or singular values) of the Jacobian are bounded
46
ℎX(𝑢) = 𝑔 𝑨X 𝑢 𝑌 ℎ 𝑍 𝛼𝑔 𝑨(𝑢) = 𝑔~(𝑨),D) ⋯ 𝑔~(𝑨),H) ⋯ ⋮ ⋮ ⋱ ⋮ ⋯ 𝑔~(𝑨),})
have derivatives that are always less than 1
– The derivative of tanh ()is never greater than 1 (and mostly less than 1)
𝛼𝑔
) 𝑨X =
𝑔
),D ~ (𝑨D)
⋯ 𝑔
),H ~ (𝑨H)
⋯ ⋮ ⋮ ⋱ ⋮ ⋯ 𝑔
),} ~ (𝑨})
47
derivatives that are always less than 1
– The derivative of tanh ()is never greater than 1 (and mostly less than 1)
48
x[•]𝑀𝑝𝑡𝑡 = 𝛼 x[‚]𝑀𝑝𝑡𝑡. 𝛼𝑔[ƒ]. 𝑋. 𝛼𝑔[ƒCD]. 𝑋 … 𝛼𝑔[)ID]. 𝑋
– The conclusion below holds for any deep network, though
x[•]𝑀𝑝𝑡𝑡 will
– Expand 𝛼
x[‚]𝑀𝑝𝑡𝑡 along directions in which the singular values of the weight
matrices are greater than 1 – Shrink 𝛼
x[‚]𝑀𝑝𝑡𝑡 in directions where the singular values are less than 1
– Repeated multiplication by the weights matrix will result in Exploding or vanishing gradients
49
– Gradients from errors at t = 𝑈 will vanish by the time they’re propagated to 𝑢 = 1
X(1)
hf(0)
Y(T)
50
– Each weights matrix and activation can shrink components of the input
h-1
𝑌(0) 𝑌(1) 𝑌(2) 𝑌(𝑈 − 2) 𝑌(𝑈 − 1) 𝑌(𝑈) 𝑍(0) 𝑍(1) 𝑍(2) 𝑍(𝑈 − 2) 𝑍(𝑈 − 1) 𝑍(𝑈)
51
– Unrolled network is very deep
52
This slide has been adopted from Socher lectures, cs224d, Stanford, 2017
53
– Can be performed with a multi-tap recursion, but how many taps? – Need an alternate way to “remember” stuff
54
– The hidden outputs can blow up, or shrink to zero depending on the Eigen values of the recurrent weights matrix – The memory is also a function of the activation of the hidden units
– The gradient of the error at the output gets concentrated into a small number of parameters in the earlier layers, and goes to zero for others
55
– RNN will “forget” pattern 1 if intermediate stuff is too long – “Jane” à the next pronoun referring to her will be “she”
– Can be performed with a multi-tap recursion, but how many taps? – Need an alternate way to “remember” stuff
1
56
– As mentioned earlier, tanh() is the generally used activation for the hidden layer
57
58
59
Largest singular value > 1: Exploding gradients Largest singular value < 1: Vanishing gradients
60
61
– High curvature walls
62
Computing gradient of h0 involves many factors
Largest singular value > 1: Exploding gradients Largest singular value < 1: Vanishing gradients Gradient clipping: Scale Computing gradient if its norm is too big Change RNN architecture
63
64
– ℎ) = 𝑀𝑇𝑈𝑁(𝑦), ℎ)CD) – ℎ) = 𝐻𝑆𝑉(𝑦), ℎ)CD)
65
– Not be directly dependent on vagaries of network parameters, but rather on input-based determination of whether it must be remembered – Replace them, e.g., by a function of the input that decides if things must be forgotten or not
66
– As mentioned earlier, tanh() is the generally used activation for the hidden layer
67
68
69
70
– And addition of history, which too is gated...
71
72
– More precisely, how much of the history to carry over – Also called the “forget” gate – Note, we’re actually distinguishing between the cell memory 𝐷 and the state ℎ that is coming over time! They’re related though
73
– A perceptron layer that determines if there’s something new and interesting in the input – A gate that decides if its worth remembering – If so its added to the current memory cell
74
– A perceptron layer that determines if there’s something interesting in the input – A gate that decides if its worth remembering – If so its added to the current memory cell
75
– Simply compress it with tanh to make it lie between 1 and -1
– Controlled by an output gate
76
77
– Input gate (current cell matter): 𝑗) = 𝜏 𝑋
X
ℎ)CD 𝑦) + 𝑐X – Forget (gate 0, forget past): 𝑔
) = 𝜏 𝑋 x
ℎ)CD 𝑦) + 𝑐x – Output (how much cell is exposed): 𝑝) = 𝜏 𝑋
“
ℎ)CD 𝑦) + 𝑐“
– New memory cell: 𝐷 ”) = tanh 𝑋
𝑦) + 𝑐• – Final memory cell: 𝐷) = 𝑗) ∘ 𝐷 ”) + 𝑔
) ∘ 𝐷)CD
– Final hidden state: ℎ) = 𝑝) ∘ tanh 𝐷)
78
– Note, we’re using both 𝐷 and ℎ
79
𝑦) ℎ)CD ℎ) 𝐷)CD 𝐷) 𝑔
)
𝑗) 𝑝) 𝐷 ”)
s() s() s()
tanh tanh
Gates Variables
80
# Assuming h(0,*) is known and C(0,*)=0 # Assuming L hidden-state layers and an output layer # Note: LSTM_cell is an indexed class with functions # [W{l},b{l}] are the entire set of weights and biases # for the lth hidden layer # Wo and bo are output layer weights and biases for t = 1:T # Including both ends of the index h(t,0) = x(t) # Vectors. Initialize hidden layer h(0) to input for l = 1:L # hidden layers operate at time t [C(t,l),h(t,l)] = LSTM_cell(t,l).forward(C(t-1,l),h(t-1,l) ,h(t,l-1)[W{l},b{l}]) zo(t) = Woh(t,L) + bo Y(t) = softmax( zo(t) )
81
# Continuing from previous slide # Note: [W,h] is a set of parameters, whose individual elements are # shown in red within the code. These are passed in # Static local variables which aren’t required outside this cell static local zf, zi, zc, f, i, o, Ci function [Co, ho] = LSTM_cell.forward(C,h,x, [W,h]) zf = WfcC + Wfhh + Wfxx + bf f = sigmoid(zf) # forget gate zi = WicC + Wihh + Wixx + bi i = sigmoid(zi) # input gate zc = WccC + Wchh + Wcxx + bc Ci = tanh(zc) # Detecting input pattern Co = f∘C + i∘Ci # “∘” is component-wise multiply zo = WocCo + Wohh + Woxx + bo
ho = o∘tanh(C) # “∘” is component-wise multiply return Co,ho
82
g in the previous slides was called 𝑑̃
83
g in the previous slides was called 𝑑̃
84
Wi Wo Wf Wg
85
86
87
– Pointless computation!
input!
88
𝑨) = 𝜏 𝑋
™
ℎ)CD 𝑦) + 𝑐™
𝑠
) = 𝜏 𝑋 +
ℎ)CD 𝑦) + 𝑐+
ℎ ›) = tanh 𝑋
b
𝑠
) ∘ ℎ)CD
𝑦) + 𝑐b
ℎ) = 𝑨) ∘ ℎ)CD + 1 − 𝑨) ∘ ℎ ›)
If reset gate unit is ~0, then this ignores previous memory and
89
90
This slide has been adopted from Socher lectures, cs224d, Stanford, 2017
– àAllows model to drop information that is irrelevant in the future
– If z close to 1, then we can copy information in that unit through many time steps! Less vanishing gradient!
91
This slide has been adopted from Socher lectures, cs224d, Stanford, 2017
92
– Greff et al. (2015), perform comparison of popular variants, finding that they’re all about the same. – Jozefowicz et al. (2015) tested more than ten thousand RNN architectures, finding some that worked better than LSTMs on certain tasks.
93
[Ann Copestake, Overview of LSTMs and word2vec, 2016.] https://arxiv.org/ftp/arxiv/papers/1611/1611.00068.pdf
94
95
Time X(t) Y(t)
96
– Memory can explode or vanish depending on the weights and activation
– Error at any time cannot affect parameter updates in the too-distant past – E.g. seeing a “close bracket” cannot affect its ability to predict an “open bracket” if it happened too long ago in the input
dependent on the input, rather than network parameters/structure
– Through a memory structure with no weights or activations, but instead direct switching and “increment/decrement” from pattern recognizers – Do not suffer from a vanishing gradient problem but do suffer from exploding gradient issue
97
– Exploding is controlled with gradient clipping. – Vanishing is controlled with additive interactions (LSTM)
98