Recurrent Neural Networks: Stability analysis and LSTMs M. - PowerPoint PPT Presentation

Linear recursions: Vector version What about at middling values of 𝑢 ? It will depend on the other eigen values • Vector linear recursion (note change of notation) – ℎ 𝑢 = 𝑋 E ℎ 𝑢 − 1 + 𝑋 I 𝑦(𝑢) If 𝜇 `1I > 1 it will blow up, otherwise it will contract 0 𝑋 – ℎ G 𝑢 = 𝑋 I 𝑦 1 and shrink to 0 rapidly E • Length of response vector to a single input at 1 is |ℎ {G} 𝑢 | For any input, for large 𝑢 the length of the hidden vector will expand or contract according to the 𝑢 th power of the E = 𝑉Λ𝑉 FG • We can write 𝑋 largest eigen value of the hidden-layer weight matrix – 𝑋 E 𝑣 V = 𝜇 V 𝑣 V Unless it has no component along the eigen vector corresponding to the – For any vector 𝑤 we can write largest eigen value. In that case it will grow according to the second largest Eigen value.. • 𝑤 = 𝑏 G 𝑣 G + 𝑏 K 𝑣 K + ⋯ + 𝑏 Z 𝑣 Z And so on.. • 𝑋 E 𝑤 = 𝑏 G 𝜇 G 𝑣 G + 𝑏 K 𝜇 K 𝑣 K + ⋯ + 𝑏 Z 𝜇 Z 𝑣 Z 0 𝑣 K + ⋯ + 𝑏 Z 𝜇 Z 0 𝑣 Z 0 𝑤 = 𝑏 G 𝜇 G 0 𝑣 G + 𝑏 K 𝜇 K • 𝑋 E 0 𝑣 ` where 𝑛 = argmax 0 𝑤 = 𝑏 ` 𝜇 ` – lim 0→_ 𝑋 𝜇 f E f 26

Linear Li r recursions • Vector linear recursion – ℎ 𝑢 = 𝑋 E ℎ 𝑢 − 1 + 𝑋 I 𝑦(𝑢) 0 𝑋 – ℎ G 𝑢 = 𝑋 I 𝑦 1 E • Response to a single input [1 1 1 1] at 1 𝜇 `1I = 0.9 𝜇 `1I = 1.1 𝜇 `1I = 1.1 𝜇 `1I = 1 𝜇 `1I = 1 27

Linear Li r recursions • Vector linear recursion – ℎ 𝑢 = 𝑋 E ℎ 𝑢 − 1 + 𝑋 I 𝑦(𝑢) 0 𝑋 – ℎ G 𝑢 = 𝑋 I 𝑦 1 E • Response to a single input [1 1 1 1] at 1 𝜇 `1I = 0.9 𝜇 `1I = 1.1 𝜇 `1I = 1.1 𝜇 KZj = 0.5 𝜇 `1I = 1 𝜇 `1I = 1 𝜇 KZj = 0.1 Complex Eigenvalues 28

Lesson.. Le • In linear systems, long-term behavior depends entirely on the eigenvalues of the hidden-layer weights matrix – If the largest Eigen value is greater than 1, the system will “blow up” – If it is lesser than 1, the response will “vanish” very quickly – Complex Eigen values cause oscillatory response • Which we may or may not want • For smooth behavior, must force the weights matrix to have real Eigen values • Symmetric weight matrix 29

Ho How w abo about ut no non-lin linear earities ities (sc (scalar) • The behavior of scalar non-linearities ℎ 𝑢 = 𝑔(𝑥 E ℎ 𝑢 − 1 + 𝑥 I 𝑦 𝑢 ) • Left: Sigmoid, Middle: Tanh, Right: Relu – Sigmoid: Saturates in a limited number of steps, regardless of 𝑥 E – Tanh: Sensitive to 𝑥 E , but eventually saturates • “Prefers” weights close to 1.0 – Relu: Sensitive to 𝑥 E , can blow up 30

How about non-linearities (sc Ho (scalar) ℎ 𝑢 = 𝑔(𝑥 E ℎ 𝑢 − 1 + 𝑥 I 𝑦 𝑢 ) • With a negative start • Left: Sigmoid, Middle: Tanh, Right: Relu – Sigmoid: Saturates in a limited number of steps, regardless of 𝑥 E – Tanh: Sensitive to 𝑥 E , but eventually saturates – Relu: For negative starts, has no response 31

� Ve Vector Process • Assuming a uniform unit vector initialization ℎ 𝑢 = 𝑔(𝑥 E ℎ 𝑢 − 1 + 𝑥 I 𝑦 𝑢 ) – 1,1,1, … / 𝑂 – Behavior similar to scalar recursion – Interestingly, RELU is more prone to blowing up (why?) • Eigenvalues less than 1.0 retain the most “memory” sigmoid tanh relu 32

St Stability Analysis • Formal stability analysis considers convergence of “Lyapunov” functions – Alternately, Routh’s criterion and/or pole-zero analysis – Positive definite functions evaluated at ℎ – Conclusions are similar: only the tanh activation gives us any reasonable behavior • And still has very short “memory” • Lessons: – Bipolar activations (e.g. tanh) have the best memory behavior – Still sensitive to Eigenvalues of 𝑋 – Best case memory is short – Exponential memory behavior • “Forgets” in exponential manner 33

St Story so far • Recurrent networks retain information from the infinite past in principle • In practice, they tend to blow up or forget – If the largest Eigen value of the recurrent weights matrix is greater than 1, the network response may blow up – If its less than one, the response dies down very quickly • The “memory” of the network also depends on the activation of the hidden units – Sigmoid activations saturate and the network becomes unable to retain new information – RELUs blow up – Tanh activations are the most effective at storing memory • But still, for not very long 34

RN RNNs.. s.. • Excellent models for time-series analysis tasks – Time-series prediction – Time-series classification – Sequence prediction.. – They can even simplify problems that are difficult for MLPs • But the memory isn’t all that great.. – Also.. 35

Th The vanishing gradient problem • A particular problem with training deep networks.. – (Any deep network, not just recurrent nets) – The gradient of the error with respect to weights is unstable.. 36

Re Reminder: Tr Training deep networks 𝑏 [t] = 𝑝𝑣𝑢𝑞𝑣𝑢 𝑏 [t] 𝑃𝑣𝑢𝑞𝑣𝑢 = 𝑏 [t] 𝑔 𝑨 [t] = 𝑔 𝑨 [t] × 𝑏 [tFG] = 𝑔 𝑋 [t] 𝑏 [tFG] 𝑋 [t] 𝑔 𝑨 [K] = 𝑔 𝑋 [t] 𝑔(𝑋 [tFG] 𝑏 [tFK] 𝑏 [G] × = 𝑔 𝑋 [t] 𝑔 𝑋 [tFG] … 𝑔 𝑋 [K] 𝑔 𝑋 [G] 𝑦 𝑔 𝑨 [G] 𝑋 [K] × 𝑋 [G] 𝑦 For convenience, we use the same activation functions for all layers. However, output layer neurons most commonly do not need activation function (they show class scores or real-valued targets.) 37

Re Reminder: Training deep deep ne networks • For 𝑀𝑝𝑡𝑡(𝑦) = 𝐹 𝑔 [t] 𝑋 [t] 𝑔 [tFG] 𝑋 [tFG] 𝑔 [tFK] … 𝑋 [G] 𝑦 • We get: x [z] 𝑀𝑝𝑡𝑡. 𝛼𝑔 [t] . 𝑋 [t] . 𝛼𝑔 [tFG] . 𝑋 [tFG] … 𝛼𝑔 [{|G] . 𝑋 [{|G] 𝛼 x [y] 𝑀𝑝𝑡𝑡 = 𝛼 • Where – 𝛼 x [y] 𝑀𝑝𝑡𝑡 is the gradient of the error w.r.t the output of the l-th layer of the network • Needed to compute the gradient of the error w.r.t 𝑋 [{] – 𝛼𝑔 [{] is jacobian of 𝑔 [{] w.r.t. to its current input – All blue terms are matrices 38

Re Reminder: Gradient pr probl blems i in de n deep ne p netw twork rks x [z] 𝑀𝑝𝑡𝑡. 𝛼𝑔 [t] . 𝑋 [t] . 𝛼𝑔 [tFG] . 𝑋 [tFG] … 𝛼𝑔 [{|G] . 𝑋 [{|G] 𝛼 x [y] 𝑀𝑝𝑡𝑡 = 𝛼 • The gradients in the lower/earlier layers can explode or vanish – Resulting in insignificant or unstable gradient descent updates – Problem gets worse as network depth increases 39

Reminder: Training deep Re deep ne networks x [z] 𝑀𝑝𝑡𝑡. 𝛼𝑔 [t] . 𝑋 [t] . 𝛼𝑔 [tFG] . 𝑋 [tFG] … 𝛼𝑔 [{|G] . 𝑋 [{|G] 𝛼 x [y] 𝑀𝑝𝑡𝑡 = 𝛼 • As we go back in layers, the Jacobians of the activations constantly shrink the derivative – After a few layers the derivative of the loss at any time is totally “forgotten” 40

The Jacobian of the hidden layers fo Th for an an RNN 𝑍 𝑔 } (𝑨 0,G ) 0 ⋯ 0 𝑔 } (𝑨 0,K ) 0 ⋯ 0 𝛼𝑔 𝑨(𝑢) = ⋮ ⋮ ⋱ ⋮ ℎ 𝑔 } (𝑨 0,€ ) 0 0 ⋯ 𝑌 ℎ V (𝑢) = 𝑔 𝑨 V 𝑢 • 𝛼𝑔() is the derivative of the output of the (layer of) hidden recurrent neurons with respect to their input – For vector activations: A full matrix – For scalar activations: A matrix where the diagonal entries are the derivatives of the activation of the recurrent hidden layer 41

Th The Jacobian 𝑍 𝑔 } (𝑨 0,G ) 0 ⋯ 0 𝑔 } (𝑨 0,K ) 0 ⋯ 0 𝛼𝑔 𝑨(𝑢) = ⋮ ⋮ ⋱ ⋮ ℎ 𝑔 } (𝑨 0,€ ) 0 0 ⋯ 𝑌 ℎ V (𝑢) = 𝑔 𝑨 V 𝑢 • The derivative (or subgradient) of the activation function is always bounded – The diagonals (or singular values) of the Jacobian are bounded • There is a limit on how much multiplying a vector by the Jacobian will scale it 42

Th The derivative of the hidden state activation • Most common activation functions, such as sigmoid, tanh() and RELU have derivatives that are always less than 1 • The most common activation for the hidden units in an RNN is the tanh() – The derivative of tanh () is never greater than 1 (and mostly less than 1) • Multiplication by the Jacobian is always a shrinking operation 43

What abo Wha bout ut the he weigh ghts x [‚] 𝑀𝑝𝑡𝑡. 𝛼𝑔 [ƒ] . 𝑋. 𝛼𝑔 [ƒFG] . 𝑋 … 𝛼𝑔 [0|G] . 𝑋 𝛼 x [•] 𝑀𝑝𝑡𝑡 = 𝛼 • In a single-layer RNN, the weight matrices are identical – The conclusion below holds for any deep network, though • The chain product for 𝛼 x [•] 𝑀𝑝𝑡𝑡 will – E xpand 𝛼 x [‚] 𝑀𝑝𝑡𝑡 along directions in which the singular values of the weight matrices are greater than 1 – S hrink 𝛼 x [‚] 𝑀𝑝𝑡𝑡 in directions where the singular values are less than 1 – Repeated multiplication by the weights matrix will result in Exploding or vanishing gradients 44

Expl Explodi ding ng/Vani nishi shing ng gr gradi dients x [‚] 𝑀𝑝𝑡𝑡. 𝛼𝑔 [ƒ] . 𝑋. 𝛼𝑔 [ƒFG] . 𝑋 … 𝛼𝑔 [0] . 𝑋 [0] 𝛼 x [•] 𝑀𝑝𝑡𝑡 = 𝛼 • Every blue term is a matrix • 𝛼 x [‚] 𝑀𝑝𝑡𝑡 is proportional to the actual loss – Particularly for L 2 and KL divergence • The chain product for 𝛼 x [•] 𝑀𝑝𝑡𝑡 will – E xpand it in directions where each stage has singular values greater than 1 – S hrink it in directions where each stage has singular values less than 1 45

Training RNN 46

Training RNN ℎ f = 𝑔 𝑋 EE ℎ fFG + 𝑋 IE 𝑦 f 𝜖ℎ f 𝜖ℎ f,` ƒ diag 𝑔 } 𝑨 = 𝑋 ƒ Z,. 𝑔 } ` = 𝑋 f EE EE 𝜖ℎ fFG,Z 𝜖ℎ fFG 𝜖ℎ f 𝑔 } 𝑨 ƒ ≤ 𝑋 ≤ 𝛾 ˆ 𝛾 E f EE 𝜖ℎ fFG 𝜖ℎ f 𝜖ℎ 0 0 0 ƒ diag 𝑔 } 𝑨 ≤ 𝛾 ˆ 𝛾 E 0FB = ‰ = ‰ 𝑋 f EE 𝜖ℎ B 𝜖ℎ fFG fŠB|G fŠB|G • This can become very small or very large quickly (vanishing/exploding gradients) [Bengio et al 1994]. 47

Recurrent nets are very deep nets Re Y(T) h f (0) X(1) • The relation between 𝑌(1) and 𝑍(𝑈) is one of a very deep network – Gradients from errors at t = 𝑈 will vanish by the time they’re propagated to 𝑢 = 1 48

Training RNNs is hard • The unrolled network can be very deep and inputs from many time steps ago can modify output – Unrolled network is very deep • Multiply the same matrix at each time step during forward prop 49

The vanishing gradient problem: Example • In the case of language modeling words from time steps far away are not taken into consideration when training to predict the next word • Example: Jane walked into the room. John walked in too. It was late in the day. Jane said hi to ____ 50 This slide has been adopted from Socher lectures, cs224d, Stanford, 2017

Th The long-te term dependency problem • Must know to “remember” for extended periods of time and “recall” when necessary – Can be performed with a multi-tap recursion, but how many taps? – Need an alternate way to “remember” stuff 51

St Story so far • Recurrent networks retain information from the infinite past in principle • In practice, they are poor at memorization – The hidden outputs can blow up, or shrink to zero depending on the Eigen values of the recurrent weights matrix – The memory is also a function of the activation of the hidden units • Tanh activations are the most effective at retaining memory, but even they don’t hold it very long • Deep networks also suffer from a “vanishing or exploding gradient” problem – The gradient of the error at the output gets concentrated into a small number of parameters in the earlier layers, and goes to zero for others 52

Vanilla RNN Gradient Flow 53

Vanilla RNN Gradient Flow 54

Vanilla RNN Gradient Flow Computing gradient of h0 involves many factors of W (and repeated tanh) Largest singular value > 1: Exploding gradients Largest singular value < 1: Vanishing gradients 55

Trick for exploding gradient: clipping trick • The solution first introduced by Mikolov is to clip gradients to a maximum value. • Makes a big difference in RNNs. 56

Gradient clipping intuition • Error surface of a single hidden unit RNN – High curvature walls • Solid lines: standard gradient descent trajectories • Dashed lines gradients rescaled to fixed size 57

Vanilla RNN Gradient Flow Computing gradient of h0 involves many factors Gradient clipping: Scale Computing of W (and repeated tanh) gradient if its norm is too big Largest singular value > 1: Exploding gradients Largest singular value < 1: Vanishing gradients Change RNN architecture 58

For vanishing gradients: Initialization + ReLus! • Initialize Ws to identity matrix I and activations to RelU • New experiments with recurrent neural nets. Le et al. A Simple Way to Initialize Recurrent Networks of Rectified Linear Unit, 2015. 59

Better units for recurrent models • More complex hidden unit computation in recurrence! – ℎ 0 = 𝑀𝑇𝑈𝑁(𝑦 0 , ℎ 0FG ) – ℎ 0 = 𝐻𝑆𝑉(𝑦 0 , ℎ 0FG ) • Main ideas: –keep around memories to capture long distance dependencies –allow error messages to flow at different strengths depending on the inputs 60

And And no now w we enter the he do domain n of.. .. 61

Expl Explodi ding ng/Vani nishi shing ng gr gradi dients • Can we replace this with something that doesn’t fade or blow up? • Can we have a network that just “remembers” arbitrarily long, to be recalled on demand? – Not be directly dependent on vagaries of network parameters, but rather on input-based determination of whether it must be remembered – Replace them, e.g., by a function of the input that decides if things must be forgotten or not 62

En Enter the he LSTM TM • Long Short-Term Memory • Explicitly latch information to prevent decay / blowup • Following notes borrow liberally from • http://colah.github.io/posts/2015-08-Understanding-LSTMs/ 63

St Standard RNN • Recurrent neurons receive past recurrent outputs and current input as inputs • Processed through a tanh() activation function – As mentioned earlier, tanh() is the generally used activation for the hidden layer • Current recurrent output passed to next higher layer and next time instant 64

Some visualization 65

Lo Long Sh Short rt-Te Term Memory • The 𝜏() are multiplicative gates that decide if something is important or not • Remember, every line actually represents a vector 66

LSTM TM: Constant Error Carousel • Key component: a remembered cell state 67

LSTM TM: CEC • 𝐷 0 is the linear history • Carries information through, only affected by a gate – And addition of history, which too is gated.. 68

LSTM TM: Gates • Gates are simple sigmoidal units with outputs in the range (0,1) • Controls how much of the information is to be let through 69

LSTM TM: Forget gate • The first gate determines whether to carry over the history or to forget it – More precisely, how much of the history to carry over – Also called the “forget” gate – Note, we’re actually distinguishing between the cell memory 𝐷 and the state ℎ that is coming over time! They’re related though 70

LSTM TM: Input gate • The second input has two parts – A perceptron layer that determines if there’s something new and interesting in the input – A gate that decides if its worth remembering – If so its added to the current memory cell 71

LSTM TM: Memory cell update • The second input has two parts – A perceptron layer that determines if there’s something interesting in the input – A gate that decides if its worth remembering – If so its added to the current memory cell 72

LSTM TM: Output and Output gate • The output of the cell – Simply compress it with tanh to make it lie between 1 and -1 • Note that this compression no longer affects our ability to carry memory forward – Controlled by an output gate • To decide if the memory contents are worth reporting at this time 73

Long-short-term-memories (LSTMs) ℎ 0FG • Input gate (current cell matter) : 𝑗 0 = 𝜏 𝑋 + 𝑐 V V 𝑦 0 ℎ 0FG • Forget (gate 0, forget past): 𝑔 0 = 𝜏 𝑋 + 𝑐 x x 𝑦 0 ℎ 0FG • Output (how much cell is exposed): 𝑝 0 = 𝜏 𝑋 + 𝑐 “ “ 𝑦 0 ℎ 0FG • New memory cell: 𝑑̃ 0 = tanh 𝑋 + 𝑐 – – 𝑦 0 • Final memory cell: 𝑑 0 = 𝑗 0 ∘ 𝑑̃ 0 + 𝑔 0 ∘ 𝑑 0FG • Final hidden state: ℎ 0 = 𝑝 0 ∘ tanh 𝑑 0 74

LSTM TM Equations • 𝒋 𝒖 : input gate, how much of the new information will be let through the memory cell. • 𝒈 𝒖 : forget gate, responsible for information should be thrown away from memory cell. • 𝒑 𝒖 : output gate, how much of the information will be passed to expose to the next time step. • 𝒉 𝒖 or 𝒅 ž 𝒖 : self-recurrent which is equal to standard RNN • 𝒅 𝒖 : internal memory of the memory cell • 𝒊 𝒖 : hidden state • 𝐳 : output 75

LSTM Gates • Gates are ways to let information through (or not): – Forget gate: look at previous cell state and current input, and decide which information to throw away. – Input gate: see which information in the current state we want to update. – Output: Filter cell state and output the filtered result. – Gate or update gate: propose new values for the cell state. • For instance: store gender of subject until another subject is seen. 76

LSTM TM: Th The “Peephole” Connection • The raw memory is informative by itself and can also be input – Note, we’re using both 𝐷 and ℎ 77

Backp Ba kpropagation ru rules: s: Forward 𝐷 0FG 𝐷 0 tanh 𝑝 0 𝑗 0 𝑔 0 ¡ 0 𝐷 s () s () s () tanh ℎ 0FG ℎ 0 𝑦 0 Gates • Forward rules: Variables 78

LSTM TM cell forward # Continuing from previous slide # Note: [W,h] is a set of parameters, whose individual elements are # shown in red within the code. These are passed in # Static local variables which aren’t required outside this cell static local z f , z i , z c , f, i, o, C i function [C o , h o ] = LSTM_cell.forward(C,h,x, [W,h]) z f = W fc C + W fh h + W fx x + b f f = sigmoid(z f ) # forget gate z i = W ic C + W ih h + W ix x + b i i = sigmoid(z i ) # input gate z c = W cc C + W ch h + W cx x + b c C i = tanh(z c ) # Detecting input pattern C o = f ∘ C + i ∘ C i # “ ∘ ” is component-wise multiply z o = W oc C o + W oh h + W ox x + b o o = sigmoid(z o ) # output gate h o = o ∘ tanh(C) # “ ∘ ” is component-wise multiply 79 return C o ,h o

LSTM TM network forward # Assuming h(0,*) is known and C(0,*)=0 # Assuming L hidden-state layers and an output layer # Note: LSTM_cell is an indexed class with functions # [W{l},b{l}] are the entire set of weights and biases # for the l th hidden layer # W o and b o are output layer weights and biases for t = 1:T # Including both ends of the index h(t,0) = x(t) # Vectors. Initialize hidden layer h(0) to input for l = 1:L # hidden layers operate at time t [C( t,l ),h( t,l )] = LSTM_cell(t,l).forward(… …C( t-1,l ),h( t-1,l ),h( t,l-1 )[W{l},b{l}]) z o (t) = W o h(t,L) + b o Y(t) = softmax( z o (t) ) 80

Long Short Term Memory (LSTM) g in the previous slides was called 𝑑̃ 81

Long Short Term Memory (LSTM) [Hochreiter et al., 1997] 82

Long Short Term Memory (LSTM) [Hochreiter et al., 1997] 83

Long Short Term Memory (LSTM) [Hochreiter et al., 1997] 84

its : Lets simplify the LSTM Ga Gated Recu current t Un Units TM • Don’t bother to separately maintain compressed and regular memories – Pointless computation! • But compress it before using it to decide on the usefulness of the current input! 85

GRUs • Gated Recurrent Units (GRU) introduced by Cho et al. 2014 • Update gate ℎ 0FG 𝑨 0 = 𝜏 𝑋 + 𝑐 ¢ ¢ 𝑦 0 • Reset gate ℎ 0FG 𝑠 0 = 𝜏 𝑋 + 𝑐 2 2 𝑦 0 • Memory 𝑠 0 ∘ ℎ 0FG ¤ 0 = tanh 𝑋 ℎ + 𝑐 ` ` 𝑦 0 • Final Memory ¤ 0 ℎ 0 = 𝑨 0 ∘ ℎ 0FG + 1 − 𝑨 0 ∘ ℎ If reset gate unit is ~0, then this ignores previous memory and only stores the new input 86

GRU intuition • Units with long term dependencies have active update gates z • Illustration: 87 This slide has been adopted from Socher lectures, cs224d, Stanford, 2017

GRU intuition • If reset is close to 0, ignore previous hidden state – à Allows model to drop information that is irrelevant in the future • Update gate z controls how much of past state should matter now. – If z close to 1, then we can copy information in that unit through many time steps! Less vanishing gradient! • Units with short-term dependencies often have reset gates very active 88 This slide has been adopted from Socher lectures, cs224d, Stanford, 2017

Other RNN Varients 89

Which of these variants is best? • Do the differences matter? – Greff et al. (2015), perform comparison of popular variants, finding that they’re all about the same. – Jozefowicz et al. (2015) tested more than ten thousand RNN architectures, finding some that worked better than LSTMs on certain tasks. 90

LSTM Achievements • LSTMs have essentially replaced n-grams as language models for speech . • Image captioning and other multi-modal tasks which were very difficult with previous methods are now feasible. • Many traditional NLP tasks work very well with LSTMs, but not necessarily the top performers: e.g., POS tagging and NER: Choi 2016. • Neural MT : broken away from plateau of SMT, especially for grammaticality (partly because of characters/subwords), but not yet industry strength. [Ann Copestake, Overview of LSTMs and word2vec, 2016.] 91 https://arxiv.org/ftp/arxiv/papers/1611/1611.00068.pdf

Multi-layer RNN 92

Mu Multi-layer LSTM TM architecture Y(t) X(t) Time • Each green box is now an entire LSTM or GRU unit • Also keep in mind each box is an array of units 93

Ex Extensi nsions ns to the he RN RNN: Bi Bidi direc ectional nal RN RNN Proposed by Schuster and Paliwal 1997 • RNN with both forward and backward recursion – Explicitly models the fact that just as the future can be predicted from the past, the past can be deduced from the future 94

Bidirectional RN Bi RNN Y(1) Y(2) Y(T-2) Y(T-1) Y(T) h f (0) X(1) X(2) X(T-2) X(T-1) X(T) h b (inf) X(1) X(2) X(T-2) X(T-1) X(T) t • A forward net process the data from t=1 to t=T • A backward net processes it backward from t=T down to t=1 95

Bidirectional RN Bi RNN: Processi ssing an input stri ring h f (0) X(1) X(2) X(T-2) X(T-1) X(T) t • The forward net process the data from t=1 to t=T – Only computing the hidden states, initially • The backward net processes it backward from t=T down to t=0 96

Bi Bidirectional RN RNN: Processi ssing an input stri ring h f (0) X(1) X(2) X(T-2) X(T-1) X(T) h b (inf) X(1) X(2) X(T-2) X(T-1) X(T) t • The backward nets processes the input data in reverse time, end to beginning – Initially only the hidden state values are computed Clearly, this is not an online process and requires the entire input data • – Note: This is not the backward pass of backprop. net processes it backward from t=T down to t=0 97

Bi Bidirectional RN RNN: Processi ssing an input stri ring Y(1) Y(2) Y(T-2) Y(T-1) Y(T) h f (0) X(1) X(2) X(T-2) X(T-1) X(T) h b (inf) X(1) X(2) X(T-2) X(T-1) X(T) t • The computed states of both networks are used to compute the final output at each time 98

Backp Ba kpropagation in BRN BRNNs 𝒊 = 𝒊, 𝒊 : represents both the past and future Y(1) Y(2) Y(T-2) Y(T-1) Y(T) h f (0) X(1) X(2) X(T-2) X(T-1) X(T) h b (inf) X(1) X(2) X(T-2) X(T-1) X(T) t • Forward pass: Compute both forward and backward networks and final output 99

Backp Ba kpropagation in BRN BRNNs Loss d 1 ..d T Loss() Y(1) Y(2) Y(T-2) Y(T-1) Y(T) h f (0) X(1) X(2) X(T-2) X(T-1) X(T) h b (inf) X(1) X(2) X(T-2) X(T-1) X(T) t • Backward pass: Define a divergence from the desired output • Separately perform back propagation on both nets 100 – From t=T down to t=0 for the forward net – From t=0 up to t=T for the backward net