Deep Learning: Theory and Practice 30-04-2019 Recurrent Neural - - PowerPoint PPT Presentation

▶

Sep 19, 2022 382 likes •781 views

Deep Learning: Theory and Practice 30-04-2019 Recurrent Neural Networks Introduction The standard DNN/CNN paradigms (x,y) - ordered pair of data vectors/images (x) and target (y) Moving to sequence data (x(t),y(t)) where this

SLIDE 1

Deep Learning: Theory and Practice

30-04-2019

Recurrent Neural Networks

SLIDE 2

Introduction

❖ The standard DNN/CNN paradigms ❖ (x,y) - ordered pair of data vectors/images (x) and

target (y)

❖ Moving to sequence data ❖ (x(t),y(t)) where this could be sequence to sequence

mapping task.

❖ (x(t),y) where this could be a sequence to vector

mapping task.

SLIDE 3

Introduction

❖ Difference between CNNs/DNNs ❖ (x(t),y(t)) where this could be sequence to sequence

mapping task.

❖ Input features / output targets are correlated in time. ❖ Unlike standard models where each pair is

independent.

❖ Need to model dependencies in the sequence over

time.

SLIDE 4

Introduction to Recurrent Networks

“Deep Learning”, Ian Goodfellow, Yoshua Bengio, Aaron Courville

SLIDE 5

Recurrent Networks

“Deep Learning”, Ian Goodfellow, Yoshua Bengio, Aaron Courville

SLIDE 6

Recurrent Networks

“Deep Learning”, Ian Goodfellow, Yoshua Bengio, Aaron Courville

SLIDE 7

Back Propagation in RNNs

Model Parameters Gradient Descent

SLIDE 8

Recurrent Networks

SLIDE 9

Back Propagation Through Time

SLIDE 10

SLIDE 11

Back Propagation Through Time

SLIDE 12

Standard Recurrent Networks

“Deep Learning”, Ian Goodfellow, Yoshua Bengio, Aaron Courville

SLIDE 13

Other Recurrent Networks

“Deep Learning”, Ian Goodfellow, Yoshua Bengio, Aaron Courville

Teacher Forcing Networks

SLIDE 14

Recurrent Networks

“Deep Learning”, Ian Goodfellow, Yoshua Bengio, Aaron Courville

Teacher Forcing Networks

SLIDE 15

Recurrent Networks

Multiple Input Single Output

SLIDE 16

Recurrent Networks

Single Input Multiple Output

SLIDE 17

Recurrent Networks

Bi-directional Networks

SLIDE 18

Recurrent Networks

Sequence to Sequence Mapping Networks

SLIDE 19

SLIDE 20

Long-term Dependency Issues

SLIDE 21

Vanishing/Exploding Gradients

❖ Gradients either vanish or explode ❖ Initial frames may not contribute to gradient

computations or may contribute too much.

SLIDE 22

Long-Short Term Memory

SLIDE 23

LSTM Cell

Input Gate Forget Gate Cell Output Gate LSTM output

f - sigmoid function g, h - tanh function

SLIDE 24

Long Short Term Memory Networks

SLIDE 25

Gated Recurrent Units (GRU)

SLIDE 26

Attention in LSTM Networks

❖ Attentions allows a mechanism to add relevance ❖ Certain regions of the audio have more importance

than the rest for the task at hand.

SLIDE 27

SLIDE 28

Encoder - Decoder Networks with Attention

SLIDE 29

Attention Models

SLIDE 30

Attention - Speech Example

From our lab [part of ICASSP 2019 paper].

SLIDE 31

Language Recognition Evaluation

SLIDE 32

End-to-end model using GRUs and Attention

SLIDE 33

Proposed End-to-End Language Recognition Model

SLIDE 34

Proposed End-to-End Language Recognition Model

SLIDE 35

Proposed End-to-End Language Recognition Model

SLIDE 36

Language Recognition Evaluation

0-3s : O...One muscle at all, it was terrible 3s-4s : .... ah .... ah .... 4s - 9s : I couldn't scream, I couldn't shout, I couldn't even move my arms up, or my legs 9s -11s : I was trying me hardest, I was really really panicking.

Attention Weight

Bharat Padi, et al. “End-to-end language recognition using hierarchical gated recurrent networks”, under review 2018.

We proposed the attention model - Attention weighs th importance of each short-term segment feature for the task. State-of-art models use the input sequence directly.

SLIDE 37

Language Recognition Evaluation

SLIDE 38