Deep Learning: Theory and Practice 30-04-2019 Recurrent Neural - PowerPoint PPT Presentation

Deep Learning: Theory and Practice 30-04-2019 Recurrent Neural Networks

Introduction ❖ The standard DNN/CNN paradigms ❖ (x,y) - ordered pair of data vectors/images (x) and target (y) ❖ Moving to sequence data ❖ (x(t),y(t)) where this could be sequence to sequence mapping task. ❖ (x(t),y) where this could be a sequence to vector mapping task.

Introduction ❖ Difference between CNNs/DNNs ❖ (x(t),y(t)) where this could be sequence to sequence mapping task. ❖ Input features / output targets are correlated in time. ❖ Unlike standard models where each pair is independent. ❖ Need to model dependencies in the sequence over time.

Introduction to Recurrent Networks “Deep Learning”, Ian Goodfellow, Yoshua Bengio, Aaron Courville

Recurrent Networks “Deep Learning”, Ian Goodfellow, Yoshua Bengio, Aaron Courville

Back Propagation in RNNs Model Parameters Gradient Descent

Recurrent Networks

Back Propagation Through Time

Standard Recurrent Networks “Deep Learning”, Ian Goodfellow, Yoshua Bengio, Aaron Courville

Other Recurrent Networks Teacher Forcing Networks “Deep Learning”, Ian Goodfellow, Yoshua Bengio, Aaron Courville

Recurrent Networks Teacher Forcing Networks “Deep Learning”, Ian Goodfellow, Yoshua Bengio, Aaron Courville

Recurrent Networks Multiple Input Single Output

Recurrent Networks Single Input Multiple Output

Recurrent Networks Bi-directional Networks

Recurrent Networks Sequence to Sequence Mapping Networks

Long-term Dependency Issues

Vanishing/Exploding Gradients ❖ Gradients either vanish or explode ❖ Initial frames may not contribute to gradient computations or may contribute too much.

Long-Short Term Memory

LSTM Cell Input Gate f - sigmoid function g, h - tanh function Forget Gate Cell Output Gate LSTM output

Long Short Term Memory Networks

Gated Recurrent Units (GRU)

Attention in LSTM Networks ❖ Attentions allows a mechanism to add relevance ❖ Certain regions of the audio have more importance than the rest for the task at hand.

Encoder - Decoder Networks with Attention

Attention Models

Attention - Speech Example From our lab [part of ICASSP 2019 paper].

Language Recognition Evaluation

End-to-end model using GRUs and Attention

Proposed End-to-End Language Recognition Model

Language Recognition Evaluation State-of-art models use the input sequence directly. We proposed the attention model - Attention weighs th importance of each short-term segment feature for the task. 0-3s : O...One muscle at all, it was terrible Attention Weight 3s-4s : .... ah .... ah .... 4s - 9s : I couldn't scream, I couldn't shout, I couldn't even move my arms up, or my legs 9s -11s : I was trying me hardest, I was really really panicking. Bharat Padi, et al. “End-to-end language recognition using hierarchical gated recurrent networks”, under review 2018.

Language Recognition Evaluation

Deep Learning: Theory and Practice 30-04-2019 Recurrent Neural - PowerPoint PPT Presentation

Deep Learning: Theory and Practice 30-04-2019 Recurrent Neural Networks Introduction The standard DNN/CNN paradigms (x,y) - ordered pair of data vectors/images (x) and target (y) Moving to sequence data (x(t),y(t)) where this

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning - Theory and Practice Deep Neural Networks 12-03-2020

Deep Learning - Theory and Practice Sriram Ganapathy sriramg@iisc.ac.in C 334, Electrical

Deep Learning - Theory and Practice [16-1-2020] Sriram Ganapathy web -

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning - Theory and Practice Linear Regression, Least Squares 13-02-2020 Classification

Deep Learning: From Theory to Algorithm Outline: 1. Overview of

Deep Learning - Theory and Practice Linear Regression, Least Squares 20-02-2020 Classification

Deep Learning - Theory and Practice Linear Regression, Least Squares 27-02-2020 Classification

Deep Learning: Theory and Practice Matrix Calculus 31-1-2019 Linear and Logistic Regression

The Ecology of Language Learning Practice to Theory - Theory to Practice DILIT - International

Reflective-based Practice Learning (RPL) A specific learning approach developed an implemented at

Industrial Level Deep Learning Training Infrastructure the Practice and Experience from

Theory or Practice? Theory : Without theory, practice is but routine born out of habit.

Deep Learning Theory with Application to Cancer Research Leonie Zeune, Stephan van Gils, Guus van

Analyzing Optimization in Deep Learning via Trajectories Nadav Cohen Institute for Advanced Study

Recurrent Neural Networks + LSTMs + Attention Surag Nair (based on slides by Xavier

Cognitive Psychology Philipp Koehn 13 February 2020 Philipp Koehn Artificial Intelligence:

2. Cognitive Perspective of Learning Cognition: Big Questions How do things out there

Class 15 - Long Short-Term Memory (LSTM) Class 15 - Long Short-Term Memory (LSTM) Study materials

Understanding LSTM Networks Recurrent Neural Networks An unrolled recurrent neural network The

GESIS Survey Guidelines Timo Lenzner and Natalja Menold These slides are based on the GESIS

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks by Kai

LSTM M Based sed Ada dapt ptive ive Fil ilterin ering g for r Redu duced ced Pre redi