Attention for Machine Comprehension Made by : Rishab Goel Based on - PowerPoint PPT Presentation

Attention for Machine Comprehension Made by : Rishab Goel Based on slides by: Alex Graves, Hien Quoc, Renjie Liao

Highway Networks

Benefits ...

Importance ... For training very deep architectures By allowing better information flow Better optimization Intuition : linear transformation/input suffice for learning, language at higher level of http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ abstraction???

Hien Quoc Dang

Idea of Maxout Hien Quoc Dang

Intuitions Inspired from dropout Similar to bagging but integrated as a part of single network Hien Quoc Dang

Idea of Maxout ... Hien Quoc Dang

Comparison to Rectifiers Hien Quoc Dang

Why Maxout Work ? Hien Quoc Dang

Slides : Santi Pascual

LSTMs ... Chris Olah’s blog

Need for Attention The embeddings not sufficient to encode information over long distances Helps to attend to important patch of data Interpretability to the model

Attentive Reader

DYNAMIC COATTENTION NETWORKS FOR QUESTION ANSWERING Authors : Caiming Xiong, Victor Zhong, Richard Socher

Introduction Machine Comprehension No knowledge base required Till SQUAD no large scale, natural dataset Cloze style datasets like CNN/Mail Daily Synthetic/small size

About SQuAD Consists questions on a set of Wikipedia articles Wh type questions The answer is a segment of text, or span Source : Rajpurkar et al.

Model in nutshell ... Socher et al

Doc and Query Encoder Socher et al

Liked ● Gagan Socher et al

Liked : all Dynamic Decoder Socher et al

Highway Maxout Network ... Socher et al

Socher et al

Disliked ● Gagan (pt. 3) Implementation ● Akshay (pt. 4) claim not proven 1. CoreNLP for preprocessing 2. GloVe word vectors pretrained on 840B Common Crawl corpus 3. OOV set to 0 4. Sentinel vectors randomly initialized, optimized during training

Iterative process visualisation ... Socher et al

Socher et al

Disliked ● Haroun (ensemble gain too Results much) Socher et al

Liked ● Barun ● Nupur Socher et al

Liked Performance across diff. types of ques. ● Shantanu Socher et al

Liked ● Prachi Ablation studies ... Socher et al

Predictions Socher et al

Logistic Regression Prediction : Theatre Museum Socher et al

Comments : Trouble decoding multiple intuitive answer Socher et al

Cons Lack error analysis, need more ablation studies[Barun, Surag] System give extractive answer and not abstractive[Nupur] Do not compare HMN and MN[all] Unintuitive decoder[Dinesh]

Doubts ... Why HMN worked out? Role of sentinel vectors?? Error propagation in argmax function Maxout for LSTMs as well (not clear) Use multiple initialisation of start and end pointers ( how ??)

Extensions ... Use approach for others datasets like CNN/Daily Mail and MS COCO QA [Barun] Use different attention, Match LSTM [Barun] Bi-directional attention [Gagan] Use iterative idea to visual QA, classification, NER, SRL etc [Akshay, Surag] Find synonyms[Haroun]

Extensions ... Combine char2vec and word2vec embeddings to represent the document and query

Thanks!

Attention for Machine Comprehension Made by : Rishab Goel Based on - PowerPoint PPT Presentation

Attention for Machine Comprehension Made by : Rishab Goel Based on slides by: Alex Graves, Hien Quoc, Renjie Liao Highway Networks Benefits ... Benefits ... Importance ... For training very deep architectures By allowing better information

COMPREHENSION Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, Hannaneh Hajishirzi Presenter: Wenda

Consensus Attention-based Neural Networks for Reading Comprehension Y IMING C UI , T ING L IU , Z

Convolutional Spatial Attention Model for Reading Comprehension with Multiple- Choice Questions Z

Multi-attention Recurrent Network for Human Communication Comprehension Amir Zadeh, Paul Pu

Using Natural Language Relations between Answer Choices for Machine Comprehension Rajkumar Pujari

Measuring Non-Expert Comprehension of Machine Learning Fairness Metrics Debjani Saha , Candice

Evaluation Metrics for Machine Reading Comprehension (RC): Prerequisite Skills and Readability

Coordinated Interplay of Scene, Utterance, and World Knowledge - Comprehension of spoken

Advanced Neural Machine Translation Gongbo Tang 21 September 2020 Outline NMT with Attention

Advanced Neural Machine Translation Gongbo Tang 23 September 2019 Outline NMT with Attention

Representa)on Learning for Reading Comprehension Russ Salakhutdinov Machine Learning Department

SQuAD:100,000+ Questions for Machine Comprehension of Text Pranav Rajpurkar, Jian Zhang,

Machine Comprehension with Discourse Relations Karthik Narasimhan Regina Barzilay CSAIL,

Sparse and Constrained Attention for Neural Machine Translation Chaitanya Malaviya 1 ,

Effective Approaches to Attention-based Neural Machine Translation Thang Luong Hieu Pham and Chris

Speech Question Answering TOEFL Listening Comprehension Test by Machine Wei Fang December 13,

FlexDNN: Input-Adaptive On-Device Deep Learning for Efficient Mobile Vision ACM/IEEE Symposium on

Coupling Technical Assistance with Student Service Learning in Mine Water Reclamation KELSEA J.

Analysis of electronic voting protocols in applied pi calculus Mark Ryan University of

III.3.1 Alluvial fans Geoscience: the Earth and its Resources

Ice-sheet dynamics: the influence of glacier sliding on ice loss and sea level Ian Hewitt,

HVP contribution of the light quarks Davide Giusti to (g -2) including QED corrections with

Adventures with AIRS: continued Tim P. Barnett David W. Pierce Eric Fetzer Andrew Gettleman

Radon monitoring in the Kamioka mine Guillaume Pronost Kamioka Observatory, ICRR, University of