SLIDE 1
Introduction
- Neural sequence-to-sequence model for direction following
Listen, Attend, and Walk: Neural Mapping of Navigational - - PowerPoint PPT Presentation
Listen, Attend, and Walk: Neural Mapping of Navigational - - PowerPoint PPT Presentation
Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences Hongyuan Mei, Mohit Bansal, Matthew R. Walter Toyota Technological Institute, Chicago Introduction Neural sequence-to-sequence model for direction
SLIDE 2
SLIDE 3
SLIDE 4
Introduction
- Learn correspondences between instruction and actions using
an alignment-based LSTM
- End-to-end differentiable sequence-to-sequence model
SLIDE 5
Model architecture
SLIDE 6
Model architecture
- Inference over a probabilistic model
- Neural encoder decoder model with attention
SLIDE 7
Model architecture
- Bidirectional LSTM to encode instruction
SLIDE 8
Model architecture
- Multi level aligner: High level (hidden states of LSTM) + low level (input words)
- One layer neural perceptron
- Intuitively, better match the salient words in input sentence (e.g., “easel”)
directly to corresponding landmarks in the current world state y(t) used in decoder
SLIDE 9
Model architecture
- LSTM decoder
- Output P is the conditional
probability distribution over actions
- E is an embedding matrix
- Trained using negative log likelihood
- f demonstrated action
SLIDE 10
Experiments
- SAIL route instructor dataset
- World state (y(t)) encodes local observable world at time t,
encoded as a concatenation of a bag-of-words vector for each direction (forward, left, and right).
SLIDE 11
Results
SLIDE 12
Ablation results
SLIDE 13