SLIDE 1
Outline
- 1. Embodied Navigation
- 2. Vision-language Navigation Task
- 3. Related Works
- 4. Our Methods
- 5. Conclusion
Vision-Language Navigation with Self-Supervised Auxiliary Reasoning - - PowerPoint PPT Presentation
Vision-Language Navigation with Self-Supervised Auxiliary Reasoning - - PowerPoint PPT Presentation
MONASH INFORMATION TECHNOLOGY Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks Fengda Zhu, Yi Zhu, Xiaojun Chang, Xiaodan Liang Outline 1. Embodied Navigation 2. Vision-language Navigation Task 3. Related Works 4.
SLIDE 2
SLIDE 3
Embodied Navigation Problem
- 1. datasets providing 3D assets with semantic annotations
- 2. simulators render these assets and simulate an embodied agent
- 3. tasks that define evaluable problems that enable us to benchmark scientific progress
SLIDE 4
Synthetic Image / Real Image
Advantage
- More data
- Faster rendering
Disadvantage
- Limited application
Advantage
- Close to real application
Disadvantage
- Less data
- Easily Overfitting
Transfer: Sim-Real Joint Reinforcement Transfer for 3D Indoor Navigation (by Zhu et al. CVPR 2019)
SLIDE 5
Matterport3D
SLIDE 6
Habitat Simulator
A flexible, high-performance 3D simulator with configurable agents, multiple sensors, and generic 3D dataset handling (with built-in support for MatterPort3D, Gibson, Replica, and other datasets). Advantage:
- Real image
- Fast rendering
- Continuous action space
Disadvantage:
- Low rendering quality
SLIDE 7
PointGoal Task
SLIDE 8
ObjectGoal Task
SLIDE 9
Vision Language Navigation (VLN) Task
- Natural Language
- More detailed description
- Require complex scene understanding
Room-to-room (R2R) dataset
- 90 houses
- 7k trajectories
- 21k instructions
Computer Vision + Natural Language Processing + Reinforcement Learning
SLIDE 10
VLN baseline (seq-to-seq)
Disadvantage:
- 1. Supervised learning is easily overfitting
- 2. Does not sufficiently exploit the panoramic view
- 3. The action space is redundant
- 4. Training-testing domain gap
SLIDE 11
Speaker-Follower Model
Speaker: trajectory to instruction Follower: instruction to trajectory
SLIDE 12
Speaker-Follower Model
SLIDE 13
Reinforced Cross-Modal Matching (RCM)
Advantage:
- 1. Use cross-modal attention
- 2. Introduce RL+ supervised learning
SLIDE 14
Environmental Dropout (Envdrop)
SLIDE 15
Self-Supervised Auxiliary Reasoning Tasks
𝑢0 𝑢1 𝑝𝑏𝑚 𝑢2
Rich information to explore:
- Semantics of the route
- Navigation Progress
- Vision Language Consistency
- Room Structure
Please turn left and walk through the living room. Exit the room and turn right into the bedroom. Navigation Node Navigation Edge Feasible Edge
SLIDE 16
Self-Supervised Auxiliary Reasoning Tasks
𝑢0 𝑢1 𝑝𝑏𝑚 𝑢2
We require the agent to:
- Interpret its actions
- Reason about the past
- Align Vision-language explicitly
- Predict the future
Please turn left and walk through the living room. Exit the room and turn right into the bedroom. Navigation Node Navigation Edge Feasible Edge
SLIDE 17
Self-Supervised Auxiliary Reasoning Tasks
SLIDE 18
Self-Supervised Auxiliary Reasoning Tasks
SLIDE 19
Self-Supervised Auxiliary Reasoning Tasks
SLIDE 20
Self-Supervised Auxiliary Reasoning Tasks
Demo Code
SLIDE 21