SLIDE 1
Image source: octomap.github.io Image source: - - PowerPoint PPT Presentation
Image source: octomap.github.io Image source: - - PowerPoint PPT Presentation
Image source: octomap.github.io Image source: pirobot.org/blog/0015/ Map from first-person images to actions Need to learn how to reason about changing observations Add explicit Camera Projection and
SLIDE 2
SLIDE 3
Image source: octomap.github.io Image source: pirobot.org/blog/0015/
SLIDE 4
- Map from first-person images to actions
- Need to learn how to reason about changing observations
SLIDE 5
- Add explicit Camera Projection and Differentiable Mapping
- Reason about the instruction on a static map
- Automatically handle changing first-person observations
SLIDE 6
Each pixel in the feature encodes an image neighbourhood
Input Image Feature Map
SLIDE 7
Feature Map Projected Features
(Map Frame) (Image Plane in Camera Frame)
SLIDE 8
Projected features (time )
Semantic Map (time )
Semantic Map (time )
SLIDE 9
Semantic Map Grounding Map Goal Map
1x1 Filter 9x9 Filter
LSTM go to the left side of plane
Inferred goal location Recognized airplane
SLIDE 10
Grounding Map Goal Map Perceptron Forward velocity
- Output the velocity
command, given Grounding and Goal maps
- Sent to quadcopter’s flight-
controller
Yaw rate
SLIDE 11
Agent
Action Image Instruction
Oracle
Ground truth action Ground truth trajectory Modified variant of DAgger Trade convergence guarantees for speed and memory efficiency
SLIDE 12
3500 Instructions + Environments Ground-truth trajectories 63 Landmarks 252 Possible Tasks Go to right side of mushroom
T
- tal number of rollouts:
3500 oracle 2000 policy
SLIDE 13
83.47 28.67 87.87
20 40 60 80 100 GSMN (Ours) NN with no Mapping Oracle
Outperform standard NN with no mapping Very close to oracle performance
SLIDE 14
Feature Extraction Mapping 1x1 Filter 9x9 Filter
MLP
LSTM
Go
- to
to th the e left left sid side e of
- f
plan plane
Action
Image Features
Instruction Embedding
Grounding Map Semantic Map Goal Map