Active Perception in Vision-Language Navigation Task - - PowerPoint PPT Presentation
Student Webinar Active Perception in Vision-Language Navigation Task (MC (MCIS Lab) (B (Beijing ng Ins nsti titute
媒体计算与智能系统实验室 媒体计算与智能系统实验室 (MC (MCIS Lab) 北京理工大学 北京理工大学 (B (Beijing ng Ins nsti titute tute of T echno hnology) Hom Home Page: https://hanqingwa wangai.github.io Em Email: hanqingwang@bit. t.edu.cn
Student Webinar
Hanqing Wang, Wenguan Wang, Tianmin Shu, Wei Liang, and Jianbing Shen
[Anderson, P., Wu, Q. et al., Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments, CVPR 2018]
[Manolis S., Abhishek K., Oleksandr M. and et al., Habitat: A Platform for Embodied AI Research, ICCV 2019]
Instruction: Leave the bathroom and walk forward along the pool. Take
a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room.
Observation:
Ground Truth Path
Observation:
Ground Truth Path
Instruction: Leave the bathroom and walk forward along the pool. Take
a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room.
Observation:
Ground Truth Path
Instruction: Leave the bathroom and walk forward along the pool. Take
a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room.
Observation:
Ground Truth Path
Instruction: Leave the bathroom and walk forward along the pool. Take
a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room.
Observation: Instruction: Leave the bathroom and walk forward along the pool.
Take a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room.
Basic Agent Door 1 Door 2
Observation:
Basic Agent
Instruction: Leave the bathroom and walk forward along the pool.
Take a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room.
Observation: Instruction: Leave the bathroom and walk forward along the pool.
Take a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room.
Our Agent Door 1 Door 2
Insufficient information
Observation:
Our Agent
Instruction: Leave the bathroom and walk forward along the pool.
Take a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room.
inconsistent
Observation:
Our Agent
Instruction: Leave the bathroom and walk forward along the pool.
Take a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room.
Observation:
Our Agent
Instruction: Leave the bathroom and walk forward along the pool. Take
a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room.
Observation:
Our Agent
Instruction: Leave the bathroom and walk forward along the pool. Take
a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room.
Starting VP Observed VP Unvisited VP !" #",%
Baseline model &",' = softmax'(#",'
1 23")
3" = LSTM( 9, !", :";% , 3";%)
#",<
Starting VP Observed VP Explored VP Navigated VP Target VP Unvisited VP !",$ %
"
&",$ Collected information
Starting VP Observed VP Explored VP Navigated VP Target VP Unvisited VP !",$ %",& ' %",$ Collected information Updated candidate feature
Starting VP Observed VP Explored VP Navigated VP Target VP Unvisited VP ! "#,% ! "#,& "#,& '#,& Updated candidate feature Collected information
Starting VP Observed VP Explored VP Navigated VP Target VP Unvisited VP ! "#,% ! "#,& Navigation decision making Updated candidate feature
Starting VP Observed VP Explored VP Navigated VP Target VP Unvisited VP
Need more exploration. Enough. STOP and return.
Starting VP Observed VP Explored VP Navigated VP Target VP Unvisited VP ℎ"
#$/&'
ℎ(
#$/&'
Exploration decision making
Stop Exploration Just make navigation.
Starting VP Observed VP Explored VP Navigated VP Target VP Unvisited VP Exploration decision making
Starting VP Observed VP Explored VP Navigated VP Target VP Unvisited VP ! "#,% "#,& Updated candidate feature
Imitation Learning Reinforcement Learning
Actor Loss Critic Loss Immediate reward
= change of distance, if != STOP, +3, if = STOP and SUCCEED,
Accumulated discount reward
Exploration-assisted relative reward shaping
Accumulated discount reward Immediate reward
Observation:
Observation:
Observation:
Observation:
Observation:
Observation:
Observation:
Observation:
Observation:
Observation:
Observation:
Observation:
by supervised learning.
between seen and new environments.
Project Page: https://github.com/HanqingWangAI/Active_VLN Email: hanqingwang@bit.edu.cn Advisor: Wei Liang, Wenguan Wang, Jianbing Shen