Active Perception in Vision-Language Navigation Task - PowerPoint PPT Presentation

Student Webinar Active Perception in Vision-Language Navigation Task 汪汗青 媒体计算与智能系统实验室 媒体计算与智能系统实验室 (MC (MCIS Lab) 北京理工大学 北京理工大学 (B (Beijing ng Ins nsti titute tute of T echno hnology) Hom Home Page: https://hanqingwa wangai.github.io Em Email: hanqingwang@bit. t.edu.cn

Outl tline • Introduction to VLN Task • Challenges of VLN Task • Baseline & Our Approach • Learning method • Conclusion

Refe ference Active Visual Information Gathering for Vision-Language Navigation Hanqing Wang , Wenguan Wang, Tianmin Shu, Wei Liang, and Jianbing Shen

VLN Task sk [Anderson, P., Wu, Q. et al., Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments, CVPR 2018]

VLN Task sk [Manolis S., Abhishek K., Oleksandr M. and et al., Habitat: A Platform for Embodied AI Research, ICCV 2019]

VLN Task sk Ground Truth Path Instruction : Leave the bathroom and walk forward along the pool. Take a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room. Observation:

VLN Task sk Ground Truth Path Instruction : Leave the bathroom and walk forward along the pool. Take a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room. Observation:

VLN Task sk Ground Truth Path Instruction : Leave the bathroom and walk forward along the pool. Take a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room. Observation:

VLN Task sk Ground Truth Path Instruction : Leave the bathroom and walk forward along the pool. Take a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room. Observation:

Challenge of f VLN Task sk Basic Agent Instruction : Leave the bathroom and walk forward along the pool . Take a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room. Observation: Door 1 Door 2

Challenge of f VLN Task sk Basic Agent Instruction : Leave the bathroom and walk forward along the pool . Take a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room. Observation: Failed

Challenge of f VLN Task sk Our Agent Instruction : Leave the bathroom and walk forward along the pool . Take a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room. Observation: Insufficient information Door 1 Door 2

Challenge of f VLN Task sk Our Agent Instruction : Leave the bathroom and walk forward along the pool . Take a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room. Observation: inconsistent

Challenge of f VLN Task sk Our Agent Instruction : Leave the bathroom and walk forward along the pool . Take a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room. Observation:

Challenge of f VLN Task sk Our Agent Instruction : Leave the bathroom and walk forward along the pool. Take a left at the stairs and go up the stairs . Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room. Observation:

Challenge of f VLN Task sk Our Agent Instruction : Leave the bathroom and walk forward along the pool. Take Succeed! a left at the stairs and go up the stairs. Go up another set of stairs, and enter the massage room first on the right. Stop in the doorway to the massage room . Observation:

Base seline Baseline model Starting VP 3 " = LSTM( 9, ! " , : ";% , 3 ";% ) Observed VP 1 23 " ) & ",' = softmax ' (# ",' # ",% Unvisited VP ! " # ",<

Naïve Ex Na Exploration Module Collected information Starting VP ! ",$ Observed VP & ",$ Explored VP % " Navigated VP Target VP Unvisited VP

Na Naïve Ex Exploration Module Collected information Starting VP ! ",$ Observed VP Updated candidate feature % ",$ ' Explored VP % ",& Navigated VP Target VP Unvisited VP

Naïve Ex Na Exploration Module Collected information Starting VP " #,% ! Observed VP Updated candidate feature Explored VP " #,& " #,& ! Navigated VP ' #,& Target VP Unvisited VP

Na Naïve Ex Exploration Module Starting VP Navigation decision making " #,% ! Observed VP Explored VP Updated candidate feature " #,& ! Navigated VP Target VP Unvisited VP

Na Naïve Ex Exploration Module Starting VP Observed VP Explored VP Navigated VP Target VP Unvisited VP

Few & Deeper Ex Exploration Exploration decision making Starting VP Need more Observed VP exploration. Explored VP #$/&' ℎ " Enough. Navigated VP STOP and return. #$/&' Target VP ℎ ( Unvisited VP

Few & Deeper Ex Exploration Stop Exploration Exploration decision making Starting VP Just make navigation. Observed VP Explored VP Navigated VP Target VP Unvisited VP

Few & Deeper Ex Exploration Updated candidate feature Starting VP " #,& Observed VP Explored VP " #,% ! Navigated VP Target VP Unvisited VP

Over vervi view ew

Tr Training Training Signals: + + + Imitation Learning Reinforcement Learning

Imit Imitat atio ion Learn Learnin ing Navigation Part: Exploration Part:

Reinfo forcement Learning Navigation Part: change of distance, if != STOP, = +3, if = STOP and SUCCEED, Immediate reward -3, if = STOP and FAIL. Accumulated discount reward Actor Loss Critic Loss

Reinfo forcement Learning Exploration Part: Exploration-assisted relative reward shaping Immediate reward Accumulated discount reward

Quantitative Resu sults

Qualitative Resu sults Observation:

Qualitative Resu sults Observation:

Qualitative Resu sults Observation:

Qualitative Resu sults Observation:

Qualitative Resu sults Observation:

Qualitative Resu sults Observation: Failed

Qualitative Resu sults Observation:

Qualitative Resu sults Observation:

Qualitative Resu sults Observation:

Qualitative Resu sults Observation:

Qualitative Resu sults Observation:

Qualitative Resu sults Observation: Succeed!

Conclusi sions 1. Active perception is effective in embodied vision tasks. 2. IL warm up + RL is powerful for some problems that are hard to learn by supervised learning. 3. VLN task is far from been solved, how to fill the performance gap between seen and new environments.

Thanks for your listening! Project Page: https://github.com/HanqingWangAI/Active_VLN Email: hanqingwang@bit.edu.cn Advisor: Wei Liang, Wenguan Wang, Jianbing Shen