Learning to Interpret Natural Language Navigation Instructions from - - PowerPoint PPT Presentation

Learning to Interpret Natural Language Navigation Instructions from Observations

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Problem

• Follow free-form natural-language navigation instructions.
• No prior linguistic knowledge.
• Training data: {(e1, a1, w1), (e2, a2, w2), …, (en, an, wn)}

○ ei - natural language instruction ○ ai - action sequence ○ wi - current state of the world

• Goal: (ej, wj) -> aj
• Three virtual worlds with same elements in different configurations.

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Example

“Go towards the coat rack and take a left at the coat rack. go all the way to the end of the hall and this is 4.” “turn so that the wall is on your right side. walk forward once. turn left. walk forward twice.” “foward to the fish. first left. go tot [sic] the end.” “Place your back to the wall of the ’T’

• intersection. Turn right. Go forward one segment

to the intersection with the yellow-tiled hall. This intersection contains a hatrack. Turn left. Go forward two segments to the end of the hall. This is Position 4.”

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Motivation

• Interpretation of human instructions
• Learning language directly from observation
• Requires both linguistic and spatial context
• Grounding of natural language
• Error recovery

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Approach

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

pi wi ai ei ej wj aj pj p’

i (refined)

Navigation Plans

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Navigation Plans

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Navigation Plans

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Lexicon Learning

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Lexicon Learning

• Score(w, g) = p(g|w) - p(g|¬w)

○ w - n-gram ○ g - graph

• A good (w, g) pair means that w should be indicative of g appearing.
• Ubiquitous g should be penalized.

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Refining Plans

for (ei, pi) repeat select highest-scoring (w, g) such that w and g appear in ei and pi, respectively remove w from ei and mark all occurrences of g in pi until no possible selection of (w, g) p’

i <- pi - unmarked nodes

end for

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Next Steps

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

pi wi ai ei ej wj aj pj p’

i (refined)

Dataset

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Evaluation - Generating Navigation Plans

• Gold-standard: hand-annotated navigation plans
• Partial correctness metric
• Leave-one-map-out for ‘refined landmarks plans’

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Evaluation - Semantic Parser

• Leave-one-map-out approach
• Precision of training data is more important than recall

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Evaluation - Executing Navigation Plans

Percentage of trials reaching the correct destination

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Example

Chen, D., & Mooney, R. (2011). . In AAAI Conference on Artificial Intelligence.

Critique

• Only the instruction used for generating navigation plan

○ Can we use the current view of the follower to assist in generating next navigation plan? ○ “Turn so that the wall is on your right side.”

• How are ambiguous words handled in lexicon learning?
• How do the results change with change in hyperparameters e.g. k, t, and n?

○ How were the mentioned choices made?

• Why does refining of plans help in error recovery?
• How well does it generalize?

○ What if new words are seen during testing? What if a new world with different items is encountered?

Discussion

• What can be some other

evaluation metrics?

• How about an end-to-end

model? Neural Networks?

• Will prior linguistic knowledge

be helpful?