Programming by Demonstration with Situated Semantic Parsing Yoav - - PowerPoint PPT Presentation

Programming by Demonstration with Situated Semantic Parsing

Yoav Artzi, Maxwell Forbes, Kenton Lee, Maya Cakmak University of Washington

Programming by Demonstration

• Teaching the robot new skills
• Kinesthetic demonstrations
• Natural language interface

[Cakmak and Takayama 2014]

Semantic Parsing

at the chair, move forward three steps past the sofa

λa.pre(a, ιx.chair(x)) ∧ move(a) ∧ len(a, 3)∧ dir(a, forward) ∧ past(a, ιy.sofa(y))

Situated Semantic Parsing

“close left hand” close-left-hand

λa.close(a) ∧ patient(a, A(λx.hand(x) ∧ left(x)))

Situated Semantic Parsing

λa.close(a)

?

close-right-arm close-right-hand close-action close-left-hand …

“close”

Situated Semantic Parsing

λa.close(a)

close-right-arm close-right-hand close-action close-left-hand …

Use the system state to disambiguate

“close”

Situated Semantic Parsing

λa.close(a)

close-right-arm close-right-hand close-action close-left-hand …

Use the system state to disambiguate

• The close action can only

be applied to hand

• bjects

“close”

Situated Semantic Parsing

λa.close(a)

Use the system state to disambiguate

• The close action can only

be applied to hand

• bjects
• Infer salient objects from

recent actions

close-right-arm close-right-hand close-action close-left-hand …

“close”

Situated Semantic Parsing

λa.close(a)

Physical Manipulation Verbal Instruction Robot command

Bi-modal interaction

close-right-arm close-right-hand close-action close-left-hand …

“close”