Programming by Demonstration with Situated Semantic Parsing Yoav - - PowerPoint PPT Presentation

programming by demonstration with situated semantic
SMART_READER_LITE
LIVE PREVIEW

Programming by Demonstration with Situated Semantic Parsing Yoav - - PowerPoint PPT Presentation

Programming by Demonstration with Situated Semantic Parsing Yoav Artzi, Maxwell Forbes, Kenton Lee, Maya Cakmak University of Washington Programming by Demonstration Teaching the robot new skills Kinesthetic demonstrations Natural


slide-1
SLIDE 1

Programming by Demonstration with Situated Semantic Parsing

Yoav Artzi, Maxwell Forbes, Kenton Lee, Maya Cakmak University of Washington

slide-2
SLIDE 2

Programming by Demonstration

  • Teaching the robot new skills
  • Kinesthetic demonstrations
  • Natural language interface

[Cakmak and Takayama 2014]

slide-3
SLIDE 3

Semantic Parsing

at the chair, move forward three steps past the sofa

λa.pre(a, ιx.chair(x)) ∧ move(a) ∧ len(a, 3)∧ dir(a, forward) ∧ past(a, ιy.sofa(y))

slide-4
SLIDE 4

Situated Semantic Parsing

“close left hand” close-left-hand

λa.close(a) ∧ patient(a, A(λx.hand(x) ∧ left(x)))

slide-5
SLIDE 5

Situated Semantic Parsing

λa.close(a)

?

close-right-arm close-right-hand close-action close-left-hand …

“close”

slide-6
SLIDE 6

Situated Semantic Parsing

λa.close(a)

close-right-arm close-right-hand close-action close-left-hand …

Use the system state to disambiguate

“close”

slide-7
SLIDE 7

Situated Semantic Parsing

λa.close(a)

close-right-arm close-right-hand close-action close-left-hand …

Use the system state to disambiguate

  • The close action can only

be applied to hand

  • bjects

“close”

slide-8
SLIDE 8

Situated Semantic Parsing

λa.close(a)

Use the system state to disambiguate

  • The close action can only

be applied to hand

  • bjects
  • Infer salient objects from

recent actions

close-right-arm close-right-hand close-action close-left-hand …

“close”

slide-9
SLIDE 9

Situated Semantic Parsing

λa.close(a)

Physical Manipulation Verbal Instruction Robot command

Bi-modal interaction

close-right-arm close-right-hand close-action close-left-hand …

“close”