Robot Learning Collaborative Manipulation Plans from YouTube Cooking - - PowerPoint PPT Presentation

robot learning collaborative manipulation plans from
SMART_READER_LITE
LIVE PREVIEW

Robot Learning Collaborative Manipulation Plans from YouTube Cooking - - PowerPoint PPT Presentation

Robot Learning Collaborative Manipulation Plans from YouTube Cooking Videos Zhang, H. and Nikolaidis, S., 2019. Robot learning and execution of collaborative manipulation plans from youtube cooking videos. Hejia Zhang and Stefanos Nikolaidis


slide-1
SLIDE 1

Robot Learning Collaborative Manipulation Plans from YouTube Cooking Videos

Hejia Zhang and Stefanos Nikolaidis

Department of Computer Science, University of Southern California

Zhang, H. and Nikolaidis, S., 2019. Robot learning and execution of collaborative manipulation plans from youtube cooking videos.

slide-2
SLIDE 2
slide-3
SLIDE 3

Research Question

How can the robot learn and execute collaborative manipulation plans from

  • nline videos?
slide-4
SLIDE 4

Key Insight

Hands contain temporal and spatial information of performed actions and manipulated objects

slide-5
SLIDE 5

Our approach

Input Video Hand Detection and Object Association Segmentation Action Recognition cut | pour | transfer |spread | grip | stir | sprinkle | wrap| coat| roll| holding | handover .... Symbolic Command Generation Action Graph Generation Robot Execution

HP CP HP AP OP OP O bowl O plate O fruit A transfer HP O spoon H RH_P2 C holding HP O plate H LH_P1

P2 P2 P1 P2 P1 P2 P1 P2

slide-6
SLIDE 6

We use deep learning techniques to detect hands and objects in the video.

slide-7
SLIDE 7

We segment the video to be several action clips based on the hand trajectories.

slide-8
SLIDE 8

We infer performed actions based on the association of hands and

  • bjects.
slide-9
SLIDE 9

We infer performed actions based on the association of hands and

  • bjects.
slide-10
SLIDE 10

We infer performed actions based on the association of hands and

  • bjects.

P(action | knife, chicken, board)

slide-11
SLIDE 11

We infer performed actions based on the association of hands and

  • bjects.

P(cut | knife, chicken, board)

Yang, Y., Li, Y., Fermuller, C. and Aloimonos, Y., 2015, March. Robot learning manipulation action plans by" watching" unconstrained videos from the world wide web. In Twenty-Ninth AAAI Conference on Artificial Intelligence.

slide-12
SLIDE 12

We infer performed actions based on the association of hands and

  • bjects.
slide-13
SLIDE 13

We infer performed actions based on the association of hands and

  • bjects.

Collaborative action: holding handover

slide-14
SLIDE 14

We infer performed actions based on the association of hands and

  • bjects.

Collaborative action: holding handover

slide-15
SLIDE 15

We infer performed actions based on the association of hands and

  • bjects.

Collaborative action: holding handover

slide-16
SLIDE 16

We infer performed actions based on the association of hands and

  • bjects.

Collaborative action: holding handover

slide-17
SLIDE 17

Example: Handover

slide-18
SLIDE 18

Example: Holding / Cut

slide-19
SLIDE 19

Based on the extracted action sequence, we construct a robot executable action graph.

slide-20
SLIDE 20

Demo: Action Graph Execution (Simulation).

WeCook: https://github.com/icaros-usc/wecook

slide-21
SLIDE 21

Demo: Action Graph Execution (Real World).

WeCook: https://github.com/icaros-usc/wecook

slide-22
SLIDE 22

When a collaborative action is about to happen? How collaborative actions emerge in close proximity interactions? Why a collaborative action happens?

Future Work

slide-23
SLIDE 23

Contribution

By leveraging hand-object and object-

  • bject associations in unconstrained

YouTube videos, robots can learn and execute human-interpretable, collaborative manipulation plans.

slide-24
SLIDE 24

Hejia Zhang and Stefanos Nikolaidis

Department of Computer Science, University of Southern California

Zhang, H. and Nikolaidis, S., 2019. Robot learning and execution of collaborative manipulation plans from youtube cooking videos.

WeCook: https://github.com/icaros-usc/wecook

Robot Learning Collaborative Manipulation Plans from YouTube Cooking Videos