Robot Learning Collaborative Manipulation Plans from YouTube Cooking - - PowerPoint PPT Presentation

▶

Oct 09, 2022 218 likes •476 views

Robot Learning Collaborative Manipulation Plans from YouTube Cooking Videos Zhang, H. and Nikolaidis, S., 2019. Robot learning and execution of collaborative manipulation plans from youtube cooking videos. Hejia Zhang and Stefanos Nikolaidis

SLIDE 1

Robot Learning Collaborative Manipulation Plans from YouTube Cooking Videos

Hejia Zhang and Stefanos Nikolaidis

Department of Computer Science, University of Southern California

Zhang, H. and Nikolaidis, S., 2019. Robot learning and execution of collaborative manipulation plans from youtube cooking videos.

SLIDE 2

SLIDE 3

Research Question

How can the robot learn and execute collaborative manipulation plans from

nline videos?

SLIDE 4

Key Insight

Hands contain temporal and spatial information of performed actions and manipulated objects

SLIDE 5

Our approach

HP CP HP AP OP OP O bowl O plate O fruit A transfer HP O spoon H RH_P2 C holding HP O plate H LH_P1

P2 P2 P1 P2 P1 P2 P1 P2

SLIDE 6

We use deep learning techniques to detect hands and objects in the video.

SLIDE 7

We segment the video to be several action clips based on the hand trajectories.

SLIDE 8

We infer performed actions based on the association of hands and

bjects.

SLIDE 9

We infer performed actions based on the association of hands and

bjects.

SLIDE 10

We infer performed actions based on the association of hands and

bjects.

P(action | knife, chicken, board)

SLIDE 11

We infer performed actions based on the association of hands and

bjects.

P(cut | knife, chicken, board)

Yang, Y., Li, Y., Fermuller, C. and Aloimonos, Y., 2015, March. Robot learning manipulation action plans by" watching" unconstrained videos from the world wide web. In Twenty-Ninth AAAI Conference on Artificial Intelligence.

SLIDE 12

We infer performed actions based on the association of hands and

bjects.

SLIDE 13

We infer performed actions based on the association of hands and

bjects.

Collaborative action: holding handover

SLIDE 14

We infer performed actions based on the association of hands and

bjects.

Collaborative action: holding handover

SLIDE 15

We infer performed actions based on the association of hands and

bjects.

Collaborative action: holding handover

SLIDE 16

We infer performed actions based on the association of hands and

bjects.

Collaborative action: holding handover

SLIDE 17

Example: Handover

SLIDE 18

Example: Holding / Cut

SLIDE 19

Based on the extracted action sequence, we construct a robot executable action graph.

SLIDE 20

Demo: Action Graph Execution (Simulation).

WeCook: https://github.com/icaros-usc/wecook

SLIDE 21

Demo: Action Graph Execution (Real World).

WeCook: https://github.com/icaros-usc/wecook

SLIDE 22

When a collaborative action is about to happen? How collaborative actions emerge in close proximity interactions? Why a collaborative action happens?

Future Work

SLIDE 23

Contribution

By leveraging hand-object and object-

bject associations in unconstrained

YouTube videos, robots can learn and execute human-interpretable, collaborative manipulation plans.

SLIDE 24

Hejia Zhang and Stefanos Nikolaidis

Department of Computer Science, University of Southern California

Zhang, H. and Nikolaidis, S., 2019. Robot learning and execution of collaborative manipulation plans from youtube cooking videos.

WeCook: https://github.com/icaros-usc/wecook