SAPIEN A SimulAted Part-based Interactive ENvironment Fanbo Xiang 1 - - PowerPoint PPT Presentation

▶
sapien
SMART_READER_LITE
LIVE PREVIEW

SAPIEN A SimulAted Part-based Interactive ENvironment Fanbo Xiang 1 - - PowerPoint PPT Presentation

Oct 24, 2023 377 likes •1.25k views

SAPIEN A SimulAted Part-based Interactive ENvironment Fanbo Xiang 1 Yuzhe Qin 1 Kaichun Mo 2 Yikuan Xia 1 Hao Zhu 1 Fangchen Liu 1 Minghua Liu 1 Hanxiao Jiang 3 Yifu Yuan 5 He Wang 2 Li Yi 4 Angel X. Chang 3 Leonidas J. Guibas 2 Hao Su 1 1 UC San

slide-1
SLIDE 1

SAPIEN

A SimulAted Part-based Interactive ENvironment

Fanbo Xiang1 Yuzhe Qin1 Kaichun Mo2 Yikuan Xia1 Hao Zhu1 Fangchen Liu1 Minghua Liu1 Hanxiao Jiang3 Yifu Yuan5 He Wang2 Li Yi4 Angel X. Chang3 Leonidas J. Guibas2 Hao Su1

1UC San Diego 2Stanford University 3Simon Fraser University 4Google Research 5UC Los Angeles

slide-2
SLIDE 2

About Me

UIUC BS CS, BS Math UCSD: MS CS

Fanbo Xiang, UCSD Advisor: Hao Su, UCSD Research: Graphics, Vision, Robotics

slide-3
SLIDE 3

Outline

  • Background

○ Intelligence, vision and robotics ○ Simulated environment

  • SAPIEN architecture

○ Physics and robotics ○ Renderer ○ Assets

  • Future research problems
slide-4
SLIDE 4
slide-5
SLIDE 5

Intelligent Agent

slide-6
SLIDE 6

Intelligent Agent

Perceive information

slide-7
SLIDE 7

Intelligent Agent

Perceive information Adapt behavior

slide-8
SLIDE 8

Perceive information Adapt behavior

slide-9
SLIDE 9

Perceive information Adapt behavior Computer vision

slide-10
SLIDE 10

Perceive information Adapt behavior Computer vision Robotics control

slide-11
SLIDE 11

Perceive information Adapt behavior Computer vision Robotics control

slide-12
SLIDE 12

Perceive information Adapt behavior Computer vision Robotics control Planning? Active perception?

slide-13
SLIDE 13
slide-14
SLIDE 14

Data driven approaches

slide-15
SLIDE 15

Learn from data?

slide-16
SLIDE 16

Real-world data collection

slide-17
SLIDE 17

Reinforcement Learning? Manual data collection?

slide-18
SLIDE 18

Reinforcement Learning

slide-19
SLIDE 19

RL Agent

slide-20
SLIDE 20

Observation

RL Agent

slide-21
SLIDE 21

Observation

RL Agent

Perception

slide-22
SLIDE 22

Observation

RL Agent

Perception Action

slide-23
SLIDE 23

Observation

RL Agent

Perception Control Action

slide-24
SLIDE 24

Observation

RL Agent

Perception Control Action Reward State update

slide-25
SLIDE 25

RL Problems

  • Expensive hardware
  • Slow data collection
  • Low sample efficiency
  • Overfit to specific agent
slide-26
SLIDE 26

Imitation Learning (Manual Data Collection)

slide-27
SLIDE 27

Observation

Human

Perception Control Action State update

slide-28
SLIDE 28

Observation

Human

Perception Control Action State update

IL Agent

Training

slide-29
SLIDE 29

IL Problems

  • Unintuitive control
  • May not be optimal
  • Human expertise
slide-30
SLIDE 30

Simulated Environment

slide-31
SLIDE 31

Simulated Environment

Manual data collection

  • More freedom in controller

design

  • Less expertise required

RL

  • Will not break hardware
  • Easy to scale
slide-32
SLIDE 32

Simulated Environment

slide-33
SLIDE 33

Simulated Environment

Real World ?

slide-34
SLIDE 34

Physics

Simulated Environment

slide-35
SLIDE 35

Physics Robotics

Simulated Environment

slide-36
SLIDE 36

Physics Robotics Rendering

Simulated Environment

slide-37
SLIDE 37

Physics Robotics Rendering Simulation Content

Simulated Environment

slide-38
SLIDE 38

SAPIEN

Physics Robotics Rendering Simulation Content

slide-39
SLIDE 39

SAPIEN Asset SAPIEN Renderer SAPIEN Engine

Physics Robotics Rendering Simulation Content

slide-40
SLIDE 40

SAPIEN Engine

slide-41
SLIDE 41

SAPIEN Engine

SAPIEN Engine

PhysX Physical Simulator

slide-42
SLIDE 42

SAPIEN Engine

SAPIEN Engine

PhysX Physical Simulator World Interface Articulation Interface

slide-43
SLIDE 43

SAPIEN Engine

SAPIEN Engine

PhysX Physical Simulator World Interface Articulation Interface

ROS Interface

Sensor Interface Controller Interface

slide-44
SLIDE 44

SAPIEN Engine

SAPIEN Engine

PhysX Physical Simulator World Interface Articulation Interface Sensor Interface Controller Interface

3D/IMU Sensor Force/Joint/Velocity Controller Trajectory Controller Inverse Kinematics Motion Planning

Client API ROS Interface

slide-45
SLIDE 45

SAPIEN Engine

3D/IMU Sensor Force/Joint/Velocity Controller Trajectory Controller Inverse Kinematics Motion Planning

Client API

slide-46
SLIDE 46

SAPIEN Engine

3D/IMU Sensor Force/Joint/Velocity Controller Trajectory Controller Inverse Kinematics Motion Planning

Client API

Reinforcement Learning

slide-47
SLIDE 47

SAPIEN Engine

3D/IMU Sensor Force/Joint/Velocity Controller Trajectory Controller Inverse Kinematics Motion Planning

Client API

Reinforcement Learning Robot Tasks

slide-48
SLIDE 48

SAPIEN Engine

slide-49
SLIDE 49

SAPIEN Renderer

3D/IMU Sensor Force/Joint/Velocity Controller Trajectory Controller Inverse Kinematics Motion Planning

Client API SAPIEN Engine

PhysX Physical Simulator World Interface Articulation Interface Sensor Interface Controller Interface

ROS Interface

slide-50
SLIDE 50

SAPIEN Renderer

3D/IMU Sensor Force/Joint/Velocity Controller Trajectory Controller Inverse Kinematics Motion Planning

Client API SAPIEN Engine

PhysX Physical Simulator World Interface Articulation Interface Sensor Interface Controller Interface

ROS Interface SAPIEN Renderer

slide-51
SLIDE 51

SAPIEN Renderer

SAPIEN Renderer

3D/IMU Sensor Force/Joint/Velocity Controller Trajectory Controller Inverse Kinematics Motion Planning

Client API SAPIEN Engine

PhysX Physical Simulator World Interface Articulation Interface Sensor Interface Controller Interface Renderer Interface

ROS Interface

slide-52
SLIDE 52

SAPIEN Engine ROS Interface

SAPIEN Renderer

3D/IMU Sensor Force/Joint/Velocity Controller Trajectory Controller Inverse Kinematics Motion Planning

Client API SAPIEN Renderer

PhysX Physical Simulator World Interface Articulation Interface Sensor Interface Controller Interface Renderer Interface Rendered Images

slide-53
SLIDE 53

SAPIEN Renderer

SAPIEN Renderer

GLSL Shaders Renderer Interface Normal RGBD Segmentation

slide-54
SLIDE 54

SAPIEN Renderer

SAPIEN Renderer

OptiX Shaders Ray Tracing GLSL Shaders Renderer Interface Normal RGBD Segmentation

slide-55
SLIDE 55

SAPIEN Renderer

SAPIEN Renderer

OptiX Shaders Ray Tracing GLSL Shaders Renderer Interface Normal RGBD Segmentation

4 spp, 3 bounce, 512x512 OptiX denoise, < 20 FPS

slide-56
SLIDE 56

Customizable Renderer/Visualizer

SAPIEN Renderer

SAPIEN Renderer

OptiX Shaders Ray Tracing GLSL Shaders Renderer Interface Normal RGBD Segmentation

slide-57
SLIDE 57

Customizable Renderer/Visualizer

SAPIEN Renderer

SAPIEN Renderer

OptiX Shaders Ray Tracing GLSL Shaders Renderer Interface Normal RGBD Segmentation

Considerations

  • Needs to run at real time (~100 FPS)

○ Rasterizer ○ Ray tracer denoise? (1spp max) ○ GPU-CPU transfer? (>10 ms) ○ Prebaking? (interaction)

slide-58
SLIDE 58

SAPIEN Renderer

3D/IMU Sensor Force/Joint/Velocity Controller Trajectory Controller Inverse Kinematics Motion Planning

Customizable Renderer/Visualizer

Client API SAPIEN Engine ROS Interface

PhysX Physical Simulator World Interface Articulation Interface Sensor Interface Controller Interface Customizable Renderer/Visualizer

SAPIEN Renderer

OptiX Shaders Ray Tracing GLSL Shaders Renderer Interface Normal RGBD Segmentation

slide-59
SLIDE 59

SAPIEN Asset PartNet-Mobility Dataset

SAPIEN Asset

ShapeNet PartNet

Motion Annotation

2K models, 14K parts

slide-60
SLIDE 60
slide-61
SLIDE 61
slide-62
SLIDE 62
slide-63
SLIDE 63

PartNet-Mobility Annotation

Shape cleaning (ShapeNet, 3D Warehouse) Motion annotation (PartNet-Mobility) Part segmentation (PartNet)

slide-64
SLIDE 64
slide-65
SLIDE 65
slide-66
SLIDE 66

Training Testing Generalize

slide-67
SLIDE 67

SAPIEN Asset

SAPIEN Asset PartNet-Mobility Dataset

slide-68
SLIDE 68

SAPIEN Asset

SAPIEN Asset PartNet-Mobility Dataset Robot Model

slide-69
SLIDE 69

SAPIEN Asset

SAPIEN Asset PartNet-Mobility Dataset Robot Model Object Layout

slide-70
SLIDE 70

SAPIEN Asset

Robot/Scene Builder

Standard URDF format Python API

SAPIEN Asset PartNet-Mobility Dataset Robot Model Object Layout

slide-71
SLIDE 71

SAPIEN

3D/IMU Sensor Force/Joint/Velocity Controller Trajectory Controller Inverse Kinematics Motion Planning

Customizable Renderer/Visualizer

SAPIEN Engine ROS Interface

PhysX Physical Simulator World Interface Articulation Interface Sensor Interface Controller Interface Customizable Renderer/Visualizer

SAPIEN Renderer

OptiX Shaders Ray Tracing GLSL Shaders Renderer Interface Normal RGBD Segmentation

SAPIEN Asset PartNet-Mobility Dataset Robot Model Object Layout Client API Robot/Scene Builder

slide-72
SLIDE 72

Task Demonstrations

slide-73
SLIDE 73

Task Demonstrations

Movable Part Segmentation

slide-74
SLIDE 74

Task Demonstrations

Movable Part Segmentation Motion Parameter Estimation

slide-75
SLIDE 75

Task Demonstrations

Movable Part Segmentation Motion Parameter Estimation Part Manipulation

slide-76
SLIDE 76

Task Demonstrations

Movable Part Segmentation Motion Parameter Estimation Long-horizon Planning Part Manipulation

slide-77
SLIDE 77

Task Demonstrations

Movable Part Segmentation

  • Standard vision problem: detection and

segmentation.

slide-78
SLIDE 78

Task Demonstrations

Motion Parameter Estimation

  • New vision task
  • Important for control tasks

○ Given the motion parameters, we can use control methods to manipulate the parts

slide-79
SLIDE 79

Task Demonstrations

Part Manipulation

  • Manipulation and control

○ Reinforcement learning ○ Imitation learning ○ robotics control

slide-80
SLIDE 80

Task Demonstrations

  • Manipulation and control

○ Reinforcement learning ○ Imitation learning ○ robotics control

slide-81
SLIDE 81

Task Demonstrations

Long-horizon Planning

  • Planning

○ Achieve meaningful tasks ○ “Home assistant”

slide-82
SLIDE 82

Future Plans/Applications

  • Benchmark (SAPIEN Challenge)
  • Vision tasks
  • Manipulation tasks
  • Education Platform
  • Control Algorithms
  • Robot Learning
slide-83
SLIDE 83

Demos

slide-84
SLIDE 84

Demos

slide-85
SLIDE 85

Summary

  • SAPIEN provides simulation and

rendering for vision and manipulation.

  • SAPIEN hosts a large-scale dataset

for articulated objects.

  • SAPIEN is open source.
  • pip install sapien

https://sapien.ucsd.edu Requirements: Python 3, Linux / Latest MacOS

slide-86
SLIDE 86

Thank you

  • Questions?

https://sapien.ucsd.edu Requirements: Python 3, Linux / Latest MacOS