SAPIEN A SimulAted Part-based Interactive ENvironment Fanbo Xiang 1 - - PowerPoint PPT Presentation

▶
sapien
SMART_READER_LITE
LIVE PREVIEW

SAPIEN A SimulAted Part-based Interactive ENvironment Fanbo Xiang 1 - - PowerPoint PPT Presentation

Jun 10, 2023 232 likes •598 views

SAPIEN A SimulAted Part-based Interactive ENvironment Fanbo Xiang 1 Yuzhe Qin 1 Kaichun Mo 2 Yikuan Xia 1 Hao Zhu 1 Fangchen Liu 1 Minghua Liu 1 Hanxiao Jiang 3 Yifu Yuan 5 He Wang 2 Li Yi 4 Angel X. Chang 3 Leonidas J. Guibas 2 Hao Su 1 1 UC San

slide-1
SLIDE 1

SAPIEN

A SimulAted Part-based Interactive ENvironment

Fanbo Xiang1 Yuzhe Qin1 Kaichun Mo2 Yikuan Xia1 Hao Zhu1 Fangchen Liu1 Minghua Liu1 Hanxiao Jiang3 Yifu Yuan5 He Wang2 Li Yi4 Angel X. Chang3 Leonidas J. Guibas2 Hao Su1

1UC San Diego 2Stanford University 3Simon Fraser University 4Google Research 5UC Los Angeles

slide-2
SLIDE 2

Intelligent Agent

Perceive information Adapt behavior

slide-3
SLIDE 3

Perceive information Adapt behavior Computer vision Robotics control

slide-4
SLIDE 4

Perceive information Adapt behavior Computer vision Robotics control Planning? Active perception?

slide-5
SLIDE 5

Data driven approaches

slide-6
SLIDE 6

Learn from data?

slide-7
SLIDE 7

Real-world data collection

slide-8
SLIDE 8

Reinforcement Learning? Manual data collection?

slide-9
SLIDE 9

Reinforcement learning?

  • 1. Expensive hardware
  • 2. Slow data collection
  • 3. Low sample efficiency
  • 4. Overfit to specific agent

Manual data collection?

  • 1. Unintuitive control
  • 2. May not be optimal
  • 3. Expertise
slide-10
SLIDE 10

Simulated Environment

slide-11
SLIDE 11

Physics Robotics Rendering Simulation Content

Simulated Environment

slide-12
SLIDE 12

Simulated Environment

Real World ?

slide-13
SLIDE 13

Physics Robotics Rendering Simulation Content

Simulated Environment

slide-14
SLIDE 14

SAPIEN

Physics Robotics Rendering Simulation Content

slide-15
SLIDE 15

SAPIEN Asset SAPIEN Renderer SAPIEN Engine

Physics Robotics Rendering Simulation Content

slide-16
SLIDE 16

SAPIEN Engine

SAPIEN Engine

PhysX Physical Simulator World Interface Articulation Interface Sensor Interface Controller Interface

3D/IMU
 Sensor Force/Joint/Velocity Controller Trajectory Controller Inverse Kinematics Motion Planning

Client API ROS Interface

slide-17
SLIDE 17

SAPIEN Engine

3D/IMU
 Sensor Force/Joint/Velocity Controller Trajectory Controller Inverse Kinematics Motion Planning

Client API

Reinforcement Learning Robot Tasks

slide-18
SLIDE 18

SAPIEN Renderer

3D/IMU
 Sensor Force/Joint/Velocity Controller Trajectory Controller Inverse Kinematics Motion Planning

Client API SAPIEN Renderer SAPIEN Engine

PhysX Physical Simulator World Interface Articulation Interface Sensor Interface Controller Interface Renderer Interface Rendered Images

ROS Interface

slide-19
SLIDE 19

SAPIEN Renderer

SAPIEN Renderer

GLSL Shaders Renderer Interface Normal RGBD Segmentation

slide-20
SLIDE 20

SAPIEN Renderer

SAPIEN Renderer

GLSL Shaders OptiX Renderer Interface Normal RGBD Segmentation Ray Tracing

slide-21
SLIDE 21

SAPIEN Renderer

GLSL Shaders OptiX Renderer Interface Normal RGBD Segmentation Ray Tracing Customizable Renderer/Visualizer

SAPIEN Renderer

slide-22
SLIDE 22

SAPIEN Engine

SAPIEN Asset

PhysX Physical Simulator World Interface Articulation Interface

SAPIEN Asset SAPIEN Renderer PartNet-Mobility Dataset

Sensor Interface Controller Interface GLSL Shaders OptiX Renderer Interface Normal RGBD Segmentation Ray Tracing

3D/IMU
 Sensor Force/Joint/Velocity Controller Trajectory Controller Inverse Kinematics Motion Planning

Customizable Renderer/Visualizer

Client API ROS Interface

slide-23
SLIDE 23

SAPIEN Asset

SAPIEN Asset PartNet-Mobility Dataset ShapeNet PartNet

Motion Annotation

2K models, 14K parts

slide-24
SLIDE 24
slide-25
SLIDE 25
slide-26
SLIDE 26
slide-27
SLIDE 27
slide-28
SLIDE 28
slide-29
SLIDE 29
slide-30
SLIDE 30

Training Testing Generalize

slide-31
SLIDE 31

SAPIEN Asset

SAPIEN Asset PartNet-Mobility Dataset Robot Model Object Layout Robot/Scene Builder

Standard URDF format Python API

slide-32
SLIDE 32

Client API SAPIEN Engine

SAPIEN

PhysX Physical Simulator World Interface Articulation Interface

SAPIEN Asset SAPIEN Renderer PartNet-Mobility Dataset Robot Model Object Layout

Sensor Interface Controller Interface GLSL Shaders OptiX Renderer Interface Normal RGBD Segmentation Ray Tracing

Robot/Scene Builder

3D/IMU
 Sensor Force/Joint/Velocity Controller Trajectory Controller Inverse Kinematics Motion Planning

Customizable Renderer/Visualizer

ROS Interface

slide-33
SLIDE 33

Task Demonstrations

Movable Part Segmentation Motion Parameter Estimation Long-horizon Planning Part Manipulation

slide-34
SLIDE 34
slide-35
SLIDE 35

Future Plans/Applications

  • Benchmark Platform
  • Vision tasks
  • Manipulation tasks
  • Education Platform
  • Control Algorithms
  • Robot Learning
slide-36
SLIDE 36

Summary

  • SAPIEN provides simulation and

rendering for vision and manipulation.

  • SAPIEN hosts a large-scale dataset

for articulated objects.

  • SAPIEN is open source.
  • pip install sapien

https://sapien.ucsd.edu Requirements: Python 3, Linux / Latest MacOS