ISAAC: A PEEK UNDER THE HOOD Adam Moravanszky Talk ID: 23386 GTC - - PowerPoint PPT Presentation

▶
isaac a peek under the hood
SMART_READER_LITE
LIVE PREVIEW

ISAAC: A PEEK UNDER THE HOOD Adam Moravanszky Talk ID: 23386 GTC - - PowerPoint PPT Presentation

Jan 08, 2023 7 likes •371 views

ISAAC: A PEEK UNDER THE HOOD Adam Moravanszky Talk ID: 23386 GTC Europe 2017 WHAT THIS TALK IS ABOUT 2 NVIDIA GAMEWORKS PHYSX FLEX DESTRUCTION HAIRWORKS FLOW CLOTH 3 GAMES 4 VR Ivan Sutherland, MIT/Utah, 70s 5 VR APP CHALLENGES

slide-1
SLIDE 1

Adam Moravanszky

Talk ID: 23386

ISAAC: A PEEK UNDER THE HOOD

GTC Europe 2017

slide-2
SLIDE 2

2

WHAT THIS TALK IS ABOUT

slide-3
SLIDE 3

3

NVIDIA GAMEWORKS

PHYSX FLEX HAIRWORKS FLOW DESTRUCTION CLOTH

slide-4
SLIDE 4

4

GAMES

slide-5
SLIDE 5

5

VR

Ivan Sutherland, MIT/Utah, 70s

slide-6
SLIDE 6

6

VR APP CHALLENGES

GRAPHICS PHYSICAL SIMULATION AUDIO TOUCH

slide-7
SLIDE 7

7

FUNHOUSE VIDEO

slide-8
SLIDE 8

8

Architectural Design

slide-9
SLIDE 9

9

AUTONOMOUS VEHICLES

slide-10
SLIDE 10

10

PROJECT HOLODECK

PHOTOREALISTIC GRAPHICS REAL-TIME COLLABORATION INTERACTIVE PHYSICS

slide-11
SLIDE 11

11

NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

ROBOTICS

example: ANYmal

slide-12
SLIDE 12

12

NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DNN REVOLUTION

by random example of Albrecht Altdorfer

slide-13
SLIDE 13

13

REINFORCEMENT LEARNING

is not rocket science

slide-14
SLIDE 14

14

WHY REINFORCEMENT LEARNING

is best in a simulated environment

slide-15
SLIDE 15

15

TRAINING A ROBOT IN HOLODECK

VR real enough for trained skill to transfer to rl

slide-16
SLIDE 16

16

FROM HOLODECK TO ISAAC

Robot Import Virtual Vision Interface to AI Server (the “brain” we are training) What we added

slide-17
SLIDE 17

17

WE ALSO NEED

A real robot Some training challenges

For a showcase

slide-18
SLIDE 18

18

slide-19
SLIDE 19

19

slide-20
SLIDE 20

20

slide-21
SLIDE 21

21

VR – ROBOT INTERFACE

slide-22
SLIDE 22

22

VR – BAXTER INTEGRATION

Block Diagram AI SERVER BAXTER BX-CONT. ISAAC VR

Domino Vision Domino Play AI Robo Drive

slide-23
SLIDE 23

23

A-I SERVER

RL DNN Domino Recognition Algorithmic Endpoint detection RL DNN Domino Play / Explicit coded Domino Play (for Hockey: RL Continuous control) This is for inference! Training backend can run on Cloud  DGX workstation.

AI SERVER

Domino Vision Domino Play AI Robo Drive

slide-24
SLIDE 24

24

DOMINO VISION

For each incoming image

Multiple Domino Detection Domino Pose Estimation Domino Type Classification

slide-25
SLIDE 25

25

VISION TECH

slide-26
SLIDE 26

26

DOMINO RECOGNITION

slide-27
SLIDE 27

27

slide-28
SLIDE 28

28

BAXTER CONTROL COMPUTER

Strapped webcam Iterative movement refiner (PID c.) Vision based human interaction

BX-CONT.

slide-29
SLIDE 29

29

VISION BASED HI BodySLAM

slide-30
SLIDE 30

30

ISAAC VR

IK arm controller (Golf/Hockey demos generate joint torques directly)

ISAAC VR

slide-31
SLIDE 31

31

VR ENVIRONMENT

A relatable robot

slide-32
SLIDE 32

32

VR RESULTS VIDEO

slide-33
SLIDE 33

33

REAL RESULT VIDEO

slide-34
SLIDE 34

34

slide-35
SLIDE 35

35

NEXT STEPS

slide-36
SLIDE 36

36

THAT’S ALL FOLKS!

Lots more to come: This is just proof of concept, and very early days. Looking for robotics collaborators Hiring like crazy…

Thanks you!