ISAAC: A PEEK UNDER THE HOOD Adam Moravanszky Talk ID: 23386 GTC - PowerPoint PPT Presentation

ISAAC: A PEEK UNDER THE HOOD Adam Moravanszky Talk ID: 23386 GTC Europe 2017

WHAT THIS TALK IS ABOUT 2

NVIDIA GAMEWORKS PHYSX FLEX DESTRUCTION HAIRWORKS FLOW CLOTH 3

GAMES 4

VR Ivan Sutherland, MIT/Utah, 70s 5

VR APP CHALLENGES PHYSICAL GRAPHICS AUDIO TOUCH SIMULATION 6

FUNHOUSE VIDEO 7

Architectural Design 8

AUTONOMOUS VEHICLES 9

PHOTOREALISTIC GRAPHICS PROJECT HOLODECK REAL-TIME COLLABORATION INTERACTIVE PHYSICS 10

ROBOTICS example: ANYmal NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 11

DNN REVOLUTION by random example of Albrecht Altdorfer NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 12

REINFORCEMENT LEARNING is not rocket science 13

WHY REINFORCEMENT LEARNING is best in a simulated environment 14

TRAINING A ROBOT IN HOLODECK VR real enough for trained skill to transfer to rl 15

FROM HOLODECK TO ISAAC What we added Robot Import Virtual Vision Interface to AI Server (the “brain” we are training) 16

WE ALSO NEED For a showcase A real robot Some training challenges 17

18

19

20

VR – ROBOT INTERFACE 21

VR – BAXTER INTEGRATION Block Diagram BAXTER ISAAC VR AI SERVER Domino Vision BX-CONT. Domino Play AI Robo Drive 22

A-I SERVER AI SERVER RL DNN Domino Recognition Algorithmic Endpoint detection Domino Vision RL DNN Domino Play / Explicit coded Domino Play (for Hockey: RL Continuous control) Domino Play AI This is for inference! Training backend can run on Cloud  DGX workstation. Robo Drive 23

DOMINO VISION For each incoming image Multiple Domino Detection Domino Pose Estimation Domino Type Classification 24

VISION TECH 25

DOMINO RECOGNITION 26

27

BAXTER CONTROL COMPUTER Strapped webcam BX-CONT. Iterative movement refiner (PID c.) Vision based human interaction 28

VISION BASED HI BodySLAM 29

ISAAC VR IK arm controller ISAAC VR (Golf/Hockey demos generate joint torques directly) 30

VR ENVIRONMENT A relatable robot 31

VR RESULTS VIDEO 32

REAL RESULT VIDEO 33

34

NEXT STEPS 35

THAT’S ALL FOLKS! Thanks you! Lots more to come: This is just proof of concept, and very early days. Looking for robotics collaborators Hiring like crazy… 36