NVIDIA DRIVE December 2019 DRIVE PLATFORM Mass production - - PowerPoint PPT Presentation

▶
nvidia drive
SMART_READER_LITE
LIVE PREVIEW

NVIDIA DRIVE December 2019 DRIVE PLATFORM Mass production - - PowerPoint PPT Presentation

Dec 16, 2022 319 likes •1.07k views

NVIDIA DRIVE December 2019 DRIVE PLATFORM Mass production INDUSTRY / MARKET SAFETY TECHNOLOGY REGULATIONS NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. 2 THE DRIVE INITIATIVE End-to-End, Open Platform for Building Autonomous Vehicles DGX

slide-1
SLIDE 1

December 2019

NVIDIA DRIVE

slide-2
SLIDE 2

2 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DRIVE PLATFORM

Mass production

SAFETY TECHNOLOGY REGULATIONS INDUSTRY / MARKET

slide-3
SLIDE 3

3 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

THE DRIVE INITIATIVE

End-to-End, Open Platform for Building Autonomous Vehicles

DGX Saturn V Constellation Xavier DRIVE AV Re-SIM DRIVE IX

The NVIDIA DRIVE™ Platform provides all major AI technologies for Driving, Development and Validation - from driving & cockpit computers & software to data collection & DNN training, to simulation tools. The system is an open platform with multiple programming points. Car makers are free to use or develop as much as they like and take advantage of an ecosystem of hundreds of partners.

slide-4
SLIDE 4

4 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

SIMULATE TRAIN MODELS COLLECT DATA DRIVE

NVIDIA DRIVE

END-TO-END PLATFORM

Lanes Lights

Path

Signs

Pedestrians Cars

Lanes Lights

Path

Signs

Pedestrians Cars

slide-5
SLIDE 5

5

DRIVE COMPUTER / ARCHITECTURE

SCALABLE NOW & INTO THE FUTURE

NCAP TO ROBOTAXI

slide-6
SLIDE 6

6

DRIVE ROADMAP

DRIVE Parker

1x Perf 1x Efficiency Samples – Q4’2015 Production – Q1’2017

DRIVE PX2

20x Perf 1x Efficiency Samples – Q2’2016 Production – Q3’2016

ORIN

DRIVE Orin-X

>100x Perf ~20x Efficiency Samples – Mid 2021 Production - Q3’2022

DRIVE Xavier

23x Perf 8x Efficiency Samples – Q1’2018 Production – Q1’2019

DRIVE Pegasus

~230x Perf 5x Efficiency Samples – Q3’2018 Production – Q3’2019

Auto-Grade Super Energy-Efficient

slide-7
SLIDE 7

7

ORIN – HIGHEST PERF SOC EVER

What is driving the need for increasing performance in AV?

Sensor Complexity - More higher resolution cameras New DNN Models- CNN, RCNN, RNN DNN GOPs/Frame increasing 20+ DNN Models required for most demanding ODD DNN Precision / Accuracy OTA new features / SW Patch Integration of functions Centralized Computer Future Proof EUNCAP With 360° Perception

More perf equates to faster time to market

slide-8
SLIDE 8

8 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

NVIDIA DRIVE

One Architecture

NVIDIA DRIVE AGX platform is an open platform; allows partners to do their own innovation

PEGASUS (L4/RoboTaxis) XAVIER (NCAP , L2+/L3)

One AV Architecture for L2 -> Robo Taxi

ORIN ORIN

slide-9
SLIDE 9

9 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DRIVE ARCHITECTURE

AUTOMOTIVE GRADE – INDUSTRIALIZED ECU’S

slide-10
SLIDE 10

10

NVIDIA DRIVE SENSOR ECOSYSTEM

slide-11
SLIDE 11

11 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

SENSOR ECOSYSTEM

https://developer.nvidia.com/drive/drive-core-sensors

GNSS/IMU RADAR LIDAR IMAGING SENSOR / CAMERA

AR0820, AR0144, AR0231, IMX390, IMX490, IMX424 ARS430 SC13S, MTi-G-710, SPAN-IGM A1

slide-12
SLIDE 12

12

NVIDIA DRIVE DRIVE SOFTWARE

slide-13
SLIDE 13

13 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

NVIDIA DRIVE SOFTWARE ARCHITECTURE

Open and modular software for AV development

DRIVE AV

DRIVE Planning DRIVE Mapping DRIVE Perception

DRIVE IX

Visualization AI CoPilot AI Assistant

DRIVE Networks DRIVE Calibration

DRIVE OS

DRIVE AGX DEVELOPER KITS

(Xavier/Pegasus)

DRIVE HYPERION

(Reference Architecture)

DRIVE Core

DRIVEWORKS

slide-14
SLIDE 14

14

NVIDIA DRIVE DRIVE OS

slide-15
SLIDE 15

15 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DRIVE AGX

AV Software Platform

Linux BSP OS Drivers, USB, File System, Network Linux Hardware Software I/O

OS/3rd Party Software NVIDIA Software Hardware

Hypervisor Radar Lidar GPS IMU

Xavier

MCU

CAN & Flexray 10G/1G/BR Ethernet

DRIVE OS

CUDA TensorRT cuDNN NvMedia

NvStreams

Open GL

ES 3.2

L1 Safety Services Foundation Services Security Services Communication Services

L3 Safety Services Diagnostics Cameras

RS 232

GMSL (via CSI-2) NOR UFS eMMC PCIe NVME PMIC EGL 1.5 NVLINK

Power Management L2 Safety Services Data Distribution (DDS)1

Notes: 1) For development only.

DriveWorks OEM/T1 Application

TU104

Optional Services

Scalability; +Xavier, NvME, or Optional Processor

slide-16
SLIDE 16

16 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DRIVE AGX

AV Software Platform

Hardware Software I/O

OS/3rd Party Software NVIDIA Software Hardware

Hypervisor Radar Lidar GPS IMU

Xavier

MCU

CAN & Flexray 10G/1G/BR Ethernet

DRIVE OS

CUDA TensorRT cuDNN1 QNX BSP OS Drivers, USB1, File System, Network NvMedia

NvStreams

Open GL1

ES 3.2

L1 Safety Services Foundation Services Security Services1 Communication Services1

L3 Safety Services Diagnostics Cameras

RS 232

GMSL (via CSI-2) NOR UFS eMMC PCIe NVME PMIC

Scalability w/dGPU

EGL 1.51 NVLINK

Scalability; +Xavier, NvME, or Optional Processor

Power Management L2 Safety Services DRIVE AV Data Distribution (DDS)1

Optional Services

Notes: 1) Not for use in production in a safety context.

slide-17
SLIDE 17

17

NVIDIA DRIVE AUTONOMOUS DRIVING APPLICATION

slide-18
SLIDE 18

18 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

NVIDIA DRIVEWORKS SDK

Data Abstraction | Calibration | Compute Modules

DRIVE AV DRIVE IX DRIVEWORKS

DRIVE Core

DRIVE OS

DRIVE AGX DEVELOPER KITS DRIVE HYPERION

Sensor Abstraction Layer + Plugins Vehicle IO DNN Frameworks Tools (Recorder) Image Processing Point Cloud Processing DRIVE Networks DriveNet OpenRoadNet SignNet LightNet ClearSightNet PathNet WaitNet MapNet LaneNet DRIVE Calibration Egomotion Self- Calibration Calibration Tools

slide-19
SLIDE 19

19 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

NVIDIA DRIVEWORKS SDK

Foundation for AV SW Development on NVIDIA DRIVE

  • Open, modularized library of functions
  • Optimized for DRIVE AGX
  • Easy to integrate and build upon

SOFTWARE MODULES

  • Software dev tools ranging from sensor

data capture to calibration and visualization

TOOLS

  • Sample apps
  • Documentation, tutorials
  • Active developer forum and community

DEVELOPER SUPPORT

  • Architected for automotive safety
  • Compliant with industry standards, (ISO

26262/21448, MISRA)

PRODUCTION GRADE

slide-20
SLIDE 20

20 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DriveWorks Drive AV RoadRunner MapWorks DNNs

Self-Calibration Obstacle Perception Path Perception Wait Perception Egomotion Estimation Localization Sensor Abstraction Low Level Processing Actuation

Cloud Mapping

Map Perception

DriveNet OpenRoadNet Motion/Structure PilotNet LaneNet PathNet Frame Packaging and Serialization Map Creation and updates DriveNet WaitNet

HD Map

MapNet

Route Planner Control World Model Lane Planner Behavior Planner

Safety Force Field Search Based Planning AI Based Prediction LSA Image Safety Checks Learned Planning SubModule

DRIVE AV

Local World Model

slide-21
SLIDE 21

21 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

SCALABLE PLATFORM

One architecture – Highway and Urban AV Driving

ON RAMP TO OFF RAMP URBAN: ADDRESS TO ADDRESS

slide-22
SLIDE 22

22 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

ADVANCED FUNCTIONS

Auto High Beam Clear Sight NCAP Parking

slide-23
SLIDE 23

23 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DriveWorks Drive AV RoadRunner MapWorks DNNs

Self-Calibration Obstacle Perception Path Perception Wait Perception Egomotion Estimation Localization Sensor Abstraction Low Level Processing Actuation

Cloud Mapping

Map Perception

DriveNet OpenRoadNet Motion/Structure PilotNet LaneNet PathNet Frame Packaging and Serialization Map Creation and updates DriveNet WaitNet

HD Map

MapNet

Route Planner Control World Model Lane Planner Behavior Planner

Safety Force Field Search Based Planning AI Based Prediction LSA Image Safety Checks Learned Planning SubModule

DRIVE AV

Local World Model

slide-24
SLIDE 24

24 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

SENSOR SELF-CALIBRATION

slide-25
SLIDE 25

25 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

EGOMOTION

IMU/Odometry Ego-motion Camera Ego-motion (SLAM) LIDAR Ego-motion Radar Ego-motion

slide-26
SLIDE 26

26 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DriveWorks Drive AV RoadRunner MapWorks DNNs

Self-Calibration Obstacle Perception Path Perception Wait Perception Egomotion Estimation Localization Sensor Abstraction Low Level Processing Actuation

Cloud Mapping

Map Perception

DriveNet OpenRoadNet Motion/Structure PilotNet LaneNet PathNet Frame Packaging and Serialization Map Creation and updates DriveNet WaitNet

HD Map

MapNet

Route Planner Control World Model Lane Planner Behavior Planner

Safety Force Field Search Based Planning AI Based Prediction LSA Image Safety Checks Learned Planning SubModule

DRIVE AV

Local World Model

slide-27
SLIDE 27

27 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

CAMERA OBSTACLE PERCEPTION

slide-28
SLIDE 28

OBJECT DETECTION

slide-29
SLIDE 29

SURROUND OBJECT TRACKING

slide-30
SLIDE 30

NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DNN DISTANCE

slide-31
SLIDE 31

31 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

slide-32
SLIDE 32

FUTURE MOTION PREDICTION (RNN)

slide-33
SLIDE 33

33 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

VELOCITY ESTIMATION

slide-34
SLIDE 34

34 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

VELOCITY ESTIMATION

slide-35
SLIDE 35

35 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

LIDAR – OBSTACLE PERCEPTION

slide-36
SLIDE 36

36 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

LIDAR – OBSTACLE PERCEPTION

slide-37
SLIDE 37

37 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

LIDAR FREESPACE

slide-38
SLIDE 38

38 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

LIDAR MOTION DETECTION

slide-39
SLIDE 39

39 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

RADAR TRACKING

slide-40
SLIDE 40

40 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

PATH REPRESENTATION (THE LANE GRAPH)

slide-41
SLIDE 41

41 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

LANENET + TRACKING + FUSION

Tracking and Fusion Lane Assignment (OIPA)

slide-42
SLIDE 42

42 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

PATHNET

slide-43
SLIDE 43

43 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

FACTORED PILOTNET TURNS

slide-44
SLIDE 44

PATH PERCEPTION ENSEMBLE

slide-45
SLIDE 45

45 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

WAIT CONDITIONS

slide-46
SLIDE 46

46

INTERSECTION DETECTION

slide-47
SLIDE 47

47

SIGN AND TRAFFIC LIGHT CLASSIFICATION

slide-48
SLIDE 48

CAMERA BLINDNESS DETECTION

slide-49
SLIDE 49

PARKING SPACE DETECTION

slide-50
SLIDE 50

LIGHT SOURCE PERCEPTION

slide-51
SLIDE 51

51 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DRIVE NETWORKS

Fully trained DNN solutions for advanced situational awareness

Obstacle Perception

DriveNet Detects objects including vehicles, pedestrians, bicycles, traffic lights & signs DepthNet Dense depth inference at pixel level OpenRoadNet Detects drivable freespace around the vehicle PredictionNet RNN to predict dynamic objects LidarNet Lidar point cloud classification

Path Perception

LaneNet Detects and classifies lanes MapNet Detects visual landmarks such as lanes & crosswalks PilotNet Learn trajectories from human driver PathNet Predicts full geometry of drivable paths

Wait Condition Perception

SignNet Classifies traffic signs detected by DriveNet, for US and EU LightNet Classifies traffic lights (color, solid, and arrows) detected by DriveNet WaitNet Detects intersections

Assistance Functions

ClearSightNet Determines if the camera view is blocked AhbNet Auto high beam detection

slide-52
SLIDE 52

52 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

NVIDIA DRIVE MAPPING

Drive, Update and Create HD Maps

Map Localization

Using DRIVE Localization API

Map Update

Using DRIVE MapStream API

Map Creation

Using DRIVE MapStream/MapServices API

slide-53
SLIDE 53

53 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

LOCALIZATION STEPS

HERE example

Detect MapNet Project HERE HD Live Map Match DLA and CUDA Drive DRIVE Software

slide-54
SLIDE 54

54 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

CAMERA LOCALIZATION

slide-55
SLIDE 55
slide-56
SLIDE 56

56 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

LIDAR/RADAR MAPPING+LOCALIZATION

slide-57
SLIDE 57

57 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

MAP UPDATE STEPS

TomTom example on I-85 in CA

Detect MapNet Create MapStreams Map update primitives Convert MapStreams to Roadgrams Update Map HD map in the Cloud

TODO from TomTom

slide-58
SLIDE 58

58 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

MAPSTREAM CREATION IN THE CAR

Traffic sign Trajectory Road boundary Dashed Solid Pole

slide-59
SLIDE 59

59 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

VISUALIZING A CREATED MAPSTREAM

slide-60
SLIDE 60
slide-61
SLIDE 61

61 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

Route Planner Lane Planner Advanced Behavior Planner Basic Behavior Planner

Behavior Planner Route Planner

Latest Safe Arrival (LSA) Image

Lane Planner

Safety Force Field Search Based Planning Prediction Learned Planning Safety Checks

PLANNING

slide-62
SLIDE 62

62 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

SAFETY FORCE FIELD

High-Level Architecture

slide-63
SLIDE 63

63 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

SAFETY FORCE FIELD

Our vehicle is always perceiving

  • ther actors and obstacles.

The Safety Force Field is a computational module that monitors for overlap between

  • ur safety procedure and that
  • f other actors.

The Safety Force Field provides a repelling constraint that prevents us from contributing to a collision.

slide-64
SLIDE 64

64 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

SAFETY FORCE FIELD

Longitudinal Lateral

slide-65
SLIDE 65

65

NVIDIA DRIVE DRIVE EXPERIENCE

slide-66
SLIDE 66

66 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE. NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DRIVE IX – CONNECT TO SELF DRIVING

Using AR and VR

Confidence View Perception view

slide-67
SLIDE 67

67 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

EXAMPLE USE CASES

Convenience Cockpit MMI Guardian Angel

Side traffic detect Drowsiness detect Distraction detect Driver/ Passenger Int face ID/ Greeting/ Settings Front cross traffic detect User Ext Face ID / Trunk opening / Car start Safe back-up PA: Personal Assistant / NLU Hand Gestures detection Auto adjust mirrors/seats Respond to sign commands Child /Pet detect Emotion detect Health/Wellness detect PA: AV Driving modes

slide-68
SLIDE 68

68 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

AI Toolkit for Intelligent Experience

Plugin Manager

TTS Emotions Face ID

API Manager

IX Server

Inference Hub

IX Core Capabilities

Remote Experience Surround View Surround Monitoring Driver Monitoring

Head Pose, Gaze, Distraction, Drowsiness Risk Assessment Stitching, Pixel Level Segmentation

AV Perceptions

IX Sensor Hub

AI CoPilot AI Assistant

IX Speech Engine Partner plugins

Cabin Sensors Partner Technology

OTA

Surround Camera

IX Sensor Hub

NLU ASR

Remote Rendering Engine

(Rich Graphics, Video, Audio)

DRIVE IX

NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

AR Viz

slide-69
SLIDE 69

69

NVIDIA DRIVE DRIVE SIM

slide-70
SLIDE 70

70

DRIVE CONSTELLATION

Hardware in the-Loop System Level Simulator Simulate Rare and Difficult Conditions Scalable Platform | Data Center Solution Timing Accurate and Bit Accurate Scenario Based Coverage

Virtual Reality AV Simulator

slide-71
SLIDE 71

71

DRIVE CONSTELLATION ARCHITECTURE

DRIVE Constellation Simulator

DRIVE Sim API

DRIVE Constellation Vehicle

Environment Model

DRIVE Constellation OS DRIVE Sim

Traffic Model Vehicle Model Sensor Model Scenario Model

DRIVE OS DRIVE CORE | DRIVE NETWORKS DRIVE AV DRIVE IX

Perception Mapping Planning

slide-72
SLIDE 72

72

CONSTELLATION DATA CENTER WORKFLOW – CONT.

slide-73
SLIDE 73

73 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

DRIVE PLATFORM

Mass production

SAFETY TECHNOLOGY REGULATIONS INDUSTRY / MARKET

slide-74
SLIDE 74