Teaching a Car to Drive: An application of End-to-End Deep Learning - - PowerPoint PPT Presentation

Teaching a Car to Drive: An application of End-to-End Deep Learning

Larry Jackel NVIDIA, Holmdel NJ 07733

ljackel@nvidia.com

arXiv.org > cs > arXiv:1604.07316

Teaching a Car to Drive: An application of End-to-End Deep Learning

Larry Jackel NVIDIA, Holmdel NJ 07733

ljackel@nvidia.com

arXiv.org > cs > arXiv:1604.07316

Status

• Mean Autonomous Distance > 50 km on highways
• Good enough that passengers doze off
• Need to get Mean Autonomous Distance > 1,000,000 km

Mapping Localization Route Planning Actuator Controllers . . . Lane Following

Lane Following

DriveAV: NVIDIA Self-Driving Software

Lane Following

LaneNet Steering Controller Path Estimation Camera

Lane Following

Use diverse systems with different strengths and weakness to get required performance

Best case: error rate = error rate(LaneNet) * error rate (PilotNet)

PilotNet LaneNet Fusion Steering Controller Path Estimation Camera

How we got to PilotNet: A long journey in Neural Nets

Bell Labs, Holmdel NJ

6000 employees in Holmdel ~300 in Research ~30 in Machine Learning

1986-Early Results on Recurrent Nets

• Smallest Associative Memory
• Extremely high density
• Months to make
• Computer Experiments

revealed problems

• Inefficient use of resources for

pattern storage

• Not as good at matched

filters

• No Learning

Initial Results: Recurrent nets were disappointing

144 “synapses” in 6x6 micron cell

1986 – The Hype Begins The Neural Net Fad

• Lots of coverage in the

press

• Minimal science
• No practical results

“Bell Labs breakthrough:

Chips that work like the brain”

1988 : First Applications

• Graf builds 54 neural

programmable chip at Holmdel

• Good results on handwritten

character recognition with hand-crafted Hubel-Wiesel like features (ConvNet without learning)

• LeCun Joins Group
• Builds first “LeNet” OCR

engine with learned features

• Early benchmarks indicate

excellent performance

Analog hardware Feature extraction

LeNet’s Feature extraction kernels are learned.

Combines the architecture of Fukushima’s Neocognitron (1981) with Gradient Descent learning of weights

1989 -1990: OCR: excellent results

But no systematic understanding of what governed the learning process

1989 -1990: OCR: excellent results

But no systematic understanding of what governed the learning process Then came Vladimir Vapnik

How we came to view learning (Largely Vladimir Vapnik’s Influence)

• Choosing the right structure
• “Structural Risk Minimization”
• Bring prior knowledge into the learning

machine

• Capacity control
• Matching learning machine complexity to available data
• Examine learning curves

With Vladimir Vapnik’s help, by the early 1990s we had a much better understanding of the learning process and its applications

• AT&T OCR has become a

product

• Holmdel group ~ 30 people
• Applications include pen-

computing, finance, network fault prediction

• SVM soft margin and kernel

classifiers invented

(This time not hype!!)

17

The 1995 Bets

In 2000 there was a fancy dinner for four paid for by Jackel (We couldn’t prove why Yann’s multi-layer nets work so well) In 2005 there was a fancy dinner for four paid for by Vapnik (Yann’s nets are still in use and keep getting better) Yann and Leon didn’t bet, but they got to eat too.

18

Autonomous Driving

The usual approach for self-driving cars

Now often combined using Convolutional Nets

Object Recognition Cost Map Path Planning Actuation

Feature Extraction

Detailed Maps

Sensors Localization

Learning a complete control loop

• Learn to steer by observing a human
• Minimal use of hand-crafted rules
• Minimal need for hand labeling

Deep Convolutional Nets Camera Steering Actuation DARPA seedling project – Yann LeCun and Urs Muller “DAVE”

Early examples

• ALVINN
• Pomerlau – CMU ~1990 low-res

image, fully-connected nets

• DAVE
• Muller, LeCun – 2003
• Higher res images
• ConvNets

Show Dave Video

Example: Road Following

Good quality lane markers and good driving conditions Traditional lane detection-based systems work well Poor quality lane markers Lane detection-based systems struggle End-to-end learning empowers the network to use additional cues

Training the network

Camera

NVIDIA PilotNet

~250,000 distinct weights ~27,000,000 connections

Neural Network Driving the Car in New Jersey

Visualization

What the network pays attention to

ATYPICAL VEHICLE CLASS

Integrating PilotNet: The Rail

45 meters

Pilot was modified and trained to have 41 outputs specifying the center of the path to be followed in 1 meter increments To drive, the 10 most recent rails (the last 1/3 sec) are averaged to get a target position at a set distance in front

• f the vehicle

The rail Computed at 30Hz

Driving the rail from Holmdel to Woodbridge

Note performance when lane markings are missing

Summary

• Good progress in driving
• A lot more work needed to get to required safety
• Looks like diversity of parallel methods is the key

ljackel@nvidia.com