Corner Cases Powered by the Jetson TX Series Sascha Hornauer, - - PowerPoint PPT Presentation

Studying Autonomous Driving Corner Cases

Powered by the Jetson TX Series Sascha Hornauer, Baladitya Yellapragada

Autonomous Driving

Marcus Erbar, https://www.youtube.com/watch?v=IHhLpB5MNTQ Vehicle Detection: Deep Learning w/ lane detection https://github.com/merbar/CarND-Vehicle-Detection

Autonomous Driving

Failures of predominant approaches

“You need to paint the bloody roads here!” - Volvos North America CEO Lex Kerssemakers to Los Angeles

Mayor Eric Garcetti while failing to drive autonomously at the Los Angeles Auto Show

Autonomous Off-Road Driving

Our Results

Platform description

Model car with Jetson TX1/TX2

• 1. Stage: Collection of Over 100 Hours of Driving

• 2. Stage Imitation Learning - Behavioral Cloning of Driving

Outdoor Domain Results

Domain Change - Indoor Results

Follow Behavior and Failure Cases

Benefits

• Complementary Solution

✔ ✔ ✔ ✔ ✔ ✔

Challenges and Further Directions

Learned behavior not completely understood Metrics reflect coarse success, better needed 𝑏𝑣𝑢𝑝𝑜𝑝𝑛𝑧 = 1 − 𝑜𝑣𝑛𝑐𝑓𝑠 𝑝𝑔 𝑗𝑜𝑢𝑓𝑠𝑤𝑓𝑜𝑢𝑗𝑝𝑜 ∙ 6 𝑡𝑓𝑑𝑝𝑜𝑒𝑡 𝑓𝑚𝑏𝑞𝑡𝑓𝑒 𝑢𝑗𝑛𝑓 𝑡𝑓𝑑𝑝𝑜𝑒𝑡 ∙ 100

• How simple can our network be? How complex must it be?
• Can we understand our network performance, before deployment/testing?
• What has our network learned?

(1)

Bojarski, M., Del Testa, D., Dworakowski, D., et al.\ 2016, arXiv:1604.07316 Bojarski, M., Del Testa, D., Dworakowski, D., et al.\ 2016, arXiv:1604.07316 Bojarski, M., Del Testa, D., Dworakowski, D., et al.\ 2016, arXiv:1604.07316

Visualizing Ego-motion Semantics Implicitly Learned by a Neural Network for Self-Driving

Baladitya Yellapragada

Outline

• Need for Visualization-Based Experiments
• Semantic Label Creation for Ego-motion Features
• Ego-motion Feature Experiments

Outline

• Need for Visualization-Based Experiments
• Semantic Label Creation for Ego-motion Features
• Ego-motion Feature Experiments

Spatiotemporal Input Novelty

General Network Our Network

Insights from Visualization – Activation Maps

Zhou, et al. “Object Detectors Emerge in Deep Scene CNNs”. 2015. ICLR

Potential Optical Flow Relevance

• Saunders. “View rotation is used to perceive path curvature from optic flow”. 2010. Journal of Vision.
• Optical flow asymptotes are cues for path centers (dashed)

Insights from Visualization – Gradient Ascent

Zeiler and Fergus. “Visualizing and Understanding Convolutional Networks”. 2013. ArXiv

Gradient Ascent with our Z2 Color

Outline

• Need for Visualization-Based Experiments
• Semantic Label Creation for Ego-motion Features
• Ego-motion Feature Experiments

Visual Representation of Network Semantics

Auto Human Bau, et al. “Network Dissection: Quantifying Interpretability of Deep Visual Representations”. 2017. CVPR.

Semantic Representations in Networks

Bau, et al. “Network Dissection: Quantifying Interpretability of Deep Visual Representations”. 2017. CVPR.

Creating Optical Flow Labels per Input Video

• Consistent True Driving Signals with Tight Standard Deviation

vs

Filtering Out Well Predicted Videos

• Reject videos whose driving signals are not predicted well by the network

○ Remaining are task-relevant videos, each with optical flow labels across shared space

Outline

• Need for Visualization-Based Experiments
• Semantic Label Creation for Ego-motion Features
• Ego-motion Feature Experiments

Implicit Optical Flow Sensitivity Experiment

Meyer, et al. “Phase-Based Frame Interpolation for Video”. 2015. Computer Vision and Pattern Recognition.

Speed Condition Examples

Third Speed Normal Speed Thrice Speed

Results

Controls

• Temporal Order
• Stereoscopic Disparity
• Domain Correlations

Compare Time in Other Self-Driving Networks

• Single Image Networks [1]
• Recurrent Units [2]
• Spatiotemporal Convolutions [3]

[1] Mengxi Wu. “Self-driving car in a simulator with a tiny neural network”. 2017. Medium. [2] Xu, et al. “End-to-end Learning of Driving Models from Large-scale Video Datasets”. 2017. ArXiv [3] Chi and Mu. “Deep Steering: Learning End-to-End Driving Model from Spatial and Temporal Visual Cues”. 2017. ArXiv

Control Condition: Temporal Order

Normal Order Random Order Reverse Order

Results with Temporal Controls

Normal Order Reverse Order Random Order

Control Condition: Stereoscopic Disparity

Normal Stereo Reverse Stereo No Stereo

Results with Stereo Controls

Normal Stereo Reverse Stereo No Stereo

Control Conditions: Domain Correlations

vs Forest Sidewalk

Confounding Domain Correlations

Forest w/ Sidewalks

Results by Domain (Trained on Tested)

Sidewalk on Sidewalk Combined on Combined Forest on Forest

Results by Domain (Trained on Tested)

Sidewalk on Forest Combined on Forest Forest on Forest

Conclusions

• Temporal variability is a consistent predictor of driving signal (bigger label = faster,

smaller label = slower).

• Even if not explicitly specified, more natural temporal order is preferred over not

○

Evidence for potential optical flow sensitivity

• Even if not explicitly specified, more natural camera differences are preferred over not

○

Evidence for potential stereo disparity sensitivity

• Optical flow may be implicitly learned, but optical flow salience may be not be

generalizable across domains, so more explicit training is better.