Ma Machine chine Lear arning ning for r Auton tonomous mous Dr - - PowerPoint PPT Presentation

Ma Machine chine Lear arning ning for r Auton tonomous mous Dr Driving ving

Nasser r Mohamm mmadi adiha ha

Senior An Analysi sis s Engine neer er at Volvo Cars/ s/Ze Zenu nuity ty & & Adjun unct ct Docent nt at Chalmer almers

Sto Stock ckholm holm (K (KTH TH), ), 2017-04 04-03 03

Zenuity!

• Develop software for autonomous

driving and driver assistance systems

• Autoliv and Volvo Cars will own Zenuity

50/50

• Starting with 200 employees from

Autoliv and Volvo Cars

• Volvo Cars and Autoliv will license and

transfer the intellectual property for their ADAS systems to Zenuity

• Headquartered in Gothenburg with

additional operations in Munich, Germany, and Detroit, USA

CEO: Dennis Nobelius

AD and AI among hottest trends

Gartner's 2016 Hype Cycle for Emerging Technologies

Förarlöst i praktiken Ai i allt

Top 10 trends from Nyteknik

Outline

• Autonomous Driving (AD)
• ML in the era of AD
• Interest in ML in the IV/ITS community
• Applications
• Some examples

Drive Me: Self driving cars for sustainable mobility

Global Trends

• Urbanisation
• Growing mega cities
• Air quality major health problem
• Traffic accident global health issue
• Time for commuting
• Desire for time efficiency
• Desire for constant connectivity

Today’s reality

SHAPING THE FUTURE MOBILITY

AD will be important for a sustainable mobility

• Improved traffic safety
• Improved environmental outcomes
• Regain time

Volvo vision 2020

No one should be killed or seriously injured in a new Volvo car by 2020

It was always about freedom

And it’s still about freedom

joint effort and proper reseach

Global challenges – Demand a joint effort Drive Me – Nordic model of collaboration Research platform – How autonomous cars can contribute to a sustainable development Pilots with real customers in real traffic

The pilots – all about learning

• Traffic environments
• Customer preferences
• Exporting the Nordic model
• f collaboration

Gothenburg – proof of concept China and London – verify our technology

Gothenburg

2013

London

2017

China

2017

well-defined commute highways

• Customer focus
• Simplification
• Risk Management

A B

F r u s t r a t i n g c o m m u t e N e i g h b o r h o o d C l o s e t o w o r k

Your neighborhood, children, no lane markings, roundabouts, ... Traffic lights, pedestrians, bicyclists, ... Well defined use case on city highways

Pilot Assist versus Autopilot

• Driver is responsible, should monitor and supervise
• Driver responsible to intervene whenever needed
• Limitations: Lane markings, road design, oncoming objects,

pedestrians, animals, restrictions in steering/braking/acceleration force that can be applied

Autopilot / UnSupervised Pilot assist / Supervised

• Manufacturer responsible
• Tested on and expects extreme situations
• Takes precautions, takes decisions
• Driver free to do something else

Unsupervised AD (Level 4) Supervised AD (Level 1&2) Manual (Level 0)

AD Car

Safety Benefits

Injury Reduction Crash Avoidance

SAFETY Impact

Low-risk driving Risky driving behaivours & situations Conflict/ Net-Crash Crash After crash

Precautionary Safety

We assume liability

“Volvo will assume liability for its autonomous technology, when used properly.”

TECHNOLOGY

• Camera
• Radar
• Laser
• Ultrasonic
• Map data
• Cloud connection
• Traffic Control Centre

“Machine learning for sensor signal processing is in the core!”

redundancy

Action Decision Perception

Sensor Fusion 1 Sensor Fusion 2 Decision & Control 1 Decision & Control 2 Vehicle Dynamics Management 1 Vehicle Dynamics Management 1 Brake Control 2 Brake Control 1 Steering Control 1 Steering Control 2 Vision Radar Lidar Ultrasonic Brake Brake Brake Brake Power steering Power steering Cloud

Interest in ML, related to AD and ITS

10 20 30 40 50 60 70 80 90 100

Records in Google Scholar ITSC 2016 (out of 430 papers)

Improving the Results over THE Years

• Object detection rate in

KITTI

• Moderate: Min. bounding

box height: 25 Px, Max.

• cclusion level: Partly
• ccluded, Max.

truncation: 30 %

Grouping the ML Methods

Where is it trained?

• Locally
• In the cloud

When is it trained?

• Offline
• Online

How is it trained?

• Supervised
• Un-supervised (semi-supervised)

Where is it executed?

• On-board
• Off-board (e.g., in the cloud)

When is it executed?

• Real-time
• Offline (or batch processing)

Continuous learning?

• Yes
• No

Application

• System design
• Verification

Training Deployment

Modular systems vs Holistic Design

• Modular systems and ML to design individual components
• Holistic and end to end learning for driving

Sensor data Vehicle control

Modular systems

(1) Well-defined task-specific modules (2) Could have parallel modules Examples:

• Semantic segmentation and free space and drivable area detection
• Object detection and tracking and information (speed, heading, type, ...)
• Road information and geometry of the routes
• Sensor fusion
• Scene Semantics such as traffic and signs, turn indicators, on-road markings etc
• Maps and updating them over time
• Positioning and localization
• Path planning
• Driving policy learning and decision making
• Other road user behavior analysis such as intention prediction
• Driver monitoring

Holistic and end to end learning

• First implemented around 28 years ago in the

ALVINN system

• Number of parameters<50,000
• Dave-2
• 9 layers (5 convolutional and 3 fully

connected)

• 250 000 parameters
• Dean A. Pomerleau, ”ALVINN, an autonomous land vehicle in a neural network”., No. AIP-77, CMU, 1989.
• Mariusz Bojarski, et al. "End to end learning for self-driving cars“, arXiv, Apr. 2016.
• Christopher Innocenti, Henrik Lindén, “Deep Learning for Autonomous Driving: A Holistic Approach for Decision Making”, thesis.

AD Verification

• Requirements: setting safety scope

for function and AD test scenarios

• Test methods
• Test track
• Test in real traffic and expeditions
• Virtual testing and simulations
• Analyzing logged data
• Need to have tools such as reference system
• Need to have advanced analysis methods
• Nidhi Kalra et al. "Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?." Transportation

Research Part A: Policy and Practice 94 (2016): 182-193.

[Kalra, 2016]

Some Examples

Object Detection

(1) Accuracy (2) Speed

• Ross Girshick et al., “Rich feature hierarchies for accurate object detection and semantic segmentation”, CVPR, 2014.
• Shaoqing Ren et al., “Faster R-CNN: Towards real-time object detection with region proposal networks”, NIPS, 2015.
• Jifeng Dai et al., “R-FCN: Object Detection via Region-based Fully Convolutional Networks”, arXiv, Jun., 2016.
• Donal Scanlan, Lucia Diego, “Robust vehicle detection using convolutional networks“, Master’s thesis, ongoing.

Semantic Segmentation

• Jonathan Long et al., “Fully Convolutional Networks for Semantic Segmentation”, CVPR 2015.
• Jifeng Dai et al., “Instance-aware Semantic Segmentation via Multi-task Network Cascades”, arXiv, 2015
• Vijay Badrinarayanan et al., “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Scene Segmentation”, PAMI 2017

FCN

Road Information

• Christian Lipski et al. "A fast and robust approach to lane marking detection and lane tracking“, SSIAI, 2008.
• Florian Janda et al., "A road edge detection approach for marked and unmarked lanes based on video and radar“, FUSION, 2013.
• Jihun Kim et al., “Robust lane detection based on convolutional neural network and random sample consensus”, ICONIP, 2014.
• Bei He et al., “Accurate and Robust Lane Detection based on Dual-View Convolutional Neutral Network”, IV, 2016.
• Gabriel L. Oliveira et al., "Efficient deep models for monocular road segmentation“, IROS, 2016.

[He 2016]

Traffic Sign Recognition

German Traffic Sign Recognition Benchmark (GTSRB) [Stallkamp 2012]

• Johannes Stallkamp et al. "Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition“, Neural networks, 2012.
• Pierre Sermanet et al., "Traffic sign recognition with multi-scale convolutional networks“, IJCNN, 2011.
• Konstantinos Mitritsakis, “Study Real World Traffic Environment Using Street View”, Master’s thesis, 2016.

Interacting with Humans

• Eshed Ohn-Bar et al., “Looking at Humans in the Age of Self-Driving and Highly Automated Vehicles”, Trans. on IV, 2015

Driver monitoring

• Kevan Yuen et al. ,”Looking at Faces in a Vehicle: A Deep CNN Based Approach and Evaluation”, ITSC, 2016.
• Akshay R. Siddharth et al., “Driver Hand Localization and Grasp Analysis: A Vision-based Real-time Approach”, ITSC, 2016.
• Tianchi Liu et al. "Driver distraction detection using semi-supervised machine learning." IEEE Transactions on ITS, 2016.

Road Friction Estimation from Connected Vehicles Data

• Prediction using both

historical friction data from the connected cars and data from weather stations

• Busy roads
• Missing data
• Ghazaleh Panahandeh, Erik Ek, Nasser Mohammadiha, “Supervised Road Friction Prediction from Fleet of Car Data”, IV 2017.

Driver Route and Destination Prediction

• History of driving for

individuals

• Use of metadata such as

driver id, the number of passengers, time-of-day, day-

• f-week
• Destination clustering
• Ghazaleh Panahandeh, “Driver Route and Destination Prediction”, IV 2017

Compressed networks

• Pruning or quantizing weights
• Devising new and smaller

architectures

• Forrest N. Iandola et al. "SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 0.5 MB model size“, arXiv, Nov. 2016
• Michael Treml et al., “Speeding up Semantic Segmentation for Autonomous Driving”, NIPS Workshop, 2016
• Adam Paszke et al. "ENet: A deep neural network architecture for real-time semantic segmentation “, arXiv, Jun. 2016
• Song Han et al., “Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding” arXiv , 2015.
• Alireza Aghasi et al., “Net-Trim: A Layer-wise Convex Pruning of Deep Neural Networks”, arXiv, Nov. 2016.

And many more ...

• Deep network understanding and interpretation
• Secure and privacy-preserving deep learning
• Transfer learning
• Pedestrian intention estimation
• Applications of generative adversarial networks
• Learning to attend
• Instance segmentation
• Object tracking
• Harsh weather conditions
• Traffic understanding such as brake light detection
• ...
• Benjamin Völz et al. “A data-driven approach for pedestrian intention estimation”, ITSC, 2016
• Arna Ghosh et al., “SAD-GAN: Synthetic Autonomous Driving using Generative Adversarial Networks”, arXiv, Nov. 2016
• Volodymyr Mnih et al. "Recurrent models of visual attention." NIPS, 2014.

Classification of 3D Point Clouds

• Patrik Nygren, Michael Jasinski, “A Comparative Study of Segmentation and Classification Methods for 3D Point Clouds”, Master’s thesis, 2016.
• Axel Bender, Elías Marel Þorsteinsson, “Object Classification using 3D Convolutional Neural Networks“, Master’s thesis, 2016.
• N. Mohammadiha, P Nygren, M. Jasinski, “A Comparison of Classification Methods for 3D Point Clouds”, Fast-zero 2017.

Ground truth for positioning

• High-accuracy

positioning in GPS- denied environments

• Scalability to new

locations

• David Bennehag, Yanuar Nugraha, “Global Positioning inside Tunnels Using Camera Pose Estimation and Point Clouds”,

Master’s thesis, 2016.

Sensor comparison framework

Data from Sensor 2 Post- processing Matching Analysis methods Results Data from Sensor 1

• J. Florbäck, L. Tornberg, N. Mohammadiha, “Offline Object Matching and Evaluation Process for Verification of Autonomous

Driving”, ITSC, 2016.

Wrap-up

• High expectations from ML to overcome some of the biggest

challenges in autonomous driving

• Successful applications of ML especially for perception already

being used in the industry

• New challenges arise in designing complete ML systems

(integration, updating, safety, interpretation...)

• Wide range of applications for ML from raw sensor data

processing to developing offline methods for verification purposes

Opportunities

• Industrial PhD position on “Reinforcement Learning for

Autonomous Driving” at Zenuity/Chalmers

• Postdoc position on “Big Sensor Data Analysis for

Verification of Autonomous Driving” at Zenuity/Chalmers

• Research Engineer position on “Big Sensor Data Analysis for

Verification of Autonomous Driving” at Zenuity/Chalmers Contact me: nasser.mohammadiha@volvocars.com More positions on:

• http://career.zenuity.com/
• http://www.volvocars.com/intl/about/our-company/careers