ADAS COMPUTER VISION AND AUGMENTED REALITY SOLUTION Sergii Bykov, - - PowerPoint PPT Presentation

www.luxoft.com

ENGINEERING SOFTWARE

TECHNOLOGY AUTOMOTIVE TRAVEL AND AVIATION ENERGY TELECOM FINANCIAL SERVICES

ADAS COMPUTER VISION AND AUGMENTED REALITY SOLUTION

Sergii Bykov, Technical Lead

www.luxoft.com

Product Vision

www.luxoft.com

Road To Autonomous Driving

Source: Intel

www.luxoft.com

Representation For The Driver

Output is an extendable metadata which describes all the augmented objects/hints and supports natural features ON/OFF

www.luxoft.com

Computer Vision and Augmented Reality Applications

COMPUTER VISION

Pattern Recognition Image Processing Physics Signal Processing Arfificial Intelligence Maths City Driving Pattern Augmented Navigation Help in low visibility mode Signs recognition Infographics

Next Gen of Adaptive Cruise Active Park Search

www.luxoft.com

Key Challenges Of Bringing AR In The Vehicle

• Usability – augmented reality subsystem should not disturb driver as it is continuously observed
• Hardware limitations – computational, power consumption, zero latency (HUD)
• Requirements for precise environmental model estimation for occlusion avoiding
• Dependency on inaccurate map and navigation data
• Distributed HW architectures, platform flexibility requirements
• High precision absolute and relative positioning requirements
• Components synchronization and latency avoidance
• Embedded memory usage limitations, different memory models
• Algorithms should be both configurable and efficient
• Specific rendering requirements, not covered by general purpose frameworks
• Variety of inputs under different platforms
• Out-of-vehicle simulation (does not support natural simulation like classical navigation)

www.luxoft.com

Framework Concept

www.luxoft.com

Augmented Navigation Structure

We offer a Unique Solution capable to create Augmented, mixed visual Reality for drivers and passengers based

• n Computer Vision, vehicle sensors, map data, telematics, navigation guidance using Data Fusion technique.

CVNAR Solution Automotive Cameras Sensors/CAN Navigation System/Map Data Vehicle displays Projection on wind shield Smart Glasses VR devices Telematics/V2X

www.luxoft.com

CVNAR Features

Natural Augmented Reality

• Basic vehicle data
• Lanes and road boundaries
• Road signs and cautions
• Navigation data and hints
• Facades highlights
• Parking places
• Narrow street infographics
• Street names and complex junctions boards
• POI and OEM specific information

Highlights

• CPU, GPU;
• OS: Linux, QNX
• HW: Intel, NVidia, TI, Renesas
• Extrapolation engines for latency avoidance
• Machine learning and deep learning

Road scene recognition and objects tracking

• Road boundaries
• Lane detection
• Vehicle detection and tracking
• Distance and time to collision estimation
• Pedestrian detection and tracking
• Facade recognition and texture extraction
• Road signs recognition
• Parking slot recognition

Positioning

• Precise relative and absolute positioning
• Flexible data fusion and smooth map matching
• Automotive constrained SLAM

Integration and Fusion

• Sensor data
• External positioner data (optional)
• External recognition engine integration
• Pupil tracking
• Telematics: V2I and V2V

www.luxoft.com

Hardware Approach: Computer Vision Box

Live data from vehicle:

• CAN data, Sensors
• Video stream

CVB

CVNAR Engine CVB Data Layer Web Interface SW Update Configuration Diagnostic

Video Stream with augmented objects

ADAS CVNAR Engine

HUD/LCD Head Unit

• Quick-install demonstration solution
• Platform for CVNAR (allows to be portable)
• Integration with Head Units
• Integration with vehicle networks
• Using of own sensors if needed

Navigation data, preprocessed sensor data, etc. Control/Settings

www.luxoft.com

Hardware Approach: Automotive Stack

Own scalable and robust Automotive Stack aimed to minimize time of project start and integration

• RTOS (OSEK, Micrium, mTRON), Microcontrollers (Renesas RH850/V850)
• Hardware Abstraction Layer (HAL), Operation System Abstraction Layer (OSAL)
• Trace Server/Client, WatchDog, IPC, Drivers (SPI, I2C, UART, Timers, etc.), SW Update
• Pre-Integrated Vector CAN/Diagnostic Stack
• Vehicle Bootloader for Renesas Microcontrollers

Domains and areas:

• System development and integration with Automotive Networks and ECUs
• Drivers development and peripheral support: Video Cameras, Automotive Sensors, external HW
• HW brings-up (BSP development)

Automotive grade technologies supported by team:

• Networks: CAN, LIN, MOST, BroadReach, Ethernet
• RTOS: OSEK, mTron, Micrium, embOS, mTRON, QNX, Linux, VxWorks
• Microcontrollers: Renesas (RH850, V850), Freescale (Bolero, i.MX6), TI (OMAP, Jacinto, MSP430)
• CAN Stacks: Vector, KPIT, own tinyCAN
• Audio/Video processing
• Media Bus: LVDS, FPDLink-III, APIX2, USB, Ethernet, BroadReach

www.luxoft.com

Perception Concept

www.luxoft.com

Sensor Fusion: Data Inference

Optimal fusion filter parameters adjustment problem statement and solution developed to fit different car models with different chassis geometries and steering wheel models/parameters. Features:

• Absolute and relative positioning
• Dead reckoning
• Fusion with available automotive grade sensors – GPS, steering wheel, steering wheel rate, wheels sensors
• Fusion with navigation data
• Rear movements support
• Complex steering wheel models identification. Ability to integrate with provided models
• GPS errors correction
• Stability and robustness against complex conditions – tunnels, urban canyons

www.luxoft.com

Sensor Fusion: Advanced Augmented Objects Positioning

Solving map accuracy problems

Placing:

• Road model
• Vehicles

detection

• Map data

Position clarification:

• Camera motion

model:

• Video-based

gyroscope

• Positioner

Component

• Road model
• Objects tracking

www.luxoft.com

Sensors Fusion: Comparing Solutions

Update frequency ~15 Hz (+extrapolation with any fps) Update frequency ~4-5 Hz

Reference solution Our solution

www.luxoft.com

Lane Detection: Adaptability and Confidence

www.luxoft.com

Lane Detection: 3D-scene Recognition Pipeline

• Low level invariant features
• Single camera
• Stereo data
• Point clouds
• Structural analysis
• Probabilistic models
• Real-world features
• Physical objects
• 3D scene reconstruction
• Road situation
• 3D space scene fusion (different sensors input)
• Backward knowledge propagation from high levels

www.luxoft.com

Lane Detection: Additional Information

• Features data base
• Low level screen (3D) features to refine position
• Points clouds
• Marking details, Road borders
• High level structural elements and real world objects
• Junctions, Facades, Signs, etc.
• Features Collection
• Existing map providers
• Real-time feature extraction and understanding from video sensors
• Satellite-view photos analysis
• Map database updates
• Routes offline processing and upload

www.luxoft.com

Lane Detection: HD Map Potential

Content

• Simplified and advanced geometry for roads, traffic lanes,

lanes boundaries etc. Application

• Precise on-road vehicle positioning
• Different weather, traffic situations
• Map matching and Path planning
• Maneuver suggestions
• Cable navigation
• Junction assistance
• Possible junction maneuvers
• Traffic lights position

Useful information:

• Road border geometry and type
• Traffic signs (position and type);
• Traffic lights (position and type);
• Type and quality of roadbed;
• Roadside POIs (gas station, store, café etc.);
• Any other additional features which can be useful for

vehicle positioning or driver.

www.luxoft.com

Lane Detection: Robustness in Normal Conditions

Graphs show error in meters between recognized lanes (curved model) and recognized road marking (distance to detected features + features accuracy) in different distance range and for different road conditions. Figure 1. Regular weather, highway, slightly curved road with lane changes Figure 2. Rainy weather, highway, straight road with lane changes

www.luxoft.com

Lane Detection: Robustness in Difficult Conditions

Figure 3. Bright sun, highway with secondary roads, straight road with turns and lane changes Figure 4. Hard rain, highway, straight road with lane changes

www.luxoft.com

Vehicle Detection

Figure – Vehicle detection examples

• Convolutional neural network for vehicle detection
• GPU Acceleration – CUDA
• Running real-time on NVidia Jetson TK1
• Inference speedup on embedded (TK1) GPU vs CPU is ~3x
• Training speedup on desktop GPU vs CPU is ~20x
• Classifier accuracy (about 50k, 960x540, ~55-60 deg HFOV):
• Positive: 99.65%
• Negative: 99.82%
• Size of detection down to 30 pix, detection range of about 60 m

www.luxoft.com

Road Scene Semantic Segmentation

Figure – Road scene segmentation examples

• Deep fully convolutional neural network for semantic

pixel-wise segmentation

• Road scene understanding use cases: model

appearance, shape, spatial-relationship between classes

• Inference speedup GPU vs CPU is ~3x

www.luxoft.com

Driver Monitoring: Haar/LBP Cascades

• Object Detection using Haar or LBP feature-based cascade

classifiers is an effective method for face detection

• It is a machine learning based approach where a cascade function

is trained from a lot of positive and negative images. It is then used to detect objects in other images.

• Applications can achieve consistent face detection in different

environments (e.g. vehicle driver, etc.)

• Can be efficiently implemented for low power embedded devices
• Speedups on mobile GPU vs CPU are
• 20x for Haar
• 3x for LBP

Figure – Haar features Figure – LBP features

www.luxoft.com

HMI Concept

www.luxoft.com

Rendering Component Structure

CVNAR Rendering

Renderer System Window System Resource Management Frame Buffer Frame Renderer HMI Renderer Augmentation Renderer

Client of rendering framework (controller)

HMI/scene update commands rendered frames Head Up Display LCD

www.luxoft.com

Renderer Output Layers

Frame renderer Augmentation renderer HMI renderer Final image

www.luxoft.com

Augmented Objects Primitives. Part 1

Barrier Lane Line Traffic Sign & POI Lane Arrow

www.luxoft.com

Augmented Objects Primitives. Part 2

Fishbone Facade Vehicle Street Name

www.luxoft.com

Demo Application Screenshot (LCD)

www.luxoft.com

Head Up Display Concept. HUD vs LCD

• Hardware limitation
• HUD devices are rarely

available on market

• FOV and object size
• Timings
• Zero latency
• Driver eye position
• Driver perception
• Virtual image distance
• Information balance

www.luxoft.com

Summary: Key Technology Advantages

• Proved understanding of pragmatic intersection and synergy between fundamental theoretical results and final

requirements

• Formal mathematical approaches are complemented by deep learning
• Solid GPU optimization expertise
• Automotive grade solutions integrated with all the data sources in vehicle – data fusion approaches
• High robustness in various weather and road conditions, confidence is estimated for efficient fusion
• Closed loops designed and implemented to enhance speed and robustness of each component
• Integration with V2X and Navigation system
• System architecture supports distributed HW setup and integration with existing in-vehicle components if required

(environmental model, objects detection, navigation, positioner etc.)

• Hierarchical Algorithmic Framework design highly optimizes computations on embedded platforms
• Collaboration with scientific groups to integrate cutting edge approaches

www.luxoft.com

THANK YOU

Sergii Bykov Technical Lead email: sergeybykov1@gmail.com