In Intelligent Vehicles using CNN-based Detection and Ori - - PowerPoint PPT Presentation

▶
in intelligent vehicles using
SMART_READER_LITE
LIVE PREVIEW

In Intelligent Vehicles using CNN-based Detection and Ori - - PowerPoint PPT Presentation

Dec 23, 2022 244 likes •579 views

Modeling Traffic Scenes for r In Intelligent Vehicles using CNN-based Detection and Ori rientation Estimation Carlos Guindel, David Martn and Jos Mara Armingol Intelligent Systems Laboratory (LSI) Universidad Carlos III de Madrid

slide-1
SLIDE 1

Modeling Traffic Scenes for r In Intelligent Vehicles using CNN-based Detection and Ori rientation Estimation

Carlos Guindel, David Martín and José María Armingol Intelligent Systems Laboratory (LSI) · Universidad Carlos III de Madrid Sevilla · 23 November 2017

slide-2
SLIDE 2

Agenda

Introduction Obstacle detection Scene modeling Results Conclusion

2

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017
slide-3
SLIDE 3

Agenda

Introduction Obstacle detection Scene modeling Results Conclusion

3

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017
slide-4
SLIDE 4

Introduction

4

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017
  • A basic

requirement for driving tasks Obstacle detection

  • An accurate

estimation of the class is essential Classification

  • Close-to-market

assemblies

  • Rich data

source

Vision-based approaches

  • Feature learning
  • The new

paradigm in computer vision

Convolutional Neural Networks

Automated vehicles

  • Highly dynamic,

semi-structured environments

  • They have to

handle complex situations

slide-5
SLIDE 5

IVVI 2.0 project

5

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017

INTELLIGENT VEHICLE BASED ON VISUAL INFORMATION 2.0

Trinocular stereo cam. Multi-layer lidar scanner Side-looking cameras Computer with GPU

+info: uc3m.es/islab

slide-6
SLIDE 6

System overview

6

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017
  • Two main branches intended to run in parallel
  • Obstacle detection
  • Features are extracted exclusively from the left stereo image
  • Scene modeling
  • Stereo-based 3D reconstruction & flat-ground assumption
slide-7
SLIDE 7

Agenda

Introduction Obstacle detection Scene modeling Results Conclusion

7

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017
slide-8
SLIDE 8

Faster R-CNN framework

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017
  • S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal

Networks,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 6, pp. 1137–1149, 2016.

Convolutional features computed only once per image A RPN generates proposals

  • wrt. a fixed set of anchors
  • Conv. features in these regions

are pooled for classification Parameters are learned through a multi-task loss

8

slide-9
SLIDE 9

Viewpoint estimation

  • Faster R-CNN framework was modified to introduce viewpoint inference

9

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017
  • C. Guindel, D. Martin, and J. M. Armingol, “Joint object detection and viewpoint estimation using CNN features,” in
  • Proc. of the IEEE International Conference on Vehicular Electronics and Safety (ICVES), 2017, pp. 145–150.
slide-10
SLIDE 10

Final estimation: Θ𝑗∗ → መ 𝜄

Discrete viewpoint inference

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017

𝑂𝑐 angle bins Θ𝑗 … Θ𝑂𝑐 𝑂𝑐 = 8 Training: 𝜄𝑗0→ Θ𝑗 Inference output: r ∈ Δ𝑂𝑐−1 𝑠

Elements

  • f 𝑠

10

  • Every object is assigned a bin
  • Inference gives a categorial distribution
slide-11
SLIDE 11

Joint detection and viewpoint estimation

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017

11

Feature map Proposal Fixed size

  • feat. vector

Fully connected (FC) layers FC layer FC layer FC layer

  • B. Box

regression Softmax Softmax Softmax Class Viewpoint

Only 𝑂𝑐 · 𝐿 · 4096 new weights

…

𝑂𝑐 · 𝐿

𝑂𝑐 · 𝐿 elements angle bins classes

slide-12
SLIDE 12

Faster R-CNN loss

Loss function and training

  • Unweighted muli-task loss with five components

Joint Object Detection and Viewpoint Estimation using CNN features Carlos Guindel · ICVES 2017

Logistic loss for RPN objectness Smooth-L1 loss for RPN b.box regression Logistic loss for class Smooth-L1 loss for b.box regression

Logistic loss for viewpoint estimation

12 …

𝑂𝑐 angle bins of the ground truth class Ground-truth angle bin Normalized

  • n the batch
slide-13
SLIDE 13

Agenda

Introduction Obstacle detection Scene modeling Results Conclusion

13

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017
slide-14
SLIDE 14

Scene modeling

14

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017

Left image Right image Disparity P.C. generator Disparity map 3D point cloud 3D RECONSTRUCTION

slide-15
SLIDE 15

Scene modeling

15

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017

Left image Right image Disparity P.C. generator Disparity map 3D point cloud 3D RECONSTRUCTION

slide-16
SLIDE 16

Scene modeling

16

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017

Left image Right image Disparity P.C. generator Disparity map 3D point cloud 3D RECONSTRUCTION SGM stereo matching

Suitable for environments with lack of texture, illumination changes, etc.

slide-17
SLIDE 17

Scene modeling

17

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017

Left image Right image Disparity P.C. generator Disparity map 3D point cloud 3D RECONSTRUCTION Pin-hole + disparity

We build a XYZRGB cloud from the left image and the disparity map

slide-18
SLIDE 18

Scene modeling

18

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017

Left image Right image Disparity P.C. generator Disparity map 3D point cloud 3D point cloud

Plane segmentation

Plane model

Calibration from plane

Camera-to- world calibration

3D RECONSTRUCTION EXTRINSIC PARAMETERS AUTO-CALIBRATION

slide-19
SLIDE 19

Scene modeling

19

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017

3D point cloud

Plane segmentation

Plane model

Calibration from plane

Camera-to- world calibration

EXTRINSIC PARAMETERS AUTO-CALIBRATION Voxel grid dowsampling The cloud from the 3D reconstruction pipeline is downsampled (grid size: 20 cm)

slide-20
SLIDE 20

Scene Modeling

20

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017

3D point cloud

Plane segmentation

Plane model

Calibration from plane

Camera-to- world calibration

EXTRINSIC PARAMETERS AUTO-CALIBRATION Planar segmentation

Using RANSAC with a 10 cm threshold, and a small angular tolerance. …Pass through filters

Vertical axis: 0-2 m Depth axis: 0-20 m

slide-21
SLIDE 21

Scene modeling

21

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017

3D point cloud

Plane segmentation

Plane model

Calibration from plane

Camera-to- world calibration

EXTRINSIC PARAMETERS AUTO-CALIBRATION

roll pitch

Plane:

slide-22
SLIDE 22

Object localization

22

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017
slide-23
SLIDE 23

Object localization

23

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017

Object ROI for localization

11 central rows

  • f the object’s

bounding box

Filtered cloud

Without points belonging to the ground or too close to the camera

Organized pointcloud Coordinates on the world’s XY plane for every point 𝒚 = (𝑦, 𝑧, 𝑨) 𝒚𝑝𝑐𝑘 = 𝑛𝑓𝑒𝑗𝑏𝑜 𝒚 𝜄𝑝𝑐𝑘 = 𝛽 − atan2(𝑧𝑝𝑐𝑘 − 𝑦𝑝𝑐𝑘)

x 𝑧𝑝𝑐𝑘 world y 𝑦𝑝𝑐𝑘 𝜄

Top-down view

yaw

slide-24
SLIDE 24

Agenda

Introduction Obstacle detection Scene modeling Results Conclusion

24

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017
slide-25
SLIDE 25

Results: Detection and viewpoint estimation

  • KITTI Object Detection Benchmark
  • 5,576 images for training and 2,065 for validation
  • Labels for class and orientation available
  • Evaluation metric
  • Average Orientation Similarity (AOS)

25

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017
slide-26
SLIDE 26

Results: Detection and viewpoint estimation

  • Two different architectures:
  • ZF (lightweight) and VGG 16-layer (more complex)
  • Three different scales (height in pixels):
  • 375, 500, 625

26

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017

88,43 66,28 63,41 N.A. N.A. N.A. 2 sec.

Top-performing comparable method in the KITTI ranking

(ms)

slide-27
SLIDE 27

Results: Scene modeling

27

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017
slide-28
SLIDE 28

Results: Scene modeling

28

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017
slide-29
SLIDE 29

Results: Scene modeling

29

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017
slide-30
SLIDE 30

Agenda

Introduction Obstacle detection Scene modeling Results Conclusion

30

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017
slide-31
SLIDE 31

Conclusion

  • Towards a full object-based scene understanding
  • CNN-based detection and viewpoint inference
  • Efficient approach: the same set of features is used for all tasks
  • Stereo-vision 3D information is included for situation assessment
  • Results validate our approach

31

Modeling Traffic Scenes for Intelligent Vehicles using CNN-based Detection and Orientation Estimation

  • C. Guindel et al. · ROBOT 2017

Code for CNN detection & viewpoints available at https://github.com/cguindel/lsi-faster-rcnn

Future work

  • New categories of traffic elements
  • Extension to the time domain
  • Tracking, filtering, etc.
  • Including information from other perception modules
  • E.g., semantic segmentation
slide-32
SLIDE 32

THANKS FOR

YOUR ATTENTION

23 November 2017 ROBOT'2017 - Third Iberian Robotics Conference

Carlos Guindel · cguindel.github.io · cguindel@ing.uc3m.es