Multibody reconstruction of the dynamic scene surrounding a vehicle - - PowerPoint PPT Presentation

Multibody reconstruction of the dynamic scene surrounding a vehicle using a wide baseline and multifocal stereo system

Laurent Mennillo1,2, ´ Eric Royer1, Fr´ ed´ eric Mondot2, Johann Mousain2, Michel Dhome1

1Pascal Institute, Clermont Auvergne University - Aubi`

ere, France

2Technocentre RENAULT - Guyancourt, France

September 24, 2017

2 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Context and scientific objectives

Introduction - Multibody reconstruction

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

3 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Context and scientific objectives

Introduction - Multibody reconstruction

Context Short baseline with an identical stereo pair is well studied Not the case of wide baseline and heterogeneous stereo Multi-camera system inspired by actual sensor implantation on current vehicles (frontal camera and AVM systems) Industrial approach with RENAULT Scientific objectives Develop a sparse, purely geometrical solution for multibody reconstruction on heterogeneous stereo systems Experimental data acquisition in a real environment

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

4 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Sparse feature extraction and matching Visual SLAM Mobile 3D points segmentation and tracking Optimization

Method - Overview

Framework

1

Offline intrinsic and extrinsic calibration using [1]

2

Feature extraction and matching

3

Visual SLAM

4

Mobile 3D points segmentation and tracking

5

Local optimization

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

5 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Sparse feature extraction and matching Visual SLAM Mobile 3D points segmentation and tracking Optimization

Method - Features

Feature sets Each frame has a corresponding set of SIFT features fi,t i ∈ 0 . . . m is the camera of observation t ∈ 0 . . . n is the time of observation Two feature matching schemes between the sets fi,t and fi′,t′ Temporal matching ⇐ ⇒ i = i′ and t = t′ Stereo matching ⇐ ⇒ i = i′ and t = t′ Matches between a feature x ∈ fi,t and another feature x′ ∈ fi′,t′ Potential feature match p(x, x′) Final feature match m(x, x′)

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

6 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Sparse feature extraction and matching Visual SLAM Mobile 3D points segmentation and tracking Optimization

Method - Feature extraction

• 1. Extracting the set of new features S1

Frame downsampling to account for the different focal lengths Frame division into blocks to ensure good spatial repartition SIFT feature detection and description for each block

• 2. Extracting the set of tracked features S2

Temporal tracking of previously triangulated features in fi,t−1 using the Lucas Kanade method [2] to compensate for block division SIFT description for each tracked feature

• 3. Merging the two sets S1 and S2 to obtain fi,t

Elimination of duplicates based on pixelwise euclidean distance

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

7 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Sparse feature extraction and matching Visual SLAM Mobile 3D points segmentation and tracking Optimization

Method - Feature matching

Locality constraint Lc for temporal matching between fi,t and fi,t+1 Potential matches ⇐ ⇒ Features at near distance (search window) Epipolar constraint Ec for stereo matching between fi,t and fi′,t Potential matches ⇐ ⇒ Features near epipolar lines If more than one potential match exists for a feature Retain the minimal L2 distance between descriptors Potential matches p(x, x′) = ⇒ Final match m(x, x′)

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

8 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Sparse feature extraction and matching Visual SLAM Mobile 3D points segmentation and tracking Optimization

Method - Visual SLAM

Estimate the ego motion parameters of the multi-camera system Bundle adjustment approach as in [3] Local optimization of selected keyframes and associated 3D points

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

9 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Sparse feature extraction and matching Visual SLAM Mobile 3D points segmentation and tracking Optimization

Method - Mobile 3D points segmentation and tracking

Set of observations oX associated to the 3D point X At least a couple of associated observations (oX

i,t, oX i′,t′)

Corresponding to either a temporal or stereo match m(x, x′) Several possible observations, in multiple frames at multiple times Determine the class C of the 3D point X from oX Static = ⇒ C X = S Mobile = ⇒ C X = M Outlier = ⇒ C X = O

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

10 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Sparse feature extraction and matching Visual SLAM Mobile 3D points segmentation and tracking Optimization

Method - Mobile 3D points segmentation and tracking

3D point consistency constraint Cc Reprojection error for all oX

i,t ∈ oX is inferior to a threshold tCc

Static 3D points are consistent for all their observations

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

11 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Sparse feature extraction and matching Visual SLAM Mobile 3D points segmentation and tracking Optimization

Method - Mobile 3D points segmentation and tracking

Mobile 3D point detection Step 1 - Stereo match and reconstruction at time t1

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

12 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Sparse feature extraction and matching Visual SLAM Mobile 3D points segmentation and tracking Optimization

Method - Mobile 3D points segmentation and tracking

Mobile 3D point detection Step 2 - Temporal matches from t1 to t2 = ⇒ Tracking

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

13 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Sparse feature extraction and matching Visual SLAM Mobile 3D points segmentation and tracking Optimization

Method - Mobile 3D points segmentation and tracking

Mobile 3D point detection Step 3 - Stereo match at time t2

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

14 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Sparse feature extraction and matching Visual SLAM Mobile 3D points segmentation and tracking Optimization

Method - Mobile 3D points segmentation and tracking

Mobile 3D point detection Consistency constraint is not satisfied for all observations of X 2

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

15 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Sparse feature extraction and matching Visual SLAM Mobile 3D points segmentation and tracking Optimization

Method - Mobile 3D points segmentation and tracking

3D point mobility constraints Mc1, Mc2 and Mc3 Mc1 = ⇒ Consistency for each individual temporality t Mc2 = ⇒ At least one stereo match per temporality Mc3 = ⇒ At least two temporalities per 3D point

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

16 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Sparse feature extraction and matching Visual SLAM Mobile 3D points segmentation and tracking Optimization

Method - Mobile 3D points segmentation and tracking

Trajectory consistency Filters erratic movements generated by false matches For mobile points that have been tracked at least 3 times Distance and elevation between each pair of consecutive points Angle formed by each triplet of consecutive points

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

17 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Sparse feature extraction and matching Visual SLAM Mobile 3D points segmentation and tracking Optimization

Method - Optimization

Optimization of camera poses and 3D points Unified optimization of all 3D points Static points and mobile points per temporality Minimization of the reprojection error with bundle adjustment

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

18 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Experimental vehicle and sequences

Experimental data

Motivations Specific camera configuration needed to reflect industrial trends No multifocal and wide baseline stereo datasets publicly available Experimental vehicle Multifocal and wide baseline multi camera system (3x 185◦, 1x 80◦) Hardware synchronization of all cameras Environment and sequences Realistic but controlled environment 8 sequences = ⇒ Different road traffic scenarios at low speed

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

19 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Experimental vehicle and sequences

Experimental data

v

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

20 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Qualitative evaluation Limitations

Results

Qualitative evaluation Green - Static points Red - Mobile points Several mobile points tracked and reconstructed

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

21 Introduction - Multibody reconstruction Method Experimental data Results Conclusion Qualitative evaluation Limitations

Results

Limitations False positives can

• ccur due to false

matches Static points on a moving object = ⇒ Not tracked for 3 consecutive frames One inconsistent

• bservation =

⇒ Dismisses the point entirely

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

22 Introduction - Multibody reconstruction Method Experimental data Results Conclusion

Conclusion and future works

Method and dataset The method works as intended on our dataset Future works Denser matching near reconstructed mobile points Scoring method to prevent outliers arising from a single false match Working on more mobile points could help their reconstruction in non-overlapped FOV of the multi-camera system

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

23 Introduction - Multibody reconstruction Method Experimental data Results Conclusion

Bibliography

• P. L´

ebraly, E. Royer, O. Ait-Aider, C. Deymier, and M. Dhome. Fast calibration of embedded non-overlapping cameras. In International Conference on Robotics and Automation, pages 221–227. IEEE, 2011.

• B. D. Lucas and T. Kanade.

An iterative image registration technique with an application to stereo vision. In Proceedings of the 7th International Joint Conference on Artificial Intelligence - Volume 2, IJCAI’81, pages 674–679, San Francisco, CA, USA, 1981. Morgan Kaufmann Publishers Inc.

• E. Mouragnon, M. Lhuillier, M. Dhome, F. Dekeyser, and P. Sayd.

Real time localization and 3d reconstruction. In Computer Vision and Pattern Recognition, volume 1, pages 363–370. IEEE, 2006.

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

24 Introduction - Multibody reconstruction Method Experimental data Results Conclusion

Questions ?

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system

25 Introduction - Multibody reconstruction Method Experimental data Results Conclusion

• L. Mennillo et al.

Multibody SLAM using an heterogeneous stereo system