Multibody reconstruction of the dynamic scene surrounding a vehicle - PowerPoint PPT Presentation

Multibody reconstruction of the dynamic scene surrounding a vehicle using a wide baseline and multifocal stereo system Laurent Mennillo 1 , 2 , ´ Eric Royer 1 , Fr´ eric Mondot 2 , Johann ed´ Mousain 2 , Michel Dhome 1 1 Pascal Institute, Clermont Auvergne University - Aubi` ere, France 2 Technocentre RENAULT - Guyancourt, France September 24, 2017

Introduction - Multibody reconstruction Method Experimental data Context and scientific objectives Results Conclusion Introduction - Multibody reconstruction 2 L. Mennillo et al. Multibody SLAM using an heterogeneous stereo system

Introduction - Multibody reconstruction Method Experimental data Context and scientific objectives Results Conclusion Introduction - Multibody reconstruction Context Short baseline with an identical stereo pair is well studied Not the case of wide baseline and heterogeneous stereo Multi-camera system inspired by actual sensor implantation on current vehicles (frontal camera and AVM systems) Industrial approach with RENAULT Scientific objectives Develop a sparse, purely geometrical solution for multibody reconstruction on heterogeneous stereo systems Experimental data acquisition in a real environment 3 L. Mennillo et al. Multibody SLAM using an heterogeneous stereo system

Introduction - Multibody reconstruction Sparse feature extraction and matching Method Visual SLAM Experimental data Mobile 3D points segmentation and tracking Results Optimization Conclusion Method - Overview Framework Offline intrinsic and extrinsic calibration using [1] 1 Feature extraction and matching 2 Visual SLAM 3 Mobile 3D points segmentation and tracking 4 Local optimization 5 4 L. Mennillo et al. Multibody SLAM using an heterogeneous stereo system

Introduction - Multibody reconstruction Sparse feature extraction and matching Method Visual SLAM Experimental data Mobile 3D points segmentation and tracking Results Optimization Conclusion Method - Features Feature sets Each frame has a corresponding set of SIFT features f i , t i ∈ 0 . . . m is the camera of observation t ∈ 0 . . . n is the time of observation Two feature matching schemes between the sets f i , t and f i ′ , t ′ ⇒ i = i ′ and t � = t ′ Temporal matching ⇐ ⇒ i � = i ′ and t = t ′ Stereo matching ⇐ Matches between a feature x ∈ f i , t and another feature x ′ ∈ f i ′ , t ′ Potential feature match p ( x , x ′ ) Final feature match m ( x , x ′ ) 5 L. Mennillo et al. Multibody SLAM using an heterogeneous stereo system

Introduction - Multibody reconstruction Sparse feature extraction and matching Method Visual SLAM Experimental data Mobile 3D points segmentation and tracking Results Optimization Conclusion Method - Feature extraction 1. Extracting the set of new features S 1 Frame downsampling to account for the different focal lengths Frame division into blocks to ensure good spatial repartition SIFT feature detection and description for each block 2. Extracting the set of tracked features S 2 Temporal tracking of previously triangulated features in f i , t − 1 using the Lucas Kanade method [2] to compensate for block division SIFT description for each tracked feature 3. Merging the two sets S 1 and S 2 to obtain f i , t Elimination of duplicates based on pixelwise euclidean distance 6 L. Mennillo et al. Multibody SLAM using an heterogeneous stereo system

Introduction - Multibody reconstruction Sparse feature extraction and matching Method Visual SLAM Experimental data Mobile 3D points segmentation and tracking Results Optimization Conclusion Method - Feature matching Locality constraint Lc for temporal matching between f i , t and f i , t +1 Potential matches ⇐ ⇒ Features at near distance (search window) Epipolar constraint Ec for stereo matching between f i , t and f i ′ , t Potential matches ⇐ ⇒ Features near epipolar lines If more than one potential match exists for a feature Retain the minimal L 2 distance between descriptors Potential matches p ( x , x ′ ) = ⇒ Final match m ( x , x ′ ) 7 L. Mennillo et al. Multibody SLAM using an heterogeneous stereo system

Introduction - Multibody reconstruction Sparse feature extraction and matching Method Visual SLAM Experimental data Mobile 3D points segmentation and tracking Results Optimization Conclusion Method - Visual SLAM Estimate the ego motion parameters of the multi-camera system Bundle adjustment approach as in [3] Local optimization of selected keyframes and associated 3D points 8 L. Mennillo et al. Multibody SLAM using an heterogeneous stereo system

Introduction - Multibody reconstruction Sparse feature extraction and matching Method Visual SLAM Experimental data Mobile 3D points segmentation and tracking Results Optimization Conclusion Method - Mobile 3D points segmentation and tracking Set of observations o X associated to the 3D point X At least a couple of associated observations ( o X i , t , o X i ′ , t ′ ) Corresponding to either a temporal or stereo match m ( x , x ′ ) Several possible observations, in multiple frames at multiple times Determine the class C of the 3D point X from o X ⇒ C X = S Static = ⇒ C X = M Mobile = ⇒ C X = O Outlier = 9 L. Mennillo et al. Multibody SLAM using an heterogeneous stereo system

Introduction - Multibody reconstruction Sparse feature extraction and matching Method Visual SLAM Experimental data Mobile 3D points segmentation and tracking Results Optimization Conclusion Method - Mobile 3D points segmentation and tracking 3D point consistency constraint Cc i , t ∈ o X is inferior to a threshold t Cc Reprojection error for all o X Static 3D points are consistent for all their observations 10 L. Mennillo et al. Multibody SLAM using an heterogeneous stereo system

Introduction - Multibody reconstruction Sparse feature extraction and matching Method Visual SLAM Experimental data Mobile 3D points segmentation and tracking Results Optimization Conclusion Method - Mobile 3D points segmentation and tracking Mobile 3D point detection Step 1 - Stereo match and reconstruction at time t 1 11 L. Mennillo et al. Multibody SLAM using an heterogeneous stereo system

Introduction - Multibody reconstruction Sparse feature extraction and matching Method Visual SLAM Experimental data Mobile 3D points segmentation and tracking Results Optimization Conclusion Method - Mobile 3D points segmentation and tracking Mobile 3D point detection Step 2 - Temporal matches from t 1 to t 2 = ⇒ Tracking 12 L. Mennillo et al. Multibody SLAM using an heterogeneous stereo system

Introduction - Multibody reconstruction Sparse feature extraction and matching Method Visual SLAM Experimental data Mobile 3D points segmentation and tracking Results Optimization Conclusion Method - Mobile 3D points segmentation and tracking Mobile 3D point detection Step 3 - Stereo match at time t 2 13 L. Mennillo et al. Multibody SLAM using an heterogeneous stereo system

Introduction - Multibody reconstruction Sparse feature extraction and matching Method Visual SLAM Experimental data Mobile 3D points segmentation and tracking Results Optimization Conclusion Method - Mobile 3D points segmentation and tracking Mobile 3D point detection Consistency constraint is not satisfied for all observations of X 2 14 L. Mennillo et al. Multibody SLAM using an heterogeneous stereo system

Introduction - Multibody reconstruction Sparse feature extraction and matching Method Visual SLAM Experimental data Mobile 3D points segmentation and tracking Results Optimization Conclusion Method - Mobile 3D points segmentation and tracking 3D point mobility constraints Mc 1, Mc 2 and Mc 3 Mc 1 = ⇒ Consistency for each individual temporality t Mc 2 = ⇒ At least one stereo match per temporality Mc 3 = ⇒ At least two temporalities per 3D point 15 L. Mennillo et al. Multibody SLAM using an heterogeneous stereo system

Introduction - Multibody reconstruction Sparse feature extraction and matching Method Visual SLAM Experimental data Mobile 3D points segmentation and tracking Results Optimization Conclusion Method - Mobile 3D points segmentation and tracking Trajectory consistency Filters erratic movements generated by false matches For mobile points that have been tracked at least 3 times Distance and elevation between each pair of consecutive points Angle formed by each triplet of consecutive points 16 L. Mennillo et al. Multibody SLAM using an heterogeneous stereo system

Introduction - Multibody reconstruction Sparse feature extraction and matching Method Visual SLAM Experimental data Mobile 3D points segmentation and tracking Results Optimization Conclusion Method - Optimization Optimization of camera poses and 3D points Unified optimization of all 3D points Static points and mobile points per temporality Minimization of the reprojection error with bundle adjustment 17 L. Mennillo et al. Multibody SLAM using an heterogeneous stereo system