Increasing the Convergence Domain of RGB-D Direct Registration - - PowerPoint PPT Presentation

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Increasing the Convergence Domain of RGB-D Direct Registration Methods for Vision-based Localization in Large Scale Environments

Renato Martins and Patrick Rives

Inria Sophia Antipolis, France http://team.inria.fr/lagadic

ITSC PPNIV’16 Workshop, Rio de Janeiro, Brazil 01 November 2016

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Outline

1

Introduction & Motivation

2

RGB-D Registration & Large Motion

3

Adaptive Formulation

4

Results

5

Conclusions & Perspectives

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Outline

1

Introduction & Motivation

2

RGB-D Registration & Large Motion

3

Adaptive Formulation

4

Results

5

Conclusions & Perspectives

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Introduction and Motivation

Main objective: the design of a robust/efficient direct RGB-D registration technique for large motions.

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Introduction and Motivation

Main objective: the design of a robust/efficient direct RGB-D registration technique for large motions. Multiple applications:

Visual odometry, mapping and SLAM; Navigation and visual servoing; Augmented reality.

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Introduction and Motivation

Main objective: the design of a robust/efficient direct RGB-D registration technique for large motions. Multiple applications:

Visual odometry, mapping and SLAM; Navigation and visual servoing; Augmented reality.

? ?

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Outline

1

Introduction & Motivation

2

RGB-D Registration & Large Motion

3

Adaptive Formulation

4

Results

5

Conclusions & Perspectives

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Direct RGB-D Registration

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Direct RGB-D Registration

F T(x) F∗ p n∗ w(p, T) F T(x) F∗ p n∗ w(p, T)

Minimisation of intensity and depth costs: C(x) =

p ρ(eI(p, T(x))) + λ2 p ρ(eD(p, T(x)))

with eI(p, T(x)) = I(w(p, T(x))) − I∗(p) eD(p, T(x))=(Rn∗(p))T (g(w(p,T(x)))−T(x)g∗(p)) g(•) is a 3D point (the inverse sensor projection) and n the normal vector.

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

RGB-D Registration & Large Motion

Direct Registration: Strengths of direct methods: Accurate (precision similar to expensive IMU’s) [A. Howard, IROS 2008]; More robust to outliers than feature based registration (essential/fundamental matrix).

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

RGB-D Registration & Large Motion

Direct Registration: Strengths of direct methods: Accurate (precision similar to expensive IMU’s) [A. Howard, IROS 2008]; More robust to outliers than feature based registration (essential/fundamental matrix). But... Direct methods are applied mostly for small displacements (small convergence domain); Real-time constraint: cannot process all frames (low frame rate or gaps); Navigation does not follow exactly the acquired 3D model (learned map).

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Direct Reg. Convergence Domain

Strategies to increase the basin of convergence In the motion estimate (prediction): Hypothetical constraints of the movement (locally 2D, non-holonomic); Prediction (assumed motion model + online estimates);

prediction initial solution

cost x

prediction initial solution

cost x Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Direct Reg. Convergence Domain

Strategies to increase the basin of convergence In the motion estimate (prediction): Hypothetical constraints of the movement (locally 2D, non-holonomic); Prediction (assumed motion model + online estimates); Acting in the sensor’s measurements: Multi-resolution (Gaussian pyramid); Correlation (often done simultaneously with assumptions in the motion); Extraction and matching of stable features (SIFT, SURF, ...). **Choice of scaling factor affects the shape of the cost.

prediction initial solution

cost x

prediction initial solution

cost x Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Outline

1

Introduction & Motivation

2

RGB-D Registration & Large Motion

3

Adaptive Formulation

4

Results

5

Conclusions & Perspectives

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Intensity and Geometric Cost Shape/Convergence

Considered data terms and modelling: Classic RGB-D formulation: C(x) = CI(x) + λ2CD(x) Scaling factor λ: Heuristically set; λ based on covariance of each point [C. Kerl & D. Cremers, ICRA 2013]; λ scaling pixels to meters [T.Tykkala & A.Comport, ICCV 2011].

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Intensity and Geometric Cost Shape/Convergence

Considered data terms and modelling: Classic RGB-D formulation: C(x) = CI(x) + λ2CD(x) Scaling factor λ: Heuristically set; λ based on covariance of each point [C. Kerl & D. Cremers, ICRA 2013]; λ scaling pixels to meters [T.Tykkala & A.Comport, ICCV 2011]. In fixed pyramidal resolution: RGB and geometric costs have different convergence properties [M. Levoy et al., 3DIM 2003] [S. Bonnabel et al., ACC 2016]; RGB term dominates when combined as in [T. Tykkala & A. Comport, ICCV 2011] and [C. Kerl & D. Cremers, ICRA 2013].

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Intensity and Geometric Cost Shape/Convergence

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Intensity and Geometric Cost Shape/Convergence

Wrong idea: set high λ during all the optimization 1) Geom. cost is flatter than RGB in the neighbourhood of the solution; 2) More sensible/unstable than RGB registration (visibility constraint); 3) Do not guarantee sub-pixel precision from intensity only cost term.

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

This Work: Adaptive Formulation

Adaptive formulation: Main idea: Exploit more geometric term when at coarse iterations; End up with classic RGB term near the solution; New cost: ˜ C(x) = (1 − µ(x))CI(x) + µ(x)CD(x) How to identify the neighbourhood where RGB is more discriminant?

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

This Work: Adaptive Formulation

Adaptive formulation: Main idea: Exploit more geometric term when at coarse iterations; End up with classic RGB term near the solution; New cost: ˜ C(x) = (1 − µ(x))CI(x) + µ(x)CD(x) How to identify the neighbourhood where RGB is more discriminant? Candidate 1 (increment norm): µ(x) = k1 + k2/ (1 + exp(−k3(||x|| − k4))) Also proposed by [L. Morency & T. Darrell, ICPR 2002];

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

This Work: Adaptive Formulation

Adaptive formulation: Main idea: Exploit more geometric term when at coarse iterations; End up with classic RGB term near the solution; New cost: ˜ C(x) = (1 − µ(x))CI(x) + µ(x)CD(x) How to identify the neighbourhood where RGB is more discriminant? Candidate 1 (increment norm): µ(x) = k1 + k2/ (1 + exp(−k3(||x|| − k4))) Also proposed by [L. Morency & T. Darrell, ICPR 2002]; But: critical tuning parameters k3, k4; Modified cost might not be convex (cross-peaks effect).

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

This Work: Adaptive Formulation

Adaptive formulation: Candidate 2 (relative conditioning): condx(C(x)) =

• C(x0 ◦ x) − C(x0)

C(x0)

• / ||x||

||x0|| The relative variation of the RGB (CI) and Geo. (CD) costs – conditioning: µ(x) = k1 + k2, if condx(CI(x))/condx(CD(x)) < k3 k1, otherwise.

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

This Work: Adaptive Formulation

Adaptive formulation: Candidate 2 (relative conditioning): condx(C(x)) =

• C(x0 ◦ x) − C(x0)

C(x0)

• / ||x||

||x0|| The relative variation of the RGB (CI) and Geo. (CD) costs – conditioning: µ(x) = k1 + k2, if condx(CI(x))/condx(CD(x)) < k3 k1, otherwise. Easier tuning (just choose k3 >> 1) and higher detectability.

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Outline

1

Introduction & Motivation

2

RGB-D Registration & Large Motion

3

Adaptive Formulation

4

Results

5

Conclusions & Perspectives

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Results: Registration Sequences

Indoor ”Corridor 1” sequence: Gap of 5 frames;

• Max. rotation ≈ 15 degrees;

State of the art method: [Tykkala, ICCV’11].

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Results: Registration Sequences

Indoor ”Office 1” sequence: Gap of 15 frames;

• Max. rotation ≈ 27 degrees;

Comparison to the method of [Tykkala, ICCV’11].

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Quantitative Results: Sponza Atrium

Sponza Atrium Sequence and Experimental Set-Up: Spherical sensor model; Indoor simulated dataset; Fast turns; Test with gaps of 15 frames (≈ 1.2 meters between frames).

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Quantitative Results: KITTI VO/SLAM

KITTI Sequence 00: Perspective sensor model (reduced FOV); Outdoor dataset in urban area; Speeds of up to 70 km/h (≈ 20 m/s). Experimental set-up: Multi-resolution: pyramid of four levels; Tests with sub-sampling (gaps) of 1,2 or 3 frames.

Table : KITTI outdoor sequence: average RRE[deg]/RTE[mm]

Gap = 1 Gap = 2 Gap = 3 [Tykkala, ICCV’11] 0.08/23.1 0.78/268 3.68/1059 Adaptive 0.06/16.4 0.37/47.5 1.05/238

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Outline

1

Introduction & Motivation

2

RGB-D Registration & Large Motion

3

Adaptive Formulation

4

Results

5

Conclusions & Perspectives

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Summary

Conclusions RGB-D direct registration for large motions; Adaptive formulation that explores convexity and convergence properties

• f intensity and geometric data terms;

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Summary

Conclusions RGB-D direct registration for large motions; Adaptive formulation that explores convexity and convergence properties

• f intensity and geometric data terms;

Exploit more geometric term when further of the minimum; End up with classic RGB term near the solution;

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Summary

Conclusions RGB-D direct registration for large motions; Adaptive formulation that explores convexity and convergence properties

• f intensity and geometric data terms;

Exploit more geometric term when further of the minimum; End up with classic RGB term near the solution; 20 times faster in simulated sequences and at least as three times fast in real sequences (fixed resolution); Applicable to any perspective/spherical RGB-D sequence (eg. depth from 3D-LIDAR, ToF, Kinect, etc.)

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Summary

Conclusions RGB-D direct registration for large motions; Adaptive formulation that explores convexity and convergence properties

• f intensity and geometric data terms;

Exploit more geometric term when further of the minimum; End up with classic RGB term near the solution; 20 times faster in simulated sequences and at least as three times fast in real sequences (fixed resolution); Applicable to any perspective/spherical RGB-D sequence (eg. depth from 3D-LIDAR, ToF, Kinect, etc.) Next Steps: Observability conditions and confidence bounds for each data term; Apply to other large FOV sensors as catadioptric and fisheye; Finding convex (quasi-convex) dual formulations for adding planes, edgelets/lines and image moments.

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods

Introduction & Motivation RGB-D Registration & Large Motion Adaptive Formulation Results Conclusions & Perspectives

Thank you very much for your attention.

Inria Lagadic Team – ITSC PPNIV’16 Increasing the Convergence Domain of RGB-D Direct Registration Methods