3D Vision Marc Pollefeys, Viktor Larsson Spring 2020 Schedule Feb - PowerPoint PPT Presentation

3D Vision Marc Pollefeys, Viktor Larsson Spring 2020

Schedule Feb 17 Introduction Feb 24 Geometry, Camera Model, Calibration Mar 2 Features, Tracking / Matching Mar 9 Project Proposals by Students Mar 16 Structure from Motion (SfM) + papers Mar 23 Dense Correspondence (stereo / optical flow) + papers Mar 30 Bundle Adjustment & SLAM + papers Apr 6 Student Midterm Presentations Apr 13 Easter break Multi-View Stereo & Volumetric Modeling + papers Apr 20 Apr 27 3D Modeling with Depth Sensors + papers May 4 3D Scene Understanding + papers May 11 4D Video & Dynamic Scenes + papers May 18 papers May 25 Student Project Demo Day = Final Presentations

3D Vision – Class 6 Bundle Adjustment and SLAM • [Triggs, McLauchlan, Hartley, Fitzgibbon, Bundle Adjustment – A Modern Synthesis, Int. Workshop on Vision Algorithms, 1999] • [Montemerio, Thrun, Koller, Wegbreit, FastSLAM: A Factored Solution to the Simultaneous Localization and Mapping Problem, AAAI 2002] • Section 2.5 from [Lee, Visual Mapping and Pose Estimation for Self-Driving Cars, PhD Thesis, ETH Zurich, 2014] Slides : Gim Hee Lee

Lecture Overview • Bundle Adjustment in Structure-from- Motion • Simultaneous Localization & Mapping (SLAM) 4

Recap: Structure-From-Motion • Two views initialization: – 5-Point algorithm (Minimal Solver) – 8-Point linear algorithm – 7-Point algorithm E → ( R,t) 5

Recap: Structure-From-Motion • Triangulation: 3D Points E → ( R,t) 6

Recap: Structure-From-Motion • Subsequent views: Perspective pose estimation (R,t) (R,t) (R,t) 7

Recap: Structure-From-Motion

Bundle Adjustment • Refinement step in Structure-from-Motion. • Refine a visual reconstruction to produce jointly optimal 3D structures P and camera poses C . • Minimize total re-projection errors .  z P Cost Function: j ( )  2 −  argmin , x P C ij j i W ( ) ij X i j  X , j C  j z ij x ij  z ij X = [ , ] P C C i 9

Bundle Adjustment • Refinement step in Structure-from-Motion. • Refine a visual reconstruction to produce jointly optimal 3D structures P and camera poses C . • Minimize total re-projection errors .  z P Cost Function: j    T argmin z W z ij ij ij ( ) X i j  X , j C j ( ) f X x ij − 1 : W  Measurement error covariance z ij ij X = [ , ] P C C i 10

Bundle Adjustment ( ) • Minimize the cost function: argmin f X X 1. Gradient Descent 2. Newton Method 3. Gauss-Newton 4. Levenberg-Marquardt 11

Bundle Adjustment 1. Gradient Descent X k = Initialization: X 0 ( )  f X = =  Compute gradient: T g Z WJ  X = Iterate until X X k convergence  −  Update: X X g k k    = J : Jacobian : Step size  X Slow convergence near minimum point! 12

Bundle Adjustment 2. Newton Method 2 nd order approximation (Quadratic Taylor Expansion): +   + +    T 1 ( ) ( ) f X f X g H = 2 = X X X X K K ( )  +  2 f X = Hessian matrix : H   2 = X X k  + )  ( Find that minimizes ! f X = X X K 13

Bundle Adjustment 2. Newton Method Differentiate and set to 0 gives: −  = − 1 H g  +  Update: X X k k Computation of H is not trivial and might get stuck at saddle point! 14

Bundle Adjustment 3. Gauss-Newton  2   = +  ij T H J WJ Z W  ij ij 2 X i j  T H J WJ Normal equation:  = −  T T J WJ J W Z  +  Update: X X k k Might get stuck and slow convergence at saddle point! 15

Bundle Adjustment 4. Levenberg-Marquardt  Regularized Gauss-Newton with damping factor . ( ) +   = −  T T J WJ I J W Z H LM  → 0 : Gauss-Newton (when convergence is rapid)  →  : Gradient descent (when convergence is slow) Adapt λ during optimization: • Decrease λ when function value decreases • Increase λ otherwise 16

Structure of the Jacobian and Hessian Matrices • Sparse matrices since 3D structures are locally observed. 17

Efficiently Solving the Normal Equation • Schur Complement: Exploit structure of H  = −  T H J W Z LM = H LM 3D Camera Structures Parameters 18

Efficiently Solving the Normal Equation • Schur Complement: Exploit structure of H  = −  T H J W Z LM H H SC s = H LM T H H C SC 3D Camera Structures Parameters 19

Efficiently Solving the Normal Equation • Schur Complement: Obtain reduced system  = −  T H J W Z LM         H H 3D Structures = S SC S S         T       Camera Parameters H H SC C C C   0 I Multiply both sides by:   − − 1 T   H H I SC S         H H = S SC S S       − − −   −  1 1 T T  0      H H H H H H C SC S SC C C S SC S 20

Efficiently Solving the Normal Equation • Schur Complement: Obtain reduced system         H H = S SC S S       − − −   −  1 1 T T 0       H H H H H H C SC S SC C C S SC S  First solve for from: C Easy to invert a block diagonal matrix − −  =  −  − 1 1 T T ( ) H H H H H H C SC S SC C C S SC S Schur Complement (Sparse and Symmetric Positive Definite Matrix)  Solve for by backward substitution. SC 21

Efficiently Solving the Normal Equation − −  =  −  −  Ax = 1 1 T T ( ) H H H H H H b C SC S SC C C S SC S Don’t solve as x=A -1 b: A is sparse, but A -1 is not! • Use sparse matrix factorization to solve system A = 1. LU Factorization LU Solve for x by forward- backward substitutions A = 2. QR factorization QR A = T 3. Cholesky Factorization LL • Iterative methods 1. Conjugate gradient 2. Gauss-Seidel 22

Problem of Fill-In 24

Problem of Fill-In • Reorder sparse matrix to minimize fill-in. ( )( ) = T T T P AP P x P b Permutation matrix to reorder A • NP-Complete problem. • Approximate solutions: 1. Minimum degree 2. Column approximate minimum degree permutation 3. Reverse Cuthill-Mckee. 4. … 25

Problem of Fill-In 26

Robust Cost Function  • Non-linear least squares:   T argmin z W z ij ij ij X ij • Maximum log-likelihood solution: argmin - ln ( | ) p Z X X • Assume that: 1. X is a random variable that follows Gaussian distribution. 2. All observations are independent.   ( )  − = − −    T  argmin ln ( | ) argmin ln exp p X Z c z W z ij ij ij ij   X X ij  =   T argmin z W z ij ij ij X ij 27

Robust Cost Function • Gaussian distribution assumption is not true in the presence of outliers! • Causes wrong convergences. 28

Robust Cost Function ( )        T argmin argmin z z S z ij ij ij ij ij X X ij ij  " Robust Cost Function W scaled with ij ij • Similar to iteratively re-weighted least-squares. • Weight is iteratively rescaled with the attenuating  " factor . ij • Attenuating factor is computed based on current error. 29

Robust Cost Function   (.) " (.) Influence from high errors Gaussian Distribution Reduced influence from high errors Cauchy Distribution 30

Robust Cost Function Outliers are taken into account in Cauchy! 31

State-of-the-Art Solvers • Google Ceres: – https://code.google.com/p/ceres-solver/ • g2o: – https://openslam.org/g2o.html • GTSAM: – https://collab.cc.gatech.edu/borg/gtsam/ • Multicore Bundle Adjustment – http://grail.cs.washington.edu/projects/mcba/ 32

Lecture Overview • Bundle Adjustment in Structure-from- Motion • Simultaneous Localization & Mapping (SLAM) 33

Simultaneous Localization & Mapping (SLAM) • Robot navigates in unknown environment: – Estimate its own pose – Acquire a map model of its environment. • Chicken-and-Egg problem: – Map is needed for localization (pose estimation). – Pose is needed for mapping. • Highly related to Structure-From-Motion. 34

Full SLAM: Problem Definition Control actions u 1 u 3 u 2 Robot poses Observations Map landmarks 35

Full SLAM: Problem Definition • Maximum a Posteriori (MAP) solution: M K ( )   = argmax , | , argmax ( ) ( | , ) ( | , ) p X L Z U p X p x x u p z x l − 0 i i 1 i k ik jk = = X,L X,L 1 1 i k 36