CS4495/6495 Introduction to Computer Vision 3C-L3 Calibrating - PowerPoint PPT Presentation

CS4495/6495 Introduction to Computer Vision 3C-L3 Calibrating cameras

Finally (last time): Camera parameters • Projection equation – the cumulative effect of all parameters:      ' 1 0 0 0 f s x    c R I T     0 M      3 1 x 3 1 x 0 ' 0 1 0 0 af y 3 3 3 3 x x      c    1 1  0   0  (3x4)     0 0 1 0 0 1 0 1 3 x 1 3 x     intrinsics projection rotation translation

Finally (last time): Camera parameters • Projection equation – the cumulative effect of all parameters:   X      * * * * sx       Y  x M X   * * * * sy       Z       * * * *  s     1 

Calibration • How to determine M?

Calibration using known points Place a known object in the scene • identify correspondence between image and scene • compute mapping from scene to image

Resectioning Estimating the camera matrix from known 3D points Projective Camera Matrix:     p K R t P M P   X      * w u m m m m  11 12 13 14      Y   * w v m m m m      21 22 23 24  Z        w   m m m m  31 32 33 34  1 

Direct linear calibration - homogeneous   X        * u w u m m m m i  Y 00 01 02 03 i i          * v w v m m m m i  Z i i 10 11 12 13       1    w   m m m m   i   1  20 21 22 23 One pair of    equations for m X m Y m Z m  00 01 02 03 i i i u each point    i m X m Y m Z m 20 i 21 i 22 i 23    m X m Y m Z m  10 i 11 i 12 i 13 v    i m X m Y m Z m 20 i 21 i 22 i 23

Direct linear calibration - homogeneous    m X m Y m Z m  00 01 02 03 i i i u    i m X m Y m Z m 20 i 21 i 22 i 23    m X m Y m Z m  One pair of 10 i 11 i 12 i 13 v    i equations for m X m Y m Z m 20 i 21 i 22 i 23 each point        ( ) u m X m Y m Z m m X m Y m Z m 20 21 22 23 00 01 02 03 i i i i i i i        ( ) v m X m Y m Z m m X m Y m Z m i 20 i 21 i 22 i 23 10 i 11 i 12 i 13

Direct linear calibration - homogeneous        ( ) u m X m Y m Z m m X m Y m Z m i 20 i 21 i 22 i 23 00 i 01 i 02 i 03        ( ) v m X m Y m Z m m X m Y m Z m 20 21 22 23 10 11 12 13 i i i i i i i   One pair of m 00   m equations for 10   m 02 each point   m  03  m   10 m   11 m   12 m   13 m   20 m   21 m   22 m   23

Direct linear calibration - homogeneous • This is a homogenous set of equations. • When over constrained, defines a least squares problem – minimize Am • Since m is only defined up to scale, solve for unit vector m* • Solution: m* = eigenvector of A T A with smallest eigenvalue • Works with 6 or more points

Direct linear calibration - homogeneous   m 00   m 10   m 02   m  03  m   10 m   11 m   12 m   13 m   20 m   21 m A 0 m   22 m   23 2n × 12 12 2n

The SVD (singular value decomposition) trick… • Find the m that minimizes 𝐵m subject to m =1. • Let 𝐵 = 𝑉𝐸𝑊 𝑈 (singular value decomposition, D diagonal, U and V orthogonal) • Therefor minimizing 𝑉𝐸𝑊 𝑈 m • But, 𝑉𝐸𝑊 𝑈 m = 𝐸𝑊 𝑈 m and m = 𝑊 𝑈 m • Thus minimize 𝐸𝑊 𝑈 m subject to 𝑊 𝑈 m = 𝟐

The SVD (singular value decomposition) trick… • Thus minimize 𝐸𝑊 𝑈 m subject to 𝑊 𝑈 m = 𝟐 • Let y = 𝑊 𝑈 m Now minimize 𝐸y subject to y = 1 . • But D is diagonal, with decreasing values. So 𝐸y minimum is when y = 0,0,0 … , 0,1 𝑈 • Since y = 𝑊 𝑈 m , m = 𝑊𝐳 since V orthogonal • Thus m = 𝑊𝐳 is the last column in V.

The SVD (singular value decomposition) trick… • Thus m = 𝑊𝐳 is the last column in V. • And, the singular values of A are square roots of the eigenvalues of A T A and the columns of V are the eigenvectors. (Show this ? Nah…) • Recap: Given Am=0, find the eigenvector of A T A with smallest eigenvalue, that’s m.

Direct linear calibration - inhomogeneous • Another approach: 1 in lower r.h. corner for 11 d.o.f   X      u m m m m  00 01 02 03      Y  v m m m m       10 11 12 13 Z       1 1    m m m  20 21 22  1 

Direct linear calibration - inhomogeneous Dangerous if m 23 is really (near) zero!

Direct linear calibration (transformation) Advantages: • Very simple to formulate and solve. Can be done, say, on a problem set • These methods are referred to as “algebraic error” minimization.

Direct linear calibration (transformation) Disadvantages: • Doesn’t directly tell you the camera parameters (more in a bit) • Approximate: e.g. doesn’t model radial distortion • Hard to impose constraints (e.g., known focal length) • Mostly: Doesn’t minimize the right error function

Direct linear calibration (transformation) For these reasons, prefer nonlinear methods: • Define error function E between projected 3D points and image positions: E is nonlinear function of intrinsics, extrinsics, and radial distortion • Minimize E using nonlinear optimization techniques e.g., variants of Newton’s method (e.g., Levenberg Marquart)

Geometric Error Predicted Image X i locations x i     ˆ minimize ( , ) E d x x i i i   min ( , ) d x M X i i M M i

“Gold Standard” algorithm (Hartley and Zisserman) Objective Given n≥6 3D to 2D point correspondences {𝑌 𝑗 ↔ 𝑦 𝑗 ’} , determine the “Maximum Likelihood Estimation” of M

“Gold Standard” algorithm (Hartley and Zisserman) Algorithm (i) Linear solution:  (a)(Optional) Normalization: x = Tx X UX i i i i (b) Direct Linear Transformation minimization (ii) Minimize geometric error: using the linear estimate as a starting point minimize the geometric error:  min ( , ) x M X d i i M i

“Gold Standard” algorithm (Hartley and Zisserman) -1 (iii) Denormalization: M = T M U

Finding the 3D Camera Center from M • M encodes all the parameters. So we should be able to find things like the camera center from M. • Two ways: pure way and easy way

Finding the 3D Camera Center from M   • Slight change in notation. Let:  | M Q b M is(3x4) – b is last column of M • The center C is the null-space camera of projection matrix. So if find C such that: M C = 0 that will be the center. Really…

Finding the 3D Camera Center from M • Proof: Let X be somewhere between any point P and C    λ (1 λ) X P C

Finding the 3D Camera Center from M • Proof: Let X be somewhere between any point P and C    λ (1 λ) X P C • And the projection:     λ (1 λ) x M X M P M C

Finding the 3D Camera Center from M • Proof: Let X be somewhere between any point P and C    λ (1 λ) X P C • And the projection:     λ (1 λ) x M X M P M C • For any P, all points on PC ray project on image of P, therefore MC must be zero. So the camera center has to be in the null space.

Finding the 3D Camera Center from M • Now the easy way. A formula! If M =[Q|b] then:     1 Q b    C 1  

Alternative: multi-plane calibration Images courtesy Jean-Yves Bouguet, Intel Corp.

Alternative: multi-plane calibration Advantages • Only requires a plane • Don’t have to know positions/orientations • Good code available online! • OpenCV library • Matlab version by Jean-Yves Bouget: http://www.vision.caltech.edu/bouguetj/calib_doc/index.html • Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/