Geometry of a single camera Slides from Derek Hoiem, Svetlana - PowerPoint PPT Presentation

Geometry of a single camera Slides from Derek Hoiem, Svetlana Lazebnik

Our goal: Recovery of 3D structure J. Vermeer, Music Lesson , 1662 A. Criminisi, M. Kemp, and A. Zisserman,Bringing Pictorial Space to Life: computer techniques for the analysis of paintings, Proc. Computers and the History of Art , 2002

Things aren’t always as they appear… http://en.wikipedia.org/wiki/Ames_room

Single-view ambiguity X? X? X? x

Single-view ambiguity

Single-view ambiguity Rashad Alakbarov shadow sculptures

Anamorphic perspective Image source

Anamorphic perspective H. Holbein The Younger, The Ambassadors , 1533 https://en.wikipedia.org/wiki/Anamorphosis

Our goal: Recovery of 3D structure • When certain assumptions • In general, we need hold, we can recover multi-view geometry structure from a single view Image source • But first, we need to understand the geometry of a single camera…

Camera calibration world coordinate system • Normalized (camera) coordinate system: camera center is at the origin, the principal axis is the z-axis, x and y axes of the image plane are parallel to x and y axes of the world • Camera calibration: figuring out transformation from world coordinate system to image coordinate system

Review: Pinhole camera model ( X , Y , Z ) ! ( f X / Z , f Y / Z ) æ ö æ ö X X ç ÷ ç ÷ æ ö é ù f X f 0 ç ÷ x = Y Y ç ÷ ç ÷ ê ú PX = ! f Y f 0 ç ÷ ç ÷ ç ÷ ê ú Z Z ç ÷ ç ÷ ç ÷ ê ú Z 1 0 è ø ë û ç ÷ ç ÷ 1 1 è ø è ø

Principal point • Principal point (p): point where principal axis intersects the image plane • Normalized coordinate system: origin of the image is at the principal point • Image coordinate system: origin is in the corner

Principal point offset We want the principal point to map to ( p x , p y ) p y instead of (0,0) p x + + ( X , Y , Z ) ! ( f X / Z p , f Y / Z p ) x y æ ö æ ö X X ç ÷ + ç ÷ æ ö é ù f X Z p f p 0 ç x ÷ x Y Y ç ÷ ç ÷ ê ú + = ! f Y Z p f p 0 ç ÷ ç ÷ ç ÷ ê ú y y Z Z ç ÷ ç ÷ ç ÷ ê ú Z 1 0 è ø ë û ç ÷ ç ÷ 1 1 è ø è ø

Principal point offset ( p x p , ) principal point: p y y p x æ ö X ç ÷ é ù f p 0 x Y ç ÷ ê ú = f p 0 ç ÷ ê ú y Z ç ÷ ê ú 1 0 ë û ç ÷ 1 è ø

Principal point offset ( p x p , ) principal point: p y y p x æ ö æ ö X X ç ÷ ç ÷ é ù ö é ù é ù f p 0 Zp f p 1 0 x ÷ x x Y Y ç ÷ ç ÷ ê ú ê ú ê ú = = Zp f p 1 0 f p 0 ÷ ç ÷ ç ÷ ê ú ê ú ê ú y y y Z Z ÷ ç ÷ ç ÷ ê ú ê ú ê ú 1 0 1 1 0 ë û ø ë û ë û ç ÷ ç ÷ 1 1 è ø è ø projection matrix calibration matrix [ I | 0] K [ ] P = K I | 0

Pixel coordinates 1 ´ 1 Pixel size: m m x y m x pixels per meter in horizontal direction, m y pixels per meter in vertical direction a b é ù é ù é ù m f p x x x x ê ú ê ú ê ú = = a b K m f p ê ú ê ú ê ú y y y y ê ú ê ú ê ú 1 1 1 ë û ë û ë û m pixels pixels/m

Camera rotation and translation • In general, the camera coordinate frame will be related to the world coordinate frame by a rotation and a translation camera coordinate system world coordinate system • Conversion from world to camera coordinate system (in non-homogeneous coordinates): ( ) ~ ~ ~ = - X R X C cam coords. of point coords. of camera center in camera frame in world frame coords. of a point in world frame

Camera rotation and translation ( ) ~ ~ ~ = - X R X C cam 3D transformation matrix (4 x 4)

Camera rotation and translation ( ) ~ ~ ~ = - X R X C cam 3D transformation matrix (4 x 4)

Camera rotation and translation 3D transformation 2D matrix (4 x 4) transformation perspective matrix (3 x 3) projection matrix (3 x 4)

Camera rotation and translation

Camera rotation and translation ~ = - t R C

[ ] P = K R t Camera parameters • Intrinsic parameters • Principal point coordinates a b é ù é ù é ù m f p x x x x • Focal length ê ú ê ú ê ú = = a b K m f p ê ú ê ú ê ú y y y y • Pixel magnification factors ê ú ê ú ê ú 1 1 1 ë û ë û ë û • Skew (non-rectangular pixels) • Radial distortion

[ ] P = K R t Camera parameters • Intrinsic parameters • Principal point coordinates • Focal length • Pixel magnification factors • Skew (non-rectangular pixels) • Radial distortion [ ] • Extrinsic parameters ~ = - P K R R C • Rotation and translation relative to world coordinate system coords. of • What is the projection of the camera center camera center? in world frame ~ [ ] é ù ~ C The camera center is the = - = PC K R R C 0 ê ú null space of the 1 ë û projection matrix!

Camera calibration [ ] X x = K R t é ù X l é ù é ù x * * * * ê ú Y ê ú ê ú ê ú l = y * * * * ê ú ê ú ê ú Z ê ú ê ú l * * * * ê ú ë û ë û 1 ë û Source: D. Hoiem

Camera calibration • Given n points with known 3D coordinates X i and known image projections x i , estimate the camera parameters X i x i P ?

Camera calibration: Linear method é ù T é ù x P X i 1 i ê ú ê ú ´ = ´ = T x PX 0 y P X 0 l x = PX ê ú ê ú i 2 i i i i i ê ú ê ú T 1 P X ë û ë û 3 i é ù - T T æ ö 0 X y X P ç ÷ i i i 1 ê ú - = T T X 0 x X P 0 ç ÷ ê ú i i i 2 ç ÷ ê ú - T T y X x X 0 P è ø ë û i i i i 3 Two linearly independent equations

Camera calibration: Linear method é ù - T T T 0 X y X 1 1 1 ê ú - æ ö T T T P X 0 x X ê ú ç ÷ 1 1 1 1 = A p 0 ê ú = P 0 ! ! ! ç ÷ 2 ê ú ç ÷ - T T T P 0 X y X è ø ê ú 3 n n n ê ú - T T T X 0 x X ë û n n n • P has 11 degrees of freedom • One 2D/3D correspondence gives us two linearly independent equations • 6 correspondences needed for a minimal solution • Homogeneous least squares: find p minimizing ||Ap|| 2 • Solution given by eigenvector of A T A with smallest eigenvalue

Camera calibration: Linear method é ù - T T T 0 X y X 1 1 1 ê ú - æ ö T T T P X 0 x X ê ú ç ÷ 1 1 1 1 Ap = 0 ê ú = P 0 ! ! ! ç ÷ 2 ê ú ç ÷ - T T T P 0 X y X è ø ê ú 3 n n n ê ú - T T T X 0 x X ë û n n n • Note: for coplanar points that satisfy Π T X =0, we will get degenerate solutions ( Π , 0 , 0 ), ( 0 , Π , 0 ), or ( 0 , 0 , Π )

Camera calibration: Linear vs. nonlinear • Linear calibration is easy to formulate and solve, but it doesn’t directly tell us the camera parameters é ù X l é ù é ù x * * * * [ ] X ê ú Y x = ê ú ê ú K R t ê ú l = y * * * * vs. ê ú ê ú ê ú Z ê ú ê ú l * * * * ê ú ë û ë û 1 ë û • In practice, non-linear methods are preferred • Write down objective function in terms of intrinsic and extrinsic parameters • Define error as sum of squared distances between measured 2D points and estimated projections of 3D points • Minimize error using Newton’s method or other non-linear optimization • Can model radial distortion and impose constraints such as known focal length and orthogonality

Homography Example Camera Center Slide from A. Efros, S. Seitz, D. Hoiem

A taste of multi-view geometry: Triangulation • Given projections of a 3D point in two or more images (with known camera matrices), find the coordinates of the point

Triangulation • Given projections of a 3D point in two or more images (with known camera matrices), find the coordinates of the point X? x 2 x 1 O 2 O 1

Triangulation • We want to intersect the two visual rays corresponding to x 1 and x 2 , but because of noise and numerical errors, they don’t meet exactly X? x 2 x 1 O 2 O 1

Triangulation: Geometric approach • Find shortest segment connecting the two viewing rays and let X be the midpoint of that segment X x 2 x 1 O 2 O 1

Triangulation: Nonlinear approach Find X that minimizes + 2 2 d ( x , P X ) d ( x , P X ) 1 1 2 2 X ? P 1 X x 2 x 1 P 2 X O 2 O 1

Triangulation: Linear approach = ´ = [x ]P X 0 l = x P X 0 x P X ´ 1 1 1 1 1 1 1 = l = ´ = [x ]P X 0 x P X x P X 0 ´ 2 2 2 2 2 2 2 Cross product as matrix multiplication: é ù - é ù 0 a a b z y x ê ú ê ú ´ = - = a b a 0 a b [ a ] b ê ú ê ú ´ z x y ê ú ê ú - a a 0 b ë û ë û y x z