Camera Matrix 16-385 Computer Vision (Kris Kitani) Carnegie Mellon - PowerPoint PPT Presentation

Camera Matrix 16-385 Computer Vision (Kris Kitani) Carnegie Mellon University

2D to 2D Transform (last session)

3D object 3D to 2D Transform (today) 2D to 2D Transform (last session)

A camera is a mapping between the 3D world and a 2D image

A camera is a mapping between the 3D world and a 2D image x = PX 2D image camera 3D world point matrix point What do you think the dimensions are?

x = PX   X     X p 1 p 2 p 3 p 4 Y    = Y p 5 p 6 p 7 p 8      Z   Z p 9 p 10 p 11 p 12 1 homogeneous Camera homogeneous image matrix world point 3 x 1 3 x 4 4 x 1

The pinhole camera y image plane X x x z camera center principal axis z = f principal point What is the equation for image coordinate x (in terms of X )?

y What is the equation for image coordinate x (in terms of X )? image plane X Y ? z f Z

y image plane X Y fY Z z f Z Z ] > 7! [ fX/Z fY/Z ] > [ X Y

Pinhole camera geometry y image plane X x x z camera center principal axis z = f principal point What is the camera matrix P for a pinhole camera model? x = PX

Relationship from similar triangles… Z ] > 7! [ fX/Z fY/Z ] > [ X Y generic camera model   X     X p 1 p 2 p 3 p 4 Y    = Y p 5 p 6 p 7 p 8      Z   Z p 9 p 10 p 11 p 12 1 What does the pinhole camera model look like?   ? ? ? ? P = ? ? ? ?   ? ? ? ?

Relationship from similar triangles… Z ] > 7! [ fX/Z fY/Z ] > [ X Y generic camera model   X     X p 1 p 2 p 3 p 4 Y    = Y p 5 p 6 p 7 p 8      Z   Z p 9 p 10 p 11 p 12 1 What does the pinhole camera model look like?   f 0 0 0 f P = 0 0 0   0 0 1 0

  f 0 0 0 f P = 0 0 0   0 0 1 0 Camera origin and image origin might be different CCD array camera coordinate p system image coordinate system

CCD array camera coordinate p system image coordinate system   f p x 0 0 f p y P = 0 0   0 0 1 0 Accounts for different origins

In general, the camera and image sensor have different coordinate systems X world point O image x image point O camera

In general, there are three different coordinate systems… X world point O image O world x image point O camera so you need the know the transformations between them

Can be decomposed into two matrices     f p x 0 1 0 0 0 f p y P = 0 0 1 0 0     0 0 1 0 0 1 0 (3 x 3) (3 x 4) P = K [ I | 0 ]   f p x 0 f p y K = 0 calibration matrix   0 0 1

Assumes that the camera and world share the same coordinate system     f p x 0 1 0 0 0 f p y P = 0 0 1 0 0     0 0 1 0 0 1 0 What if they are different? How do we align them? y c x c y w Camera coordinate system x w z c World z w coordinate system

Assumes that the camera and world share the same coordinate system     f p x 0 1 0 0 0 f p y P = 0 0 1 0 0     0 0 1 0 0 1 0 What if they are different? How do we align them? y c x c y w Camera coordinate system x w z c R t World z w coordinate 3R rotation and translation to align axis system

y c X w x c y w Camera coordinate system x w z c z w World coordinate system

y c X w x c y w Camera coordinate system x w z c C z w World Coordinate of the coordinate camera center in the system world coordinate frame

X c y c X w x c y w Camera coordinate system x w z c C z w World Coordinate of the coordinate camera center in the ( X w − C ) system world coordinate frame ( X w − C ) Translate

Why aren’t X c the points aligned? y c X w x c y w Camera coordinate system x w z c C z w World coordinate system ( X w − C ) Translate

What happens to points after alignment? Rotate X c R y c X w x c y w Camera coordinate system x w z c C z w World coordinate system R ( X w − C ) Rotate Translate

In inhomogeneous coordinates: X c = R ( X w − C ) Optionally in homogeneous coordinates: 2 3 2 3 X c X w  R � − RC Y c Y w 6 7 6 7 5 = 6 7 6 7 0 Z c 1 Z w 4 4 5 1 1 (4 × 4) General mapping of a pinhole camera P = KR [ I | − C ]

Quiz What is the meaning of each matrix of the camera matrix decomposition? P = KR [ I | − C ] 3x3 intrinsics

Quiz What is the meaning of each matrix of the camera matrix decomposition? P = KR [ I | − C ] 3x3 3x3 intrinsics 3D rotation

Quiz What is the meaning of each matrix of the camera matrix decomposition? P = KR [ I | − C ] 3x3 3x3 3x3 intrinsics 3D rotation identity

Quiz What is the meaning of each matrix of the camera matrix decomposition? P = KR [ I | − C ] 3x3 3x3 3x3 3x1 intrinsics 3D rotation identity 3D translation

General mapping of a pinhole camera P = KR [ I | − C ] (translate first then rotate) Another way to write the mapping P = K [ R | t ] where t = − RC (rotate first then translate)

Quiz The camera matrix relates what two quantities?

Quiz The camera matrix relates what two quantities? x = PX

Quiz The camera matrix relates what two quantities? x = PX 3D points to 2D image points

Quiz The camera matrix relates what two quantities? x = PX 3D points to 2D image points The camera matrix can be decomposed into?

Quiz The camera matrix relates what two quantities? x = PX 3D points to 2D image points The camera matrix can be decomposed into? P = K [ R | t ]

Quiz The camera matrix relates what two quantities? x = PX 3D points to 2D image points The camera matrix can be decomposed into? P = K [ R | t ] intrinsic and extrinsic parameters

Generalized pinhole camera model P = K [ R | t ]     f p x r 1 r 2 r 3 t 1 0 f p y r 4 r 5 r 6 t 2 P = 0     r 7 r 8 r 9 t 3 0 0 1 intrinsic parameters extrinsic parameters     t 1 r 1 r 2 r 3 R = t = t 2 r 4 r 5 r 6     t 3 r 7 r 8 r 9 3D rotation 3D translation

Why do we need P ? to properly relate world points to image points (by taking into account different coordinate systems)