Problem Definition Using Shape Spaces for Structure from Motion Can - PowerPoint PPT Presentation

Motivation Factorization Non-Rigid Motion Occlusion Motivation Factorization Non-Rigid Motion Occlusion Problem Definition Using Shape Spaces for Structure from Motion Can we understand motion using a single camera? Sharat Chandran ViGIL Indian Institute of Technology Bombay http://www.cse.iitb.ac.in/ ∼ sharat Given 2D point tracks of landmark points from a single view March 2012 point , recover 3D pose and orientation Assumptions Joint Work: Appu Shaji, Yashoteja Prabhu, Pascal Fua, S. Ladha & other ViGIL students 2D tracks of major landmark points are provided Scaled-projective/orthographic projection model. Note: These slides are best seen with accompanying video

Motivation Factorization Non-Rigid Motion Occlusion Motivation Factorization Non-Rigid Motion Occlusion Problem Definition Problem Definition Can we understand motion using a single camera? Can we understand motion using a single camera? Given 2D point tracks of landmark points from a single view Given 2D point tracks of landmark points from a single view point , recover 3D pose and orientation point , recover 3D pose and orientation Assumptions Assumptions 2D tracks of major landmark points are provided 2D tracks of major landmark points are provided Scaled-projective/orthographic projection model. Scaled-projective/orthographic projection model.

Motivation Factorization Non-Rigid Motion Occlusion Motivation Factorization Non-Rigid Motion Occlusion Rigid Body Geometry and Motion Rank Theorem Define ˜ x ij = x ij − ¯ x i and ˜ y ij = y ij − ¯ y i where the bar notation refers to the centroid of the points in the i th frame. We have the measurement matrix  ˜ ˜  x 11 · · · x 1 p y 11 · · · y 1 p    . . .  ¯ . . . W 2F × P =   . . .     ˜ ˜ x f 1 · · · x fp   · · · y f 1 y fp The matrix ¯ W has rank 3 Object centroid based World Co-ordinate System (WCS)

Motivation Factorization Non-Rigid Motion Occlusion Motivation Factorization Non-Rigid Motion Occlusion Rank Theorem Rank Theorem Proof n 1 � i T y ij = j T x ij = i ( P j − T i ) , i ( P j − T i ) , P j = 0 n j = 1 Define ˜ x ij = x ij − ¯ x i and ˜ y ij = y ij − ¯ y i where the bar notation n refers to the centroid of the points in the i th frame. We have the i ( P j − T i ) − 1 � i T i T ˜ = i ( P m − T i ) x ij measurement matrix n m = 1 n  ˜ ˜  x 11 · · · x 1 p i ( P j − T i ) − 1 � j T j T ˜ y ij = i ( P m − T i ) y 11 · · · y 1 p n    . . .  m = 1 ¯ . . . W 2F × P =   . . . i T y ij = j T   ˜ x ij = i P j ˜ i P j   ˜ ˜ x f 1 · · · x fp   ¯ W = RS · · · y f 1 y fp i T   1 j T The matrix ¯ W has rank 3   1   � � R = . . . S = P 1 P 2 . . . P N     i T   N j T N

Motivation Factorization Non-Rigid Motion Occlusion Motivation Factorization Non-Rigid Motion Occlusion Rigid Body Geometry and Motion Rigid Body Geometry and Motion Without noise W is atmost of rank three Without noise W is atmost of rank three Using SVD, W = O 1 Σ O 2 where, Using SVD, W = O 1 Σ O 2 where, O 1 , O 2 are column orthogonal matrices and Σ is a diagonal O 1 , O 2 are column orthogonal matrices and Σ is a diagonal matrix with singular values in non-decreasing order matrix with singular values in non-decreasing order ′ ′ O ′ ′′ ′′ O ′′ ′ ′ O ′ ′′ ′′ O ′′ O 1 Σ O 2 = O 1 Σ 2 + O 1 Σ 2 where, O 1 Σ O 2 = O 1 Σ 2 + O 1 Σ 2 where, ′ ′ ′ ′ O 1 has first three columns of O 1 , O 2 has first three rows of O 1 has first three columns of O 1 , O 2 has first three rows of ′ is 3 × 3 matrix with 3 largest non-singular values. ′ is 3 × 3 matrix with 3 largest non-singular values. O 2 and Σ O 2 and Σ The second term is completely due to noise and can be The second term is completely due to noise and can be eliminated eliminated ′ � 1 / 2 ′ � 1 / 2 ′ � 1 / 2 ′ � 1 / 2 � � � � ˆ and ˆ ˆ and ˆ ′ ′ ′ ′ R = O Σ S = Σ O R = O Σ S = Σ O 1 2 1 2

Motivation Factorization Non-Rigid Motion Occlusion Motivation Factorization Non-Rigid Motion Occlusion Rigid Body Geometry and Motion Rigid Body Geometry and Motion Without noise W is atmost of rank three Without noise W is atmost of rank three Using SVD, W = O 1 Σ O 2 where, Using SVD, W = O 1 Σ O 2 where, O 1 , O 2 are column orthogonal matrices and Σ is a diagonal O 1 , O 2 are column orthogonal matrices and Σ is a diagonal matrix with singular values in non-decreasing order matrix with singular values in non-decreasing order ′ ′ O ′ ′′ ′′ O ′′ ′ ′ O ′ ′′ ′′ O ′′ O 1 Σ O 2 = O 1 Σ 2 + O 1 Σ 2 where, O 1 Σ O 2 = O 1 Σ 2 + O 1 Σ 2 where, ′ ′ ′ ′ O 1 has first three columns of O 1 , O 2 has first three rows of O 1 has first three columns of O 1 , O 2 has first three rows of ′ is 3 × 3 matrix with 3 largest non-singular values. ′ is 3 × 3 matrix with 3 largest non-singular values. O 2 and Σ O 2 and Σ The second term is completely due to noise and can be The second term is completely due to noise and can be eliminated eliminated ′ � 1 / 2 ′ � 1 / 2 ′ � 1 / 2 ′ � 1 / 2 � � � � ˆ and ˆ ˆ and ˆ ′ ′ ′ ′ R = O Σ S = Σ O R = O Σ S = Σ O 1 2 1 2

Motivation Factorization Non-Rigid Motion Occlusion Motivation Factorization Non-Rigid Motion Occlusion Rigid Body Geometry and Motion Rigid Body Geometry and Motion Without noise W is atmost of rank three Solution is not unique any invertible 3 × 3 , Q matrix can be RQ ) and S = ( Q − 1 ˆ written as R = (ˆ Using SVD, W = O 1 Σ O 2 where, S ) O 1 , O 2 are column orthogonal matrices and Σ is a diagonal R is a linear transformation of R , similarly ˆ ˆ S is a linear matrix with singular values in non-decreasing order transformation of S . ′ ′ O ′ ′′ ′′ O ′′ O 1 Σ O 2 = O 1 Σ 2 + O 1 Σ 2 where, Using the following orthonormality constraints we can find ′ ′ O 1 has first three columns of O 1 , O 2 has first three rows of R and S ′ is 3 × 3 matrix with 3 largest non-singular values. O 2 and Σ ˆ f QQ T ˆ i T i f = 1 The second term is completely due to noise and can be ˆ f QQ T ˆ j T j f = 1 eliminated ′ � 1 / 2 ′ � 1 / 2 ˆ f QQ T ˆ i T � � j f = 0 (1) ˆ and ˆ ′ ′ R = O Σ S = Σ O 1 2

Motivation Factorization Non-Rigid Motion Occlusion Tomasi Kanade Factorisation (Recap) Rigid Body Geometry and Motion . . . Solution is not unique any invertible 3 × 3 , Q matrix can be RQ ) and S = ( Q − 1 ˆ written as R = (ˆ S ) R is a linear transformation of R , similarly ˆ ˆ S is a linear transformation of S . Using the following orthonormality constraints we can find R and S ˆ f QQ T ˆ i T i f = 1 ˆ f QQ T ˆ j T j f = 1 ˆ f QQ T ˆ i T j f = 0 (1)

Tomasi Kanade Factorisation (Recap) Tomasi Kanade Factorisation (Recap) . . . . . .     27 61 96 27 61 96 · · · · · · 97 53 122 97 53 122 · · · · · ·         28 62 97 28 62 97 · · · · · ·         97 53 122 97 53 122 · · · · · ·      . . . .   . . . .  . . . . . . . .     . . . . . . . .         . . . . . . . . . . . . . . . .     . . . . . . . .         94 ? 131 94 ? 131  · · ·   · · ·  109 ? 135 109 ? 135 · · · · · · W W

Tomasi Kanade Factorisation (Recap) Tomasi Kanade Factorisation (Recap) . . . . . .     27 61 96 27 61 96 · · · · · · 97 53 122 97 53 122 · · · · · ·         28 62 97 28 62 97 · · · · · ·         97 53 122 97 53 122 · · · · · ·      . . . .   . . . .  . . . . . . . .     . . . . . . . .         . . . . . . . . . . . . . . . .     . . . . . . . .         94 ? 131 94 ? 131  · · ·   · · ·  109 ? 135 109 ? 135 · · · · · · W W

Tomasi Kanade Factorisation (Recap) Tomasi Kanade Factorisation (Recap) . . . . . .     27 61 96 27 61 96 · · · · · · 97 53 122 97 53 122 · · · · · ·         28 62 97 28 62 97 · · · · · ·         97 53 122 97 53 122 · · · · · ·      . . . .   . . . .  . . . . . . . .     . . . . . . . .         . . . . . . . . . . . . . . . .     . . . . . . . .         94 ? 131 94 ? 131  · · ·   · · ·  109 ? 135 109 ? 135 · · · · · · W W