Motion Lecture: Motion Juan Carlos Niebles and Ranjay Krishna Stanford Vision and Learning Lab 19-Nov-2019 1 St Stanfor ord University
CS 131 Roadmap Motion Pixels Segments Images Videos Web Neural networks Convolutions Recognition Resizing Motion Convolutional Edges Detection Segmentation Tracking neural networks Descriptors Machine learning Clustering 19-Nov-2019 2 St Stanfor ord University
What will we learn today? • Optical flow • Lucas-Kanade method • Pyramids for large motion Motion • Horn-Schunk method • Common fate • Applications 19-Nov-2019 Reading: [Szeliski] Chapters: 8.4, 8.5 [Fleet & Weiss, 2005] http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf 3 St Stanfor ord University
What will we learn today? • Optical flow • Lucas-Kanade method • Pyramids for large motion Motion • Horn-Schunk method • Common fate • Applications 19-Nov-2019 Reading: [Szeliski] Chapters: 8.4, 8.5 [Fleet & Weiss, 2005] http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf 4 St Stanfor ord University
From images to videos • A video is a sequence of frames captured over time • Now our image data is a function of space (x, y) and time (t) Motion 19-Nov-2019 5 St Stanfor ord University
Why is motion useful? Motion 19-Nov-2019 6 St Stanfor ord University
Why is motion useful? Motion 19-Nov-2019 7 St Stanfor ord University
Optical flow • Definition: optical flow is the apparent motion of brightness patterns in the image Motion • Note: apparent motion can be caused by lighting changes without any actual motion – Think of a uniform rotating sphere under fixed lighting vs. a stationary sphere under moving illumination 19-Nov-2019 Source: Silvio Savarese GOAL: Recover image motion at each pixel from optical flow 8 St Stanfor ord University
Optical flow Vector field function of the spatio-temporal image brightness variations Motion 19-Nov-2019 Picture courtesy of Selim Temizer - Learning and Intelligent Systems (LIS) Group, MIT 9 St Stanfor ord University
Estimating optical flow Motion I ( x , y , t ) I ( x , y , t+1 ) • Given two subsequent frames, estimate the apparent motion field u(x,y), v(x,y) between them • Key assumptions 19-Nov-2019 Source: Silvio Savarese • Brightness constancy: projection of the same point looks the same in every frame • Small motion: points do not move very far • Spatial coherence: points move like their neighbors 10 St Stanfor ord University
Key Assumptions: small motions Motion 19-Nov-2019 11 St Stanfor ord University * Slide from Michael Black, CS143 2003
Key Assumptions: spatial coherence Motion 19-Nov-2019 12 St Stanfor ord University * Slide from Michael Black, CS143 2003
Key Assumptions: brightness Constancy Motion 19-Nov-2019 𝐽 𝑦, 𝑧, 𝑢 = 𝐽(𝑦 + 𝑣 𝑦, 𝑧 , 𝑧 + 𝑤 𝑦, 𝑧 , 𝑢 + 1) 13 St Stanfor ord University * Slide from Michael Black, CS143 2003
The brightness constancy constraint I ( x , y , t ) I ( x , y , t+1 ) Motion • Brightness Constancy Equation: 𝐽 𝑦, 𝑧, 𝑢 = 𝐽(𝑦 + 𝑣, 𝑧 + 𝑤, 𝑢 + 1) Linearizing the right side using Taylor expansion: Image derivative along x Image derivative along t 19-Nov-2019 𝐽 𝑦 + 𝑣, 𝑧 + 𝑤, 𝑢 + 1 ≈ 𝐽 𝑦, 𝑧, 𝑢 + 𝐽 . / 𝑣 + 𝐽 0 / 𝑤 + 𝐽 1 Source: Silvio Savarese 𝐽 𝑦 + 𝑣, 𝑧 + 𝑤, 𝑢 + 1 − 𝐽 𝑦, 𝑧, 𝑢 ≈ 𝐽 . / 𝑣 + 𝐽 0 / 𝑤 + 𝐽 1 T + I t = 0 × + × + » Hence, I u I v I 0 [ ] → ∇ I ⋅ u v x y t 14 St Stanfor ord University
Filters used to find the derivatives Motion 𝐽 . 𝐽 0 𝐽 1 19-Nov-2019 15 St Stanfor ord University
The brightness constancy constraint Can we use this equation to recover image motion (u,v) at each pixel? T + I t = 0 [ ] ∇ I ⋅ u v Motion • How many equations and unknowns per pixel? •One equation (this is a scalar equation!), two unknowns (u,v) The component of the flow perpendicular to the gradient (i.e., parallel to the edge) cannot be measured ∇𝐽 gradient 19-Nov-2019 ( u , v ) Source: Silvio Savarese If ( u , v ) satisfies the equation, ( u + u ’, v + v ’) ( u ’, v ’) so does ( u+u’ , v+v’ ) if T = 0 edge [ ] ∇ I ⋅ u ' v ' 16 Stanfor St ord University
The aperture problem Motion 19-Nov-2019 Source: Silvio Savarese Actual motion 17 St Stanfor ord University
The aperture problem Motion 19-Nov-2019 Source: Silvio Savarese 18 St Stanfor ord University
The aperture problem Motion 19-Nov-2019 Source: Silvio Savarese Perceived motion 19 St Stanfor ord University
The aperture problem Motion 19-Nov-2019 Source: Silvio Savarese Actual motion 20 St Stanfor ord University
The aperture problem Motion 19-Nov-2019 Source: Silvio Savarese Perceived motion 21 St Stanfor ord University
The barber pole illusion Motion 19-Nov-2019 http://en.wikipedia.org/wiki/Barberpole_illusion 22 St Stanfor ord University
The barber pole illusion Motion 19-Nov-2019 Source: Silvio Savarese http://en.wikipedia.org/wiki/Barberpole_illusion 23 St Stanfor ord University
What will we learn today? • Optical flow • Lucas-Kanade method • Pyramids for large motion Motion • Horn-Schunk method • Common fate • Applications 19-Nov-2019 Reading: [Szeliski] Chapters: 8.4, 8.5 [Fleet & Weiss, 2005] http://www.cs.toronto.edu/pub/jepson/teaching/vision/2503/opticalFlow.pdf 24 St Stanfor ord University
Solving the ambiguity… • How to get more equations for a pixel? • Spatial coherence constraint: Assume the pixel’s neighbors have the same (u,v) • – If we use a 5x5 window, that gives us 25 equations per pixel Motion 19-Nov-2019 Source: Silvio Savarese B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo 25 vision. In Proceedings of the International Joint Conference on Artificial Intelligence , pp. 674– St Stanfor ord University 679, 1981.
Lucas-Kanade flow • Overconstrained linear system: Motion 19-Nov-2019 Source: Silvio Savarese 26 St Stanfor ord University
Lucas-Kanade flow • Overconstrained linear system Motion Least squares solution for d given by 19-Nov-2019 Source: Silvio Savarese The summations are over all pixels in the K x K window 27 St Stanfor ord University
Conditions for solvability – Optimal (u, v) satisfies Lucas-Kanade equation Motion When is This Solvable? • A T A should be invertible • A T A should not be too small due to noise 19-Nov-2019 – eigenvalues l 1 and l 2 of A T A should not be too small Source: Silvio Savarese • A T A should be well-conditioned – l 1 / l 2 should not be too large ( l 1 = larger eigenvalue) Does this remind anything to you? 28 St Stanfor ord University
M = A T A is the second moment matrix ! (Harris corner detector…) Motion • Eigenvectors and eigenvalues of A T A relate to edge direction and magnitude • The eigenvector associated with the larger eigenvalue points in 19-Nov-2019 the direction of fastest intensity change Source: Silvio Savarese • The other eigenvector is orthogonal to it 29 St Stanfor ord University
Interpreting the eigenvalues Classification of image points using eigenvalues of the second moment matrix: l 2 “Edge” l 2 >> l 1 Motion “Corner” l 1 and l 2 are large, l 1 ~ l 2 19-Nov-2019 Source: Silvio Savarese l 1 and l 2 are small “Edge” “Flat” l 1 >> l 2 region l 1 30 St Stanfor ord University
Edge Motion 19-Nov-2019 Source: Silvio Savarese – gradients very large or very small – large l 1 , small l 2 31 St Stanfor ord University
Low-texture region Motion 19-Nov-2019 Source: Silvio Savarese – gradients have small magnitude – small l 1 , small l 2 32 St Stanfor ord University
High-texture region Motion 19-Nov-2019 Source: Silvio Savarese – gradients are different, large magnitudes – large l 1 , large l 2 33 St Stanfor ord University
Errors in Lukas-Kanade What are the potential causes of errors in this procedure? – Assumed A T A is easily invertible Motion – Assumed there is not much noise in the image • When our assumptions are violated – Brightness constancy is not satisfied – The motion is not small 19-Nov-2019 – A point does not move like its neighbors • window size is too large • what is the ideal window size? 34 Stanfor St ord University * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Improving accuracy • Recall our small motion assumption I t-1 (x,y) I t-1 (x,y) Motion • This is not exact – To do better, we need to add higher order terms back in: I t-1 (x,y) • This is a polynomial root finding problem 19-Nov-2019 – Can solve using Newton’s method (out of scope for this class) – Lukas-Kanade method does one iteration of Newton’s method • Better results are obtained via more iterations 35 St Stanfor ord University * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Recommend
More recommend
