Lecture 20: Motion estimation Most slides from S. Lazebnik, which - PowerPoint PPT Presentation

Lecture 20: Motion estimation Most slides from S. Lazebnik, which are based on other slides from S. Seitz, R. Szeliski, M. Pollefeys 1

Announcements • PS9 and PS10 deadlines extended to April 21 2

Optical flow What moved where? Source: S. Lazebnik 3

Motion is a powerful perceptual cue Source: S. Lazebnik 4

Motion is a powerful perceptual cue G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973. Source: S. Lazebnik 5

Motion is a powerful perceptual cue G. Johansson, “Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973. Source: S. Lazebnik 6

Motion field Source: S. Lazebnik 7

Estimating optical flow I ( x , y , t –1) I ( x , y , t ) • Given two subsequent frames, estimate the motion field u(x,y) and v(x,y) between them • Common assumptions • Brightness constancy: projection of the same point looks the same in every frame • Small motion: points do not move very far • Spatial coherence: points move like their neighbors Source: S. Lazebnik 8

<latexit sha1_base64="DhWrnIE7gdWIgy9PNwT2+fX+E=">ACOXicbVBLSwMxEM76tr6qHr0MFqHFWnZV0IsgelHwUMGq0K4lm6YazD5IJsWy9G958V94E7x4UMSrf8BsreBrYMj3fTPDZL4gkUKj6z4Q8Mjo2PjE5O5qemZ2bn8/MKpjo1ivMZiGavzgGouRcRrKFDy80RxGgaSnwX+1n9rMOVFnF0gt2E+yG9jERbMIpWauarjZDiFaMyPeoVTRk6JdiBhjZhM70pQ7dXPyxmbxlwzSvBGlgKq2D6YsmJZ0vgiX/Yr2ZL7gVtx/wF3gDUCDqDbz941WzEzI2Sal3AT9lCoUTPJermE0Tyi7pe8bmFEQ679tH95D1as0oJ2rGxGCH31+0RKQ627YWA7szv171om/lerG2xv+6mIEoM8Yp+L2kYCxpDZC2hOEPZtYAyJexfgV1Rlas3PWBO/3yX/B6XrF26i4x5uF3b2BHRNkiSyTIvHIFtklB6RKaoSRW/JInsmLc+c8Oa/O2frkDOYWSQ/wn/ALA2pq8=</latexit> The brightness constancy constraint I ( x , y , t –1) I ( x , y , t ) Simple loss function [Lucas & Kanade 1981]. Find flow that minimizes: X [ I ( x, y, t − 1) − I ( x + u ( x, y ) , y + v ( x, y ) , t )] 2 L ( u, v ) = x,y Source: S. Lazebnik

The brightness constancy constraint I ( x , y , t –1) I ( x , y , t ) Brightness Constancy Equation: I ( x , y , t 1 ) I ( x u , y v ) ( x , y ) ( x , y ), t − = + + Derivative in y direction Linearizing the right side using Taylor expansion: I ( x , y , t 1 ) I ( x , y , t ) I u ( x , y ) I v ( x , y ) − ≈ + + x y I u I v I 0 + + ≈ Hence, x y t Source: S. Lazebnik 10

The brightness constancy constraint I u I v I 0 + + = x y t • How many equations and unknowns per pixel? • One equation, two unknowns • What does this constraint mean? I ( u , v ) I 0 ∇ ⋅ + = t • The component of the flow perpendicular to the gradient (i.e., parallel to the edge) is unknown! gradient ( u , v ) If ( u , v ) satisfies the equation, 
 so does ( u+u’ , v+v’ ) if I ( u ' , v ' ) 0 ( u ’, v ’) ∇ ⋅ = ( u + u ’, v + v ’) edge Source: S. Lazebnik 11

The aperture problem Perceived motion Source: S. Lazebnik 12

The aperture problem Actual motion Source: S. Lazebnik 13

The barber pole illusion http://en.wikipedia.org/wiki/Barberpole_illusion Source: S. Lazebnik 14

The barber pole illusion http://en.wikipedia.org/wiki/Barberpole_illusion Source: S. Lazebnik 15

Solving the aperture problem • How to get more equations for a pixel? • Spatial coherence constraint: assume the pixel’s neighbors have the same (u,v) • E.g., if we use a 5x5 window, that gives us 25 equations per pixel I ( x ) [ u , v ] I ( x ) 0 ∇ ⋅ + = i t i I ( x ) I ( x ) I ( x ) ⎡ ⎤ ⎡ ⎤ x 1 y 1 t 1 ⎢ ⎥ ⎢ ⎥ I ( x ) I ( x ) u I ( x ) ⎡ ⎤ x 2 y 2 ⎢ ⎥ t 2 ⎢ ⎥ = − ⎢ ⎥ v … … … ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ � � � ⎢ ⎥ ⎢ ⎥ I ( x ) I ( x ) I ( x ) ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ x n y n t n [Lucas & Kanade 1981] Source: S. Lazebnik 16

Lucas-Kanade flow Least squares problem: I ( x ) I ( x ) I ( x ) ⎡ ⎤ ⎡ ⎤ x 1 y 1 t 1 ⎢ ⎥ ⎢ ⎥ I ( x ) I ( x ) I ( x ) u ⎡ ⎤ x 2 y 2 ⎢ ⎥ t 2 ⎢ ⎥ = − ⎢ ⎥ v ⎢ ⎥ … ⎢ ⎥ … … ⎣ ⎦ � � � ⎢ ⎥ ⎢ ⎥ I ( x ) I ( x ) I ( x ) ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ x n y n t n When is this system solvable? [Lucas & Kanade 1981] Source: S. Lazebnik 17

Lucas-Kanade optical flow I ( x ) I ( x ) I ( x ) ⎡ ⎤ ⎡ ⎤ x 1 y 1 t 1 A d b ⎢ ⎥ ⎢ ⎥ = I ( x ) I ( x ) I ( x ) u ⎡ ⎤ x 2 y 2 ⎢ ⎥ t 2 ⎢ ⎥ = − ⎢ ⎥ n 2 2 1 n 1 v ⎢ ⎥ × × × ⎢ ⎥ … … … ⎣ ⎦ � � � ⎢ ⎥ ⎢ ⎥ I ( x ) I ( x ) I ( x ) ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ x n y n t n T T • Solution given by ( A A)d A b = I I I I I I u ∑ ∑ ∑ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ x x x y x t M = A T A is the “second = − ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ I I I I I I v ∑ ∑ ∑ moment” matrix ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ x y y y y t (summations are over all pixels in the window) [Lucas & Kanade 1981] Source: S. Lazebnik 18

Analyzing the second moment matrix Estimation of optical flow is well-conditioned • precisely for regions with high “cornerness”: λ 2 “Edge” 
 λ 2 >> λ 1 “Corner” 
 λ 1 and λ 2 are large, 
 λ 1 ~ λ 2 Eigenvalues “Edge” 
 λ 1 and λ 2 are small “Flat” λ 1 >> λ 2 region λ 1 Source: S. Lazebnik 19

Conditions for solvability Source: S. Lazebnik 20

Conditions for solvability Source: S. Lazebnik 21

Lucas-Kanade flow example Input frames Output Source: MATLAB Central File Exchange Source: S. Lazebnik 22

Fixing the errors in Lucas-Kanade • The motion is large (larger than a pixel) • Iterative refinement • Multi-resolution (coarse-to-fine) estimation 
 • Local ambiguity • Smooth using graphical model refinement Source: S. Lazebnik 23

Large motions Source: Khurram Hassan-Shafique CAP5415 Computer Vision 2003 24