Tracking Feature Windows COMPSCI 527 Computer Vision COMPSCI 527 - PowerPoint PPT Presentation

Tracking Feature Windows COMPSCI 527 — Computer Vision COMPSCI 527 — Computer Vision Tracking Feature Windows 1 / 19

Outline 1 Local Motion Estimation 2 Window Tracking 3 The Lucas-Kanade Tracker 4 Good Features to Track COMPSCI 527 — Computer Vision Tracking Feature Windows 2 / 19

Local Motion Estimation Motion Estimation • Given: Two (black-and-white) images f ( x ) and g ( x ) of the same scene • Assumption 1: Constant appearance • Assumption 2: All displacements d ( x ) between corresponding points are small (more on this later) • Want: Displacements d ( x ) , wherever they can be computed (more on this later) • Aperture problem: Image derivatives at x only yield one scalar equation in the two unknowns in d ( x ) • Therefore, we can only estimate several displacement vectors d simultaneously, under an additional constraint that relates them COMPSCI 527 — Computer Vision Tracking Feature Windows 3 / 19

Local Motion Estimation Local Estimation Methods • Global methods estimate all displacements in an image • They assume some version of smoothness of motion in space • Local methods: • The image displacement d in a small window around a pixel x is assumed to be constant (extreme local smoothness) • Write one constancy-of-appearance equation for every pixel in the window • Solve for the one displacement that satisfies all these equations as much as possible (in the LSE sense) • These techniques are called (feature) window tracking methods COMPSCI 527 — Computer Vision Tracking Feature Windows 4 / 19

Local Motion Estimation Key Questions • Which windows can be tracked? • How can these windows be tracked? • Logically, it’s best to answer the second question first • The answer to the first question is then “wherever the tracking method works well” • We’ll come back to the first question later COMPSCI 527 — Computer Vision Tracking Feature Windows 5 / 19

Window Tracking Window Tracking • Given images f ( x ) and g ( x ) , a point x f in image f , and a square window W ( x f ) of side-length 2 h + 1 centered at x f , what are the coordinates x g = x f + d ∗ ( x f ) of the corresponding window’s center in image g ? • d ∗ ( x f ) 2 R 2 is the displacement of that point feature • Assumption 1: The whole window translates • Assumption 2: d ∗ ( x f ) ⌧ h got f t E COMPSCI 527 — Computer Vision Tracking Feature Windows 6 / 19

Window Tracking General Window Tracking Strategy • Let w ( x ) be the indicator function of W ( 0 ) • Measure the dissimilarity between W ( x f ) and a candidate window in g with the loss o_0 [ g ( x + d ) � f ( x )] 2 w ( x � x f ) X L ( x f , d ) = s x m • Minimize L ( x f , d ) over d : d ∗ ( x f ) = arg min d ∈ R L ( x f , d ) • The search range R ✓ R 2 is a square centered at the origin • Half-side of R is ⌧ h COMPSCI 527 — Computer Vision Tracking Feature Windows 7 / 19

Window Tracking Obvious Failure Points • Multiple motions in the same window ME 70 (Less dramatic cases arise as well) • Actual motion large compared with h (We’ll come back to this later) COMPSCI 527 — Computer Vision Tracking Feature Windows 8 / 19

Window Tracking A Softer Window • Make w ( x ) a (truncated) Gaussian rather than a box ( 2 k x k 1 2 ( σ ) e if | x 1 |  h and | x 2 |  h w ( x ) / 0 otherwise O x [ g ( x + d ) � f ( x )] 2 w ( x � x f ) • Dissimilarity L ( x f , d ) = P depends more on what’s around the window center • Reduces the effects of multiple motions • Does not eliminate them I EA COMPSCI 527 — Computer Vision Tracking Feature Windows 9 / 19

The Lucas-Kanade Tracker How to Minimize L ( x f , d ) ? • Method 1: Exhaustive search over a grid of d • Advantages: Unlikely to be trapped in local minima imma LET • Disadvantage: Fixed resolution • Accurate motion is sometimes necessary • Example: 3D reconstruction is ill-posed • Small errors accumulate: drift • Using a very fine grid would be very expensive • Exhaustive search may provide a good initialization COMPSCI 527 — Computer Vision Tracking Feature Windows 10 / 19

The Lucas-Kanade Tracker How to Minimize L ( x f , d ) ? • Method 2: Use a gradient-descent method • Search space is small ( d 2 R 2 ), so we can use Newton’s method for faster convergence • Compute gradient and Hessian of sa x [ g ( x + d ) � f ( x )] 2 w ( x � x f ) 0 L ( d ) = P (omitted x f from arguments of L for simplicity) • Take Newton steps • Technical difficulty: the unknown d appears inside g ( x + d ) , and computing a Hessian would require computing second-order derivatives of an image, which is available only through its pixels • Second derivatives of images are very sensitive to noise COMPSCI 527 — Computer Vision Tracking Feature Windows 11 / 19

The Lucas-Kanade Tracker The Lucas-Kanade Tracker, 1981 • Instead of computing the Hessian of x [ g ( x + d ) � f ( x )] 2 w ( x � x f ) , L ( d ) = P 0 linearize g ( x + d ) ⇡ g ( x ) + [ r g ( x )] T d • This brings d “outside g ” • L ( d ) is now quadratic in d , and we can find a minimum in closed form • The derivatives needed for that no longer go through g , since d appears linearly • Only differentiate the image once to get r g ( x ) • Since the solution relies on an approximation, we iterate • This method works for losses that are sums of squares, and is called the Newton-Raphson method COMPSCI 527 — Computer Vision Tracking Feature Windows 12 / 19

The Lucas-Kanade Tracker AT 5 d f d o Lucas-Kanade Derivation O • Let d t the solution at iteration t . We seek d t + 1 = d t + s by minimizing the following over s (with d 0 = 0 ): dtt_NOTATION x [ g ( x + d t + s ) � f ( x )] 2 w ( x � x f ) L ( d t + s ) = P def • For simplicity, define g t ( x ) = g ( x + d t + s ) so that g t ( x + s ) ⇡ g t ( x ) + [ r g t ( x )] T s (linearization) and O [ g t ( x + s ) � f ( x )] 2 w ( x � x f ) X L ( d t + s ) = x [ g t ( x ) + [ r g t ( x )] T s � f ( x )] 2 w ( x � x f ) , X 0 ⇡ x a quadratic function of s COMPSCI 527 — Computer Vision Tracking Feature Windows 13 / 19

The Lucas-Kanade Tracker Lucas-Kanade Derivation, Cont’d • Gradient of 0 x [ g t ( x ) + [ r g t ( x )] T s � f ( x )] 2 w ( x � x f ) is L ( d t + s ) ⇡ P x r g t ( x ) { g t ( x )+[ r g t ( x )] T s � f ( x ) } w ( x � x f ) r L ( d t + s ) ⇡ 2 P • Setting to zero yields tEEI E w E xf q0geE IfE gtc b COMPSCI 527 — Computer Vision Tracking Feature Windows 14 / 19

The Lucas-Kanade Tracker GRAM MATRIX The Core System of Lucas-Kanade Linear, 2 ⇥ 2 system A is 2 2 A s = b b is 2 1 where r g t ( x )[ r g t ( x )] T w ( x � x f ) X A = g tEH gftdi.is 2772 x and O O X b = r g t ( x )[ f ( x ) � g t ( x )] w ( x � x f ) . x s A 5 0 • Iteration t yields s t • Image g t is shifted by s t by subpixel interpolation • This shift makes f and g t more similar within the windows • Repeat until convergence • Overall displacement is d ∗ = P t s t COMPSCI 527 — Computer Vision Tracking Feature Windows 15 / 19

Good Features to Track Good Features to Track f • How to select which windows to track? • We keep solving A s = b where x r g t ( x )[ r g t ( x )] T w ( x � x f ) 0 A = P • So we want A to be far from degenerate g • That is, λ min ( A ) � λ 0 for some λ 0 def • The matrix A keeps changing: g t ( x ) = g ( x + d t + s ) • g t is g shifted to match f • So the A matrices are not far from def x r f ( x )[ r f ( x )] T w ( x � x f ) A f ( x f ) = P • Pick x f for tracking if A f ( x f ) is far from degenerate • That is, λ min ( A f ( x f )) � λ 0 COMPSCI 527 — Computer Vision Tracking Feature Windows 16 / 19

Good Features to Track Examples of Degenerate A f ( x f )  g 1 ( x )  γ 11 ( x ) � � 0 r f ( x )[ r f ( x )] T = r f ( x ) = ) 0 0 0  a 11 ( x f ) � 0 x r f ( x )[ r f ( x )] T w ( x � x f ) = A f ( x f ) = P 0 0 COMPSCI 527 — Computer Vision Tracking Feature Windows 17 / 19

Good Features to Track Feature Selection Algorithm I • Compute λ min ( A f ( x )) for all x (eigenvalues of 2 ⇥ 2 matrix can be computed in closed form) • While max λ min > λ 0 and want more points: • Pick ˆ x with highest λ min • Remove x whose windows contain ˆ x (video) COMPSCI 527 — Computer Vision Tracking Feature Windows 18 / 19

Good Features to Track What if Motion is Large? FEET E COMPSCI 527 — Computer Vision Tracking Feature Windows 19 / 19