Tracking Feature Windows COMPSCI 527 Computer Vision COMPSCI 527 - - PowerPoint PPT Presentation

Tracking Feature Windows

COMPSCI 527 — Computer Vision

Outline

1 Local Motion Estimation 2 Window Tracking 3 The Lucas-Kanade Tracker 4 Good Features to Track

Motion Estimation

• Given: Two (black-and-white) images f(x) and g(x) of the

same scene

• Assumption 1: Constant appearance
• Assumption 2: All displacements d(x) between

corresponding points are small (more on this later)

• Want: Displacements d(x), wherever they can be computed

(more on this later)

• Aperture problem: Image derivatives at x only yield one

scalar equation in the two unknowns in d(x)

• Therefore, we can only estimate several displacement

vectors d simultaneously, under an additional constraint that relates them

Local Estimation Methods

• Global methods estimate all displacements in an image
• They assume some version of smoothness of motion in

space

• Local methods:
• The image displacement d in a small window around a pixel

x is assumed to be constant (extreme local smoothness)

• Write one constancy-of-appearance equation for every pixel

in the window

• Solve for the one displacement that satisfies all these

equations as much as possible (in the LSE sense)

• These techniques are called (feature) window tracking

methods

Key Questions

• Which windows can be tracked?
• How can these windows be tracked?
• Logically, it’s best to answer the second question first
• The answer to the first question is then “wherever the

tracking method works well”

• We’ll come back to the first question later

Window Tracking

• Given images f(x) and g(x), a point xf in image f, and a

square window W(xf) of side-length 2h + 1 centered at xf, what are the coordinates xg = xf + d∗(xf) of the corresponding window’s center in image g?

• d∗(xf) 2 R2 is the displacement of that point feature
• Assumption 1: The whole window translates
• Assumption 2: d∗(xf) ⌧ h

got

f

E

t

General Window Tracking Strategy

• Let w(x) be the indicator function of W(0)
• Measure the dissimilarity between W(xf) and a candidate

window in g with the loss L(xf, d) = X

x

[g(x + d) f(x)]2 w(x xf)

• Minimize L(xf, d) over d:

d∗(xf) = arg mind∈R L(xf, d)

• The search range R ✓ R2 is a square centered at the origin
• Half-side of R is ⌧ h

• _0

s

m

Obvious Failure Points

• Multiple motions in the same window

(Less dramatic cases arise as well)

• Actual motion large compared with h

(We’ll come back to this later)

70

ME

A Softer Window

• Make w(x) a (truncated) Gaussian rather than a box

w(x) / ( e

if |x1|  h and |x2|  h

• therwise
• Dissimilarity L(xf, d) = P

x[g(x + d) f(x)]2 w(x xf)

depends more on what’s around the window center

• Reduces the effects of multiple motions
• Does not eliminate them

O

EA

I

How to Minimize L(xf, d)?

• Method 1: Exhaustive search over a grid of d
• Advantages: Unlikely to be trapped in local minima
• Disadvantage: Fixed resolution
• Accurate motion is sometimes necessary
• Example: 3D reconstruction is ill-posed
• Small errors accumulate: drift
• Using a very fine grid would be very expensive
• Exhaustive search may provide a good initialization

imma

LET

How to Minimize L(xf, d)?

• Method 2: Use a gradient-descent method
• Search space is small (d 2 R2), so we can use Newton’s

method for faster convergence

• Compute gradient and Hessian of

L(d) = P

x[g(x + d) f(x)]2 w(x xf)

(omitted xf from arguments of L for simplicity)

• Take Newton steps
• Technical difficulty: the unknown d appears inside g(x + d),

and computing a Hessian would require computing second-order derivatives of an image, which is available

• nly through its pixels
• Second derivatives of images are very sensitive to noise

sa

The Lucas-Kanade Tracker, 1981

• Instead of computing the Hessian of

L(d) = P

x[g(x + d) f(x)]2 w(x xf),

linearize g(x + d) ⇡ g(x) + [rg(x)]Td

• This brings d “outside g”
• L(d) is now quadratic in d, and we can find a minimum in

closed form

• The derivatives needed for that no longer go through g,

since d appears linearly

• Only differentiate the image once to get rg(x)
• Since the solution relies on an approximation, we iterate
• This method works for losses that are sums of squares, and

is called the Newton-Raphson method

Lucas-Kanade Derivation

• Let dt the solution at iteration t. We seek dt+1 = dt + s by

minimizing the following over s (with d0 = 0): L(dt + s) = P

x[g(x + dt + s) f(x)]2 w(x xf)

• For simplicity, define gt(x)

def

= g(x + dt + s) so that gt(x + s) ⇡ gt(x) + [rgt(x)]Ts (linearization) and L(dt + s) = X

x

[gt(x + s) f(x)]2 w(x xf) ⇡ X

x

[gt(x) + [rgt(x)]Ts f(x)]2 w(x xf) , a quadratic function of s

do

f

d

AT 5

O

dtt_NOTATION

O

Lucas-Kanade Derivation, Cont’d

• Gradient of

L(dt + s) ⇡ P

x[gt(x) + [rgt(x)]Ts f(x)]2 w(x xf) is

rL(dt+s) ⇡ 2 P

x rgt(x){gt(x)+[rgt(x)]Tsf(x)} w(xxf)

• Setting to zero yields

E

tEEI

q0geE IfE

gtc

w E xf

b

The Core System of Lucas-Kanade

Linear, 2 ⇥ 2 system As = b where A = X

x

rgt(x)[rgt(x)]T w(x xf) and b = X

x

rgt(x)[f(x) gt(x)] w(x xf) .

• Iteration t yields st
• Image gt is shifted by st by subpixel interpolation
• This shift makes f and gt more similar within the windows
• Repeat until convergence
• Overall displacement is d∗ = P

t st

GRAM MATRIX

A is 2 2

b is 2 1

2772

gtEHgftdi.is

O O

s A 5

Good Features to Track

• How to select which windows to track?
• We keep solving As = b where

A = P

x rgt(x)[rgt(x)]T w(x xf)

• So we want A to be far from degenerate
• That is, λmin(A) λ0 for some λ0
• The matrix A keeps changing: gt(x)

def

= g(x + dt + s)

• gt is g shifted to match f
• So the A matrices are not far from

Af(xf)

def

= P

x rf(x)[rf(x)]T w(x xf)

• Pick xf for tracking if Af(xf) is far from degenerate
• That is, λmin(Af(xf)) λ0

f

g

Examples of Degenerate Af(xf)

rf(x) =  g1(x)

• )

rf(x)[rf(x)]T =  γ11(x)

• Af(xf) = P

x rf(x)[rf(x)]T w(x xf) =

 a11(xf)

• COMPSCI 527 — Computer Vision

Feature Selection Algorithm

• Compute λmin(Af(x)) for all x

(eigenvalues of 2 ⇥ 2 matrix can be computed in closed form)

• While max λmin > λ0 and want more points:
• Pick ˆ

x with highest λmin

• Remove x whose windows contain ˆ

x

I

What if Motion is Large?

E

FEET