[PPT] - Image Motion COMPSCI 527 Computer Vision COMPSCI 527 Computer PowerPoint Presentation

SLIDE 1

Image Motion

COMPSCI 527 — Computer Vision

COMPSCI 527 — Computer Vision Image Motion 1 / 12

SLIDE 2

Outline

1 Image Motion 2 Occlusion, Correspondence, Motion Boundaries 3 Constancy of Appearance 4 Motion Field and Optical Flow 5 The Aperture Problem 6 Estimating the Motion Field

COMPSCI 527 — Computer Vision Image Motion 2 / 12

SLIDE 3

Image Motion

Sensor Irradiance → Pixel Values

sensor lens aperture

t T Te P

s

P y x

Irradiance is the patterns of colors on the image sensor,

compressed as an (r, g, b) triple e(x, t)

A pixel value is a noisy and quantized version of the integral
f irradiance over a volume of size Ps × Ps × Te:

f(i, n) = Q ´ nT+Te/2

nT−Te/2

˜ iP+Ps/2

iP−Ps/2 e(x, t) dx

dt + ν(i, n)
COMPSCI 527 — Computer Vision

Image Motion 3 / 12

SLIDE 4

Image Motion

Motion Field and Displacement

We are interested in real-valued positions x and times t
We have to infer those from integer-valued positions i and

times n

Motion field at x and at time t is the instantaneous velocity
f the image point that is visible at x at time t
The true image velocity of a world point
Displacement at x between times s and t is the difference

between the position of a point at time t and the position of the same point at time s < t

Displacement is the integral of image velocity between two

times

We typically approximate image velocity with displacement

between consecutive image frames

COMPSCI 527 — Computer Vision Image Motion 4 / 12

SLIDE 5

Occlusion, Correspondence, Motion Boundaries

The Displacement Field is not a 1 − 1 Map

Point visible at x at time s

that becomes hidden at time t (with s < t) forms an occlusion

When s > t, this is called a disocclusion
If points x at time s and y at time t do not form an occlusion

and are projections of the same point in the world, they correspond to each other

The displacement field is generally not integer-valued, so

we cannot compute a 1 − 1 map between image pixels even if no occlusions or disocclusions exist

A displacement field is typically given as a map Z2 → R2,

undefined at occlusions

COMPSCI 527 — Computer Vision Image Motion 5 / 12

SLIDE 6

Constancy of Appearance

What is assumed to remain constant across images?
Motion estimation is impossible without such an assumption
Most generic assumption: The appearance of a point does

not change with time or viewpoint

If two image points in two images correspond, they look the

same

If x at time s and y at time t correspond, then

e(x, s) = e(y, t) (finite-displacement formulation)

Equivalently,

de(x(t),t) dt

= 0 (differential formulation)

This is the key constraint for motion estimation

COMPSCI 527 — Computer Vision Image Motion 6 / 12

SLIDE 7

Motion Field and Optical Flow

Extreme violations of constancy of appearance:
B. K. P

. Horn, Robot Vision, MIT Press, 1986

Ill-defined distinction:
Motion field ≈ true motion
Optical flow ≈ locally observed motion

COMPSCI 527 — Computer Vision Image Motion 7 / 12

SLIDE 8

Motion Field and Optical Flow

The Optical Flow Constraint Equation

The appearance of a point does not change with time or

viewpoint:

de(x(t),t) dt

= 0

Total derivative, not partial:

de(x(t), t) dt def

= lim∆t→0

e(x(t+∆t), t+∆t)−e(x(t), t) ∆t

Use chain rule on

de(x(t),t) dt

= 0 to obtain the Optical Flow Constraint Equation (OFCE) ∂e ∂xT dx dt + ∂e ∂t = 0

v

def

=

dx dt is the unknown motion field

This is the key constraint for motion estimation

COMPSCI 527 — Computer Vision Image Motion 8 / 12

SLIDE 9

The Aperture Problem

Issues arise even when the appearance is constant

OFCE: ∂e ∂xT v + ∂e ∂t = 0

Three equations in two unknowns
However, changes in irradiance are often caused by

shading or shadows, which affects r, g, b similarly

The Jacobian

∂e ∂xT def

=       

∂r ∂x1 ∂r ∂x2 ∂g ∂x1 ∂g ∂x2 ∂b ∂x1 ∂b ∂x2

      

has often rank close to 1 and the components of ∂e

∂t are close to each other

This degeneracy is called the aperture problem

COMPSCI 527 — Computer Vision Image Motion 9 / 12

SLIDE 10

The Aperture Problem

The Aperture Problem for Black-and-White Video

The aperture problem is extreme for black-and-white

images, for which e ∈ R: ∂e ∂xT v + ∂e ∂t = 0 (OFCE is one scalar equation in the two unknowns in v)

We cannot recover motion based on local measurements

alone

Only recover the normal component along the gradient

∇e(x) =

∂e ∂xT (if the gradient is nonzero):

v(x)

def

= ∇e(x)−1 [∇e(x)]T v(x)

In practice, this is very often the case also with color video

COMPSCI 527 — Computer Vision Image Motion 10 / 12

SLIDE 11

Estimating the Motion Field

Smoothness and Motion Boundaries

The assumption of constancy of appearance yields about
ne equation in two unknowns at every point in the image
To solve for v, we need further assumptions
The motion field v : R2 → R2 is usually modeled as

piecewise smooth (in space)

OFCE is solved in the LSE sense, and an additional

regularization term is added to penalize deviations from smoothness

Smoothness holds almost everywhere, but not everywhere
Motion discontinuities are smooth image curves called

motion boundaries

COMPSCI 527 — Computer Vision Image Motion 11 / 12

SLIDE 12

Estimating the Motion Field

Because of the aperture problem, we can only estimate

several displacement vectors d or motion field vectors v simultaneously, not each individually

Local methods
The image displacement d in a small window around a pixel

x is assumed to be constant (extreme local smoothness)

Write one constancy of appearance equation for every pixel

in the window

Solve for the one displacement that satisfies all these

equations as much as possible (in the LSE sense)

Global methods
A data term measures deviations from constancy of

appearance at every pixel in the image

A smoothness term measures deviations of the motion field

v(x) from smoothness

Minimize a linear combination of the two types of terms

COMPSCI 527 — Computer Vision Image Motion 12 / 12