Motion Estimation (I) Ce Liu celiu@microsoft.com Microsoft - - PowerPoint PPT Presentation

Motion Estimation (I)

Ce Liu celiu@microsoft.com Microsoft Research New England

We live in a moving world

• Perceiving, understanding and predicting motion is an

important part of our daily lives

Motion estimation: a core problem of computer vision

• Related topics:

– Image correspondence, image registration, image matching, image alignment, …

• Applications

– Video enhancement: stabilization, denoising, super resolution – 3D reconstruction: structure from motion (SFM) – Video segmentation – Tracking/recognition – Advanced video editing (label propagation)

Contents (today)

• Motion perception
• Motion representation
• Parametric motion: Lucas-Kanade
• Dense optical flow: Horn-Schunck
• Robust estimation
• Applications (1)

Contents (next time)

• Discrete optical flow
• Layer motion analysis
• Contour motion analysis
• Obtaining motion ground truth
• SIFT flow: generalized optical flow
• Applications (2)

Readings

• Rick’s book: Chapter 8
• Ce Liu’s PhD thesis (appendix A & B)
• S. Baker and I. Matthews. Lucas-Kanade 20 years on: a

unifying framework. IJCV 2004

• Horn-Schunck (wikipedia)
• A. Bruhn, J. Weickert, C. Schnorr. Lucas/Kanade meets

Horn/Schunk: combining local and global optical flow

• methods. IJCV 2005

Contents

• Motion perception
• Motion representation
• Parametric motion: Lucas-Kanade
• Dense optical flow: Horn-Schunck
• Robust estimation
• Applications (1)

Seeing motion from a static picture?

http://www.ritsumei.ac.jp/~akitaoka/index-e.html

More examples

How is this possible?

• The true mechanism is to be

revealed

• FMRI data suggest that

illusion is related to some component of eye movements

• We don’t expect computer

vision to “see” motion from these stimuli, yet

What do you see?

In fact, …

We still don’t touch these areas

Motion analysis: human vs. computer

• Computers can only analyze motion for opaque

and solid objects

• Challenges:

– Shapeless or transparent scenes

• Key: motion representation

Contents

• Motion perception
• Motion representation
• Parametric motion: Lucas-Kanade
• Dense optical flow: Horn-Schunck
• Robust estimation
• Applications (1)

Motion forms

• Mapping: 𝑦1, 𝑧1 → (𝑦2, 𝑧2)
• Global parametric motion: 𝑦2, 𝑧2 = 𝑔(𝑦1, 𝑧1; 𝜄)
• Motion types

– Translation: 𝑦2 𝑧2 = 𝑦1 + 𝑏 𝑧1 + 𝑐 – Similarity: 𝑦2 𝑧2 = 𝑡 cos 𝛽 sin 𝛽 − sin 𝛽 cos 𝛽 𝑦1 + 𝑏 𝑧1 + 𝑐 – Affine: 𝑦2 𝑧2 = 𝑏𝑦1 + 𝑐𝑧1 + 𝑑 𝑒𝑦1 + 𝑓𝑧1 + 𝑔 – Homography: 𝑦2 𝑧2 =

1 𝑨

𝑏𝑦1 + 𝑐𝑧1 + 𝑑 𝑒𝑦1 + 𝑓𝑧1 + 𝑔 , 𝑨 = 𝑕𝑦1 + 𝑖𝑧1 + 𝑗

Illustration of motion types

Translation

Optical flow field

• Parametric motion is limited and cannot describe the

motion of arbitrary videos

• Optical flow field: assign a flow vector 𝑣 𝑦, 𝑧 , 𝑤 𝑦, 𝑧

to each pixel (𝑦, 𝑧)

• Projection from 3D world to 2D

Optical flow field visualization

• Too messy to plot flow vector for every pixel
• Map flow vector to color

– Magnitude: saturation – Orientation: hue

Visualization code [Baker et al. 2007]

Ground-truth flow field Input

Matching criterion

• Brightness constancy assumption

𝐽1 𝑦, 𝑧 = 𝐽2 𝑦 + 𝑣, 𝑧 + 𝑤 + 𝑠 + 𝑕 𝑠 ∼ 𝑂 0, 𝜏2 , 𝑕 ∼ 𝑉 −1,1 Noise 𝑠, outlier 𝑕 (occlusion, lighting change)

• Matching criteria

– What’s invariant between two images?

• Brightness, gradients, phase, other features…

– Distance metric (L2, L1, truncated L1, Lorentzian) 𝐹 𝑣, 𝑤 = 𝜍 𝐽1 𝑦, 𝑧 − 𝐽2 𝑦 + 𝑣, 𝑧 + 𝑤

𝑦,𝑧

– Correlation, normalized cross correlation (NCC)

Error functions

• 2
• 1.5
• 1
• 0.5

0.5 1 1.5 2 0.5 1 1.5 2 2.5 3

• 2
• 1.5
• 1
• 0.5

0.5 1 1.5 2 0.5 1 1.5 2 2.5 3

• 2
• 1.5
• 1
• 0.5

0.5 1 1.5 2 0.5 1 1.5 2 2.5 3

• 2
• 1.5
• 1
• 0.5

0.5 1 1.5 2 0.5 1 1.5 2 2.5 3

L2 norm 𝜍 𝑨 = 𝑨2 L1 norm 𝜍 𝑨 = |𝑨| Truncated L1 norm 𝜍 𝑨 = min⁡ ( 𝑨 , 𝜃) Lorentzian 𝜍 𝑨 = log⁡ (1 + 𝛿𝑨2)

Robust statistics

• Traditional L2 norm: only noise, no outlier
• Example: estimate the average of

⁡⁡⁡⁡⁡⁡⁡⁡⁡⁡⁡⁡⁡⁡⁡⁡⁡⁡⁡⁡⁡0.95, 1.04, 0.91, 1.02, 1.10, 20.01

• Estimate with minimum error

𝑨∗ = arg min

𝑨

𝜍 𝑨 − 𝑨𝑗

𝑗

– L2 norm: 𝑨∗ = 4.172 – L1 norm: 𝑨∗ = 1.038 – Truncated L1: 𝑨∗ = 1.0296 – Lorentzian: 𝑨∗ = 1.0147

• 2
• 1.5
• 1
• 0.5

L2 norm 𝜍 𝑨 = 𝑨2

• 2
• 1.5
• 1
• 0.5

L1 norm 𝜍 𝑨 = |𝑨|

• 2
• 1.5
• 1
• 0.5

Truncated L1 norm 𝜍 𝑨 = min⁡ ( 𝑨 , 𝜃)

• 2
• 1.5
• 1
• 0.5

Lorentzian 𝜍 𝑨 = log⁡ (1 + 𝛿𝑨2)

Contents

• Motion perception
• Motion representation
• Parametric motion: Lucas-Kanade
• Dense optical flow: Horn-Schunck
• Robust estimation
• Applications (1)

Lucas-Kanade: problem setup

• Given two images 𝐽1(𝑦, 𝑧) and 𝐽2(𝑦, 𝑧), estimate a parametric

motion that transforms 𝐽1 to 𝐽2

• Let 𝐲 = 𝑦, 𝑧 𝑈 be a column vector indexing pixel coordinate
• Two typical transforms

– Translation: 𝑋 x; p = 𝑦 + 𝑞1 𝑧 + 𝑞2 – Affine: 𝑋 x; p = 𝑞1𝑦 + 𝑞2𝑧 + 𝑞3 𝑞4𝑦 + 𝑞5𝑧 + 𝑞6 = 𝑞1 𝑞2 𝑞3 𝑞4 𝑞5 𝑞6 𝑦 𝑧 1

• Goal of the Lucas-Kanade algorithm

p∗ = arg min

p 𝐽2 𝑋 x; p

− 𝐽1 x

2 x

An incremental algorithm

• Difficult to directly optimize the objective function

p∗ = arg min

p 𝐽2 𝑋 x; p

− 𝐽1 x

2 x

• Instead, we try to optimize each step

Δp∗ = arg min

Δp 𝐽2 𝑋 x; p + Δp

− 𝐽1 x

2 x

• The transform parameter is updated:

p ← p + Δp∗

Taylor expansion

• The term 𝐽2 𝑋 x; p + Δp

is highly nonlinear

• Taylor expansion:

𝐽2 𝑋 x; p + Δp ≈ 𝐽2 𝑋 𝑦; 𝑞 + ∇𝐽2 𝜖𝑋 𝜖p Δp

• 𝜖𝑋

𝜖p⁡: Jacobian of the warp

• If 𝑋 x; p = 𝑋

𝑦 x; p , 𝑋 𝑧 x; p 𝑈⁡, then

𝜖𝑋 𝜖p = 𝜖𝑋

𝑦

𝜖𝑞1 … 𝜖𝑋

𝑦

𝜖𝑞𝑜 𝜖𝑋

𝑧

𝜖𝑞1 … 𝜖𝑋

𝑧

𝜖𝑞𝑜

Jacobian matrix

• For affine transform: 𝑋 x; p = 𝑞1

𝑞2 𝑞3 𝑞4 𝑞5 𝑞6 𝑦 𝑧 1 The Jacobian is ⁡

𝜖𝑋 𝜖p = 𝑦

𝑧 0⁡⁡⁡⁡⁡1 𝑦⁡⁡⁡⁡⁡0 𝑧 1

• For translation : 𝑋 x; p = 𝑦 + 𝑞1

𝑧 + 𝑞2 The Jacobian is ⁡

𝜖𝑋 𝜖p = 1

1⁡

Taylor expansion

• 𝛼𝐽2 = 𝐽𝑦⁡𝐽𝑧 is the gradient of image 𝐽2 evaluated at

𝑋(x; p): compute the gradients in the coordinate of 𝐽2 and warp back to the coordinate of 𝐽1

• For affine transform

𝜖𝑋 𝜖p = 𝑦

𝑧 0⁡⁡⁡⁡⁡1 𝑦⁡⁡⁡⁡⁡0 𝑧 1 ∇𝐽2 𝜖𝑋 𝜖p = 𝐽𝑦𝑦 𝐽𝑦𝑧 𝐽𝑦⁡⁡⁡⁡⁡𝐽𝑧𝑦 𝐽𝑧𝑧 𝐽𝑧

• Let matrix 𝐂 = [𝐉𝑦𝐘⁡⁡𝐉𝑦𝐙 𝐉𝑦 𝐉𝑧𝐘⁡⁡𝐉𝑧𝐙 𝐉𝑧] ∈ ℝ𝑜×6, 𝐉𝑦 and

𝐘⁡are both column vectors. 𝐉𝑦𝐘⁡is element-wise vector multiplication.

Gauss-Newton

• With Taylor expansion, the objective function becomes

Δp∗ = arg min

Δp 𝐽2 𝑋 𝑦; 𝑞

+ ∇𝐽2 𝜖𝑋 𝜖p Δp − 𝐽1 x

2 x

Or in a vector form: Δp∗ = arg min

Δp 𝐉𝑢 + 𝐂Δp 𝑈(𝐉𝑢 + 𝐂Δp)

Where 𝐂 = [𝐉𝑦𝐘⁡⁡𝐉𝑦𝐙 𝐉𝑦 𝐉𝑧𝐘⁡⁡𝐉𝑧𝐙 𝐉𝑧] ∈ ℝ𝑜×6 𝐉𝑢 = 𝐉2 𝐗 p − 𝐉1

• Solution:

Δp∗ = − 𝐂𝑈𝐂 −1𝐂𝑈𝐉𝑢

Translation

• Jacobian: ⁡

𝜖𝑋 𝜖p = 1

1⁡

• ∇𝐽2

𝜖𝑋 𝜖p = 𝐽𝑦

𝐽𝑧

• 𝐂 = [𝐉𝑦⁡⁡𝐉𝑧] ∈ ℝ𝑜×2
• Solution:

Δp∗ = − 𝐂𝑈𝐂 −1𝐂𝑈𝐉𝑢 = − 𝐉𝑦

𝑈𝐉𝑦

𝐉𝑦

𝑈𝐉𝑧

𝐉𝑦

𝑈𝐉𝑧

𝐉𝑧

𝑈𝐉𝑧 −1 𝐉𝑦 𝑈𝐉𝑢

𝐉𝑧

𝑈𝐉𝑢

How it works

Coarse-to-fine refinement

• Lucas-Kanade is a greedy algorithm that converges to local

minimum

• Initialization is crucial: if initialized with zero, then the

underlying motion must be small

• If underlying transform is significant, then coarse-to-fine is

a must

Smooth & down- sampling (𝑣2, 𝑤2) (𝑣1, 𝑤1) (𝑣, 𝑤) × 2 × 2

Variations

• Variations of Lucas Kanade:

– Additive algorithm [Lucas-Kanade, 81] – Compositional algorithm [Shum & Szeliski, 98] – Inverse compositional algorithm [Baker & Matthews, 01] – Inverse additive algorithm [Hager & Belhumeur, 98]

• Although inverse algorithms run faster (avoiding re-

computing Hessian), they have the same complexity for robust error functions!

From parametric motion to flow field

• Incremental flow update (𝑒𝑣, 𝑒𝑤) for pixel 𝑦, 𝑧 ⁡
• We obtain the following function within a patch
• The flow vector of each pixel is updated independently
• Median filtering can be applied for spatial smoothness

d𝑣 d𝑤 = − 𝐉𝑦

𝑈𝐉𝑦

𝐉𝑦

𝑈𝐉𝑧

𝐉𝑦

𝑈𝐉𝑧

𝐉𝑧

𝑈𝐉𝑧 −1 𝐉𝑦 𝑈𝐉𝑢

𝐉𝑧

𝑈𝐉𝑢

⁡⁡⁡⁡⁡𝐽2 𝑦 + 𝑣 + 𝑒𝑣, 𝑧 + 𝑤 + 𝑒𝑤 − 𝐽1 𝑦, 𝑧 = 𝐽2 𝑦 + 𝑣, 𝑧 + 𝑤 + 𝐽𝑦 𝑦 + 𝑣, 𝑧 + 𝑤 𝑒𝑣 + 𝐽𝑧 𝑦 + 𝑣, 𝑧 + 𝑤 𝑒𝑤 − 𝐽1 𝑦, 𝑧 𝐽𝑦𝑒𝑣 + 𝐽𝑧𝑒𝑤 + 𝐽𝑢 = 0

Example

Input two frames Coarse-to-fine LK Coarse-to-fine LK with median filtering Flow visualization

Contents

• Motion perception
• Motion representation
• Parametric motion: Lucas-Kanade
• Dense optical flow: Horn-Schunck
• Robust estimation
• Applications (1)

Motion ambiguities

• When will the Lucas-Kanade algorithm fail?
• The inverse may not exist!!!
• How?

– All the derivatives are zero: flat regions – X- and y- derivatives are linearly correlated: lines d𝑣 d𝑤 = − 𝐉𝑦

𝑈𝐉𝑦

𝐉𝑦

𝑈𝐉𝑧

𝐉𝑦

𝑈𝐉𝑧

𝐉𝑧

𝑈𝐉𝑧 −1 𝐉𝑦 𝑈𝐉𝑢

𝐉𝑧

𝑈𝐉𝑢

The aperture problem

Corners Lines Flat regions

Dense optical flow with spatial regularity

• Local motion is inherently ambiguous

– Corners: definite, no ambiguity – Lines: definite along the normal, ambiguous along the tangent – Flat regions: totally ambiguous

• Solution: imposing spatial smoothness to the flow field

– Adjacent pixels should move together as much as possible – Horn & Schunck equation 𝑣, 𝑤 = arg min 𝐽𝑦𝑣 + 𝐽𝑧𝑤 + 𝐽𝑢

2 + 𝛽 ∇𝑣 2 + ∇𝑤 2 𝑒𝑦𝑒𝑧

– ∇𝑣 2 =

𝜖𝑣 𝜖𝑦 2

+

𝜖𝑣 𝜖𝑧 2

= 𝑣𝑦

2 + 𝑣𝑧 2

– 𝛽: smoothness coefficient

Data term Smoothness term

2D Euler Lagrange

• 2D Euler Lagrange: the functional

𝑇 = 𝑀 𝑦, 𝑧, 𝑔, 𝑔

𝑦, 𝑔 𝑧 𝑒𝑦𝑒𝑧 Ω

is minimized only if 𝑔 satisfies the partial differential equation (PDE)

𝜖𝑀 𝜖𝑔 − 𝜖 𝜖𝑦 𝜖𝑀 𝜖𝑔

𝑦

− 𝜖 𝜖𝑧 𝜖𝑀 𝜖𝑔

𝑧

= 0

• In Horn-Schunck

– 𝑀 𝑣, 𝑤, 𝑣𝑦, 𝑣𝑧, 𝑤𝑦, 𝑤𝑧 = 𝐽𝑦𝑣 + 𝐽𝑧𝑤 + 𝐽𝑢

2 + 𝛽 𝑣𝑦 2 + 𝑣𝑧 2 + 𝑤𝑦 2 + 𝑤𝑧 2

–

𝜖𝑀 𝜖𝑣 = 2 𝐽𝑦𝑣 + 𝐽𝑧𝑤 + 𝐽𝑢 𝐽𝑦

–

𝜖𝑀 𝜖𝑣𝑦 = 2𝛽𝑣𝑦, 𝜖 𝜖𝑦 𝜖𝑀 𝜖𝑣𝑦 = 2𝛽𝑣𝑦𝑦, 𝜖𝑀 𝜖𝑣𝑧 = 2𝛽𝑣𝑧, 𝜖 𝜖𝑧 𝜖𝑀 𝜖𝑣𝑧 = 2𝛽𝑣𝑧𝑧

Linear PDE

• The Euler-Lagrange PDE for Horn-Schunck is

𝐽𝑦𝑣 + 𝐽𝑧𝑤 + 𝐽𝑢 𝐽𝑦 − 𝛽 𝑣𝑦𝑦 + 𝑣𝑧𝑧 = 0 𝐽𝑦𝑣 + 𝐽𝑧𝑤 + 𝐽𝑢 𝐽𝑧 − 𝛽 𝑤𝑦𝑦 + 𝑤𝑧𝑧 = 0

• 𝑣𝑦𝑦 + 𝑣𝑧𝑧 can be obtained by a Laplacian operator:

−1 −1 4 −1 −1

• In the end, we solve a large linear system

𝐉𝑦

2 + 𝛽𝐌

𝐉𝑦𝐉𝑧 𝐉𝑦𝐉𝑧 𝐉𝑧

2 + 𝛽𝐌

𝑉 𝑊 = − 𝐉𝑦𝐽𝑢 𝐉𝑧𝐽𝑢

How to solve a large linear system?

• With 𝛽 > 0, this system is positive definite!
• You can use your favorite solver

– Gauss-Seidel, successive over-relaxation (SOR) – (Pre-conditioned) conjugate gradient

• No need to wait for the solver to converge

completely

𝐉𝑦

2 + 𝛽𝐌

𝐉𝑦𝐉𝑧 𝐉𝑦𝐉𝑧 𝐉𝑧

2 + 𝛽𝐌

𝑉 𝑊 = − 𝐉𝑦𝐽𝑢 𝐉𝑧𝐽𝑢

Condition for convergence

• In the objective function

𝑣, 𝑤 = arg min 𝐽𝑦𝑣 + 𝐽𝑧𝑤 + 𝐽𝑢

2 + 𝛽 ∇𝑣 2 + ∇𝑤 2 𝑒𝑦𝑒𝑧

The displacement (𝑣, 𝑤) has to be small for the Taylor expansion to be valid.

• More practically, we can estimate the optimal incremental

change

𝐽𝑦𝑒𝑣 + 𝐽𝑧𝑒𝑤 + 𝐽𝑢

2 + 𝛽 ∇ 𝑣 + 𝑒𝑣 2 + ∇ 𝑤 + 𝑒𝑤 2 𝑒𝑦𝑒𝑧

• The solution becomes

𝐉𝑦

2 + 𝛽𝐌

𝐉𝑦𝐉𝑧 𝐉𝑦𝐉𝑧 𝐉𝑧

2 + 𝛽𝐌

𝑒𝑉 𝑒𝑊 = − 𝐉𝑦𝐽𝑢 + 𝛽𝐌𝑉 𝐉𝑧𝐽𝑢 + 𝛽𝐌𝑊

Examples

Input two frames Coarse-to-fine LK with median filtering Flow visualization Horn-Schunck Coarse-to-fine LK

The source of over-smoothness

• Horn-Schunck is a Gaussian Markov

random field (GMRF)

• Spatial over-smoothness is caused by

quadratic smoothness term

• Nevertheless, optical flow fields are sparse!

Horn-Schunck Ground truth

𝐽𝑦𝑣 + 𝐽𝑧𝑤 + 𝐽𝑢

2 + 𝛽 ∇𝑣 2 + ∇𝑤 2 𝑒𝑦𝑒𝑧

𝑣 𝑣𝑦 𝑣𝑧 𝑤 𝑤𝑦 𝑤𝑧

Continuous Markov Random Fields

• Horn-Schunck started 30 years of research on continuous

Markov random fields

– Optical flow estimation – Image reconstruction, e.g. denoising, super resolution – Shape from shading, inverse rendering problems – Natural image priors

• Why continuous?

– Many signals are differentiable – More complicated spatial relationships

• Fast solvers

– Multi-grid – Preconditioned conjugate gradient – FFT + annealing

Contents

• Motion perception
• Motion representation
• Parametric motion: Lucas-Kanade
• Dense optical flow: Horn-Schunck
• Robust estimation
• Applications (1)

Modification to Horn-Schunck

• Let x = (𝑦, 𝑧, 𝑢), and w x = (𝑣 x , 𝑤 x , 1) be the flow

vector

• Horn-Schunck (recall)

𝐽𝑦𝑣 + 𝐽𝑧𝑤 + 𝐽𝑢

2 + 𝛽 𝛼𝑣 2 + 𝛼𝑤 2 𝑒𝑦𝑒𝑧

• Robust estimation

𝜔 𝐽 x + w − 𝐽 x

2 + 𝛽𝜚 ∇𝑣 2 + ∇𝑤 2 𝑒𝑦𝑒𝑧

• Robust estimation with Lucas-Kanade

𝑕 ∗ 𝜔 𝐽 x + w − 𝐽 x

2 + 𝛽𝜚 𝛼𝑣 2 + 𝛼𝑤 2 𝑒𝑦𝑒𝑧

Robust functions

• Various forms of robust functions

– L1 norm: 𝜔 𝑨2 = 𝑨2 + 𝜁2, 𝜚 𝑨2 = 𝑨2 + 𝜁2 – Sub L1: 𝜔 𝑨2; 𝜃 = 𝑨2 + 𝜁2 𝜃, 𝜃 < 0.5 – Lorentzian: 𝜔 𝑨2 = log(1 + 𝑨2)

• 1
• 0.8
• 0.6
• 0.4
• 0.2

0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 1.2

−⁡|𝑨| −⁡ 𝑨2 + 𝜁2

• 1
• 0.8
• 0.6
• 0.4
• 0.2

0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 1.2

=0.5 =0.4 =0.3 =0.2

Special cases

• The robust objective function

𝑕 ∗ 𝜔 𝐽 x + w − 𝐽 x

2 + 𝛽𝜚 𝛼𝑣 2 + 𝛼𝑤 2 𝑒𝑦𝑒𝑧

– Lucas-Kanade: 𝛽 = 0, 𝜔 𝑨2 = 𝑨2 – Robust Lucas-Kanade: 𝛽 = 0, 𝜔 𝑨2 = 𝑨2 + 𝜁2 – Horn-Schunck: 𝑕 = 1, 𝜔 𝑨2 = 𝑨2, 𝜚 𝑨2 = 𝑨2

• One can also learn the filters (other than gradients), and

robust function 𝜔 ⋅ , 𝜚 ⋅ [Roth & Black 2005]

Derivation strategies

• Euler-Lagrange

– Derive in continuous domain, discretize in the end – Nonlinear PDE’s – Outer and inner fixed point iterations – Cannot generalize to general filters

• Variational optimization
• Iterative reweighted least square (IRLS)

– Discretize first and derive in matrix form – Easy to understand and derive

• These three approaches are equivalent!

Iterative reweighted least square (IRLS)

• Let 𝜚 𝑨 = 𝑨2 + 𝜁2 𝜃 be a robust function
• We want to minimize the objective function

Φ 𝐁𝑦 + 𝑐 = 𝜚 𝑏𝑗

𝑈𝑦 + 𝑐𝑗 2 𝑜 𝑗=1

where 𝑦 ∈ ℝ𝑒, 𝐁 = 𝑏1⁡𝑏2 ⋯ 𝑏𝑜 𝑈 ∈ ℝ𝑜×𝑒, 𝑐 ∈ ℝ𝑜

• By setting 𝜖Φ

𝜖𝑦 = 0, we can derive

𝜖Φ 𝜖𝑦 = 𝜚′ 𝑏𝑗

𝑈𝑦 + 𝑐𝑗 2

𝑏𝑗

𝑈𝑦 + 𝑐𝑗 𝑏𝑗 𝑜 𝑗=1

⁡⁡⁡⁡⁡⁡= w𝑗𝑗𝑏𝑗

𝑈𝑦𝑏𝑗 + w𝑗𝑗𝑐𝑗𝑏𝑗 𝑜 𝑗=1

⁡⁡⁡⁡⁡⁡= 𝑏𝑗

𝑈w𝑗𝑗𝑦𝑏𝑗 + 𝑐𝑗w𝑗𝑗𝑏𝑗 𝑜 𝑗=1

⁡⁡⁡⁡⁡⁡= 𝐁𝑈𝐗𝐁𝑦 + 𝐁𝑈𝐗𝑐

w𝑗𝑗 = 𝜚′ 𝑏𝑗

𝑈𝑦 + 𝑐𝑗 2

𝐗 = diag Φ′ 𝐁𝑦 + 𝑐

Iterative reweighted least square (IRLS)

• Derivative:

𝜖Φ 𝜖𝑦 = 𝐁𝑈𝐗𝐁𝑦 + 𝐁𝑈𝐗𝑐

• Iterate between reweighting and least square
• 1. Initialize 𝑦 = 𝑦0
• 2. Compute weight matrix 𝐗 = diag Φ′ 𝐁𝑦 + 𝑐
• 3. Solve the linear system 𝐁𝑈𝐗𝐁𝑦 = −𝐁𝑈𝐗𝑐
• 4. If 𝑦 converges, return; otherwise, go to 2
• Convergence is guaranteed (local minima)

IRLS for robust optical flow

• Objective function

𝑕 ∗ 𝜔 𝐽 x + w − 𝐽 x

2 + 𝛽𝜚 𝛼𝑣 2 + 𝛼𝑤 2 𝑒𝑦𝑒𝑧

• Discretize, linearize and increment

𝑕 ∗ 𝜔( 𝐽𝑢 + 𝐽𝑦𝑒𝑣 + 𝐽𝑍𝑒𝑤 2)

𝑦,𝑧

+ 𝛽𝜚( ∇ 𝑣 + 𝑒𝑣

2 + ∇ 𝑤 + 𝑒𝑤 2)

• IRLS (initialize 𝑒𝑣 = 𝑒𝑤 = 0)

– Weight: – Least square: 𝛀𝑦𝑦

′

= diag 𝑕 ∗ ψ′𝐉𝑦𝐉𝑦 , 𝛀𝑦𝑧

′

= diag 𝑕 ∗ ψ′𝐉𝑦𝐉𝑧 , 𝛀𝑧𝑧

′

= diag 𝑕 ∗ ψ′𝐉𝑧𝐉𝑧 , 𝛀𝑦𝑢

′ = diag(𝑕 ∗ ψ′𝐉𝑦𝐉𝑢),

𝛀𝑧𝑢

′ = diag(𝑕 ∗ ψ′𝐉𝑧𝐉𝑢), 𝐌 = 𝐄𝑦 𝑈𝚾′𝐄𝑦 + 𝐄𝑧 𝑈𝚾′𝐄𝑧

𝛀𝑦𝑦

′ + 𝛽𝐌

𝛀𝑦𝑧

′

𝛀𝑦𝑧

′

𝛀𝑧𝑧

′

+ 𝛽𝐌 𝑒𝑉 𝑒𝑊 = − 𝛀𝑦𝑢

′ + 𝛽𝐌𝑉

𝐉𝑧𝐽𝑢 + 𝛽𝐌𝑊

Examples

Input two frames Coarse-to-fine LK with median filtering Flow visualization Horn-Schunck Robust optical flow

Contents

• Motion perception
• Motion representation
• Parametric motion: Lucas-Kanade
• Dense optical flow: Horn-Schunck
• Robust estimation
• Applications (1)

Video stabilization

Video denoising

• Use multiple frames for temporal coherence
• Non-local mean

Video denoising

Video super resolution

• Merge information from adjacent frames
• Reconstruction depends on flow accuracy

Summary

• Lucas-Kanade

– Parametric motion – Dense flow field (with median filtering)

• Horn-Schunck

– Gaussian Markov random field – Euler-Lagrange

• Robust flow estimation

– Robust function

• Account for outliers in data term
• Encourage piecewise smoothness

– IRLS (= nonlinear PDE = variational optimization)

Next time

• Discrete optical flow
• Layer motion analysis
• Contour motion analysis
• Obtaining motion ground truth
• SIFT flow: generalized optical flow
• Applications (2)