Aggregation Methods for Optical Flow Computation Denis Fortun, - - PowerPoint PPT Presentation

SIAM Conference on Imaging Science, Philadelphia, 2012

Aggregation Methods for Optical Flow Computation

Denis Fortun, Patrick Bouthemy, Charles Kervrann

SERPICO Team Inria Rennes - Bretagne Atlantique Campus universitaire de Beaulieu 35 042 Rennes Cedex, France

Framework : patch-wise approach and aggregation method for large and small displacement optical flow

Aggregation Methods for Optical Flow Computation 2

Outline

• 1. Local and global methods for computing optical flow

⊲ data conservation ⊲ local smoothing ⊲ global approach and spatial regularization

• 2. Semi-local estimation and global aggregation

⊲ semi-local estimation of flow fields ⊲ global aggregation

• 3. Experimental results and application

⊲ comparisons of aggregation procedures ⊲ comparison of parametric and variational methods ⊲ algorithm parameters Discussion and conclusion

Aggregation Methods for Optical Flow Computation 3

• 1. Local and global methods for computing optical flow

Aggregation Methods for Optical Flow Computation 4

1.

Local and global methods for computing

• ptical flow

• 1. Local and global methods for computing optical flow

⊲ data conservation

Data conservation constraint

Fundamental assumption to find correspondences : conservation

• f brightness or image gradient over time

I2(x + w(x)) = I1(x), x ∈ Ω Insufficient constraint :

◮ ”Aperture problem” : 1 equation for 2 unknowns

w(x) = (u(x), v(x))T

◮ Uniform regions : no image gradient ◮ Assumption violations : occlusion, brightness changes, . . .

→ Adding a spatial constraint to make the problem well-posed !

Aggregation Methods for Optical Flow Computation 5

• 1. Local and global methods for computing optical flow

⊲ local smoothing

Local smoothing and spatial neighborhoods

Assume a coherent motion in a neighborhood V (x0) ⊂ Ω Coherent motion : parametric motion model wθ(x)

◮ translation :

wθ(x) = (θ1, θ4)T

◮ Affine motion : wθ(x) =

θ1 θ2 θ3 θ4 θ5 θ6 1 xT

• Local motion estimation of θ over V (x0) (Lucas & Kanade, 1981) :

ELK(wθ(x0)) =

• v(x0)

ρdata(x, wθ) dx where ρdata(x, wθ) = ψ(I2(x + wθ(x)) − I1(x)) denotes the data potential (penalizes deviations from the data constraint)

Aggregation Methods for Optical Flow Computation 6

• 1. Local and global methods for computing optical flow

⊲ local smoothing

Limitations of the local smoothing

Problem : choice of the region V V must fulfill two criteria :

◮ Delineating a coherently moving region to ensure the validity of

the parametric assumption

◮ Collecting enough trustful gradient information for reliable optical

flow computation

Aggregation Methods for Optical Flow Computation 7

• 1. Local and global methods for computing optical flow

⊲ local smoothing

Limitations of the local smoothing

square fixed-size windows regions with variable shapes

◮ parametric assumption violated

• ver motion boundaries

◮ ”aperture problem” for small

windows

→ fast optimization at each pixel

◮ joint motion estimation and

image segmentation problem

→ minimization of non-convex

functionals (e.g. ”motion competition” , Cremers & Soatto, 2005)

Aggregation Methods for Optical Flow Computation 8

• 1. Local and global methods for computing optical flow

⊲ global approach and spatial regularization

Global approach and spatial regularization

Spatial regularization : penalization of high gradients |∇w| Minimization of a global energy (Brox et al., 2004) : EG(w) =

• Ω

ρdata(x, w) + λρreg(x, w) dx =

• Ω

φ(|I2(x + w(x)) − I1(x)|2)) + φ(∇w(x)2) dx where φ(z2) = √ z2 + ǫ2 with ǫ = 0.001 Optimization method : resolution of Euler-Lagrange equations → state-of-the-art methods

Aggregation Methods for Optical Flow Computation 9

• 1. Local and global methods for computing optical flow

⊲ global approach and spatial regularization

Limitation of the global approach

Variational optimization :

◮ Difficulty to handle non-convex functionals :

⊲ restriction to convex penalty functions ⊲ over-smoothing of discontinuities

◮ Coarse-to-fine warping to cope with large displacements :

⊲ over-smoothing of fast moving details

Discrete optimization :

◮ Discretization of the search space and multi-label assignment ◮ Quantization of the flow field range :

⊲ compromise between computational cost and accuracy

Aggregation Methods for Optical Flow Computation 10

• 2. Semi-local estimation and global aggregation

Aggregation Methods for Optical Flow Computation 11

2.

Semi-local estimation and global aggregation

• 2. Semi-local estimation and global aggregation

⊲ semi-local estimation of flow fields

Semi-local estimations of flow fields

Decomposition of I1 into overlapping square patches with several sizes : Set of patches containing x : Vs2(x) = {V1, . . . , V4} V(x) =

• i=1...4

Vsi(x)

◮ Computation of flow fields over overlapping patches ◮ Combining adaptively the patches of V(x) for pointwise motion

estimation

Aggregation Methods for Optical Flow Computation 12

• 2. Semi-local estimation and global aggregation

⊲ semi-local estimation of flow fields

Block matching to cope with large displacements

For each patch of I1, find the N most similar patches MV = {M1, . . . , MN} in I2 (normalized cross-correlation) :

◮ Crude estimation of translation (i.e. pixel accuracy) ◮ Handle large displacements of small structures (small patch sizes)

Aggregation Methods for Optical Flow Computation 13

• 2. Semi-local estimation and global aggregation

⊲ semi-local estimation of flow fields

Hierarchical motion estimation

For each pair of registered patches (V , Mi), compute optical flow w0

V ,Mi with a variational (Brox et al., 2004) or parametric

(Odobez & Odobez, 1995) method (sub-pixel accuracy, optional

coarse-to-fine warping) V M1 xV − xM1 w0

V ,M1

wV ,M1 At each pixel x, we collect several hundreds of candidates : W(x) = {wV ,M(x) = (xM − xV )

• translation

+ w0

V ,M(x)

• ptical flow

: M ∈ MV , V ∈ V(x)}

Aggregation Methods for Optical Flow Computation 14

• 2. Semi-local estimation and global aggregation

⊲ global aggregation

Global aggregation

Motivations :

◮ Optimization of a variational energy or parametric estimation for

each patch

◮ “Selection of the best” competing estimators at each location

Aggregation principle : minimization of a global energy to compute a unique flow field w : Ω → R2 from the set W =

x∈Ω W(x) : ◮ Discrete optimization (W considered as a discrete space)

w = arg min

w∈W EDA(w) ◮ Continuous optimization (W used to derive a new data term)

w = arg min

w∈R2|Ω| ECA(w, W)

Aggregation Methods for Optical Flow Computation 15

• 2. Semi-local estimation and global aggregation

⊲ global aggregation

Global aggregation : discrete optimization

Finite set of candidates W(x) at each pixel : quantization of the space of motion vectors Adaptive selection by global energy minimization EDA(w) =

• x∈Ω

ΨDA

data(x, w) + β ΨDA reg(x, w)

Data term : ΨDA

data(x, w) =

ρdata(x, w) 1 − NCCσ(x, w) (normalized cross-correlation) Regularization term : (N(x) : spatial neighborhood of pixel x) ΨDA

reg(x, w) =

• y∈N(x)

φ(w(x) − w(y)2)

Aggregation Methods for Optical Flow Computation 16

• 2. Semi-local estimation and global aggregation

⊲ global aggregation

Global aggregation : discrete optimization (contd’)

Discrete optimization : no restriction on the form of EDA(w) (differentiability, convexity) “Fusion-Move” transforms a multi-label problem into a succession of binary labeled problems (Lempitsky et al., 2008) : application to the fusion of independent pre-computed motion fields

(e.g. Horn & Schunk, Lucas & Kanade, ...)

Aggregation Methods for Optical Flow Computation 17

• 2. Semi-local estimation and global aggregation

⊲ global aggregation

Global aggregation : continuous optimization

Limitations of discrete optimization : minimizer is found in the finite set of candidates Continuous aggregation : aggregated motion field is allowed to deviate from the local candidates to ensure global smoothness ECA(w)

• x∈Ω

ΨCA

data(x, W, w) + β φ(∇w(x)2) dx

A new data term to compare the candidates w0(x) ∈ W(x) : ΨCA

data(x, W, w)

=

• w0(x)∈W(x)

α(x, w0)w(x) − w0(x)p → variational minimization

Aggregation Methods for Optical Flow Computation 18

• 2. Semi-local estimation and global aggregation

⊲ global aggregation

Global aggregation : continuous optimization

Remark : if β = 0, the aggregated pointwise solution is easily found if the weights α(x, w0) are uniform :

◮ Mean

: α(x, w0) = |W(x)|−1, p = 2

◮ Median : α(x, w0) = |W(x)|−1, p = 1

→ experimentally insufficient Weighting according to a “confidence measure” : α(x, w0) = 1 ρdata(x, w0) = 1 φ(|I2(x + w0(x)) − I1(x)|2) → α(x, w0) as a function of Var(w0) is possible in the parametric case (Odobez & Bouthemy, 1995)

Aggregation Methods for Optical Flow Computation 19

• 3. Experimental results and application

Aggregation Methods for Optical Flow Computation 20

3.

Experimental results and application

• 3. Experimental results and application

Middlebury dataset and evaluation criterion

”grove” ”urban” ”rubberwhale” ”dimetrodon” Average Angular Error (AAE) : angle between the normalized estimated flow

1 √ u2+v2+1 (u, v, 1) and true flow 1

√

u2

true+v2 true+1

(utrue, vtrue, 1)

Aggregation Methods for Optical Flow Computation 21

• 3. Experimental results and application

⊲ comparisons of aggregation procedures

Comparisons of aggregation procedures (“grove”)

discrete approach continuous approach → continuous approach : over-smoothing of flow fields, low robustness to outliers

Aggregation Methods for Optical Flow Computation 22

• 3. Experimental results and application

⊲ comparisons of aggregation procedures

Performance of aggregation (“grove”)

global variational method patch-wise variational method (Brox et al., 2004) and aggregation AAE = 5.97 AAE = 5.67 → patch-wise approach : sharpening of the discontinuities, small detail recovering

Aggregation Methods for Optical Flow Computation 23

• 3. Experimental results and application

⊲ comparisons of aggregation procedures

Performance of aggregation (“rubberwhale”)

global variational method patch-wise variational method (Brox et al., 2004) and aggregation AAE = 3.92 AAE = 3.34

Aggregation Methods for Optical Flow Computation 24

• 3. Experimental results and application

⊲ comparison of parametric and variational methods

Parametric and variational methods (“grove”)

patch-wise parametric patch-wise variational method method AAE = 5.45 AAE = 5.67 → parametric approach : preservation of sharp discontinuities and small details

Aggregation Methods for Optical Flow Computation 25

• 3. Experimental results and application

⊲ comparison of parametric and variational methods

Parametric and variational methods (“dimetrodon”)

patch-wise parametric patch-wise variational method method AAE = 2.91 AAE = 1.79 → variational approach : smooth temporal-varying image sequences

Aggregation Methods for Optical Flow Computation 26

• 3. Experimental results and application

⊲ algorithm parameters

Algorithm parameters

Size of patches :

◮ Several sizes to capture large and small motions ◮ Typical values : S = {9, 19, 39, 59}

Overlapping rate : not critical when α > 0.8 Block matching :

◮ Typical value : N = 2 ◮ Minor improvements for N > 2

Aggregation Methods for Optical Flow Computation 27

Discussion and conclusion

Discussion and conclusion

Regularization parameter and window sizes : λ and β are constant and spatial patch-wise or pointwise adaptivity did not improve the experimental results (window size adaptation) Complexity of the discrete aggregation (“best” algorithm) :

◮ Number of candidates : 200-500 vectors ◮ Time computings : 2-3 hours (Brox et al., 2004) or several dozens

• f minutes (Odobez & Bouthemy, 1995) (PC Linux 3.6 Ghz)

◮ Acceleration : computation of competing vectors

(multi-threading), “Fusion-Move” (to be investigated) Perspectives :

◮ SURE and local risk minimization need to be studied more ◮ Occlusion detection is currently investigated

Aggregation Methods for Optical Flow Computation 28