Improving the Robustness of Variational Optical Flow through - - PowerPoint PPT Presentation

Improving the Robustness

• f

Variational Optical Flow

through

Tensor Voting

by: Hatem A. Rashwan, Domenec Puig, Miguel Angel Garcia presented by:

Merlin Lang langmerlin@stud.uni-sb.de Milestones And Advatages in Image Analysis Mathematical Image Analysis Group

Saarland University

Contents

• 1. Introduction
• 2. Complementary Optic Flow Model
• 3. Proposed Model
• 4. Adapted optical flow model
• 5. Experiment and Results
• 6. Summary

Contents

• 1. Introduction

Motivation

• 2. Complementary Optic Flow Model
• 3. Proposed Model
• 4. Adapted optical flow model
• 5. Experiment and Results
• 6. Summary

3 / 26

Introduction

Motivation

• Variational methods outperform other methods
• State of the art method: complementary optic flow
• Improvement with tensor voting

4 / 26

Contents

• 1. Introduction
• 2. Complementary Optic Flow Model

Data Term Smoothness Term Constraint Adaptive Regularizer (CAR)

• 3. Proposed Model
• 4. Adapted optical flow model
• 5. Experiment and Results
• 6. Summary

5 / 26

Complementary Optic Flow Model

• Given and image sequence f(x) with x := (x, y, t) and displacement

w = (u, v, 1)

• Energy functional formulation:

E(w) =

• Ω

(M(w, f)

• data term

+ α V (∇2u, ∇2v, f

• smoothness term

) dxdy

• Minimization with Euler-Lagrange-Equations:

0 = ∂uM − α(∂x(∂uxV ) + ∂y(∂uyV )) 0 = ∂vM − α(∂x(∂vxV ) + ∂y(∂vyV ))

6 / 26

Complementary Optic Flow Model

Data Term

• Given grey value constancy

f(x + w) = f(x)

• can be linearized as

fxu + fyv + ft = wT ∇3f = 0

• Rewriting to a least squares data term

M = (wT ∇3f)2 = wT ∇3f(∇3f)T w = wT J0w

• Where J0 is called the motion tensor

7 / 26

Complementary Optic Flow Model

• J0 unsufficient since aperture problem present
• Remedy: Gradient constancy

∇3f(x + w) = ∇3f(x)

• One can use the final Motion Tensor

J = ∇3f(∇3f)T + γ(∇3fx(∇3fx)T + ∇3fy(∇3fy)T )

• With postponing the linearisation:

M(u, v) =ΨM((f(x + w) − f(x))2) +γΨM((∇2f(x + w) − ∇2f(x))2)

• Using the robust penalizer

ΨM(s2) =

• s2 + ξ2

8 / 26

Complementary Optic Flow Model

Smoothness Term

• Classical homogenious regularisation

V (∇2u, ∇2v) = |∇2u|2 + |∇2v2| = (u2

x + u2 y) + (v2 x + v2 y)

• Compute eigenvectors of structure tensor

Sρ = Kρ ∗ (∇2f∇2f T )

• Results in joint image- and flow-driven regularisation

V (∇2u, ∇2v) = (e1∇2u)2 + (e2∇2u)2 + (e1∇2v)2 + (e1∇2v)2

• Yields the rubustified smoothness term

V (∇2u, ∇2v) = ΨV ((sT

1 ∇2u)2) + (sT 1 ∇2v)2)

+ΨV ((sT

2 ∇2u)2) + (sT 2 ∇2v)2) 9 / 26

Complementary Optic Flow Model

• Results in new Euler Lagrange Equations:

0 = ∂uM − α(divDu(s1, s2, ∇2u)∇2u) 0 = ∂vM − α(divDv(s1, s2, ∇2v)∇2v)

• with

Dp(s1, s2, ∇2p) = (s1, s2)

• Ψ′

V ((sT 1 ∇2p)2)

Ψ′

V ((sT 2 ∇2p)2)

s1 s2

• called the “diffusion tensor”

10 / 26

Complementary Optic Flow Model

Constraint Adaptive Regularizer (CAR)

• Regularisation Tensor.

Rp = Kρ ∗

• ∇2f(∇2f)T + γ
• ∇2fx(∇2fx)T + ∇2fy(∇2fy)T
• Single Robust Penalisation.

V (∇2u, ∇2v) = ΨV ((rT

1 ∇2u)2) + (rT 1 ∇2v)2)

+(rT

2 ∇2u)2) + (rT 2 ∇2v)2)

• gives final diffusion tensor

Dp(s1, s2, ∇2p) = (r1, r2)

• Ψ′

V ((rT 1 ∇2u)2) + rT 1 ∇2v)2)

1 r1 r2

• 11 / 26

Contents

• 1. Introduction
• 2. Complementary Optic Flow Model
• 3. Proposed Model

Pre-segmentation of image pixels Approach overview Tensor Voting Smoothing image gradients

• 4. Adapted optical flow model
• 5. Experiment and Results
• 6. Summary

12 / 26

Proposed Model

Pre-segmentation of image pixels

Homogeneous and textured regions

• Compute signal to noise ratio

SNR = 20log10(µ/σ)

• Classify as homogenious if SNR > τ and

cos(β) = 1

• 1 + ||∇3f||

≈ 0

• else classify as texture moving if SNR ≤ τ, above holds and

cos(δ) = ft ||∇3f|| + ǫ ≈ 1

• else as non moving

13 / 26

Proposed Model

Approach overview

Overview of the model using tensor voting 14 / 26

Proposed Model

Tensor Voting

• Tensor Voting for pixel p:

TV (p) =

• q∈Θ(p)

SV (v, Sq) + PV (v, Pq) + BV (v, Bq)

• Where SV stick, PV plate and BV ball tensor votes
• Stick voting by rotation oround surface normal and applying

f(Θ) =

• exp
• l(Θ)+bk(Θ)

σ

• if Θ ≤ π

4

else

• BV and PV obtained by integration

15 / 26

Proposed Model

Smoothing image gradients

• Apply Tensor Voting after segmentation to TM and HM pixels
• Only applied to the same class of pixels
• No voting for pixels with huge gradient difference

16 / 26

Contents

• 1. Introduction
• 2. Complementary Optic Flow Model
• 3. Proposed Model
• 4. Adapted optical flow model
• 5. Experiment and Results
• 6. Summary

17 / 26

Adapted optical flow model

• Replace Gaussion Convolution with TV

T = TV (∇3f) + γ(TV (∇3fx) + TV (∇3fy))

• Change CAR to:

R =TV (∇2f) + γ(TV (∇2fx) + TV (∇2fy))

• With additional regularisation

M(w, f) = wT Tw V (∇2u, ∇2v) = ΨV (R) + R

18 / 26

Contents

• 1. Introduction
• 2. Complementary Optic Flow Model
• 3. Proposed Model
• 4. Adapted optical flow model
• 5. Experiment and Results
• 6. Summary

19 / 26

Experiment and Results

(a)Frame at time t in sequence OPEN-HOTEL.(b) Frame at time t + dt. (c) Classified pixels: red pixels are textured-moving regions, green pixels are homogeneous- moving regions and blue pixels are stationary (not moving) regions. (d) Frame at time t in sequence STREET-CROSS. (e) Frame at time t + dt. (f) Classified pixels: red pixels are textured-moving regions, green pixels are homogeneous-moving regions and blue pixels are stationary (not moving) regions. 20 / 26

Experiment and Results Results for some Middlebury sequences with corresponding ground-truth. (1st column and 2nd column) Frames 10 and 11. (3rd column) Ground-truths (black points correspond to pixels without available ground-truth). (4th column) Optical flow fields obtained with the proposed approach. 21 / 26

Experiment and Results Results for some Middlebury and MIT sequences with associated ground-truths. (1st column and 2nd column) Two consecutive frames. (3rd column) Ground-truths. (4th Column) Optical flow fields obtained with the proposed approach. 22 / 26

Contents

• 1. Introduction
• 2. Complementary Optic Flow Model
• 3. Proposed Model
• 4. Adapted optical flow model
• 5. Experiment and Results
• 6. Summary

23 / 26

Summary

• Proposed method enhances Complementary model with Tensor voting
• Separately applied to homogeneous-moving and textured-moving regions
• Proposed model yields flow fields with lower quantitative errors
• Drawback: Computational Complexity

24 / 26

References

• RASHWAN Hatem A., PUIG Domenec , GARCIA Miguel A.:

Improving the Robustness of Variational Optical Flow Through Tensor Voting. In: Computer Vision and Image Understanding (2012)

• ZIMMER Henning, BRUHN Andr´

es, WEICKERT Joachim, VALGAERTS Levi, SALGADO Agust’ın, ROSENHAHN Bodo , SEIDEL Hans P .: Complementary Optic Flow. In: Proceedings of the 7th International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition. Berlin, Heidelberg : Springer-Verlag, 2009 (EMMCVPR ’09), p. 207–220

25 / 26

Thank you for your attention!

26 / 26