Dense optical flow estimation in image sequences and disparity map - - PowerPoint PPT Presentation

Universidad de Las Palmas de Gran Canaria Departamento de Informática y Sistemas Grupo de Análisis Matemático de Imágenes

Dense optical flow estimation in image sequences and disparity map computation for stereo pairs applied to 3D reconstruction

Javier Sánchez Pérez

Contents

• Optical flow methods

– Standard approach – Considering colour images – Making the method symmetric

• Stereoscopic methods

– Epipolar geometry – Standard approach – Colour method – Symmetrical method

• Main contributions and conclusions

Optical flow Standard approach

• Variational approach
• Energy minimization
• Large displacements
• Numerical scheme
• Experimental results

) , ( y x h r

Variational approach

• Energy

( )

t

y x v y x u h dw h dw h L h E ) , ( ), , ( ) ( ) ( ) ( = ⋅ Φ + ⋅ =

∫ ∫

Ω Ω

r r r r α

( )

2 2 1

) ) , ( ), , ( ( ) , ( ) ( y x v y y x u x I y x I h L + + − = r

( )

h I D h h

t

r r r ∇ ⋅ ∇ ⋅ ∇ = Φ ) (

1

Regularizing term

• Nagel-Enkelmann operator

( )

⎪ ⎭ ⎪ ⎬ ⎫ ⎪ ⎩ ⎪ ⎨ ⎧ + ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ − − + ∇ = ∇ Id I I I I I I I I D

x y x y x y 2 2 2 2 2

2 1 γ γ

Energy minimization

• Associated Euler-Lagrange equations

( ) ( ) (

)

( ) ( ) (

)

) ( ) ( ) ( ) (

2 2 1 1 2 2 1 1

h x y I h x I I v I D div C h x x I h x I I u I D div C r r r r r r r r + ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ + − + ∇ ∇ = + ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ ∂ ∂ + − + ∇ ∇ =

Energy minimization

• Gradient descent

( ) ( )

( )

( ) ( )

( )

) ( ) ( ) ( ) (

2 2 1 1 2 2 1 1

h x y I h x I I v I D div C t v h x x I h x I I u I D div C t u r r r r r r r r + ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ + − + ∇ ∇ = ∂ ∂ + ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ ∂ ∂ + − + ∇ ∇ = ∂ ∂

• Anisotropic diffusion – diffusion tensor

( )

⎪ ⎩ ⎪ ⎨ ⎧ ∇ ↓ ∇ ∇ ↑ ∇ ⇒ ∇ ∇ = ∂ ∂

⊥

∇

) ( 2 1 ) ) ( ( u div I u div I u I D div t u

I

Large displacements

• Pyramidal approach

( ) ( ) (

)

( ) ( ) (

)

) ( ) ( ) ( ) (

2 2 1 1 2 2 1 1 z z z z z z z z z z z z z z z z

h x y I h x I I v I D div C t v h x x I h x I I u I D div C t u r r r r r r r r + ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ + − + ∇ ∇ = ∂ ∂ + ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ + − + ∇ ∇ = ∂ ∂

( )

( ) ( ) ( ) ( )

( )

v u v u v u v u z z z y x I y x I

n n

z z z z z z n z z z

, , , , 2 , 2 ,

1 1

1

→ → → → → > > > = K K

Large displacements

Large displacements

• Scale-Space approach

( ) ( ) (

)

( ) ( ) (

)

) ( ) ( ) ( ) (

2 2 1 1 2 2 1 1 σ σ σ σ σ σ σ σ σ σ σ σ σ σ σ σ

h x y I h x I I v I D div C t v h x x I h x I I u I D div C t u r r r r r r r r + ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ + − + ∇ ∇ = ∂ ∂ + ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ + − + ∇ ∇ = ∂ ∂

) , ( ..... ) , ( ) , ( .... *

1 1

1

n n v

u v u v u I G I

n σ σ σ σ σ σ σ σ

σ σ σ → → → → > > > =

Large displacements

Invariance

• Invariance under gray level shifts

dz z H s y x I C

I y x

) ( ) , ( max

) , ( 2

∫

∇

= ∇ =

ν

α

Numerical scheme

( ) ( ) ( ) ( )

k j i j i k j i j i k j i j i k j i j i j i j i k j i k j i j i j i k j i k j i j i j i k j i k j i j i j i k j i k j i j i j i k j i k j i j i j i k j i k j i j i j i k j i k j i j i j i k j i k j i j i j i k j i k j i

v y u x x I v y u x I y x I h u u e e h u u e e h u u e e h u u e e h u u f f h u u f f h u u d d h u u d d C t u u

, , , , , 2 , , , , , 2 , , , 1 2 1 1 , 1 1 , 1 , 1 , 1 2 1 1 , 1 1 , 1 , 1 , 1 2 1 1 , 1 1 , 1 , 1 , 1 2 1 1 , 1 1 , 1 , 1 , 1 2 1 1 , 1 1 , , 1 , 2 1 1 , 1 1 , , 1 , 2 1 1 , 1 , 1 , , 1 2 1 1 , 1 , 1 , , 1 , 1 ,

, , , 2 2 2 2 2 2 2 2 2 2 2 2 + + ∂ ∂ + + − + ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ ⎛ − + − − + − − − + + − + + + − + + − + + − + + − + = ∆ −

+ + + − + − + + − + − + + + − − − − + + + + + + + + − − + + + + + + − − + + + + + σ σ σ

Experimental results

• Squares sequence

Experimental results

( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )

⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ + + + + = ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − + − = 1 ~ ~ 1 , ~ , ~ 1 1 , , arccos ~ ~

2 , 2 , , , 2 , 2 , , , 2 , , 2 , , j i j i t j i j i j i j i j i j i j i j i j i j i

v u v u v u v u Average Error v v u u Average Error

σ σ σ σ σ σ

Experimental results

• Square sequence

Experimental results

Comparison with other methods in Barron et al. 100% dense methods Method

• Ang. Error
• Stand. Dev.

Horn y Schunk 32,81º 13,67º Nagel 34,57º 14,38º Anandan 31,46º 18,31º Singh 46,12º 18,64º Ours 10,97º 9,60º

Experimental results

• Taxi sequence

Experimental results

• Yosemite sequence

Experimental results

Comparison with other methods in Barron et al. 100% dense methods Method

• Ang. Error
• Stand. Dev.

Horn y Schunk 9,78º 16,19º Nagel 10,22º 16,51º Anandan 13,36º 15,64º Singh 10,03º 13,13º Ours 5,53º 7,40º

Optical flow for color images

• Energy

( )

( )

∫ ∑∫

∇ ∇ ∇ + + − =

=

h I D h h x I x I h E

t i i i

r r r r r r

max , 1 3 1 2 , 2 , 1

) ( ) ( ) ( α

{ }

3 , 2 , 1 ) , (

max

= ∀ ∇ ≥ ∇ ∇ = ∇ i I I I y x I

i i i

Results

• Cube sequence

Symmetrical optic flow

2

h r

1

h r ) , ( ) , ( ) , (

2 1 2 2 1 1 2 1

h h E h h E h h E r r r r r r + =

• Composed energy

Variational approach

• Energies

( )

∫ ∫ ∫

Ω Ω Ω

⎟ ⎠ ⎞ ⎜ ⎝ ⎛ + + Φ + =

2 2 1 1 1 2 1 1

1

) ( ) ( ) , (

h

h h h h L h h E

r

r r r r r r ψ β α

( )

∫ ∫ ∫

Ω Ω Ω

⎟ ⎠ ⎞ ⎜ ⎝ ⎛ + + Φ + =

2 1 2 2 2 2 1 2

2

) ( ) ( ) , (

h

h h h h L h h E

r

r r r r r r ψ β α

Energy minimization

• Choosing the ψ function

) , ( ) , ( ) , (

1 2 1 2 2 1 1 1 1 2 1 1 + + + +

+ =

n n n n n n

h h E h h E h h E r r r r r r ⎪ ⎩ ⎪ ⎨ ⎧ =

− γ s

e γ s s s

1

) ( ψ

• Iterative scheme

Experimental results

• Squares sequence

Occlusions

1

h r

• Euclid. error
• Ang. error

Non- symmetric 0.16 0.081 Symmetric 0.50º 0.18º

Experimental results

1

h r

• Marble blocks (H. H. Nagel)
• Euclid. error
• Ang. error

Non- symmetric 0.37 0.17 Symmetric 9.4º 5.2º

Stereoscopic vision

• Epipolar geometry
• Variational approach
• Energy minimization
• Experimental results

Epipolar geometry

I2

C2

I1

C1

M

Epipolar lines Fundamental matrix mt

2 F m1=0

m1 m2

Variational approach

• Embedding the epipolar geometry

I2 I1

m1 m2

h r

λ m1

1

m F c b a ⋅ = ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ = )) , ( ( )) , ( ( y x v y x u h λ λ r

Variational approach

• Energy

( )

( )

∫ ∫

∇ ⋅ ∇ ⋅ ∇ + + − = λ λ α λ λ

1 2 2 1

)) ( ( ) ( ) ( I D h x I x I E r r r

( ) ( ) ( )

( )

2 2 2 2 2 1 2

) ( ) ( ) ( ) ( b a x x I b x y I a x I x I I D div C t + ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ ∂ ∂ − ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ − + ∇ ∇ = ∂ ∂ r r r r

λ λ λ

λ λ

Experimental results

• Hervé’s face

Experimental results

• Inria Library

Stereo for color images

( )

( )

∫ ∑∫

∇ ∇ ∇ + + − =

=

λ λ λ λ

max , 1 3 1 2 , 2 , 1

)) ( ( ) ( ) ( I D h x I x I E

t i i i

r r r

{ }

3 , 2 , 1 ) , (

max

= ∀ ∇ ≥ ∇ ∇ = ∇ i I I I y x I

i i i

( ) ( )

( )

( )

2 2 , 2 , 2 3 1 , 2 , 1 max , 1

) ( ) ( ) ( ) ( b a x x I b x y I a x I x I I D div C t

i i i i i

+ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ − ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ∂ ∂ − + ∇ ∇ = ∂ ∂

∑∫

=

r r r r

λ λ λ

λ λ

Experimental results

Experimental results

Symmetrical stereo method

• Energy

) , ( ) , ( ) , (

2 1 2 2 1 1 2 1

h h E h h E h h E r r r r r r + =

( )

∫ ∫ ∫

Ω Ω Ω

⎟ ⎠ ⎞ ⎜ ⎝ ⎛ + + Φ + =

2 2 1 1 1 2 1 1

1

) ( ) ( ) , (

h

h h h h L h h E

r

r r r r r r ψ β α

( )

∫ ∫ ∫

Ω Ω Ω

⎟ ⎠ ⎞ ⎜ ⎝ ⎛ + + Φ + =

2 1 2 2 2 2 1 2

2

) ( ) ( ) , (

h

h h h h L h h E

r

r r r r r r ψ β α

( ) ( )

t t

v u h v u h ) ( ), ( ) ( ), (

2 2 2 1 1 1

λ λ λ λ = = r r

Experimental results

Main contributions

• Common framework for optical flow and

stereoscopic vision

• Improvement of Nagel-Enkelmann method

– Consistent centralization of both terms – Grey level shift invariance

• Scale space approach to estimate large

displacements

• Extensions to colour images
• Symmetrization of results

Conclusions

• Good accuracy for large

displacements

• Improvement of dense methods

studied in the work by Barron et al.

• High accuracy of 3D reconstructions
• Coherence in both directions
• Natural way of detecting occlusions