Image Processing using Partial Differential Equations (PDE) - PowerPoint PPT Presentation

Image Processing using Partial Differential Equations (PDE) Restoration, segmentation, tracking, optical flow estimation, registration Pierre Kornprobst NeuroMathComp project team INRIA Sophia Antipolis - M´ editerran´ ee Vision Student Talks [ViST] April 2014 Kornprobst (INRIA) PDEs April 2014 1 / 48

My goals today Introduce methodology Kornprobst (INRIA) PDEs April 2014 2 / 48

My goals today Introduce methodology �� Show links between formulations (min, ∂. ∂ t , ) Kornprobst (INRIA) PDEs April 2014 2 / 48

My goals today Introduce methodology �� Show links between formulations (min, ∂. ∂ t , ) Show a success story: level-sets Kornprobst (INRIA) PDEs April 2014 2 / 48

1 Can we use PDEs to do some interesting image processing? Definitions From Gaussian filtering to the heat equation Kornprobst (INRIA) PDEs April 2014 3 / 48

1 Can we use PDEs to do some interesting image processing? Definitions From Gaussian filtering to the heat equation 2 Three solutions to go further Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem Kornprobst (INRIA) PDEs April 2014 3 / 48

1 Can we use PDEs to do some interesting image processing? Definitions From Gaussian filtering to the heat equation 2 Three solutions to go further Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem 3 A success story: Levels-sets Kornprobst (INRIA) PDEs April 2014 3 / 48

1 Can we use PDEs to do some interesting image processing? Definitions From Gaussian filtering to the heat equation 2 Three solutions to go further Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem 3 A success story: Levels-sets 4 Conclusion Kornprobst (INRIA) PDEs April 2014 3 / 48

1 Can we use PDEs to do some interesting image processing? Definitions From Gaussian filtering to the heat equation 2 Three solutions to go further Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem 3 A success story: Levels-sets 4 Conclusion Kornprobst (INRIA) PDEs April 2014 4 / 48

An image is seen as a function defined in continuous space R 2 → I u : Ω ⊂ I R Kornprobst (INRIA) PDEs April 2014 5 / 48

A PDE defines an evolution u ( t , x ) / ∂ u ∂ t = H ( t , x , u , ∇ u , ∇ 2 u ) � v ( x ) ≡ u ( ∞ , x ) Kornprobst (INRIA) PDEs April 2014 6 / 48

1 Can we use PDEs to do some interesting image processing? Definitions From Gaussian filtering to the heat equation 2 Three solutions to go further Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem 3 A success story: Levels-sets 4 Conclusion Kornprobst (INRIA) PDEs April 2014 7 / 48

Gaussian filtering Let u 0 an image, we define : � � − | x | 2 1 u σ ( x ) = ( G σ ∗ u 0 )( x ) avec G σ ( x ) = 2 π σ 2 exp . 2 σ 2 σ = 0 σ = 5 σ = 11 σ = 17 Kornprobst (INRIA) PDEs April 2014 8 / 48

Heat equation A linear PDE � ∂ u ∂ t ( t , x ) = ∆ u ( t , x ) , t ≥ 0 , u (0 , x ) = u 0 ( x ) . t = 0 t = 12 . 5 t = 60 . 5 t = 93 . 5 A notion of scale Kornprobst (INRIA) PDEs April 2014 9 / 48

Solution of the heat equation is a convolution u ( t , x ) = ( G √ 2 t ∗ u 0 )( x ) Gaussian filtering One operation in a large neighbourhood σ = 0 σ = 5 σ = 11 σ = 17 t = 0 t = 12 . 5 t = 60 . 5 t = 93 . 5 Heat equation A succession of local operations Kornprobst (INRIA) PDEs April 2014 10 / 48

Take home messages PDE appear as a natural way to smooth images. When it is linear, a PDE (or equivalently the convolution) do not preserve edges. Kornprobst (INRIA) PDEs April 2014 11 / 48

1 Can we use PDEs to do some interesting image processing? Definitions From Gaussian filtering to the heat equation 2 Three solutions to go further Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem 3 A success story: Levels-sets 4 Conclusion Kornprobst (INRIA) PDEs April 2014 12 / 48

1 Can we use PDEs to do some interesting image processing? Definitions From Gaussian filtering to the heat equation 2 Three solutions to go further Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem 3 A success story: Levels-sets 4 Conclusion Kornprobst (INRIA) PDEs April 2014 13 / 48

Solution 1: Make convolution ”nonlinear” Bilateral filtering (Tomasi, Manduchi [1998]) Given u 0 , we define u by : �� 1 d ( x − ξ )˜ u ( x ) = d ( u 0 ( x ) − u 0 ( ξ )) u 0 ( ξ ) d ξ with w ( x ) �� d ( x − ξ )˜ w ( x ) = d ( u 0 ( x ) − u 0 ( ξ )) d ξ Kornprobst (INRIA) PDEs April 2014 14 / 48

Denoising and Simplification before after Kornprobst (INRIA) PDEs April 2014 15 / 48

Denoising and Simplification before after Computer Graphics community like it! Numerous improvements, extensions, efficient implementations and great applications Kornprobst (INRIA) PDEs April 2014 15 / 48

Do your own comics Winnemoller, Olsen, Gooch [2006] (video-spiderman.avi) before after Kornprobst (INRIA) PDEs April 2014 16 / 48

Tone management for photographic look Bae, Paris, Durand [2006] Kornprobst (INRIA) PDEs April 2014 17 / 48

Tone management for photographic look Bae, Paris, Durand [2006] ”Clearing winter storm”, Ansel Adams Kornprobst (INRIA) PDEs April 2014 17 / 48

Tone management for photographic look Bae, Paris, Durand [2006] Our classical picture Kornprobst (INRIA) PDEs April 2014 18 / 48

Tone management for photographic look Bae, Paris, Durand [2006] model before Kornprobst (INRIA) PDEs April 2014 19 / 48

Tone management for photographic look Bae, Paris, Durand [2006] model after Kornprobst (INRIA) PDEs April 2014 20 / 48

1 Can we use PDEs to do some interesting image processing? Definitions From Gaussian filtering to the heat equation 2 Three solutions to go further Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem 3 A success story: Levels-sets 4 Conclusion Kornprobst (INRIA) PDEs April 2014 21 / 48

Solution 2: Modify the heat equation Heat equation ∂ u ∂ t = ∆ u = div ( ? ∇ u ) Kornprobst (INRIA) PDEs April 2014 22 / 48

Solution 2: Modify the heat equation Heat equation ∂ u ∂ t = ∆ u = div ( ? ∇ u ) Perona and Malik model [1990] � 1 / √ 1 + s ∂ u ∂ t = div ( c ( |∇ u | 2 ) ∇ u ) with c ( s ) = exp ( − s ) scalar Kornprobst (INRIA) PDEs April 2014 22 / 48

Solution 2: Modify the heat equation Heat equation ∂ u ∂ t = ∆ u = div ( ? ∇ u ) Perona and Malik model [1990] � 1 / √ 1 + s ∂ u ∂ t = div ( c ( |∇ u | 2 ) ∇ u ) with c ( s ) = exp ( − s ) scalar Buades, Coll et Morel [2005]: Bilateral filter is related to Perona and Malik model Kornprobst (INRIA) PDEs April 2014 22 / 48

Diffusion acts on isophotes Kornprobst (INRIA) PDEs April 2014 23 / 48

Diffusion can interpreted w.r.t. local image structures Nonlinear diffusion is non only a ”controlled” diffusion but it is also related to a directional diffusion depending on local image structures Most diffusion operators can be rewritten as: � � � � . . . u TT + . . . u NN Kornprobst (INRIA) PDEs April 2014 24 / 48

How to better take into account local image structure? Limitation of previous models: Estimation of T , N is local Kornprobst (INRIA) PDEs April 2014 25 / 48

How to better take into account local image structure? Limitation of previous models: Estimation of T , N is local Solution: Use structure tensor � u σ xx � u σ xy k ρ ∗ ∇ u σ ∇ u t σ = k ρ ∗ u σ xy u σ yy Kornprobst (INRIA) PDEs April 2014 25 / 48

How to better take into account local image structure? Limitation of previous models: Estimation of T , N is local Solution: Use structure tensor � u σ xx � u σ xy k ρ ∗ ∇ u σ ∇ u t σ = k ρ ∗ u σ xy u σ yy Weickert [1996] ∂ u ∂ t = div ( D ( k ρ ∗ ∇ u σ ∇ u t σ ) ∇ u ) matrix Kornprobst (INRIA) PDEs April 2014 25 / 48

Example Original Chaleur Perona-Malik Weickert Kornprobst (INRIA) PDEs April 2014 26 / 48

1 Can we use PDEs to do some interesting image processing? Definitions From Gaussian filtering to the heat equation 2 Three solutions to go further Solution 1: Make convolution ”nonlinear” Solution 2: Modify the heat equation Solution 3: Define an optimisation problem 3 A success story: Levels-sets 4 Conclusion Kornprobst (INRIA) PDEs April 2014 27 / 48

When a PDE is a gradient descent of an optimisation problem (also called variational problem) � F ( x , u , ∇ u , ∇ 2 u ) dx v ( x ) = Argmin E ( u ) = u ( x ) Ω Kornprobst (INRIA) PDEs April 2014 28 / 48

Solution 3: Define an optimisation problem Let us start from a model of formation of images, for example : u 0 = R u + η where η is a white Gaussian noise and R is a linear operator. Kornprobst (INRIA) PDEs April 2014 29 / 48