Space-variant directional regularisation for image restoration - PowerPoint PPT Presentation

Space-variant directional regularisation for image restoration problems Luca Calatroni CMAP, ´ Ecole Polytechnique CNRS joint work with: M. Pragliola, A. Lanza, F. Sgallari (University of Bologna) Computational Imaging workshop, Semester program in Computer Vision ICERM March 18-22 2019 Providence, USA

Introduction

Imaging inverse problems Inverse problem formulation Given f , seek u such that f = Ku + n where K (known) models blur and n noise in the data. Task : compute reconstruction u ∗ . Different viewpoints. . . 1

Imaging inverse problems Inverse problem formulation Given f , seek u such that f = Ku + n where K (known) models blur and n noise in the data. Task : compute reconstruction u ∗ . Different viewpoints. . . Optimisation Statistics PDEs Maximise posterior: Evolution problem MAP estimation u ∗ ∈ argmax u ∗ ∈ argmin  P ( u ; f , K ) − log ( P ( u ) P ( f , K ; u )) u t = −∇ R ( u ) − µ ∇ Φ( Ku ; f )  u = ⇒ u = ⇒   u (0) = f Use Bayes’ formula Variational regularisation:   b.c.  u ∗ ∈ argmin P ( u ) P ( f , K ; u ) u ∗ ∈ argmax R ( u )+ µ Φ( Ku ; f ) Get u ∗ as the steady state u P ( f ) u of the system above Discrete/continuous framework. Analogies : → P ( u ), R ( u ) encode prior assumptions on u ( topic of this talk! ) - P ( f , K ; u ), Φ( f , K ; u ) describe noise statistics 1

Imaging inverse problems Inverse problem formulation Given f , seek u such that f = Ku + n where K (known) models blur and n noise in the data. Task : compute reconstruction u ∗ . Different viewpoints. . . Optimisation Statistics Maximise posterior: MAP estimation u ∗ ∈ argmax u ∗ ∈ argmin P ( u ; f , K ) − log ( P ( u ) P ( f , K ; u )) u u Use Bayes’ formula Variational regularisation: u ∗ ∈ argmin P ( u ) P ( f , K ; u ) u ∗ ∈ argmax R ( u )+ µ Φ( Ku ; f ) u P ( f ) u Discrete/continuous framework. 1

Tailored image regulariser? Image model adapted to local features? 2

Statistical viewpoint: Markov Random Fields Standard image priors are based on stationary Markov Random Field modelling n n � � P ( u ) = 1 exp ( − α V N i ( u ) ) = 1 � � − α Z exp V N i ( u ) , Z i =1 i =1 where α > 0, N i is the clique around i and V N i is the Gibbs’ potential in N i . 3

Statistical viewpoint: Markov Random Fields Standard image priors are based on stationary Markov Random Field modelling n � � � � P ( u ) = 1 = 1 � − α TV ( u ) − α � ( ∇ u ) i � 2 Z exp Z exp , i =1 where α > 0 and V N i ( u ) = � ( ∇ u ) i � 2 . 3

Statistical viewpoint: Markov Random Fields Standard image priors are based on stationary Markov Random Field modelling � � � n � P ( u ) = 1 = 1 � Z exp − α TV ( u ) Z exp − α � ( ∇ u ) i � 2 , i =1 where α > 0 and V N i ( u ) = � ( ∇ u ) i � 2 . Interpretation q := �∇ u � 2 is distributed locally as a α - half-Laplacian distribution:  α exp( − α q i ) for q i ≥ 0 ,  P ( q i ; α ) =  0 for q i < 0 , and α describes image scales. One-parameter family describing all pixel features. . . too restrictive? 3

Variational and PDE approach: TV regularisation n � TV ( u ) = � ( ∇ u ) i � 2 i =1 1 Rudin, Osher, Fatemi, ’92 4

Variational and PDE approach: TV regularisation n � TV ( u ) = � ( ∇ u ) i � 2 i =1 Variational model PDE model (non-linear diffusion) E ( u ) = TV ( u ) + λ 2 � u − f � 2 u t = p + λ ( f − u ) , p ∈ ∂ TV ( u ) 2 Seek u fitting the data with low TV: edges preservation & noise removal 1 ! (a) f (b) Tikhonov (c) TV Edge-preserving non-linear diffusion PDE. . . 1 Rudin, Osher, Fatemi, ’92 4

Introduction Describing anisotropy

Modelling anisotropy Anisotropy operators For all x ∈ Ω, let λ : Ω → R 2 be a positive vector field and let θ : Ω → [0 , 2 π ) describe local orientation. � cos θ ( x ) � λ 1 ( x ) � � 0 sin θ ( x ) Λ ( x ) := , R θ ( x ) := . 0 λ 2 ( x ) − sin θ ( x ) cos θ ( x ) 5

Modelling anisotropy Anisotropy operators For all x ∈ Ω, let λ : Ω → R 2 be a positive vector field and let θ : Ω → [0 , 2 π ) describe local orientation. � cos θ ( x ) � λ 1 ( x ) � � 0 sin θ ( x ) Λ ( x ) := , R θ ( x ) := . 0 λ 2 ( x ) − sin θ ( x ) cos θ ( x ) M λ ,θ := Λ R θ is the anisotropic metric and W λ ,θ := M ⊺ λ ,θ M λ ,θ . Note : if z ( x ) := (cos θ ( x ) , sin θ ( x )), then: � � λ 1 ( x ) ∂ z ( x ) u ( x ) M λ ,θ ∇ u ( x ) = ⇒ locally weighted gradient along z λ 2 ( x ) ∂ z ( x ) ⊥ u ( x ) 5

Modelling anisotropy Anisotropy operators For all x ∈ Ω, let λ : Ω → R 2 be a positive vector field and let θ : Ω → [0 , 2 π ) describe local orientation. � cos θ ( x ) � λ 1 ( x ) � � 0 sin θ ( x ) Λ ( x ) := , R θ ( x ) := . 0 λ 2 ( x ) − sin θ ( x ) cos θ ( x ) M λ ,θ := Λ R θ is the anisotropic metric and W λ ,θ := M ⊺ λ ,θ M λ ,θ . Note : if z ( x ) := (cos θ ( x ) , sin θ ( x )), then: � � λ 1 ( x ) ∂ z ( x ) u ( x ) M λ ,θ ∇ u ( x ) = ⇒ locally weighted gradient along z λ 2 ( x ) ∂ z ( x ) ⊥ u ( x ) Using this formalism: • Anisotropic diffusion: � � u t = div W λ ,θ ∇ u • Corresponding directional energy: � � � � 2 dx E λ ,θ ( u ) := � M λ ,θ ∇ u Ω 5

Directional regularisation for imaging: previous work - Statistics of natural images : Generalised Gaussian PDFs (Mallat, ’89), Laplace (Green, ’92), non-stationary MRFs & optimisation (Lanza, Morigi, Pragliola, Sgallari, ’16, ’18),. . . ; - Directional variational regularisation : Variable exponent (Chen, Levine, Rao, ’06), DT(G)V (Bayram, ’12, Kongskov, Dong, Knudsen, ’17, Parisotto, Sch¨ onlieb, Masnou, ’18), . . . - Application to inverse problems : Limited Angle Tomography (Tovey, Benning et al., ’19), . . . - PDEs : structure tensor modelling (Weickert, ’98).  u t = div ( D ( J ρ ( ∇ u σ )) ∇ u ) in Ω × (0 , T ]    � D ( J ρ ( ∇ u σ ) ∇ u ) , n � = 0 on ∂ Ω   u (0 , x ) = f ( x ) in Ω ,  where for convolution kernels K ρ , K σ � � J ρ ( ∇ u σ ) := K ρ ∗ ∇ u σ ⊗ ∇ u σ , u σ := K σ ∗ u . and D is a smooth and symmetric diffusion tensor. - Consistent numerical schemes (Fehrenbach, Mirebeau, ’14,. . . ). 6

Directional regularisation for imaging: previous work - Statistics of natural images : Generalised Gaussian PDFs (Mallat, ’89), Laplace (Green, ’92), non-stationary MRFs & optimisation (Lanza, Morigi, Pragliola, Sgallari, ’16, ’18),. . . ; - Directional variational regularisation : Variable exponent (Chen, Levine, Rao, ’06), DT(G)V (Bayram, ’12, Kongskov, Dong, Knudsen, ’17, Parisotto, Sch¨ onlieb, Masnou, ’18), . . . - Application to inverse problems : Limited Angle Tomography (Tovey, Benning et al., ’19), . . . - PDEs : structure tensor modelling (Weickert, ’98).  u t = div ( D ( J ρ ( ∇ u σ )) ∇ u ) in Ω × (0 , T ]    � D ( J ρ ( ∇ u σ ) ∇ u ) , n � = 0 on ∂ Ω   u (0 , x ) = f ( x ) in Ω ,  where for convolution kernels K ρ , K σ � � J ρ ( ∇ u σ ) := K ρ ∗ ∇ u σ ⊗ ∇ u σ , u σ := K σ ∗ u . and D is a smooth and symmetric diffusion tensor. - Consistent numerical schemes (Fehrenbach, Mirebeau, ’14,. . . ). Common problem Tailored regularisation adapted to local image orientation/structures? 6

Introduction Variational space-variant regularisation

Space-variant and directional regularisation models Discrete formulation. Let Ω be the image domain with | Ω | = n . n � � ( ∇ u ) i � 2 TV ( u ) = i =1 7

Space-variant and directional regularisation models Discrete formulation. Let Ω be the image domain with | Ω | = n . n � � ( ∇ u ) i � p TV p ( u ) = 2 , i =1 Enforcing sparsity : - p = 2 : Tikhonov regularisation (Tikhonov, Arsenin, ’77). - p = 1 : Total Variation (Rudin, Osher, Fatemi, ’92). - 0 < p < 1 : Non-convex regularisation (Hinterm¨ uller, Wu, Valkonen, ’13, ’15, Nikolova, Ng, Tam, ’10). 7

Space-variant and directional regularisation models Discrete formulation. Let Ω be the image domain with | Ω | = n . n � TV sv � ( ∇ u ) i � p i p ( u ) = 2 i =1 Space-variant modelling : - 1 ≤ p i ≤ 2 : Convex, space-variant regularisation (Blomgren, Chan, Mulet, Wong, ’97, Chen, Levine, Rao, ’06). - p i ∈ ( 0 , 2 ] : Non-convex space-variant regularisation (Lanza, Morigi, Pragliola, Sgallari, ’16, ’18). 7

Space-variant and directional regularisation models Discrete formulation. Let Ω be the image domain with | Ω | = n . n � � M λ i ,θ i ( ∇ u ) i � p DTV p ( u ) = 2 , θ i ∈ [0 , 2 π ) i =1 where M λ i ,θ i = Λ i R θ i as before. Directional modelling : - p = 2 : Anisotropic diffusion (Weickert, ’98). - p = 1 : Directional Total (Generalised) Variation for dominant direction θ i ≡ ¯ θ (Kongskov, Dong, Knudsen, ’17) and inverse problems (Tovey, Benning et al., ’19). Combine (possibly non-convex) space-variance AND directional modelling? 7

A flexible directional & space-variant regularisation

A flexible directional & space-variant regularisation Statistical motivation