Nonlocal methods for image processing Lecture note, Xiaoqun Zhang - PowerPoint PPT Presentation

Nonlocal methods for image processing Nonlocal methods for image processing Lecture note, Xiaoqun Zhang Oct 30, 2009 1/29

Nonlocal methods for image processing Outline Local smoothing Filters 1 Nonlocal means filter 2 Nonlocal operators 3 Applications 4 References 5 2/29

Nonlocal methods for image processing General Model v ( x ) = u ( x ) + n ( x ) , x ∈ Ω v ( x ) observed image u ( x ) true image n ( x ) i.i.d gaussian noise (white noise) Gaussian kernel 4 πh 2 e − | x | 2 1 x → G h ( x ) = 4 h 2 3/29

Nonlocal methods for image processing Local smoothing Filters Outline Local smoothing Filters 1 Nonlocal means filter 2 Nonlocal operators 3 Denoising by nonlocal functionals Inverse problems by nonlocal regularization Nonlocal regularization with Bregmanized methods Applications 4 Compressive sampling Deconvolution Wavelet Inpainting References 5 4/29

Nonlocal methods for image processing Local smoothing Filters Linear low-pass filter Idea: average in a local spatial neighborhood � y − x � 2 1 � GF h ( v )( x ) = G h ∗ v ( x ) = v ( y ) exp 4 h 2 dy C ( x ) y ∈ Ω where C ( x ) = 4 πh 2 Pro: work well for harmonic function (homogenous region) Con: perform poorly on singular part, namely edge and texture 5/29

Nonlocal methods for image processing Local smoothing Filters Anisotropic filter Idea: average only in the direction orthogonal to Dv ( x )( ∂v ( x ) ∂x , ∂v ( y ) ∂y ) . v ( x + f Dv ( x ) ⊥ 1 � − t 2 h 2 dt AF h ( v )( x ) = | Dv ( x ) | ) exp C ( x ) t where C ( x ) = 4 πh 2 . Pro: Avoid blurring effect of Gaussian filter, maintaining edges. Con: perform poorly on flat region, worse there than a Gaussian blur. 6/29

Nonlocal methods for image processing Local smoothing Filters Neighboring filter Spatial neighborhood B ρ ( x ) = { y ∈ Ω |� y − x � ≤ ρ } Gray-level neighborhood B ( x, h ) = { y ∈ Ω |� v ( y ) − v ( x ) � ≤ ρ } for a given image v . Yaroslavsky filter u ( y ) e − | u ( y ) − u ( x ) | 2 1 � Y NF h,ρ = dy 4 h 2 C ( x ) B ρ ( x ) Bilateral(SUSAN) filter − | y − x | 2 1 u ( y ) e − | u ( y ) − u ( x ) | 2 � 4 ρ 2 dy SUSAN h,ρ = 4 h 2 e C ( x ) Behave like weighted heat equation, enhancing the edges 7/29

Nonlocal methods for image processing Local smoothing Filters Denoising example 8/29

Nonlocal methods for image processing Nonlocal means filter Outline Local smoothing Filters 1 Nonlocal means filter 2 Nonlocal operators 3 Denoising by nonlocal functionals Inverse problems by nonlocal regularization Nonlocal regularization with Bregmanized methods Applications 4 Compressive sampling Deconvolution Wavelet Inpainting References 5 9/29

Nonlocal methods for image processing Nonlocal means filter Nonlocal mean filter 1 Idea: Take advantage of high degree of redundancy of natural images. 10/29

Nonlocal methods for image processing Nonlocal means filter Denoising formula 1 � NLM ( v )( x ) := w ( x, y ) v ( y ) dy, C ( x ) Ω where w ( x, y ) = exp {− G a ∗ ( || v ( x + · ) − v ( y + · ) || 2 )(0) } , 2 h 2 0 � C ( x ) = w v ( x, y ) dy Ω 11/29

Nonlocal methods for image processing Nonlocal means filter Weight from clean image 12/29

Nonlocal methods for image processing Nonlocal means filter Weight from noisy image 13/29

Nonlocal methods for image processing Nonlocal means filter Example 14/29

Nonlocal methods for image processing Nonlocal means filter Comparison with other methods 15/29

Nonlocal methods for image processing Nonlocal operators Outline Local smoothing Filters 1 Nonlocal means filter 2 Nonlocal operators 3 Denoising by nonlocal functionals Inverse problems by nonlocal regularization Nonlocal regularization with Bregmanized methods Applications 4 Compressive sampling Deconvolution Wavelet Inpainting References 5 16/29

Nonlocal methods for image processing Nonlocal operators Nonlocal operators 2 /Graph based Regularization Given a nonnegative and symmetric weight function w ( x, y ) for each pair of points ( x, y ) ∈ Ω × Ω : Nonlocal gradient of an image u ( x ) : � ∇ w u ( x, y ) = ( u ( y ) − u ( x )) w ( x, y ) : Ω × Ω → Ω Nonlocal divergence of a gradient filed p ( x, y ) : Ω × Ω → R is defined by < ∇ w u, p > = − < u, div w p >, ∀ u ( x ) , p ( x, y ) � � = ⇒ div w p ( x ) = ( p ( x, y ) − p ( y, x )) w ( x, y ) dy. Ω Nonlocal functionals of u : 1 � 1 � � |∇ w u ( x ) | 2 : |∇ w u ( x, y ) | 2 J NL/H 1 ( f ) = 4 4 Ω x y �� � � |∇ w u ( x, y ) | 2 . J NL/TV ( f ) = |∇ w u ( x ) | 1 : Ω x y 17/29

Nonlocal methods for image processing Nonlocal operators Denoising by nonlocal functionals Nonlocal H 1 regularization by non-local means Model: min J NL/H 1 ( u ) + µ 2 || u − f || 2 Euler-Lagrange equation: L w ( u ) u + µ ( u − f ) = 0 , where L w is unnormalized graph laplacian : � L w ( u ) = w ( x, y )( u ( x ) − u ( y )) . Ω We can replace L w ( u ) by normalized graph laplacian 3 1 L 0 w = C ( x ) L w = Id − NLM w ( u ) . Semi-explicit iteration: for a time step τ > 0 , s = 1 + τ + τµ, α 1 = τ s , α 2 = τµ s : u k +1 = (1 − α 1 ) u k + α 1 NLM w ( u k ) + α 2 f. 3 When N → ∞ and h 0 → 0 , then L 0 w converges to the continuous manifold Laplace - Beltrami operator. 18/29

Nonlocal methods for image processing Nonlocal operators Denoising by nonlocal functionals Nonlocal TV regularization by Chambolle’s algorithm Model: min u J NL/TV,w ( u ) + µ 2 || u − f || 2 Extension of Chambolle’s projection method for Nonlocal TV: � < ∇ w u, p > + µ 2 || u − f || 2 , inf sup u || p ||≤ 1 Ω × Ω where the solution can be solved by a projected solution u ∗ = f − 1 µ ÷ w p ∗ . and the dual variable p ∗ is obtained by � < ∇ w u, p > + 1 2 µ || div w p || 2 . sup || p ||≤ 1 Ω × Ω Algorithm: p n +1 = p n + τ ∇ w ( div w p n − µf ) 1 + τ |∇ w ( div w p n − µf ) | , τ > 0 19/29

Nonlocal methods for image processing Nonlocal operators Inverse problems by nonlocal regularization Deblurring by Nonlocal Means 4 Problem: f = Au + n , A linear operator, n Gaussian noise. Idea: Use initial blurry and noisy image f to compute the weight. J NLM ,w ( f ) := min || u − NLM f u || 2 + λ 2 || Au − f || 2 (1) which is equivalent to w f ( u ) || 2 + λ J NLM ,w ( f ) := min || L 0 2 || Au − f || 2 (2) where L 0 w f is the normalized graph laplacian with the weight computed from f . Gradient descents flow: (( L 0 w f ) ∗ L 0 w f ) u + λA ∗ ( Au − f ) = 0 4 A. Buades, B. Coll, and J-M. Morel. 2006 20/29

Nonlocal methods for image processing Nonlocal operators Inverse problems by nonlocal regularization Image recovery via nonlocal operators Idea: Use a deblurred image to compute the weight. 1 Preprocessing: Compute a deblurred image via a fast method: u 0 = min 1 2 || Au − f || 2 + δ || u || 2 ⇐ ⇒ u 0 = ( A ∗ A + δ ) − 1 A ∗ f. where δ is chosen optimally by respecting the condition σ 2 = || Au 0 − f || 2 where σ 2 is the noise level in blurry image. Compute the nonlocal weight w 0 by using u 0 as a reference image (set h 0 = σ 2 || ( A ∗ A + δ ) − 1 A ∗ || 2 .) 2 Nonlocal regularization with the fixed weight w 0 : min J w 0 ( u ) + λ 2 || Au − f || 2 by gradient descent. 21/29

Nonlocal methods for image processing Nonlocal operators Inverse problems by nonlocal regularization Nonlocal regularization for inverse problems Idea : nonlocal weight updating during nonlocal regularization by operator splitting. Model : u J w ( u ) ( u ) + λ 2 || Au − v || 2 min Approximated Algorithm: u k + 1 v k +1 µ A ∗ ( f − Au k ) =   w k +1 w ( v k +1 )( optional ) = (3) arg min J NL/TV,w k +1 + λµ u k +1 2 || u − v k +1 || 2 =  where u k +1 is solved by Chamobelle’s method for NLTV. 22/29

Nonlocal methods for image processing Nonlocal operators Nonlocal regularization with Bregmanized methods Nonlocal regularization with Bregmanized methods With/without weight updating: Algorithm: f k + f − Au k f k +1  = u k + 1 µ A ∗ ( f k +1 − Au k )  v k +1  =  (4) w k +1 w ( v k +1 )( optional ) =   arg min J NL/TV,w k +1 + λµ u k +1 2 || u − v k +1 || 2  = 23/29

Nonlocal methods for image processing Applications Outline Local smoothing Filters 1 Nonlocal means filter 2 Nonlocal operators 3 Denoising by nonlocal functionals Inverse problems by nonlocal regularization Nonlocal regularization with Bregmanized methods Applications 4 Compressive sampling Deconvolution Wavelet Inpainting References 5 24/29

Nonlocal methods for image processing Applications Compressive sampling Compressive sampling : Au = RFu True Image Initial guess TV NLTV Figure: Data: 30% random Fourier measurements 25/29

Nonlocal methods for image processing Applications Deconvolution Deconvolution: Au = k ∗ u True Image Blurry and noisy Image Fix weight Update weight Figure: 9 × 9 box average blur kernel, σ = 3 26/29

Nonlocal methods for image processing Applications Wavelet Inpainting Wavelet Inpainting: Au = RWu Original Received, PSNR= 17.51 TV, PSNR=28.64 NLTV, PSNR= 36.06 Figure: Block loss(including low-low frequencies loss). For both TV and NLTV, the initial guess is the received image 27/29