Practical denoising of clipped or overexposed noisy images - PowerPoint PPT Presentation

1 TAMPERE UNIVERSITY OF TECHNOLOGY Department of Signal Processing — Transforms and Spectral Methods Group Practical denoising of clipped or overexposed noisy images Alessandro Foi www.cs.tut.fi/~foi EUSIPCO-2008 - Lausanne, Switzerland, 27.9.2008

2 Clipped noisy data original added noise and then clipped

3 What is proposed? Complete framework for the Þ ltering of clipped observations with signal-dependent noise Ingredients: A good (transform-based) denoising algorithm for AWGN Four nonlinear homomorphic transformations: 1) variance-stabilizing transformation [and its 3) inverse] 2) debiasing 4) declipping

4 Preliminaries

5 Modeling clipped signal-dependent observations x ∈ X ⊂ Z 2 , z ( x ) = max { 0 , min { z ( x ) , 1 }} , ˜ z ( x ) = y ( x ) + σ ( y ( x )) ξ ( x ) y : X → Y ⊆ R unknown original image (deterministic) σ ( y ( x )) ξ ( x ) zero-mean random error σ : R → R + standard-deviation function (deterministic) ξ ( x ) random variable E { ξ ( x ) } = 0 var { ξ ( x ) } = 1 y ( x ) = E { z ( x ) } expectation σ ( y ( x )) = std { z ( x ) } standard deviation

6 Modeling clipped signal-dependent observations x ∈ X ⊂ Z 2 , z ( x ) = max { 0 , min { z ( x ) , 1 }} , ˜ z ( x ) = y ( x ) + σ ( y ( x )) ξ ( x ) y ( x )) ˜ z ( x ) = ˜ ˜ y ( x ) + ˜ σ (˜ ξ ( x ) y ( x ) = E { ˜ ˜ z ( x ) } expectation σ : [0 , 1] → R + ˜ standard-deviation function (of expectation) ˜ σ (˜ y ( x )) = std { ˜ z ( x ) } standard deviation

7 Modeling clipped signal-dependent observations x ∈ X ⊂ Z 2 , z ( x ) = max { 0 , min { z ( x ) , 1 }} , ˜ z ( x ) = y ( x ) + σ ( y ( x )) ξ ( x ) y ( x )) ˜ z ( x ) = ˜ ˜ y ( x ) + ˜ σ (˜ ξ ( x ) y ( x ) = E { ˜ ˜ z ( x ) } expectation σ : [0 , 1] → R + ˜ standard-deviation function (of expectation) ˜ σ (˜ y ( x )) = std { ˜ z ( x ) } standard deviation Goal : estimate y from ˜ z

8 Modeling raw-data The random error (before clipping) is composed of two mutually independent parts: σ ( y ( x )) ξ ( x ) = η p ( y ( x )) + η g ( x ) η p Poissonian signal-dependent component η g Gaussian signal-independent component ¡ ¢ y ( x ) + η p ( y ( x )) χ ∼ P ( χy ( x )) , χ > 0 η g ( x ) ∼ N (0 , b ) , b > 0 σ 2 ( y ( x )) = ay ( x ) + b, a = χ − 1 p Normal approximation σ ( y ( x )) ξ ( x ) = ay ( x ) + bξ ( x ) , ξ ( x ) ∼ N (0 , 1) Foi et al., IEEE TIP, Oct. 2008.

9 (Generalized) Probability distributions Before clipping: ³ ´ ζ − y 1 ℘ z ( ζ ) = σ ( y ) φ σ ( y ) After clipping: ³ ´ ³ ´ ³ ³ ´´ − y ζ − y 1 − y 1 ℘ ˜ z ( ζ ) = Φ δ 0 ( ζ ) + σ ( y ) φ χ [0 , 1] + 1 − Φ δ 0 (1 − ζ ) σ ( y ) σ ( y ) σ ( y ) φ and Φ are p.d.f. and c.d.f. of N (0 , 1)

10 a = 0 . 02 2 , 0 . 06 2 , 0 . 10 2 b = 0 . 04 2 a = 0 . 04 2 b = 0 . 02 2 , 0 . 06 2 , 0 . 10 2

11 Clipped ¿ non-clipped Direct transformations (non-clipped to clipped) ³ ´ ³ ´ y 1 − y y = A ( y, σ ( y )) = σ ( y ) E m ˜ − y + 1 − σ ( y ) E m σ ( y ) σ ( y ) ³ ´ ³ ´ y 1 − y σ (˜ ˜ y ) = B ( y, σ ( y )) = σ ( y ) S m S m σ ( y ) σ ( y ) where E m ( µ ) = Φ ( µ ) µ + φ ( µ ) S 2 Φ ( µ ) + E m ( µ ) µ − E 2 m ( µ ) = m ( µ )

12 Clipped ¿ non-clipped Inverse transformations (clipped to non-clipped) ³ ´ ³ ´ y ˜ 1 − ˜ y y = C (˜ y, ˜ σ (˜ y )) = ˜ y E r − ˜ y + 1 − (1 − ˜ y ) E r σ (˜ ˜ y ) σ (˜ ˜ y ) σ (˜ ˜ y ) ³ ´ ³ ´ σ ( y ) = D (˜ y, ˜ σ (˜ y )) = y ˜ 1 − ˜ y S r S r σ (˜ ˜ y ) ˜ σ (˜ y ) where ρ = E m ( µ ) E r ( ρ ) = µ/ E m ( µ ) S r ( ρ ) = S m ( µ ) S m ( µ )

13 Algorithm

14 Algorithm: Step 0, Noise estimation If σ (or ˜ σ ) is known, go ahead. Otherwise, estimate noise. Estimation is possible provided that σ is described by few parameters (e.g., a and b for the raw-data), using algorithm in [ Foi et al., IEEE TIP, Oct. 2008 ].

15 Algorithm: Step 1, Variance stabilization Idea: transform ˜ z to f (˜ z ) such that var { f (˜ z ) } is approximately constant. Z t c f ( t ) = y ) d ˜ y, t, t 0 ∈ [0 , 1] ˜ σ (˜ t 0 Main result: analytical proof that the integral does not diverge (i.e. f is bounded) if b 6 = 0 , − a (i.e., for all cases of practical interest).

16 Algorithm: Step 2, Denoising z ) assuming constant variance c 2 . Denoise f (˜ D ho ( f (˜ z )) ≈ E { f (˜ z ) } We use BM3D algorithm [ Dabov et al., IEEE TIP, Aug. 2007 ].

17 Algorithm: Step 3, Debiasing D ho ( f (˜ z )) ≈ E { f (˜ z ) } 6 = f ( E { ˜ z } )

18 Algorithm: Step 3, Debiasing D ho ( f (˜ z )) ≈ E { f (˜ z ) } 6 = f ( E { ˜ z } ) ³ ´ ³ ³ ´´ R 1 R 1 1 ζ − y 1 − y E { f (˜ z ) } = 0 f ( ζ ) ℘ ˜ z ( ζ ) dζ = 0 f ( ζ ) σ ( y ) φ dζ + f (1) 1 − Φ σ ( y ) σ ( y ) ³R 1 ´ ¡R 1 ¢ ¡ ¡ − y ¢ ¡ ζ − y ¢ ¡ ¡ 1 − y ¢¢ ¢ 1 f ( E { ˜ z } ) = f 0 ζ℘ ˜ z ( ζ ) dζ = f 0 ζ Φ δ 0 ( ζ )+ σ ( y ) φ χ [0 , 1] + 1 − Φ δ 0 (1 − ζ ) dζ σ ( y ) σ ( y ) σ ( y )

19 Algorithm: Step 3, Debiasing D ho ( f (˜ z )) ≈ E { f (˜ z ) } 6 = f ( E { ˜ z } ) ³ ´ ³ ³ ´´ R 1 R 1 1 ζ − y 1 − y E { f (˜ z ) } = 0 f ( ζ ) ℘ ˜ z ( ζ ) dζ = 0 f ( ζ ) σ ( y ) φ dζ + f (1) 1 − Φ σ ( y ) σ ( y ) ³R 1 ´ ¡R 1 ¢ ¡ ¡ − y ¢ ¡ ζ − y ¢ ¡ ¡ 1 − y ¢¢ ¢ 1 f ( E { ˜ z } ) = f 0 ζ℘ ˜ z ( ζ ) dζ = f 0 ζ Φ δ 0 ( ζ )+ σ ( y ) φ χ [0 , 1] + 1 − Φ δ 0 (1 − ζ ) dζ σ ( y ) σ ( y ) σ ( y ) Let h : [0 , f (1)] → [0 , f (1)] be the function de Þ ned (implicitly varying y ) by h f ( E { ˜ z } ) 7 − → E { f (˜ z ) }

20 Algorithm: Step 3, Debiasing D ho ( f (˜ z )) ≈ E { f (˜ z ) } 6 = f ( E { ˜ z } ) ³ ´ ³ ³ ´´ R 1 R 1 1 ζ − y 1 − y E { f (˜ z ) } = 0 f ( ζ ) ℘ ˜ z ( ζ ) dζ = 0 f ( ζ ) σ ( y ) φ dζ + f (1) 1 − Φ σ ( y ) σ ( y ) ³R 1 ´ ¡R 1 ¢ ¡ ¡ − y ¢ ¡ ζ − y ¢ ¡ ¡ 1 − y ¢¢ ¢ 1 f ( E { ˜ z } ) = f 0 ζ℘ ˜ z ( ζ ) dζ = f 0 ζ Φ δ 0 ( ζ )+ σ ( y ) φ χ [0 , 1] + 1 − Φ δ 0 (1 − ζ ) dζ σ ( y ) σ ( y ) σ ( y ) Let h : [0 , f (1)] → [0 , f (1)] be the function de Þ ned (implicitly varying y ) by h f ( E { ˜ z } ) 7 − → E { f (˜ z ) } h − 1 ( D ho ( f (˜ z ))) ≈ f ( E { ˜ z } ) Debiasing

21 Algorithm: Step 3, Debiasing D ho ( f (˜ z )) ≈ E { f (˜ z ) } 6 = f ( E { ˜ z } ) ³ ´ ³ ³ ´´ R 1 R 1 1 ζ − y 1 − y E { f (˜ z ) } = 0 f ( ζ ) ℘ ˜ z ( ζ ) dζ = 0 f ( ζ ) σ ( y ) φ dζ + f (1) 1 − Φ σ ( y ) σ ( y ) ³R 1 ´ ¡R 1 ¢ ¡ ¡ − y ¢ ¡ ζ − y ¢ ¡ ¡ 1 − y ¢¢ ¢ 1 f ( E { ˜ z } ) = f 0 ζ℘ ˜ z ( ζ ) dζ = f 0 ζ Φ δ 0 ( ζ )+ σ ( y ) φ χ [0 , 1] + 1 − Φ δ 0 (1 − ζ ) dζ σ ( y ) σ ( y ) σ ( y ) Let h : [0 , f (1)] → [0 , f (1)] be the function de Þ ned (implicitly varying y ) by h f ( E { ˜ z } ) 7 − → E { f (˜ z ) } h − 1 ( D ho ( f (˜ z ))) ≈ f ( E { ˜ z } ) Debiasing Caution: in image processing literature it is always assumed E { f (˜ z ) } = f ( E { ˜ z } ) Arsenault and Denis, “Integral expression for transforming signal-dependent noise..”, Optics Letters , 1981. Prucnal and Saleh, “Transformation of image-signal-dependent noise into...”, Optics Letters , 1981. Kasturi, Walkup, and Krile, “Image restoration by transformation of signal-dependent noise...”, App.Optics , 1983. Hirakawa and Parks, “Image denoising using total least squares”, IEEE Trans. Image Process ., 2006.

22 Algorithm: Step 4, Inversion of f f − 1 ¡ ¢ ≈ f − 1 ¡ ¢ h − 1 ( D ho ( f (˜ z ))) f ( E { ˜ z } ) = E { ˜ z } This provides an estimate b ˜ y of ˜ y

23 Algorithm: Step 4, Inversion of f f − 1 ¡ ¢ ≈ f − 1 ¡ ¢ h − 1 ( D ho ( f (˜ z ))) f ( E { ˜ z } ) = E { ˜ z } Algorithm: Step 5, Declipping y = f − 1 ¡ ¢ y )) and de Þ ne b h − 1 ( D ho ( f (˜ Recall y = C (˜ y, ˜ σ (˜ ˜ z ))) ³ ³ ´´ b b y = C ˆ ˜ y, ˜ σ y ˜ Declipping

24 Experiments

25 Observations ˜ z µ 2 π ¶ y ( x 1 , x 2 ) = 0 . 7 sin 512 x 1 + 0 . 5

26 Noise estimation a = 0 . 0038 ˆ estimation and Þ tting ˆ b = 0 . 022 st.dev.-function ˜ σ

27 f and h − 1 debiasing h − 1 variance-stabilizing f

28 Denoised estimate before declipping

29 Declipped estimate

30 Declipped estimate

31 Cross section

32 Observations ˜ z Raw-data from Fuji Þ lm FinePix S9600, ISO 1600

33 Noise estimation a = 0 . 0043 ˆ estimation and Þ tting ˆ b = 0 . 00038 st.dev.-function ˜ σ

34 f and h − 1 debiasing h − 1 variance-stabilizing f

35 Denoised estimate before declipping

36 Declipped estimate

37 Declipped estimate

38 Cross section

39 Related works Noise estimation A. Foi, M. Trimeche, V. Katkovnik, and K. Egiazarian, “Practical Poissonian-Gaussian noise modeling and Þ tting for single-image raw-data”, to appear in IEEE Trans. Image Process . (October 2008) Multiframe raw-data G. Boracchi and A. Foi, “Multiframe raw-data denoising based on block-matching and 3-D Þ ltering for low-light imaging and stabilization”, Proc. LNLA 2008 , August 2008.

40 Matlab software www.cs.tut.fi/~foi/sensornoise.html