Image Denoising and Enhancement Karen Egiazarian (TUT , NI) - - PowerPoint PPT Presentation

▶
image denoising and enhancement
SMART_READER_LITE
LIVE PREVIEW

Image Denoising and Enhancement Karen Egiazarian (TUT , NI) - - PowerPoint PPT Presentation

Nov 04, 2022 424 likes •1.41k views

1 Image Denoising and Enhancement Karen Egiazarian (TUT , NI) Department of Signal Processing 2 Image denoising: motivating example Images are inevitably corrupted by various degradations and particularly by noise. Megapixels

slide-1
SLIDE 1

Department of Signal Processing

1

Image Denoising and Enhancement

Karen Egiazarian (TUT , NI)

slide-2
SLIDE 2

Department of Signal Processing

2

Image denoising: motivating example

  • Images are inevitably corrupted by various degradations and

particularly by noise.

  • Megapixels race: Pixels are getting smaller, and images even noisier

= _

image noise denoised image

Canon Powershot A590IS ISO 800

slide-3
SLIDE 3

Department of Signal Processing

3

Imaging Sensors: Exposure-time/noise trade-off

Digital imaging sensors can have very different performance Different acquisition settings result in different noise levels in the image “Exposure-time/noise trade-off “

slide-4
SLIDE 4

Department of Signal Processing

  • Intro
  • Signal-dependent noise modeling and removal for digital imaging

sensors

  • Local polynomial approximations (LPA-ICI)
  • Advanced image processing techniques:
  • shape-adaptive methods
  • nonlocal transform-based methods
  • Applications:
  • denoising
  • deblurring
  • deblocking
  • super-resolution/zooming

4

Outline

slide-5
SLIDE 5

Department of Signal Processing

5

Outline

  • Signal-dependent noise modeling and removal for digital imaging

sensors

  • Local polynomial approximations (LPA-ICI)
  • Advanced image processing techniques:
  • shape-adaptive methods
  • nonlocal transform-based methods
  • Applications:
  • denoising
  • deblurring
  • deblocking
  • super-resolution/zooming
slide-6
SLIDE 6

Department of Signal Processing

6

Outline

  • Signal-dependent noise modeling and removal for digital imaging

sensors

  • Local polynomial approximations (LPA-ICI)
  • Advanced image processing techniques:
  • shape-adaptive methods
  • nonlocal transform-based methods
  • Applications:
  • denoising
  • deblurring
  • deblocking
  • super-resolution/zooming
slide-7
SLIDE 7

Department of Signal Processing

7

Outline

  • Signal-dependent noise modeling and removal for digital imaging

sensors

  • Local polynomial approximations equipped with ICI rule (LPA-ICI)
  • Advanced image processing techniques:
  • shape-adaptive methods
  • nonlocal transform-based methods
  • Applications:
  • denoising
  • deblurring
  • deblocking
  • super-resolution/zooming
slide-8
SLIDE 8

Department of Signal Processing

8

Outline

  • Signal-dependent noise modeling and removal for digital imaging

sensors

  • Local polynomial approximations equipped with ICI rule (LPA-ICI)
  • Advanced image processing techniques:
  • shape-adaptive methods
  • nonlocal transform-based methods
  • Applications:
  • denoising
  • deblurring
  • deblocking
  • super-resolution/zooming
slide-9
SLIDE 9

Department of Signal Processing

9

Outline

  • Signal-dependent noise modeling and removal for digital imaging

sensors

  • Local polynomial approximations (LPA-ICI)
  • Advanced image processing techniques:
  • shape-adaptive methods
  • nonlocal transform-based methods
  • Applications:
  • denoising
  • deblurring
  • deblocking
  • super-resolution/zooming
slide-10
SLIDE 10

Department of Signal Processing

10

Outline

  • Signal-dependent noise modeling and removal for digital imaging

sensors

  • Local polynomial approximations (LPA-ICI)
  • Advanced image processing techniques:
  • shape-adaptive methods
  • nonlocal transform-based methods
  • Applications:
  • denoising
  • deblurring
  • deblocking
  • super-resolution/zooming
slide-11
SLIDE 11

Department of Signal Processing

11

Intro

slide-12
SLIDE 12

Department of Signal Processing

12

Intro

slide-13
SLIDE 13

Department of Signal Processing

13

Intro

slide-14
SLIDE 14

Department of Signal Processing

14

Intro

slide-15
SLIDE 15

Department of Signal Processing

15

Intro

slide-16
SLIDE 16

Department of Signal Processing

16

Karen Egiazarian

Intro

slide-17
SLIDE 17

Department of Signal Processing

17

Karen Egiazarian

Intro

slide-18
SLIDE 18

Department of Signal Processing

18

Karen Egiazarian

Intro

slide-19
SLIDE 19

Department of Signal Processing

19

Karen Egiazarian

Intro

slide-20
SLIDE 20

Department of Signal Processing

20

Karen Egiazarian

Intro

slide-21
SLIDE 21

Department of Signal Processing

21

Karen Egiazarian

Intro

slide-22
SLIDE 22

Department of Signal Processing

22

Karen Egiazarian

Intro

slide-23
SLIDE 23

Department of Signal Processing

23

Statistical analysis of raw data

slide-24
SLIDE 24

Department of Signal Processing

24 8.4.2016

Statistical analysis of raw data

slide-25
SLIDE 25

Department of Signal Processing

25

Statistical analysis of raw data

slide-26
SLIDE 26

Department of Signal Processing

26

Statistical analysis of raw data

slide-27
SLIDE 27

Department of Signal Processing

27

Statistical analysis of raw data

slide-28
SLIDE 28

Department of Signal Processing

28

Statistical analysis of raw data

slide-29
SLIDE 29

Department of Signal Processing

29

Statistical analysis of raw data

slide-30
SLIDE 30

Department of Signal Processing

30

The analysis of experimental data demonstrates that:

  • 1. The model of noise is close to the

Poissonian one

  • 2. Model parameters depend neither on the

color channel nor on the exposure time

Statistical analysis of raw data

slide-31
SLIDE 31

Department of Signal Processing

31

Parametric signal-dependent noise-modelling: Poissonian-Gaussian with clipping

slide-32
SLIDE 32

Department of Signal Processing

32

Parametric signal-dependent noise-modelling: automatic estimation from single-image raw- data (http://www.cs.tut.fi/~foi/sensornoise.html)

slide-33
SLIDE 33

Department of Signal Processing

33

Practical modeling for raw data: idea

  • Model photon-to-electron conversion using Poisson distributions (signal

dependent);

  • Model the other noise sources as signal-independent and Gaussian (central-

limit theorem);

  • Exploit normal approximation of Poisson distributions;
  • The acquisition/dynamic range is limited: too dark or too bright signals are

clipped;

  • There can be a pedestal;
  • Spatial dependencies can be ignored for normal operating conditions (go for

independent noise). Eventually, only two parameters are sufficient to describe the noise model where the raw data is described as clipped signal-dependent observations.

slide-34
SLIDE 34

Department of Signal Processing

Variance stabilization

34

Karen Egiazarian

slide-35
SLIDE 35

Department of Signal Processing

35

Karen Egiazarian

Variance stabilization

slide-36
SLIDE 36

Department of Signal Processing

Inversion for Poisson stabilized by Anscombe Mäkitalo, Foi (TIP, 2011)

36

Karen Egiazarian

slide-37
SLIDE 37

Department of Signal Processing

Experiment: clipped noisy data

37

Original image : y (x1, x2) = 0.7 sin (2π x1/512)+ 0.5

slide-38
SLIDE 38

Department of Signal Processing

Experiment: Noise Estimation

38

estimation and fitting a = 0.0038, b = 0.022 st.dev.-function .σ

slide-39
SLIDE 39

Department of Signal Processing

Experiment: denoised estimate after variance stabilization before declipping

39

slide-40
SLIDE 40

Department of Signal Processing

Experiment: declipped estimate

40

slide-41
SLIDE 41

Department of Signal Processing

Experiment: declipped estimate (crosssection)

41

slide-42
SLIDE 42

Department of Signal Processing

Real experiment: (Raw-data from Fujiflm FinePix S9600, ISO 1600)

42

slide-43
SLIDE 43

Department of Signal Processing

Real experiment: Denoising before declipping

43

slide-44
SLIDE 44

Department of Signal Processing

Real experiment: Denoising after declipping

44

slide-45
SLIDE 45

Department of Signal Processing

Real experiment: Denoising after declipping (crossection)

45

slide-46
SLIDE 46

Department of Signal Processing

46

LASIP

www.cs.tut.fi/~lasip/

  • Local Approximation Signal and Image Processing

(LASIP) Project

LASIP project is dedicated to investigations in a wide class of novel efficient adaptive signal processing techniques.

slide-47
SLIDE 47

Department of Signal Processing

47

LASIP

LPA estimates, bias and variance, and asymptotic MSE

slide-48
SLIDE 48

Department of Signal Processing

48

LASIP: Intersection of Confidence Intervals (ICI) rule Goldenshluger & Nemirovski, 1997

slide-49
SLIDE 49

Department of Signal Processing

49

Karen Egiazarian

slide-50
SLIDE 50

Department of Signal Processing

Anisotropy: motivation

50

Karen Egiazarian

slide-51
SLIDE 51

Department of Signal Processing

Anisotropic estimator based on directional adaptive-scale: idea

51

Karen Egiazarian

slide-52
SLIDE 52

Department of Signal Processing

Directional LPA

52

Karen Egiazarian

slide-53
SLIDE 53

Department of Signal Processing

53

LASIP: HOW LPA-ICI WORKS

slide-54
SLIDE 54

Department of Signal Processing

Anisotropic LPA-ICI: Kernels used in practice

54

Karen Egiazarian

slide-55
SLIDE 55

Department of Signal Processing

55

Karen Egiazarian

slide-56
SLIDE 56

Department of Signal Processing

56

Karen Egiazarian

slide-57
SLIDE 57

Department of Signal Processing

57

Karen Egiazarian

slide-58
SLIDE 58

Department of Signal Processing

xi

B

yi

B

DCT DCT-1

xi

B

yi

B

KxK

xi

B

yi

B

Sliding DCT denoising

  • K. Egiazarian, J. Astola, M. Helsingius, and P.

Kuosmanen (1999) “Adaptive denoising and lossy compression of images in transform domain”, J. Electronic Imaging

slide-59
SLIDE 59

Department of Signal Processing

59

Shape-adaptive DCT image filtering

By demanding the local fit of a polynomial model, we are able to avoid the presence of singularities or discontinuities within the transform support. In this way, we ensure that data are represented sparsely in the transform domain, significantly improving the effectiveness of shrinkage (e.g., thresholding). noisy image and noisy data after hard-thresholding adaptive-shape extracted from in SA-DCT domain neighborhood the neighborhood

slide-60
SLIDE 60

Department of Signal Processing

60

Shape-adaptation: use directional LPA-ICI

slide-61
SLIDE 61

Department of Signal Processing

61

Shape-adaptive DCT image filtering

Pointwise SA-DCT: anisotropic neighborhoods

slide-62
SLIDE 62

Department of Signal Processing

62

Shape-adaptive DCT image filtering

  • Direct generalization of the classical block-DCT (B-DCT);
  • On rectangular domains (e.g., squares) the SA-DCT and B-DCT coincide;
  • Comparable computational complexity as the separable B-DCT (fast

algorithms);

  • SA-DCT is part of the MPEG-4 standard;
  • Efficient (low-power) hardware implementations available.

Before our work on SA-DCT filtering, the SA-DCT had been used

  • nly for image and video compression.
slide-63
SLIDE 63

Department of Signal Processing

63

Pointwise SA-DCT: denoising results

A fragment of Cameraman: noisy observation (σ=25, PSNR=20.14dB), BLS- GSM estimate (Portilla et al.) (PSNR=28.35dB), and the proposed Pointwise SA-DCT estimate (PSNR=29.11dB).

slide-64
SLIDE 64

Department of Signal Processing

64

Pointwise SA-DCT: deblocking results

JPEG coded Cameraman with 2 different quality levels and the results of post-filtering using SA-DCT

slide-65
SLIDE 65

Department of Signal Processing

65

Pointwise SA-DCT: deblurring results

Images blurred & noisy are deblurred & denoised by SA-DCT filter.

slide-66
SLIDE 66

Department of Signal Processing

66

Pointwise SA-DCT: extension to color, motivation

Luminance-chrominance decompositions: structural correlation

slide-67
SLIDE 67

Department of Signal Processing

67

Pointwise SA-DCT: structural contraint in luminance-chrominance space

Use for all three channels the adaptive neighborhoods defined by the anisotropic LPA-ICI for the luminance channel.

slide-68
SLIDE 68

Department of Signal Processing

68

Pointwise SA-DCT: deblocking results

JPEG-compressed Pointwise SA-DCT deblocking (Q=10, 0.25bpp, PSNR=26.87dB) (PSNR=28.30dB)

slide-69
SLIDE 69

Department of Signal Processing

69

Pointwise SA-DCT: deblocking results

slide-70
SLIDE 70

Department of Signal Processing

70

Pointwise SA-DCT: denoising results

Fragments of the noisy F-16 (σ=30, PSNR=18.59dB), of ProbShrink-MB (Pizurica et al.) estimate (PSNR=30.50dB), and of Pointwise SA-DCT estimate (PSNR=31.59dB).

slide-71
SLIDE 71

Department of Signal Processing

71

Block-Matching and 3D filtering (BM3D) denoising algorithm

  • Generalizes NL-means and overcomplete transform methods
  • Current state-of-the-art denoising method
  • K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image

denoising with block-matching and 3D filtering”, Proc. SPIE Electronic Imaging 2006, Image Process.: Algorithms and Systems V, no. 6064A-30, San Jose (CA), USA, Jan. 2006.

  • -- , “Image denoising by sparse 3D transform-domain collaborative

filtering”, IEEE Trans. Image Process., vol. 16, no. 8, pp. 2080- 2095, Aug. 2007.

slide-72
SLIDE 72

Department of Signal Processing

72

Block-matching and grouping

Groups are characterized by both:

  • intra-block correlation between the pixels of each grouped block (natural

images);

  • inter -block correlation between the corresponding pixels of different blocks

(grouped block are similar);

slide-73
SLIDE 73

Department of Signal Processing

73

BM3D: Collaborative filtering

  • Each grouped block collaborates for the filtering of all others, and vice versa.
  • Provides individual estimates for all grouped blocks (not necessarily equal).
  • Realized as shrinkage in a 3-D transform domain.
slide-74
SLIDE 74

Department of Signal Processing

74

BM3D with Shape-Adaptive PCA (BM3D- SAPCA)

Main ingredients:

  • Local Polynomial Approximation - Intersection of Confidence Intervals (LPA-

ICI) to adaptively select support for 2-D transform;

  • Block-Matching to enable non-locality;
  • Shape-Adaptive PCA (SA-PCA);
  • Shape-Adaptive DCT low-complexity 2-D transform on arbitrarily-shaped

domains (when SA-PCA is not feasible).

  • K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, .BM3D Image Denoising

with Shape-Adaptive Principal Component Analysis., Proc. Workshop on Signal Processing with Adaptive Sparse Structured Representations (SPARS.09), Saint- Malo, France, April 2009.

slide-75
SLIDE 75

Department of Signal Processing

75

BM3D-SAPCA

slide-76
SLIDE 76

Department of Signal Processing

76

Comparison of BM3D-SAPCA with other filters

slide-77
SLIDE 77

Department of Signal Processing

77

Comparison of BM3D-SAPCA with other filters

slide-78
SLIDE 78

Department of Signal Processing

78

Comparison of BM3D-SAPCA with other filters

slide-79
SLIDE 79

Department of Signal Processing

79

Comparison of BM3D-SAPCA with other filters (PSNR, SSIM)

Original Noisy, σ = 35 BM3D (27.82, 0.8207) P.SADCT (27.51, 0.8143) SA-BM3D (28.02, 0.8228) BM3D-SAPCA (28.16, 0.8269)

slide-80
SLIDE 80

Department of Signal Processing

80

Interpolation for Bayer Pattern

Original scene Color Filter Array Observation Color Interpolation

slide-81
SLIDE 81

Department of Signal Processing

81

Competitiveness with state-of-the-art techniques

The proposed CFAI technique adapts to spatial properties of an image

slide-82
SLIDE 82

Department of Signal Processing

82

Conventional Approach for Noiseless Data (Hamilton-Adams)

slide-83
SLIDE 83

Department of Signal Processing

83

Karen Egiazarian

Proposed Approach for Noiseless Data (Spatially-Adaptive LPA-ICI)

slide-84
SLIDE 84

Department of Signal Processing

84

Karen Egiazarian

Simulation of Radon reconstruction from sparse projections (approximating Radon projections as radial lines in FFT domain: Sparse projections: 11 radial lines) Compressed Sensing Image Reconstruction via Recursive BM3D

Egiazarian, K., A. Foi, and V. Katkovnik, “Compressed Sensing Image Reconstruction via Recursive Spatially Adaptive Filtering, ICIP 2007

slide-85
SLIDE 85

Department of Signal Processing

Compressed Sensing Image Reconstruction via Recursive BM3D

Egiazarian, K., A. Foi, and V. Katkovnik, “Compressed Sensing Image Reconstruction via Recursive Spatially Adaptive Filtering, ICIP 2007

85

Karen Egiazarian

Simulation of Radon reconstruction from sparse projections (approximating Radon projections as Limited-angle in FFT domain)

slide-86
SLIDE 86

Department of Signal Processing

86

BM3D for upsampling and super-resolution

Image upsampling or zooming, can be de.ned as the process of resampling a single low-resolution (LR) image on a high-resolution grid. The process of combining a sequence of undersampled and degraded low- resolution images in order to produce a single high-resolution image is commonly referred to as a Super-resolution (SR) reconstruction. Modern SR methods (e.g., Protter et al. 2008, Ebrahimi and Vrscay 2008) are based on the nonlocal means (NLM) filtering paradigm (Buades-Coll-Morel, 2005).

  • No explicit registration: one-to-one pixel mapping between frames is replaced by

a one-to-many mapping. The BM3D and V-BM3D algorithms share with the NLM the idea of exploiting nonlocal similarity between blocks. However, in (V-)BM3D a more powerful transform-domain modeling is used.

slide-87
SLIDE 87

Department of Signal Processing

87

BM3D based superresolution

slide-88
SLIDE 88

Department of Signal Processing

88

Image upsampling x 4 (pixel replication)

slide-89
SLIDE 89

Department of Signal Processing

89

Image upsampling x 4 in wavelet domain (Danielyan et al. EUSIPCO 2008)

slide-90
SLIDE 90

Department of Signal Processing

90

Video superresolution comparison with (Protter et. al.)

  • 1. M. Protter, M. Elad, H. Takeda, and P. Milanfar, .Generalizing the Non-Local-Means to

Super-Resolution Reconstruction., IEEE Trans. Image Process., 2008.

  • 2. A. Danielyan, A. Foi, V. Katkovnik, and K. Egiazarian, .Image upsampling via spatially

adaptive block-matching filtering, EUSIPCO2008, Lausanne, Switzerland, Aug. 2008.

slide-91
SLIDE 91

Department of Signal Processing

91

Examples: Video denoising using V-BM3D

slide-92
SLIDE 92

Department of Signal Processing

92

Examples: Video denoising using V-BM3D

slide-93
SLIDE 93

Department of Signal Processing

93

Examples: Video denoising using V-BM3D

slide-94
SLIDE 94

Department of Signal Processing

94

Conclusions

Our algorithms have been licensed to major digital camera manufacturers and are already in use by various research institutes for processing and enhancing their images.

Tomographic reconstruction of mouse embryo with BM3D filtering of axial slices (Harvard Medical School, Boston MA, 2010)

BM3D

slide-95
SLIDE 95

Department of Signal Processing

95

Conclusions

slide-96
SLIDE 96

Department of Signal Processing

96

Thanks to collaborators and former students:

Jaakko Astola (TUT, Finland) Leonid Yaroslavsky (Israel) –Sliding DCT denoising (1997-2000) Dmitiy Paliy (NOKIA, Finland) LPA-ICI CFAI (2005-2007) Rusen Oktem (Berkeley, USA) Sliding DCT Aram Danielyan (Noiseless Imaging) Enrique Sánchez-Monge (Noiseless Imaging) super-resolution using BM3D (2008-) Alessandro Foi (TUT, Finland) SA DCT, BM3D,… Vladimir Katkovnik (TUT, Finland) LPA-ICI, SA-DCT, BM3D,… Kostadin Dabov (Apple, USA), BM3D Dmytro Rusanovsky (LG, USA) Video BM3D and many-many others…