IMAGE RESTORATION WITH NEURAL NETWORKS Orazio Gallo Work with Hang - - PowerPoint PPT Presentation

▶
image restoration with neural networks
SMART_READER_LITE
LIVE PREVIEW

IMAGE RESTORATION WITH NEURAL NETWORKS Orazio Gallo Work with Hang - - PowerPoint PPT Presentation

Apr 16, 2023 8 likes •469 views

IMAGE RESTORATION WITH NEURAL NETWORKS Orazio Gallo Work with Hang Zhao, Iuri Frosio, Jan Kautz MOTIVATION The long path of images AF/AE Bad Pixel Black Correction Level Lens Demosaic Denoise Metering Correction Image Tone

slide-1
SLIDE 1

Work with Hang Zhao, Iuri Frosio, Jan Kautz

IMAGE RESTORATION WITH NEURAL NETWORKS

Orazio Gallo

slide-2
SLIDE 2
slide-3
SLIDE 3
slide-4
SLIDE 4

MOTIVATION

The long path of images…

Demosaic Denoise Bad Pixel Correction Image Enhancing Tone Mapping Lens Correction Black Level Metering

AF/AE

Image Signal Processor (ISP)

slide-5
SLIDE 5

DEMOSAICING

colors by interpolation

Image credit: Wikipedia Image credit: Marc Levoy

slide-6
SLIDE 6

DENOISING

Several types of noise involved in the image formation:

  • Photon shot noise
  • Dark current (AKA thermal noise)
  • Photo-response non-uniformity
  • Vignetting
  • Readout noise:
  • Reset noise (charge-to-voltage transfer)
  • White noise (during voltage amplification amplification)
  • Quantization noise (ADC)
slide-7
SLIDE 7

DENOISING

slide-8
SLIDE 8

MOTIVATION

Demosaic Denoise Bad Pixel Correction Image Enhancing Tone Mapping Lens Correction Black Level Metering

AF/AE

Demosaicing before denoising changes the statistics of the noise. And the best de-noising algorithms require to know what the noise looks like.

slide-9
SLIDE 9

MOTIVATION

Denoise Demosaic Bad Pixel Correction Image Enhancing Tone Mapping Lens Correction Black Level Metering

AF/AE

Denoising first can change the color reproduction accuracy as the three channels may be denoised differently.

slide-10
SLIDE 10

PSF CFA Noise [1] Heide et al., ACM SIGGRAPH Asia 2012 (ToG)

FLEXISP1

A Flexible Camera Image Processing Framework

slide-11
SLIDE 11

CAN WE DO IT WITH A NEURAL NETWORK?

Can we do it with a neural network, which moves the heavy lifting to the training stage and inference is very quick?

slide-12
SLIDE 12

JOINT DEMOSAICING AND DENOISING

slide-13
SLIDE 13

JOINT DEMOSAICING AND DENOISING

Network architecture

convolution convolution convolution bilinear interpolation

slide-14
SLIDE 14

MEASURING IMAGE QUALITY

Image adapted from https://ece.uwaterloo.ca/~z70wang/research/ssim/

Original

slide-15
SLIDE 15

Higher sensitivity to errors in texture-less regions!

MEASURING IMAGE QUALITY

Wang, et al. "Image quality assessment: from error visibility to structural similarity." IEEE TIP (2004)

slide-16
SLIDE 16

MEASURING IMAGE QUALITY

Image adapted from https://ece.uwaterloo.ca/~z70wang/research/ssim/

Original

0.988 0.662

slide-17
SLIDE 17

MEASURING IMAGE QUALITY

Higher sensitivity to errors in texture-less regions!

slide-18
SLIDE 18

JOINT DEMOSAICING AND DENOISING

Network architecture

convolution convolution convolution bilinear interpolation

slide-19
SLIDE 19

JOINT DEMOSAICING AND DENOISING

Network training

Training data 31 x 31 patches from 700, 999x666 RGB images (MIT-Adobe FiveK dataset) Input

  • noisy image (realistic noise model)
  • bilinear interpolation

Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM

slide-20
SLIDE 20

Ground truth

slide-21
SLIDE 21

Noisy

slide-22
SLIDE 22
slide-23
SLIDE 23
slide-24
SLIDE 24

RESULTS

Visual comparison (+ unsharp masking)

Noisy BM3D (state of the art) Ground truth

slide-25
SLIDE 25

Noisy

slide-26
SLIDE 26
slide-27
SLIDE 27
slide-28
SLIDE 28

RESULTS

Visual comparison (+ unsharp masking)

Noisy BM3D (state of the art) Ground truth

slide-29
SLIDE 29

JOINT DEMOSAICING AND DENOISING: RESULTS

Average image quality metrics on the testing dataset

slide-30
SLIDE 30

DOES IT GENERALIZE? JPEG ARTIFACT REMOVAL & SUPER-RESOLUTION

slide-31
SLIDE 31

JPEG ARTIFACT REMOVAL

Network training

Training data 31 x 31 patches from 700, 999x666 RGB images (MIT-Adobe FiveK dataset) Input JPEG compressed image, 25% quality Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM

slide-32
SLIDE 32

JPEG ARTIFACT REMOVAL: RESULTS

Visual comparison (+ unsharp masking)

Ground truth L1 + MS-SSIM L2 JPEG

slide-33
SLIDE 33

JPEG ARTIFACT REMOVAL: RESULTS

Numerical comparison

Average image quality metrics on the testing dataset

slide-34
SLIDE 34

SUPER-RESOLUTION

Network training

Training data 31 x 31 patches from 700, 999x666 RGB images (MIT-Adobe FiveK dataset) Input 2x downsampled image + upsampled with bilinear interpolation Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM

slide-35
SLIDE 35

SUPER-RESOLUTION: RESULTS

Visual comparison (+ unsharp masking)

L1 + MS-SSIM L2 Low rez

slide-36
SLIDE 36

SUPERRESOLUTION: RESULTS

Numerical comparison and literature

slide-37
SLIDE 37

LEARNINGS?

slide-38
SLIDE 38

LEARNINGS

A closer look at the different losses

slide-39
SLIDE 39

LEARNINGS

and

slide-40
SLIDE 40

LEARNINGS

and

0.3939 0.3896

  • seems to have more convergence issues.
  • converges faster and speeds up the convergence or other losses, too.
slide-41
SLIDE 41

LEARNINGS

A closer look at the different losses

slide-42
SLIDE 42

LEARNINGS

SSIM and MS-SSIM

“Higher sensitivity to errors in texture-less regions!”

  • Multi-scale is helpful when dealing with transition regions.
slide-43
SLIDE 43

LEARNINGS

A closer look at the different losses

slide-44
SLIDE 44

RESULTS

Why mixing MS-SSIM and ?

slide-45
SLIDE 45

CONCLUSIONS

  • Even a shallow network can produce state-of-the-art results…
  • …if you train it carefully.
  • Perceptually-motivated loss functions can help!
  • But you have to be aware of their limitations!

What have we learnt?

slide-46
SLIDE 46

Thanks!

Zhao, Gallo, Frosio, and Kautz, “Loss Functions for Image Restoration with Neural Networks”, IEEE Trans. on Comp. Imaging, 2017