IMAGE RESTORATION WITH NEURAL NETWORKS Orazio Gallo Work with Hang - PowerPoint PPT Presentation

IMAGE RESTORATION WITH NEURAL NETWORKS Orazio Gallo Work with Hang Zhao, Iuri Frosio, Jan Kautz

MOTIVATION The long path of images… AF/AE Bad Pixel Black Correction Level Lens Demosaic Denoise Metering Correction Image Tone Enhancing Mapping Image Signal Processor (ISP)

Image credit: Wikipedia colors by interpolation DEMOSAICING Image credit: Marc Levoy

DENOISING Several types of noise involved in the image formation: • Photon shot noise Dark current (AKA thermal noise) • • Photo-response non-uniformity Vignetting • Readout noise: • • Reset noise (charge-to-voltage transfer) White noise (during voltage amplification amplification) • • Quantization noise (ADC)

DENOISING

MOTIVATION AF/AE Bad Pixel Black Correction Level Lens Demosaic Denoise Metering Correction Image Tone Enhancing Mapping Demosaicing before denoising changes the statistics of the noise. And the best de-noising algorithms require to know what the noise looks like.

MOTIVATION AF/AE Bad Pixel Black Correction Level Lens Denoise Demosaic Metering Correction Image Tone Enhancing Mapping Denoising first can change the color reproduction accuracy as the three channels may be denoised differently.

FLEXISP 1 A Flexible Camera Image Processing Framework PSF CFA Noise [1] Heide et al., ACM SIGGRAPH Asia 2012 (ToG)

CAN WE DO IT WITH A NEURAL NETWORK? Can we do it with a neural network, which moves the heavy lifting to the training stage and inference is very quick?

JOINT DEMOSAICING AND DENOISING

JOINT DEMOSAICING AND DENOISING bilinear interpolation Network architecture convolution convolution convolution

MEASURING IMAGE QUALITY Original Image adapted from https://ece.uwaterloo.ca/~z70wang/research/ssim/

MEASURING IMAGE QUALITY Higher sensitivity to errors in texture-less regions! Wang, et al. "Image quality assessment: from error visibility to structural similarity." IEEE TIP (2004)

MEASURING IMAGE QUALITY Original 0.662 0.988 Image adapted from https://ece.uwaterloo.ca/~z70wang/research/ssim/

MEASURING IMAGE QUALITY Higher sensitivity to errors in texture-less regions!

JOINT DEMOSAICING AND DENOISING bilinear interpolation Network architecture convolution convolution convolution

JOINT DEMOSAICING AND DENOISING Network training Training data 31 x 31 patches from 700, 999x666 RGB images (MIT-Adobe FiveK dataset) Input - noisy image (realistic noise model) - bilinear interpolation Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM

Ground truth

Noisy

RESULTS Visual comparison (+ unsharp masking) Noisy BM3D (state of the art) Ground truth

Noisy

RESULTS Visual comparison (+ unsharp masking) Noisy BM3D (state of the art) Ground truth

JOINT DEMOSAICING AND DENOISING: RESULTS Average image quality metrics on the testing dataset

DOES IT GENERALIZE? JPEG ARTIFACT REMOVAL & SUPER-RESOLUTION

JPEG ARTIFACT REMOVAL Network training Training data 31 x 31 patches from 700, 999x666 RGB images (MIT-Adobe FiveK dataset) Input JPEG compressed image , 25% quality Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM

JPEG ARTIFACT REMOVAL: RESULTS Visual comparison (+ unsharp masking) JPEG L2 L1 + MS-SSIM Ground truth

JPEG ARTIFACT REMOVAL: RESULTS Numerical comparison Average image quality metrics on the testing dataset

SUPER-RESOLUTION Network training Training data 31 x 31 patches from 700, 999x666 RGB images (MIT-Adobe FiveK dataset) Input 2x downsampled image + upsampled with bilinear interpolation Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM

SUPER-RESOLUTION: RESULTS Visual comparison (+ unsharp masking) Low rez L2 L1 + MS-SSIM

SUPERRESOLUTION: RESULTS Numerical comparison and literature

LEARNINGS?

LEARNINGS A closer look at the different losses

LEARNINGS and

LEARNINGS and 0.3939 0.3896 • seems to have more convergence issues. converges faster and speeds up the convergence or other losses, too. •

LEARNINGS A closer look at the different losses

LEARNINGS SSIM and MS-SSIM “Higher sensitivity to errors in texture - less regions!” Multi-scale is helpful when dealing with transition regions. •

LEARNINGS A closer look at the different losses

RESULTS Why mixing MS-SSIM and ?

CONCLUSIONS What have we learnt? • Even a shallow network can produce state-of-the- art results… …if you train it carefully. • Perceptually-motivated loss functions can help! • • But you have to be aware of their limitations!

Thanks! Zhao, Gallo, Frosio, and Kautz, “Loss Functions for Image Restoration with Neural Networks”, IEEE Trans. on Comp. Imaging, 2017