Image Quality Assessment: Unifying Structure and Texture Similarity - PowerPoint PPT Presentation

Image Quality Assessment: Unifying Structure and Texture Similarity Kede Ma May 27, 2020

Collaborators Eero P. Simoncelli Keyan Ding Shiqi Wang PhD student Assistant Professor Professor City University of Hong Kong City University of Hong Kong New York University

Outline ➢ Review of Full-Reference Image Quality Assessment (FR-IQA) ➢ Deep Image Structure and Texture Similarity Metric (DISTS) ➢ Model Comparison by “Perceptual Optimization”

Full-Reference IQA Review • Error visibility methods MSE (PSNR) , VSNR, MAD, PAMSE, NLPD, … • Structural similarity methods SSIM, MS-SSIM, IW- SSIM, FSIM, GSIM, GMSD, VSI, … • Information theoretic methods IFC, VIF , … • Learning based methods DNN-based: WaDIQaM-FR, DeepQA, LPIPS , PieAPP, … • Fusion based methods MAE + VGG Loss, …

MSE? Image Credit: Berardino

MSE? Image Credit: Wang and Simoncelli

SSIM? • Not “accurate” enough MS-SSIM, IW- SSIM, VIF, MAD, FSIM, VSI, NLPD, LPIPS, … • Not “computational efficient” enough PAMSE, GMSD, … • Not misalignment-aware Adaptive linear system, CW-SSIM, GTI- IQA, … • Not color-aware Adaptive linear system, FSIM_c, LPIPS, PieAPP , … • Not texture-aware STSIM , …

Texture Similarity Existing full-reference IQA models are over-sensitive to texture resampling Reference Blurred Resampled ✓ LPIPS, DISTS × PSNR, SSIM

Texture Similarity Existing full-reference IQA models are over-sensitive to texture resampling High-resolution EDSR SRGAN ✓ DISTS × PSNR, SSIM, LPIPS

A Common Problem of Recent Full-Reference IQA Models They do not satisfy the uniqueness property (identity of indiscernibles): × D ( x, y ) = 0 x = y FSIM, VSI, GMSD SSIM, MSE VIF, CW-SSIM, MAD Bijective DeepIQA, PieAPP MS-SSIM, NLPD, DISTS Injective Surjective Uniqueness is very important for “perceptual optimization”!

Reference Image Recovery Recovered images Initialization SSIM FSIM VIF GMSD Reference NLPD PieAPP LPIPS DISTS

Deep Image Structure and Texture Similarity (DISTS) Goal: Develop a full-reference IQA metric that is 1) sensitive to structural distortions (e.g., artifacts due to noise, blur, or compression) 2) tolerant to texture resampling (exchanging a texture region with a new sample) Two steps: 1. Transform an image to a perceptual representation 2. Measure the distance on the representation

DISTS — Representation • Use pretrained VGG features VGG 𝑦 = 𝑔(𝑦) ෤ features • Replace Max pooling with L 2 pooling (translation-invariant) Conv_5 Conv_4 Hanning window • Satisfy the injective property Conv_3 (distinct inputs should map to distinct outputs) Conv_2 Conv_1 𝑦

DISTS — Quality Measurements 1. Design texture similarity using global means We synthesize textures by solving Global mean of each feature map (a) Statistics of wavelet subbands 710 parameters (b) Gram matrices of VGG features ~306Kparameters (c) Global means of VGG features 1,475 parameters Reference (a) Portilla & (b) Gatys et al. (c) Ours Simoncelli

DISTS — Quality Measurements Use normalized “global mean”: 2. Design structure similarity using global covariance (inspired by SSIM) Positive learnable weights (1475*2) 3. Combine texture and structure terms:

DISTS — Transferring to a Metric l s (𝑗) (𝑗) 𝑦 𝑘 ෤ 𝑧 𝑘 ෤ 𝑧 𝑦 𝛽 𝑗𝑘 𝛾 𝑗𝑘 𝐸 𝑦, 𝑧 Texture comparsion Structure comparsion

DISTS — Training are jointly optimized for human perception of image quality ( KADID-10k dataset ) and texture invariance (two patches (z 1 , z 2 ) sampled from the same texture image) The final objective: Code is available at https://github.com/dingkeyan93/DISTS

DISTS — Connections to Existing IQA Measures • SSIM and its variants MS-SSIM, CW-SSIM • The adaptive linear system framework (Wang and Simoncelli, 2005) Separating structural and non-structural distortions • Content and style losses MSE on VGG features, Gram matrix • Image restoration losses Weighted sum of L1/L2 distances computed on the raw pixels and several stages of VGG feature maps

DISTS — Performance on Quality Prediction • Three standard IQA databases

DISTS — Performance on Quality Prediction • Image generation/restoration quality databases

DISTS — Performance on Texture Similarity • Two texture quality databases

DISTS — Texture Classification and Retrieval • Brodatz texture dataset

DISTS — Invariance to Geometric Transformations • A visual example DISTS Translation, 5% Dilation, 1.05 Cloud movement Reference PSNR SSIM FSIM Blur JPEG JP2K

DISTS — Summary • A new full-reference IQA method, which is the first of its kind with built-in invariance to texture resampling • DISTS unifies structure and texture similarity, is robust to mild geometric distortions, and performs well in texture relevant tasks • DISTS can be employed as an objective function in various optimization problems

A Perceptual Optimization Tour of Full- Reference IQA Models

IQA Model Comparison 1. Compute correlation with human judgments (PLCC, SRCC) 1) Huge budget to build a large-scale database 2) With potential risk of overfitting 2. MAximum Differentiation competition (MAD) methodology 1) MAD (Wang and Simoncelli, 2008) synthesizes counter-examples to falsify ify a model (the generated images may be highly unnatural) 2) gMAD (Ma et al., 2016) searches counter-examples from a large unlabeled image set 3. Compare the IQA-based optimization results “Analysis by Synthesis”

“Perceptual Optimization” • Diagram of IQA-based Optimization: Feedback Input Output Reference Image processing IQA model system evaluation Denoising MSE Compression SSIM … … A highly promising but relatively under-studied application of objective IQA measures

Optimization Objective • Select 11 representative IQA models: MAE, MS-SSIM, VIF, CW-SSIM, MAD, FSIM, GMSD, VSI, NLPD, LPIPS, DISTS • Four low-level vision tasks: – Image denoising – Blind image deblurring – Single image super-resolution Code is available at – Lossy image compression https://github.com/dingkeyan93/IQA-optimization

Optimization Network • Denoising and Deblurring: ResBlock ResBlock … Conv Conv + Input Output ReLU Conv Conv + ResBlock

Optimization Network • Super-resolution: Upsample Upsample ResBlock ResBlock … Conv Conv Conv Conv + Input Output • Compression: × 𝑜 𝑜 × Downsample ResBlock ResBlock ResBlock ResBlock Upsample … … Conv Conv Conv Conv Q Output Input Analysis Transform Synthesis Transform

Optimization Performance • Subjective Testing Two-alternative forced choice (2AFC) method The Bradley-Terry model is employed to convert paired comparison results to global rankings The paired t -test is conducted to investigate whether the optimization results of the IQA models are statistically significant Test images (from the validation set of DIV2K)

Optimization Performance • Performance ranking and grouping: MS-SSIM MAE MAD LPIPS DISTS NLPD CW-SSIM VSI VIF FSIM GMSD Denoising 0.70 0.65 0.45 0.45 0.39 0.37 0.36 -0.44 -0.51 -0.58 -2.04 DISTS LPIPS MAD MS-SSIM MAE CW-SSIM VIF NLPD FSIM VSI GMSD Deblurring 3.23 3.10 0.48 0.32 0.20 0.16 -0.79 -0.94 -1.54 -1.73 -2.75 DISTS LPIPS MS-SSIM MAE NLPD MAD FSIM VIF VSI GMSD CW-SSIM Super-res 2.50 1.88 1.20 1.02 0.65 0.53 -0.70 -1.37 -1.81 -1.85 -2.04 DISTS LPIPS MS-SSIM MAE MAD NLPD FSIM VIF VSI GMSD CW-SSIM Compression 2.61 2.35 1.58 1.53 0.68 0.29 -0.37 -1.64 -2.00 -2.06 -4.26 Best worst

Visual Example — Denoising

Visual Example — Deblurring

Visual Example — Super-Resolution

Visual Example — Compression

Artifacts Analysis Blurring MAE , MS-SSIM and NLPD , relying on simple injective mappings, prefer to make a more conservative estimate, producing something akin to a superposition of all possible outcomes GT MAE MS-SSIM NLPD Super-resolution

Artifacts Analysis Ringing FSIM , VSI and GMSD , rely heavily on local gradient magnitude for feature similarity comparison. This leads to enormous “fake edge” lines that are imperceptible to gradient operator GT FSIM VSI GMSD Deblurring