Image Sharpness Metric Based on MaxPol Convolution Kernels Mahdi S. - - PowerPoint PPT Presentation

▶
image sharpness metric based on maxpol convolution kernels
SMART_READER_LITE
LIVE PREVIEW

Image Sharpness Metric Based on MaxPol Convolution Kernels Mahdi S. - - PowerPoint PPT Presentation

Sep 22, 2022 239 likes •438 views

Human Visual System (HVS) Response Modelling Numerical Framework by MaxPol Convolution Kernels Natural Image Frequency Falloff Modelling No-Reference (NR) Focus Quality Assessment (FQA) University of Toronto Experiment-I: Synthetic Blur Imaging

slide-1
SLIDE 1

Human Visual System (HVS) Response Modelling Numerical Framework by MaxPol Convolution Kernels Natural Image Frequency Falloff Modelling No-Reference (NR) Focus Quality Assessment (FQA) Experiment-I: Synthetic Blur Imaging Experiment-II: Natural Blur Imaging Experiment-III: Whole Slide Imaging in Digital Pathology

University of Toronto

Image Sharpness Metric Based on MaxPol Convolution Kernels

Mahdi S. Hosseini and Konstantinos N. Plataniotis

mahdi.hosseini@mail.utoronto.ca kostas@ece.utoronto.ca

Multimedia Laboratory The Edward S. Rogers Dept. of Electrical and Computer Engineering University of Toronto, Ontario, Canada

2018 IEEE International Conference on Image Processing (ICIP) Paper#2842, Session: MQ.L3: Visual Quality Assessment I Monday, 17:40-18:00, October 8, 2018, Athens, Greece

Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 1 / 18

slide-2
SLIDE 2

Objective and Contribution

Main objective: Propose a computational model to Human Visual System (HVS) response to assess natural image blur

1 Synthesize visual sensitivity response by a convolutional filter 2 Use HVS convolution filter to perceive image blur features 3 Implement algorithmic workflow to quantize image blur

Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 2 / 18

slide-3
SLIDE 3

Human Visual System (HVS) Response Modelling Numerical Framework by MaxPol Convolution Kernels Natural Image Frequency Falloff Modelling No-Reference (NR) Focus Quality Assessment (FQA) Experiment-I: Synthetic Blur Imaging Experiment-II: Natural Blur Imaging Experiment-III: Whole Slide Imaging in Digital Pathology

University of Toronto

Outline

Human Visual System (HVS) Response Modelling Numerical Framework by MaxPol Convolution Kernels Natural Image Frequency Falloff Modelling No-Reference (NR) Focus Quality Assessment (FQA) Experiment-I: Synthetic Blur Imaging Experiment-II: Natural Blur Imaging Experiment-III: Whole Slide Imaging in Digital Pathology

Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 3 / 18

slide-4
SLIDE 4

Frequency Response of Natural Images

  • Natural images follow a decay response ∝ 1/ωγ
  • ω is spatial frequency, γ > 1 is energy tuning factor
  • Amplitude response of high-frequency is lower than low-frequency

Natural Image

I2D(x, y)

Frequency Spectrum

|ˆ I2D(ωx, ωy)|

Radial Freq. Binning Amplitude Spectrum

1 2 3 Frequency ( ) 0.5 1 1.5 2 2.5 3 Amplitude Spectrum

|ˆ I2D(ωr)|

Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 4 / 18

slide-5
SLIDE 5

Visual Sensitivity in Human Visual System (HVS)

  • HVS analyzes visual inputs in frequency domain
  • Energy of all amplitude frequencies are perceives equally in HVS
  • HVS introduces a sensitivity response to compensate the

energy-loss of high frequency information

  • Neurones in visual cortex automatically tune the frequency

amplitudes to balance out the falloff of high-frequency range1

Natural Image Perception in Human Vision System (HVS)

1[Field-OSA1987], [FieldBrady-Elsevier1995], [FieldBrady-Elsevier1997] Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 5 / 18

slide-6
SLIDE 6

Modelling HVS as a Linear Operator

  • Visual sensitivity response boosts high frequencies to balance out

wide spectrum of input visuals

  • Model HVS as a linear convolution process

¯ I ≈ IInput ∗ hHVS

1

¯ I - Output image signal perceived by human visual cortex

2 IInput

  • Input image signal

3 hHVS

  • Convolution filter emulating visual sensitivity response
  • Goal: synthesize a convolution filter hHVS(x) to boost

high-frequency amplitudes such that hfalloff(x) ∗ hHVS(x) = δ(x)

  • hfalloff(x) simulates falloff frequency of input image |ˆ

I2D(ωr)|

  • What is the main merit? If all frequencies are balanced, the

features corresponding to different edge types can be visually compared in a meaningful way

Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 6 / 18

slide-7
SLIDE 7

Design of HVS Convolution Filter

  • HVS filter response should satisfy ˆ

hHVS(ω) = ˆ hfalloff(ω)

−1

  • Define HVS as a linear combination of even-derivative operators

hHVS(x) ≡ c1d2(x) + c2d4(x) + . . . + cNd2N(x) where d2n(x) = d2n/dx2n

  • Fourier transform of even derivatives is F{d2n(x)} = (jω)2n
  • So, Fourier transform of HVS filter gives

ˆ hHVS(ω) ≡

N

  • n=1

cn ˆ d2n(ω) =

N

  • n=1

(−1)ncnω2n

  • Unknown coefficients cn are inferred by fitting the model into the

inverse falloff response

N

  • n=1

(−1)ncnω2n ≡ ˆ hfalloff(ω)

−1

Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 7 / 18

slide-8
SLIDE 8

Numerical Approximation via MaxPol Convolution Kernels

  • HVS attenuates frequencies close to Nyquist band
  • Once coefficients cn are obtained, we design lowpass filter

ˆ hHVS(ω) =   

N

  • n=1

(−1)ncnω2n, 0 ≤ ω ≤ ωc 0, ω ≥ ωc

  • ωc is cutoff frequency and is tuned for optimum performance
  • MaxPol2 library is used for numerical implementation of lowpass

derivative filters ω2n

hfalloff(x)

  • 30
  • 20
  • 10

10 20 30 Discrete node (x) 0.05 0.1 0.15 0.2 0.25 Amplitude

ˆ hfalloff(ω)

  • 3
  • 2
  • 1

1 2 3 0.2 0.4 0.6 0.8 1 Amplitude spectrum

hHV S(x)

  • 1
  • 0.5

0.5 1 1.5 Amplitude response

  • 32-28-24-20-16-12 -8 -4

4 8 12 16 20 24 28 32 Discrete node (x)

ˆ hHVS(ω)

  • 3
  • 2
  • 1

1 2 3 2 4 6

2[MaxPol Package] [HosseiniPlataniotis-IEEE2017] [HosseiniPlataniotis-SIAM2017] Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 8 / 18

slide-9
SLIDE 9

Natural Image Frequency Falloff Modeling

The falloff frequency ˆ hfalloff(ω) is related to imaging application

Synthetic Imaging Blur [HosseiniPlataniotis-ICIP2018]

  • hfalloff(x) = 1/ωp, blur is dominant in p ∈ {1, 3}

Natural Imaging Blur [HosseiniPlataniotis-arXive2018]

  • Using generalized Gaussian (GG) as a frequency falloff distribution
  • hfalloff(x) = c exp −|

x A(β,α)|β, Scale α = 1.7, Shape β = 1.4

Microscopic Out-of-Focus Blur [HosseiniPlataniotis-2018]

  • Encode out-of-focus blur in digital microscopy
  • hfalloff(x) =
  • C

1

0 J0(k NA n xρ)e− 1

2 ikρ2z( NA n )2ρdρ

  • 2

Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 9 / 18

slide-10
SLIDE 10

No-Reference Sharpness Metric Development

Images can now be convolved with HVS filter to identify balanced features for NR-FQA metric development

Algorithm for Sharpness Scoring

1 Exclude background pixels 2 Decompose image using HVS filter Fx = I ∗ hHVS, Fy = I ∗ hHVST 3 Activate features by ReLu R(x) = max(x, 0) 4 Construct sparse feature map in ℓ1/2-norm MHVS =

  • |R(Fx)|1/2 + |R(Fy)|1/22

. 5 Keep a subset Ω of feature pixels MHVS = sortd(MHVS)k, k ∈ Ω, 6 Measure the mth central moment µm = E

  • (MHVS − µ0)m

7 Record the final score Sharpness Score = − log µm

I Fx Fy MHVS R(Fx) R(Fy)

Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 10 / 18

slide-11
SLIDE 11

Experiment-I: Synthetic Blur Imaging

  • Images are synthetically blurred for quality assessment (IQA)
  • Images are subjectively evaluated for mean opinion score (MOS)
  • Database examples: LIVE, CSIQ, TID2008, and TID2013
  • Terms of evaluation

1 Pearson linear correlation coefficient (PLCC) 2 Spearman rank order correlation (SRCC)

Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 11 / 18

slide-12
SLIDE 12

Overall Performance

  • Developed metrics based on MaxPol meet both

1 High correlation accuracy 2 Fast speed calculation CPU time vs image size

6 4 x 6 4 1 2 8 x 1 2 8 2 5 6 x 2 5 6 5 1 2 x 5 1 2 1 2 4 x 1 2 4 2 4 8 x 2 4 8 10-2 100 102 Computation time (sec)

ARISM SPARISH RISE S3 MLV MaxPol GPC HVS MaxPol-2 HVS MaxPol-1

PLCC vs CPU Time: Synthetic

10-8 10-7 10-6 10-5 10-4 CPU Time/Pixel (sec) 0.89 0.9 0.91 0.92 0.93 0.94 0.95 0.96 plcc S3 MLV ARISM GPC SPARISH RISE HVS MaxPol-1 HVS MaxPol-2 MaxPol

PLCC vs CPU Time: Natural

10-8 10-7 10-6 10-5 10-4 CPU Time/Pixel (sec) 0.2 0.3 0.4 0.5 0.6 0.7 0.8 plcc S3 MLV ARISM GPC SPARISH RISE HVS MaxPol-1 HVS MaxPol-2 MaxPol

Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 12 / 18

slide-13
SLIDE 13

Experiment-II: FocusPath Natural Blur Database

  • Out-of-focus is common problem in whole slide imaging (WSI)
  • FocusPath3 is 864 digital pathology image patches from 9 WSIs
  • FocusPath images are scanned by Huron TissueScope LE1.2
  • 16 Z-stack scans collected from each slide to cover all focus levels

3download from https://sites.google.com/view/focuspathuoft/home Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 13 / 18

slide-14
SLIDE 14

Experiment-II: Natural Blur Imaging

  • Images are natural blurred for

quality assessment (IQA)

1 BID (586 images) 2 CID2013 (474 images) 3 FocusPath (864 images)

Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 14 / 18

slide-15
SLIDE 15

Experiment-III: Whole Slide Imaging in Digital Pathology

  • Tissue slides in digital microscopy are mapped to obtain best

focus level for scanning

  • Sharpness assessment can be used in quality control of WSI scan

Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 15 / 18

slide-16
SLIDE 16

Experiment-III: Whole Slide Imaging in Digital Pathology

  • Image patches from different WSI are shown bellow
  • Image patches are sorted based on different focus levels (bins)
  • Notice the robustness of focus levels across different slides

Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 16 / 18

slide-17
SLIDE 17

Concluding remarks

  • We implemented a no-reference image sharpness assessment

based on HVS response design

  • We implemented convolutional kernel simulating HVS response
  • Visual sensitivity response is modelled by linear combination of

high order derivatives

  • Numerical implementation of derivative provided by MaxPol library
  • Sharpness quality metric development based on MaxPol is

1 Highly accurate 2 High speed calculation with minimum computation complexity

  • Diverse imaging applications in

1 Synthetic blur 2 Natural blur 3 Microscopic out-of-focus

Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 17 / 18

slide-18
SLIDE 18

Human Visual System (HVS) Response Modelling Numerical Framework by MaxPol Convolution Kernels Natural Image Frequency Falloff Modelling No-Reference (NR) Focus Quality Assessment (FQA) Experiment-I: Synthetic Blur Imaging Experiment-II: Natural Blur Imaging Experiment-III: Whole Slide Imaging in Digital Pathology

University of Toronto

Thank You!

Hosseini and Plataniotis October 2018 Sharpness Metric via MaxPol Convolution Kernels 18 / 18