The EXQUIRES (EXtensible QUantitative Image RESampling) Test Suite: - - PowerPoint PPT Presentation

The EXQUIRES (EXtensible QUantitative Image RESampling) Test Suite: Impact of the Downsampler, Difference Metric, Test Image, Resampling Ratio and Colour Space on Upsampler Rank

Adam Turcotte

Department of Mathematics and Computer Science Laurentian University

Sudbury, October 12, 2012

EXQUIRES (EXtensible QUantitative Image RESampling) 1

Image Resampling

• Image resampling is performed when:
• zooming in on a digital photograph on a tablet or computer

screen

• watching a classic sitcom in HD (high definition)
• registering (aligning) two medical images taken at different

times with slightly different positioning and state of the patient in order to compare them

• using the software zoom in a digital camera
• warping a portrait for artistic purposes
• assembling multiple satellite views of the Earth or the Sun
• producing image thumbnails
• making an image fit a browser window
• processing one’s desktop so that it is shown within a

computer monitor taking into account the type and configuration of the screen’s physical pixels

• etc.

EXQUIRES (EXtensible QUantitative Image RESampling) 2

Overview

• What is Image Resampling?
• Combining known pixel values in order to obtain:
• pixel values at locations where none are provided (between

the original pixel locations, as in image enlargement)

• a pixel value suitable for different viewing conditions (as in

image reduction)

EXQUIRES (EXtensible QUantitative Image RESampling) 3

Overview

• Geometric Conventions:
• align (image) corners [ImageMagick]
• align (corner pixel) centres [old versions of VIPS]
• linear algebra (define a zero-coordinate and stretch)
• align top-left corner pixel centre [current VIPS]
• others...

EXQUIRES (EXtensible QUantitative Image RESampling) 4

Overview

• Image Quality:
• Subjective (Perceptual) VS Objective (Physical)

EXQUIRES (EXtensible QUantitative Image RESampling) 5

Overview

• Re-Enlargement (Reconstruction) Test:
• An objective test to evaluate image upsampling methods

EXQUIRES (EXtensible QUantitative Image RESampling) 6

Re-Enlargement Tests

• We need small versions of the test image so we can

upsample them

• In theory, take a step back and take the picture from further

away

• In practice:
• changes in lighting, conditions, focus, alignment, etc.

EXQUIRES (EXtensible QUantitative Image RESampling) 7

Re-Enlargement Tests

• Box filtering:
• Reasonable approximation of "stepping back"
• Introduces artifacts detectable by professional

photographers

• The specific artifacts identify the image as being box filtered
• Lesson: use several different downsamplers

EXQUIRES (EXtensible QUantitative Image RESampling) 8

STEP 1: Downsample the original image

Original Reduced

EXQUIRES (EXtensible QUantitative Image RESampling) 9

STEP 2: Upsample the reduced image

Original Reduced Re-Enlarged

EXQUIRES (EXtensible QUantitative Image RESampling) 10

STEP 3: Compare the re-enlargement and original

Original Difference Reduced Re-Enlarged

EXQUIRES (EXtensible QUantitative Image RESampling) 11

Image Comparison Metrics

• Standard physical (pixel-by-pixel) comparison:
• ℓ1(x, y) = AAE(x, y) = 1

N N

• i=1

|xi − yi|

• ℓ2(x, y) = RMSE(x, y) =
• 1

N N

• i=1

(xi − yi)2

• PSNR = 20 log10( MAX

RMSE )

• Non-standard physical (pixel-by-pixel) comparison:
• ℓ4(x, y) =

4

• 1

N N

• i=1

(xi − yi)4

• ℓ∞(x, y) = MAE(x, y) = max1≤i≤N |xi − yi|

EXQUIRES (EXtensible QUantitative Image RESampling) 12

Image Comparison Metrics

• Perceptual:
• SSIM(x, y) =

(2µx µy +C1)(2σxy +C2) (µ2

x +µ2 y +C1)(σ2 x +σ2 y +C2)

• µx and µy are the sample means
• σx and σy are the standard deviations
• σxy is the correlation coefficient between images x and y
• Typically:
• µx = Gaussian_blur(x)
• µy = Gaussian_blur(y)
• σ2

x = Gaussian_blur(x 2) − µ2 x

• σ2

y = Gaussian_blur(y 2) − µ2 y

• σxy = Gaussian_blur(xy) − µxµy

EXQUIRES (EXtensible QUantitative Image RESampling) 13

The Issue of Colour Spaces

• sRGB is a perceptual colour space
• Approximates the frequency response of the HVS
• RMSE is a physical measurement
• So why use straight sRGB values?
• They aren’t in linear light...
• Solution: convert to a linear colour space

EXQUIRES (EXtensible QUantitative Image RESampling) 14

Should we always use linear light?

• Charles Poynton (chose size of HDTVs)
• Eric Brasseur’s Dalai Lama example (gamma correction):

Original sRGB Linear

EXQUIRES (EXtensible QUantitative Image RESampling) 15

Implemented Image Comparison Metrics

• Pixel-by-pixel comparison in different colour spaces:
• sRGB (perceptual; approximates frequency response of

HVS)

• XYZ (physical; linear)

EXQUIRES (EXtensible QUantitative Image RESampling) 16

Implemented Image Comparison Metrics

• Delta-E CMC(1:1)
• Defined by the Colour Measurement Committee of the

Society of Dyers and Colourists

• Color tolerancing system
• Based on the CIE LCH (lightness, chroma, hue) colour

space

EXQUIRES (EXtensible QUantitative Image RESampling) 17

Implemented Image Comparison Metrics

• Mean SSIM (MSSIM)
• Refactored computation requires 20% fewer FLOPs
• Removed a Gaussian blur
• MATLAB version handles images 4x larger than the official

release

• Additional metric (called "blur")
• SSIM-inspired pixel-by-pixel comparison (grayscale)
• Apply Gaussian blur to both images
• Compute ℓ1, ℓ2, ℓ4, and ℓ∞

EXQUIRES (EXtensible QUantitative Image RESampling) 18

Aggregation

• Aggregate in a way that is consistent with the metric
• Average of ℓ1 errors
• Square root of sum of squares of ℓ2 errors
• Fourth root of sum of quads of ℓ4 errors
• Maximum of ℓ∞ errors
• Average of MSSIM values

EXQUIRES (EXtensible QUantitative Image RESampling) 19

Rank Correlation

• Standard Competition Ranking (1, 2, 2, 4)
• Fractional Ranks (1, 2.5, 2.5, 4):
• Ties are resolved by assigning to all tied upsamplers the

average rank they would obtain if the ties were resolved arbitrarily without affecting the ranks of the other upsamplers.

• Spearman’s Rank Correlation:
• Measures how uncorrelated rankings are
• Not a very accurate measure of how strongly correlated

rankings are

• It shows, just not quite as well as one would want
• With lots of data, probably not so different from Kendall
• Kendall’s Rank Correlation:
• Better choice for measuring rank correlation strength
• More complex and difficult to implement

EXQUIRES (EXtensible QUantitative Image RESampling) 20

EXQUIRES Overview

• 5 main programs
• exquires-new
• exquires-run
• exquires-update
• exquires-report
• exquires-correlate

EXQUIRES (EXtensible QUantitative Image RESampling) 21

EXQUIRES Overview

• 2 support programs
• exquires-compare
• exquires-aggregate

EXQUIRES (EXtensible QUantitative Image RESampling) 22

EXQUIRES Overview

• Sphinx documentation
• Can compile into many formats (HTML, PDF, L

ATEX, etc...)

• HTML version is online at http://exquires.ca

EXQUIRES (EXtensible QUantitative Image RESampling) 23

Natural Images

• We used natural images:
• different cameras
• different photographers
• different demosaicing
• different subject matter
• different conversion from RAW
• different denoising

EXQUIRES (EXtensible QUantitative Image RESampling) 24

Test Images

• Images are high quality:
• Taken with reasonably recent high quality DSLRs
• 840x840 colour images
• TIFF file format
• 16 bits per sample (48 bits per pixel)
• sRGB_IEC61966-2-1_black_scaled.icc ICC profile
• ICC = International Color Consortium

wave

EXQUIRES (EXtensible QUantitative Image RESampling) 25

Test Images

apartments baby boy cabins cat curios dragon footbridge

EXQUIRES (EXtensible QUantitative Image RESampling) 26

Test Images

frog garland horse man paint shed tower wreck

EXQUIRES (EXtensible QUantitative Image RESampling) 27

Dumic, Grgic & Grgic

“Hidden influences on image quality when comparing interpolation methods”

• Published in 2008
• Performed a re-enlargement test
• Did not specify the colour space(s) they used

(except that they used one greyscale image)

• Looked at upsampler ranks and compared the results

"by hand"

• Didn’t use correlations
• Determined that the choice of downsampler had the largest

impact on upsampler rankings

EXQUIRES (EXtensible QUantitative Image RESampling) 28

Improving Upon Dumic, Grgic & Grgic

• They used the following components:
• 4 test images
• 3 downsamplers
• 3 resampling ratios (2, 4, 8)
• 3 upsamplers
• 4 metrics (SNR, PSNR, PQS, SSIM)
• 1 colour space = 1 toolchain
• TOTAL DATA POINTS = 432

EXQUIRES (EXtensible QUantitative Image RESampling) 29

Improving Upon Dumic, Grgic & Grgic

• This thesis uses the following components:
• 17 test images
• 5 downsamplers
• 7 resampling ratios (2, 3, 4, 5, 6, 7, 8)
• 64 upsamplers
• 17 metrics
• 2 colour spaces = 4 toolchains
• TOTAL DATA POINTS = 2,589,440
• Each data point is an error measure resulting from

comparing two 840x840 images with 3 colour channels = 2,116,800 pixel value differences

EXQUIRES (EXtensible QUantitative Image RESampling) 30

Definitions

• Tensor:
• 2-pass: can be done as two 1D filtering passes
• Sinc:
• Ideal low pass filter in 1D
• Gives a square in Fourier space
• Jinc:
• Ideal low pass filter in 2D
• Gives a disc in Fourier space

EXQUIRES (EXtensible QUantitative Image RESampling) 31

Definitions

• Windowing:
• Pre-filtering (more or less)
• Example window functions:
• Welch
• Cosine
• Lanczos
• Hamming
• Lobes:
• Sinc and Jinc have infinite extent in “physical” space
• Not directly usable in practice
• So we chop the filter to a specific number of lobes
• EWA (Elliptical Weighted Averaging):
• Disc in output image mapped back to disc in input image

when preserving aspect ratio

EXQUIRES (EXtensible QUantitative Image RESampling) 32

Upsampler Ranks (in linear light using RMSE)

• Top 5 (Nearest Neighbour downsampling excluded):

1 Welch-windowed Sinc 4-Lobe 2 Cosine-windowed Sinc 4-Lobe 3 Lanczos-windowed Sinc 4-Lobe 4 Hamming-windowed Sinc 4-Lobe 5 EWA Catmull-Rom

• Top 4 are standard interpolatory linear tensor methods

EXQUIRES (EXtensible QUantitative Image RESampling) 33

upsampler rank upsampler rank welch4 1 bartlett2 33 cosine4 2 lanczos2 34 lanczos4 3 ewa_lanczos3sharp 35 hamming4 4 catmull_rom 36 ewa_catmull_rom 5 ewa_lanczos4 37 welch3 6 ewa_lanczos3 38 cosine3 7 kaisersharp2 39 ewa_lanczos4sharpest 8 parzen3 40 hann4 9 ewa_lanczos2sharpest 41 lanczos3 10 ewa_lanczos_radius2 42 bartlett4 11 hamming2 43 ewa_lanczos3sharpest 12 kaiser2 44 blackman4 13 ewa_mitchell_netravali 45 bartlett3 14 ewa_lanczos2sharp 46 bohman4 15 kaisersoft2 47 hann3 16 hann2 48 hamming3 17 ewa_robidoux 49 ewa_lanczos_radius3 18 ewa_lanczos2 50 parzen4 19 blackman2 51 ewa_lanczos_radius4 20 bohman2 52 kaisersharp4 21 parzen2 53 welch2 22 mitchell_netravali 54 kaiser4 23 cubic_hermite 55 kaisersoft4 24 ewa_hermite 56 cosine2 25 ewa_teepee 57 kaisersharp3 26 bilinear 58 blackman3 27 ewa_quadratic_b_spline 59 ewa_robidouxsharp 28 quadratic_b_spline 60 bohman3 29 nearest 61 kaiser3 30 ewa_cubic_b_spline 62 ewa_lanczos4sharp 31 cubic_b_spline 63 kaisersoft3 32 eanbqh 64 EXQUIRES (EXtensible QUantitative Image RESampling) 34

Upsampler Ranks (in linear light using RMSE)

• Top 5 (Nearest Neighbour downsampling only):

1 Mitchell-Netravali cubic spline 2 Bilinear interpolation 3 Hann-windowed Sinc 2-Lobe 4 EWA Robidoux 5 kaisersoft-windowed Sinc 2-Lobe

EXQUIRES (EXtensible QUantitative Image RESampling) 35

upsampler rank upsampler rank mitchell_netravali 1 ewa_quadratic_b_spline 33 bilinear 2 kaisersharp4 34 hann2 3 parzen4 35 ewa_robidoux 4 welch2 36 kaisersoft2 5 quadratic_b_spline 37 ewa_teepee 6 ewa_lanczos3 38 blackman2 7 ewa_lanczos3sharp 39 bohman2 8 bohman4 40 kaiser2 9 hamming3 41 hamming2 10 hann3 42 ewa_lanczos2sharp 11 bartlett3 43 cubic_hermite 12 blackman4 44 ewa_lanczos2 13 ewa_lanczos_radius3 45 ewa_mitchell_netravali 14 ewa_lanczos4 46 parzen2 15 ewa_lanczos4sharp 47 parzen3 16 bartlett4 48 kaisersharp2 17 ewa_lanczos_radius4 49 catmull_rom 18 lanczos3 50 kaisersoft3 19 ewa_lanczos3sharpest 51 lanczos2 20 hann4 52 ewa_lanczos_radius2 21 cosine3 53 bartlett2 22 hamming4 54 kaiser3 23 welch3 55 bohman3 24 ewa_lanczos4sharpest 56 ewa_lanczos2sharpest 25 lanczos4 57 blackman3 26 cosine4 58 kaisersharp3 27 ewa_cubic_b_spline 59 ewa_hermite 28 welch4 60 kaisersoft4 29 cubic_b_spline 61 ewa_robidouxsharp 30 ewa_catmull_rom 62 cosine2 31 nearest 63 kaiser4 32 eanbqh 64 EXQUIRES (EXtensible QUantitative Image RESampling) 36

Rank Comparison

Ranks are out of 64.

• Top 5 (Nearest Neighbour downsampling excluded):
• 60. Welch-windowed Sinc 4-Lobe
• 58. Cosine-windowed Sinc 4-Lobe
• 57. Lanczos-windowed Sinc 4-Lobe
• 54. Hamming-windowed Sinc 4-Lobe
• 62. EWA Catmull-Rom
• Top 5 (Nearest Neighbour downsampling only):
• 54. Mitchell-Netravali cubic spline
• 58. Bilinear interpolation
• 48. Hann-windowed Sinc 2-Lobe
• 49. EWA Robidoux
• 47. kaisersoft-windowed Sinc 2-Lobe

EXQUIRES (EXtensible QUantitative Image RESampling) 37

Downsampler Correlations

EWA Lanczos Lanczos nearest box Gaussian 3-lobe 3-lobe neighbour downsampler filtering blur filtering filtering decimation box filtering 1 .966 .957 .951

• .483

Gaussian blur .966 1 .945 .974

• .564

Lanczos 3-lobe filtering .957 .945 1 .986

• .471

EWA Lanczos 3-lobe filtering .951 .974 .986 1

• .527

nearest neighbour decimation

• .483
• .564
• .471
• .527

1

EXQUIRES (EXtensible QUantitative Image RESampling) 38

Resampling Ratio Correlations

downsampling ratio 2 4 3 6 8 5 7 2 1 .989 .974 .964 .939 .933 .907 4 .989 1 .993 .989 .972 .966 .943 3 .974 .993 1 .995 .983 .978 .958 6 .964 .989 .995 1 .994 .989 .973 8 .939 .972 .983 .994 1 .998 .988 5 .933 .966 .978 .989 .998 1 .994 7 .907 .943 .958 .973 .988 .994 1

EXQUIRES (EXtensible QUantitative Image RESampling) 39

Test Image Correlations

test image curios frog baby garland wave apartments dragon paint man wreck footbridge cabins boy tower horse cat shed curios 1 .998 .997 .992 .990 .989 .988 .986 .978 .975 .973 .973 .970 .967 .965 .959 .817 frog .998 1 .995 .992 .993 .989 .991 .988 .972 .979 .973 .968 .965 .963 .963 .963 .824 baby .997 .995 1 .985 .986 .982 .985 .985 .981 .977 .978 .974 .978 .973 .972 .962 .824 garland .992 .992 .985 1 .983 .985 .980 .974 .968 .963 .960 .966 .956 .957 .950 .944 .797 wave .990 .993 .986 .983 1 .994 .999 .997 .946 .985 .961 .938 .938 .935 .946 .980 .849 apartments .989 .989 .982 .985 .994 1 .992 .990 .947 .971 .946 .935 .936 .927 .932 .971 .834 dragon .988 .991 .985 .980 .999 .992 1 .998 .945 .987 .962 .936 .936 .934 .946 .983 .854 paint .986 .988 .985 .974 .997 .990 .998 1 .942 .990 .964 .932 .936 .932 .949 .989 .868 man .978 .972 .981 .968 .946 .947 .945 .942 1 .932 .956 .990 .990 .983 .958 .906 .744 wreck .975 .979 .977 .963 .985 .971 .987 .990 .932 1 .979 .934 .932 .941 .966 .987 .897 footbridge .973 .973 .978 .960 .961 .946 .962 .964 .956 .979 1 .971 .962 .981 .996 .948 .853 cabins .973 .968 .974 .966 .938 .935 .936 .932 .990 .934 .971 1 .987 .995 .974 .894 .753 boy .970 .965 .978 .956 .938 .936 .936 .936 .990 .932 .962 .987 1 .985 .968 .907 .764 tower .967 .963 .973 .957 .935 .927 .934 .932 .983 .941 .981 .995 .985 1 .987 .901 .774 horse .965 .963 .972 .950 .946 .932 .946 .949 .958 .966 .996 .974 .968 .987 1 .933 .842 cat .959 .963 .962 .944 .980 .971 .983 .989 .906 .987 .948 .894 .907 .901 .933 1 .918 shed .817 .824 .824 .797 .849 .834 .854 .868 .744 .897 .853 .753 .764 .774 .842 .918 1 EXQUIRES (EXtensible QUantitative Image RESampling) 40

Metric Correlations

metric xyz2 xyz4 xyz1 blur1 blur2 mssim srgb1 srgb2 blur4 cmc1 srgb4 cmc2 cmc4 srgb∞ xyz∞ blur∞ cmc∞ xyz2 1 .958 .957 .865 .721 .691 .514 .512 .221 .177 .027 −.049 −.300 −.649 −.650 −.734 −.801 xyz4 .958 1 .959 .893 .781 .732 .560 .562 .274 .213 .097 −.010 −.244 −.591 −.531 −.695 −.740 xyz1 .957 .959 1 .949 .851 .837 .674 .669 .383 .347 .191 .113 −.150 −.526 −.609 −.617 −.687 blur1 .865 .893 .949 1 .947 .867 .733 .736 .534 .415 .275 .182 −.083 −.459 −.567 −.493 −.597 blur2 .721 .781 .851 .947 1 .927 .864 .871 .740 .595 .483 .370 .104 −.300 −.458 −.339 −.413 mssim .691 .732 .837 .867 .927 1 .944 .937 .740 .735 .605 .528 .260 −.165 −.395 −.280 −.312 srgb1 .514 .560 .674 .733 .864 .944 1 .994 .890 .893 .800 .731 .488 .022 −.264 −.089 −.102 srgb2 .512 .562 .669 .736 .871 .937 .994 1 .898 .881 .810 .721 .482 .017 −.258 −.095 −.106 blur4 .221 .274 .383 .534 .740 .740 .890 .898 1 .896 .863 .798 .609 .141 −.147 .132 .108 cmc1 .177 .213 .347 .415 .595 .735 .893 .881 .896 1 .938 .953 .802 .331 −.026 .239 .242 srgb4 .027 .097 .191 .275 .483 .605 .800 .810 .863 .938 1 .942 .854 .407 .119 .318 .350 cmc2 −.049 −.010 .113 .182 .370 .528 .731 .721 .798 .953 .942 1 .939 .506 .157 .426 .442 cmc4 −.300 −.244 −.150 −.083 .104 .260 .488 .482 .609 .802 .854 .939 1 .668 .379 .603 .638 srgb∞ −.649 −.591 −.526 −.459 −.300 −.165 .022 .017 .141 .331 .407 .506 .668 1 .696 .929 .946 xyz∞ −.650 −.531 −.609 −.567 −.458 −.395 −.264 −.258 −.147 −.026 .119 .157 .379 .696 1 .650 .736 blur∞ −.734 −.695 −.617 −.493 −.339 −.280 −.089 −.095 .132 .239 .318 .426 .603 .929 .650 1 .960 cmc∞ −.801 −.740 −.687 −.597 −.413 −.312 −.102 −.106 .108 .242 .350 .442 .638 .946 .736 .960 1 EXQUIRES (EXtensible QUantitative Image RESampling) 41

Toolchains

• sRGB VS Linear RGB Toolchain:
• The ranks are roughly the same
• sRGB toolchain gives more accurate results (smaller error)
• This is surprising...
• We convert using gamma correction built into ImageMagick

instead of using colour profiles

• Could it be the fault of the conversion?

EXQUIRES (EXtensible QUantitative Image RESampling) 42

Toolchains

• Mixed sRGB and Linear RGB Toolchains:
• Even bigger surprise:
• These toolchains produce more accurate results!
• Suggests that the conversion is not bad
• Leads to sigmoidization, which is now available in

ImageMagick and already being tested by a commercial video rendering program (madVR)

EXQUIRES (EXtensible QUantitative Image RESampling) 43

Summary of Results

• The measured relative accuracy of an upsampler depends

strongly on the characteristics of the image being enlarged

• Downsampler choice not only has the biggest impact, it can

completely flip the ranks

EXQUIRES (EXtensible QUantitative Image RESampling) 44

Summary of Results

• Commonly used metrics give fairly well correlated rankings
• But in general, the choice of metric significantly affects the

rank of an image upsampler

EXQUIRES (EXtensible QUantitative Image RESampling) 45

Summary of Results

• The rankings are mostly independent of the choice of

resampling ratio and the subject matter of test images

EXQUIRES (EXtensible QUantitative Image RESampling) 46

Summary of Results

• Resampling through linear light or directly using sRGB

pixel values has insignificant impact on rankings

• However, mixed toolchains give significantly more accurate

results

• This suggests that the choice of colour space when

enlarging is a topic worth investigating

EXQUIRES (EXtensible QUantitative Image RESampling) 47

Future Directions

• Add noise (e.g. compress with JPEG) to the downsamples

EXQUIRES (EXtensible QUantitative Image RESampling) 48

Thanks

• Dr. John Cupitt (Imperial College, London)
• Anthony Thyssen (Griffith University, Brisbane)
• Mark Thompson
• Advisory committee:
• Dr. Nicolas Robidoux (supervisor)
• Dr. Ralf Meyer (co-supervisor)
• Dr. Julien Dompierre
• Dr. Torsten Möller (Simon Fraser University)

EXQUIRES (EXtensible QUantitative Image RESampling) 49

Thanks

• Test image photographers:
• Henry HO
• Dr. Minglun Gong
• Dr. John Cupitt
• Anthony Barnett
• Luiz E. Vasconcellos
• Jean-Francois Avon
• Michael Muré
• Jana Duncan
• Mukund Sivaraman
• Dr. Kirk Martinez
• Holly Graham
• Friends, family, and colleagues

EXQUIRES (EXtensible QUantitative Image RESampling) 50