Image and Video Coding: Human Visual Perception Y 1 X 1 1 Z - - PowerPoint PPT Presentation

Image and Video Coding: Human Visual Perception

Y X Z 1 1 1

The Human Eye / Structure of the Human Eye

The Human Eye

Human Eye: Similar components as a camera Two-lens system: Cornea and crystalline lens Variable aperture: Pupil / iris Light-sensitive surface: Retina Data processing: Visual cortex of the brain

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 2 / 58

The Human Eye / Human Photoreceptors

Light-Sensitive Cells in Retina

Rods About 100 million rods in retina (each eye) More light-sensitive than cone cells (≈ 100 times) Responsible for scotopic vision (night vision) Low visual acuity, no color perception Cones About 5 million cones in retina (each eye) Less light-sensitive than rod cells Responsible for photopic vision (well-lit conditions) High visual acuity Responsible for color vision

Colorized picture of rods (yellow) and cones (blue) as seen through a scanning electron microscope

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 3 / 58

The Human Eye / Human Photoreceptors

Distribution of Rods and Cones

2 4 6 8 10 12 14 16 18 20

• 60
• 40
• 20

20 40 blind spot receptor density [104 / mm2] visual angle relative to center of fovea [degree] 2 4 6 8 10 12 14 16 18 20

• 60
• 40
• 20

20 40 nasal retina temporal retina rods cones

Scotopic vision mainly at outer parts of the retina Photopic and color vision centered at the fovea (by far highest visual acuity) No receptors at blind spot (optic nerve)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 4 / 58

The Human Eye / Spectral Sensitivity

Spectral Sensitivity of Human Vision

0.2 0.4 0.6 0.8 1 400 450 500 550 600 650 700 750 luminous efficiency wavelength λ [nm] photopic vision scotopic vision

Luminous Efficiency Functions Characterize spectral sensitivity of brightness perception Experimentally determined in brightness matching experiments V (λ) for photopic vision (standardized in 1924), V ′(λ) for scotopic vision (standardized in 1951) Visible light: Electromagnetic radiation in spectrum 390 nm−700 nm

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 5 / 58

The Human Eye / Spectral Sensitivity

Radiometric vs Photometric Quantities

Two parallel systems of quantities for characterizing electromagnetic radiation Radiometric quantities: Unweighted absolute energy / power measures Photometric quantities: Every wavelength is weighted with luminous efficiency function V (λ) Example: Radiance Φ – Luminance I Given is a radiance spectrum Φ(λ) [ unit: W/(sr · m3) ] Radiometric quantity: Radiance [ unit: W/(sr · m2) ] Φ = ∞ Φ(λ) dλ Equivalent photometric quantity: Luminance [ unit: cd/m2 ] I = K ∞ V (λ) · Φ(λ) dλ with K = 683 cd · sr W

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 6 / 58

The Human Eye / Spectral Sensitivity

Trichromatic Vision

Three Cone Types: Postulated by Young and Helmholtz (confirmed later by measurements) Responsible for color vision under photopic conditions

colorized picture of cone mosaic in central region of fovea

S-cones (S for short wavelength) Most sensitive to blue light (≈ 430 nm) Only 5-6% of the cones are S-cones (blue light is blurred due to chromatic aberration) M-cones (M for medium wavelength) Most sensitive to green light (≈ 530 nm) L-cones (L for long wavelength) Most sensitive to yellow-green light (≈ 560 nm) Ratio of L/M cones highly varies for different individuals

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 7 / 58

The Human Eye / Spectral Sensitivity

Spectral Sensitivity of Human Cones

0.2 0.4 0.6 0.8 1 400 450 500 550 600 650 700 750 S-cones rods M-cones L-cones normalized sensitivity wavelength λ [nm]

Cone fundamentals Spectral sensitivities with respect to light entering the cornea Estimated by comparing color-matching data (discussed later)

• f individuals with normal vision and individuals lacking one or two cone types

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 8 / 58

The Human Eye / Comparison to Other Species

Color Perception of Other Species

Monochromatic vision: Seals, sea lions, walruses, dolphins, whales Dichromatic vision: Most mammals (except sea mammals and some primates) Trichromatic vision: Humans and closely related primates, also bees Tetrachromatic vision: Many species of birds, fish, amphibians, reptiles, arachnids, and insects Pentachromatic vision: Some butterflies and birds (e.g., pigeons)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 9 / 58

The Human Eye / Comparison to Other Species

The King of Color Vision: The Mantis Shrimp

16 different types of photoreceptors Can detect polarized light Depth vision with a single eye (3 focal points per eye)

[ Marshall, et al, 2007 ]

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 10 / 58

The Human Eye / Luminance Sensitivity

Luminance Sensitivity

Sensing Capabilities of Human Eye From 10−6 cd/m2 (visual threshold) to 105 cd/m2 (sunny day) In each moment, only 2-3 orders of magnitude Adaptation to Lighting Conditions Pupillary light reflex: Fast but rather small effect Main factor: Photochemical reactions in the pigments of rods and cones Sensitivities of cones are nearly independently adjusted Weber-Fechner Law Ability to distinguish areas with different luminances depends on background brightness For a wide range of luminance levels I ∆I I ≈ const (approx. 1-2%)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 11 / 58

The Human Eye / Neural Processing in Retina

Opponent Colors

• 1
• 0.5

0.5 1 400 450 500 550 600 650 700 750 yellow-blue red-green achromatic normalized sensitivity wavelength λ [nm]

Hering (1920): Colors never look reddish-green or yellowish-blue Neurons in retina transform cone responses into opponent signals (effective decorrelation): Achromatic signal + red-green difference + yellow-blue difference Opponent signals could be measured in stimulated retina tissue

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 12 / 58

The Human Eye / Summary

Intermediate Summary

Human Eye Similar components as a camera Photopic vision: Three types of photorecptors (cones) with different spectral characteristics Responsible for color vision Think about the following ...

For representing color images, we use three different color components (e.g., RGB). How many color components would we need, if we wanted to design an image communication system for your ... dog (dichromatic vision) goldfish (tetrachromatic vision)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 13 / 58

Human Color Perception / What is Color ?

The Nature of Light and Color

Isaac Newton (1672): Color is a property of light, not of objects Sun light consists of 7 colored particles

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 14 / 58

Human Color Perception / What is Color ?

Electromagnetic Waves

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 15 / 58

Human Color Perception / What is Color ?

What Is Color ?

what we see what our dogs/cats might see

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 16 / 58

Human Color Perception / What is Color ?

Color: Interaction of Electromagnetic Waves with Human Cones

Θ(λ)

l(λ) ⋅ Θ(λ) l(λ)

L cones

m(λ) ⋅ Θ(λ) m(λ)

M cones

s(λ) ⋅ Θ(λ) s(λ)

S cones L =

∞

• ¯

l(λ) Φ(λ) dλ M =

∞

• ¯

m(λ) Φ(λ) dλ S =

∞

• ¯

s(λ) Φ(λ) dλ

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 17 / 58

Human Color Perception / What is Color ?

Human Color Perception

What is Color? A certain spectral composition induces the perception of a particular color Color is not a physical quantity Color is a sensation in the viewer’s mind (interaction of electromagnetic waves with cones) Response of Human Cones to Electromagnetic Waves Electromagnetic spectra are mapped to a 3D vector   L M S   =

∞

• 

 ¯ ℓ(λ) ¯ m(λ) ¯ s(λ)   Φ(λ) Φ0 dλ Φ(λ) – Observed radiance spectrum Φ0 – Arbitrarily chosen reference radiance (Φ0 > 0) ¯ l(λ), ¯ m(λ), ¯ s(λ) – Cone fundamentals (normalized spectral sensitivities)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 18 / 58

Human Color Perception / Color Reproduction

Metamers

400 450 500 550 600 650 700 750 Φ1(λ) Φ2(λ) Φ3(λ) Φ4(λ) spectral radiance Φ(λ) wavelength λ [nm]

perceived color:

• range

Cannot distinguish light stimuli that yield the same cone excitation response (L, M, S) Light stimuli with that property are called metamers

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 19 / 58

Human Color Perception / Color Reproduction

Characterization of Metamers

Metameric Color Matches Consider two lights with radiance spectra Φ1(λ) and Φ2(λ) Cone excitation responses (L,M,S) are given by   L M S  

1

=

∞

• 

 ¯ ℓ(λ) ¯ m(λ) ¯ s(λ)   Φ1(λ) Φ0 dλ and   L M S  

2

=

∞

• 

 ¯ ℓ(λ) ¯ m(λ) ¯ s(λ)   Φ2(λ) Φ0 dλ The lights Φ1(λ) and Φ2(λ) are metamers if and only if   L M S  

1

=   L M S  

2 ∞

• 

 ¯ ℓ(λ) ¯ m(λ) ¯ s(λ)   Φ1(λ) Φ0 dλ =

∞

• 

 ¯ ℓ(λ) ¯ m(λ) ¯ s(λ)   Φ2(λ) Φ0 dλ Interesting Question Can we always construct a metameric spectrum by using 3 primary lights ?

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 20 / 58

Human Color Perception / Color Reproduction

Mixing Three Primary Lights

400 450 500 550 600 650 700 750 pC(λ) pA(λ) pB(λ) spectral radiance Φ(λ) wavelength λ [nm] 400 450 500 550 600 650 700 750 ( 0, 0.5, 0.9 ) ( 0.9, 0.2, 0.2 ) spectral radiance Φ(λ) wavelength λ [nm]

Three primary lights with spectra pA(λ), pB(λ), and pC(λ) Spectrum of mixture with mixing factors A, B, and C Φmix(λ) = A · pA(λ) + B · pB(λ) + C · pC(λ) =   pA(λ) pB(λ) pC(λ)  

T

·   A B C   = p(λ)T ·   A B C  

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 21 / 58

Human Color Perception / Color Reproduction

Metameric Match with Three Primary Lights

Cone excitation response for mixture Φmix(λ) of the three primary lights pA(λ), pB(λ), and pC(λ)   L M S  

ABC

=

∞

• 

 ¯ ℓ(λ) ¯ m(λ) ¯ s(λ)   Φmix(λ) Φ0 dλ =  

∞

• 

 ¯ ℓ(λ) ¯ m(λ) ¯ s(λ)   p(λ)T Φ0 dλ   ·   A B C   = T ·   A B C   Mixing Factors for Generating a Metamer Consider any spectrum Φ(λ) that has the cone excitation response (L, M, S)Φ Mixing factors for metameric match with primary system are given by   A B C  

Φ

= T −1 ·   L M S  

Φ

with T = 1 Φ0

∞

• 

 ¯ ℓ(λ) pA(λ) ¯ ℓ(λ) pB(λ) ¯ ℓ(λ) pC(λ) ¯ m(λ) pA(λ) ¯ m(λ) pB(λ) ¯ m(λ) pC(λ) ¯ s(λ) pA(λ) ¯ s(λ) pB(λ) ¯ s(λ) pC(λ)   dλ

T must be invertible: Primaries are perceived as having different colors None of the primaries can be presented as mixture of the other two

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 22 / 58

Human Color Perception / Color Description

Color-Matching Functions

Reconsider mixing factors for a given spectrum Φ(λ)   A B C  

Φ

= T −1   L M S  

Φ

= T −1

∞

• 

 ¯ ℓ(λ) ¯ m(λ) ¯ s(λ)   Φ(λ) Φ0 dλ =

∞

• 

 ¯ a(λ) ¯ b(λ) ¯ c(λ)   Φ(λ) Φ0 dλ =

∞

• ¯

c(λ) Φ(λ) Φ0 dλ Color-Matching Functions Vector function ¯ c(λ) = ( ¯ a(λ), ¯ b(λ), ¯ c(λ) )T is similar to cone fundamentals, but for primaries   ¯ a(λ) ¯ b(λ) ¯ c(λ)   = T −1   ¯ ℓ(λ) ¯ m(λ) ¯ s(λ)   =   1 Φ0

∞

• 

 ¯ ℓ(λ) pA(λ) ¯ ℓ(λ) pB(λ) ¯ ℓ(λ) pC(λ) ¯ m(λ) pA(λ) ¯ m(λ) pB(λ) ¯ m(λ) pC(λ) ¯ s(λ) pA(λ) ¯ s(λ) pB(λ) ¯ s(λ) pC(λ)   dλ  

−1 

 ¯ ℓ(λ) ¯ m(λ) ¯ s(λ)   The functions ¯ a(λ), ¯ b(λ), and ¯ c(λ) are called color-matching functions for the primary lights If we know ¯ c(λ) for a set of three primaries, we can uniquely describe all perceivable colors by the corresponding tristimulus values (A, B, C)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 23 / 58

Human Color Perception / Color Description

Linearity of Color Mixing

Consider two lights with spectra Φ1(λ) and Φ2(λ) Associated tristimulus values for a system of three primaries   A B C  

1

=

∞

• ¯

c(λ) Φ1(λ) Φ0 dλ and   A B C  

2

=

∞

• ¯

c(λ) Φ2(λ) Φ0 dλ Grassmann’s Law of Linear Mixing Experimentally discovered by Grassmann in 1853 Tristimulus (A, B, C) values for a mixture Φ(λ) = α Φ1(λ) + β Φ2(λ)   A B C   =

∞

• ¯

c(λ) Φ(λ) Φ0 dλ =

∞

• ¯

c(λ) α Φ1(λ) + β Φ2(λ) Φ0 dλ = α   A B C  

1

+ β   A B C  

2

Tristimulus values are weighted with mixing factors

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 24 / 58

Human Color Perception / Color Description

Determination of Color-Matching Functions

Consider monochromatic lights for a given wavelength λ Φλ(λ′) = Φλ δ(λ′ − λ) where the dirac delta distribution δ(x) has the properties δ(x) = ∞ : x = 0 : x = 0 and ∞

−∞

δ(x) dx = 1

λ′ Φλ(λ′) λ

• Φλ(λ′) dλ′ = Φλ

Measuring Color-Matching Functions Determine tristimulus values (A, B, C)λ for all monochromatic lights Φλ(λ′)   A B C  

λ

=

∞

• ¯

c(λ′) Φλ(λ′) Φ0 dλ′ = Φλ Φ0

∞

• ¯

c(λ′) δ(λ′ − λ) dλ′ = Φλ Φ0 · ¯ c(λ) Note: Ratio Φλ/Φ0 has to be known for all wavelengths λ

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 25 / 58

Human Color Perception / Color Description

Color-Matching Experiments

Experimental Determination of Color-Matching Functions Adjust amounts of primary lights to match a monochromatic test light Not all test lights could be matched Possibly to move any of the primaries to the side of the test light (counted as negative value)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 26 / 58

Human Color Perception / Color Description

Experiments for Defining Standard Colorimetric Observer

Color-Matching Experiments of Wright (1928) and Guild (1925) Experimental data converted into common primary system (700 nm, 546.1 nm, 435.8 nm) Ratios A:B :C were combined with already determined V (λ) Φλ/Φ0 not required Scaling Factors for Color-Matching Functions ¯ r(λ), ¯ g(λ), ¯ b(λ) Equal-energy spectrum is represented by equal amounts of the three primaries ∞ ¯ r(λ) dλ = ∞ ¯ g(λ) dλ = ∞ ¯ b(λ) dλ Luminance ratios IR : IG : IB = 1 : 4.5907 : 0.0601 Radiance ratios ΦR : ΦG : ΦB = 1 : 0.0191 : 0.0137 Normalization factor (ΦR/Φ0) chosen so that V (λ) = ¯ r(λ) + IG IR · ¯ g(λ) + IB IR · ¯ b(λ)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 27 / 58

Human Color Perception / Color Description

CIE 1931 RGB Color-Matching Functions

• 0.1

0.1 0.2 0.3 0.4 400 450 500 550 600 650 700 750 b

• (λ)

g

• (λ)

r

• (λ)

B G R tristimulus amplitudes wavelength λ [nm]

monochromatic primaries: red: 700.0 nm green: 546.1 nm blue: 435.8 nm Obtained using experimental data of Wright and Guild Basis of today’s color specification (CIE 1931 Standard Colorimetric Observer)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 28 / 58

Human Color Perception / The XYZ Color Space

Changing the Set of Primaries

Given: Tristimulus values (A, B, C)1 for a first set of primaries p1(λ) Question: What are the tristimulus values (A, B, C)2 for a second set of primaries p2(λ) ? Conversion between Primary Systems Tristimulus values (A, B, C)2 in second primary system are given by   A B C  

2

=

∞

• ¯

c2(λ) Φ1(λ) Φ0 dλ =  

∞

• ¯

c2(λ) p1(λ)T Φ0 dλ   ·   A B C  

1

= T 12 ·   A B C  

1

Linear transform with matrix T 12 = 1 Φ0

∞

• 

  ¯ a2(λ) pA1(λ) ¯ a2(λ) pB1(λ) ¯ a2(λ) pC1(λ) ¯ b2(λ) pA1(λ) ¯ b2(λ) pB1(λ) ¯ b2(λ) pC1(λ) ¯ c2(λ) pA1(λ) ¯ c2(λ) pB1(λ) ¯ c2(λ) pC1(λ)    dλ Columns of T 12 specify tristimulus vectors for a primary of set 1 in primary set 2

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 29 / 58

Human Color Perception / The XYZ Color Space

CIE 1931 XYZ Color Space

CIE 1931 XYZ Color-Matching Functions Primary conversion also valid for color-matching function (special tristimulus values) Color-matching function have to be linearly related to ¯ r(λ), ¯ g(λ), ¯ b(λ)   ¯ x(λ) ¯ y(λ) ¯ z(λ)   = T XYZ ·   ¯ r(λ) ¯ g(λ) ¯ b(λ)   Design criteria for choosing matrix T XYZ All values of ¯ x(λ), ¯ y(λ) and ¯ z(λ) shall be non-negative Function ¯ y(λ) equal to luminous efficiency function V (λ) For equal-energy spectrum: X = Y = Z Function ¯ z(λ) vanishes for long wavelengths Maximize area of meaningful radiance spectra inside plane X + Y + Z = 1

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 30 / 58

Human Color Perception / The XYZ Color Space

CIE 1931 XYZ Color-Matching Functions

0.5 1 1.5 2 400 450 500 550 600 650 700 750 z

• (λ)

y

• (λ) = V(λ)

x

• (λ)

tristimulus amplitudes wavelength λ [nm]

Color of any spectrum Φ(λ) is characterized by   X Y Z   =

∞

• 

 ¯ x(λ) ¯ y(λ) ¯ z(λ)  Φ(λ) Φ0 dλ Chosen transform matrix T XYZ = 1 0.17697

  0.49000 0.31000 0.20000 0.17697 0.81240 0.01063 0.00000 0.01000 0.99000  

Color-matching functions ¯ x(λ), ¯ y(λ), ¯ z(λ) form basis for color description

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 31 / 58

Human Color Perception / The XYZ Color Space

Radiance Spectra in XYZ Color Space

Location of monochromatic lights & imaginary purple plane (connects tangents)

Y X Z

All possible radiance spectra are located in a cone (apex in origin)

Y X Z 1 1 1

Discriminate between luminance and quality of color (hue & saturation) Quality of color can be described by normalized chromaticity coordinates x = X X + Y + Z and y = Y X + Y + Z (z = 1 − x − y is redundant) Chromaticity coordinates x and y specify the values of X and Y inside the plane X + Y + Z = 1

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 32 / 58

Human Color Perception / The XYZ Color Space

The CIE xy Chromaticity Diagram

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 chromaticity y chromaticity x 460 4 7 480 490 5 510 520 530 540 550 5 6 5 7 5 8 5 9 6 6 1 6 2 E W R G B

spectral locus purple line

Y X Z 1 1 1 R B G

Equal-energy white (E): x = y = 1/3 Primaries and white point of sRGB color space

R: x = 0.6400 y = 0.3300 G: x = 0.3000 y = 0.6000 B: x = 0.1500 y = 0.0600 W: x = 0.3127 y = 0.3290

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 33 / 58

Human Color Perception / Camera and Display Color Spaces

Linear Color Spaces

Color spaces that are linearly related to cone excitation space LMS CIE 1931 XYZ color space, RGB color spaces used in practice Typically specified by chromaticity coordinates of primaries and white point Conversion matrix to XYZ space   X Y Z   =   Xr Xg Xb Yr Yg Yb Zr Zg Zb  ·   R G B   consider white point: R = G = B = 1 Y = 1 Use X = Y · x/y and Z = Y · (1 − x − y)/y for white point  

xw yw

1

1−xw−yw yw

 =   

Yr xr

yr

Yg

xg yg

Yb xb

yb

Yr Yg Yb Yr 1−xr−yr

yr

Yg

1−xg−yg yg

Yb 1−xb−yb

yb

   =   

xr yr xg yg xb yb

1 1 1

1−xr−yr yr 1−xg−yg yg 1−xb−yb yb

     Yr Yg Yb   Solve linear equation system for Yr, Yg, and Yb and calculate conversion matrix

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 34 / 58

Human Color Perception / Camera and Display Color Spaces

Primary Paradoxon

Have shown: Can calculate color-matching functions for given primaries (knowing cone fundamentals) Primary spectra p(λ) for given color-matching functions ¯ c(λ) ? Not uniquely defined, but have to fulfill condition 1 Φ0

∞

• ¯

c(λ) p(λ)T dλ = 1 Φ0

∞

• 

 ¯ a(λ) ¯ b(λ) ¯ c(λ)  

• pA(λ) pB(λ) pC(λ)
• dλ = I

(identity matrix) Primary Paradoxon There are no primary lights for non-negative color-matching functions (e.g., XYZ or LMS) For all real systems of primaries, the associated color-matching functions have negative entries Reason: Cone fundamentals overlap (cannot excite M cones without also exciting L/S cones)

0.2 0.4 0.6 0.8 1 400 450 500 550 600 650 700 750 S-cones M-cones L-cones normalized sensitivity wavelength λ [nm]

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 35 / 58

Human Color Perception / Camera and Display Color Spaces

Primary Paradoxon: Consequence for Acquisition and Display

Cameras All colors can be captured using three color filters Filter functions have same meaning as color-matching functions Design problem: Color filters = Linear combination of cone fundamentals Displays Not all colors perceivable by humans can be represented as mixture of three primary lights Color Gamut: Set of representable colors (determined by choice of primaries) Displaying Images Captured by Camera Always require conversion from camera color space into display color space   R G B  

display

= M3×3 ·   R G B  

camera

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 36 / 58

Human Color Perception / Camera and Display Color Spaces

Color Gamut – Examples

ITU-R ITU-R BT.709 BT.2020 xr 0.6400 0.7080 red yr 0.3300 0.2920 xg 0.3000 0.1700 green yg 0.6000 0.7970 xb 0.1500 0.1310 blue yb 0.0600 0.0460 white xw 0.3127 0.3127 (D65) yw 0.3290 0.3290 D65 white BT.709 (HD) BT.2020 (UHD) human gamut x y

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 37 / 58

Human Color Perception / Chromatic Adaptation

Illumination

0.2 0.4 0.6 0.8 1 400 450 500 550 600 650 700 daylight tungsten light bulb wavelength λ [nm] incident spec. radiance S(λ) / Smax 0.2 0.4 0.6 0.8 1 400 450 500 550 600 650 700 flower "veronica fruticans" wavelength λ [nm] spectral reflectance R(λ) 0.2 0.4 0.6 0.8 1 400 450 500 550 600 650 700 for daylight for tungsten light bulb wavelength λ [nm] reflected spectral radiance Φ(λ) / Φmax

Light entering the eye depends on illumination source and object properties Φ(λ) = S(λ) · R(λ) with Φ(λ) : Radiance spectrum entering the eye S(λ) : Incident spectral radiance reaching the surface point R(λ) : Reflectance spectrum of object surface

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 38 / 58

Human Color Perception / Chromatic Adaptation

Color Temperature

Color temperature Absolute temperature of ideal black body radiators (Planck’s law) Figure illustrates chromaticity coordinates of black body radiators Planckian locus Correlated color temperature Equivalent measure for non-incandescent sources Color temperature of black body radiator that best matches the color

• f the illumination source

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 chromaticity y chromaticity x

460 470 480 490 500 510 520 530 540 550 5 6 570 580 590 600 610 620

1000 K 2000 K 3000 K 4000 K 5 K 6000 K 8000 K 12000 K

∞

E

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 39 / 58

Human Color Perception / Chromatic Adaptation

Chromatic Adaptation: Stare at the white circle for 15 sec

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 40 / 58

Human Color Perception / Chromatic Adaptation

Chromatic Adaptation: Do you see the afterimage?

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 40 / 58

Human Color Perception / Chromatic Adaptation

Chromatic Adaptation

Chromatic adaptation To a large extent, human eye adjusts to lighting conditions Example: White paper appears white for various illumination sources Simple von Kries model (1902) Cone sensitivities are independently adjusted to illumination source   L2 M2 S2   =   α β γ   ·   L1 M1 S1   Allows prediction of color appearance for different illumination sources Research on color appearance Chromatic adaptation cannot solely described by an independent scaling of cone sensitivity functions Nonetheless, simple variations of the von Kries model are still widely used

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 41 / 58

Human Color Perception / Chromatic Adaptation

Generalized Linear Chromatic Adaptation Model

Chromatic adaptation in XYZ color space   X2 Y2 Z2   = M−1

CAT ·

  α β γ   · MCAT ·   X1 Y1 Z1   Suggestion for chromatic adaption transform MCAT in color appearance model CIECAM02 MCAT =

• 0.7328

0.4296 −0.1624 −0.7036 1.6974 0.0061 0.0030 −0.0136 0.9834

• Determination of scaling factors α, β, γ

α = Aw2/Aw1 β = Bw2/Bw1 γ = Cw2/Cw1 with   Awk Bwk Cwk   = MCAT ·  

xwk ywk

1

1−xwk−ywk ywk

  with (xw1, yw1) and (xw2, yw2) being the chromaticity coordinates

• f the white points for the considered illumination sources

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 42 / 58

Human Color Perception / Chromatic Adaptation

White Balancing

camera data assuming equal-energy white (photo taken after sunset) after suitable white balancing

Camera sensors do not adjust to illumination sources Sensor data have to be processed to simulate chromatic adaptation White Balancing Often combined with color space conversion (camera to representation format)   R G B  

rep

= M−1

rep · M−1 CAT ·

  α β γ   · MCAT · Mcam ·   R1 G1 B1  

cam

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 43 / 58

Human Color Perception / Summary

Intermediate Summary

What is Color ? No physical quantity, but sensation in viewer’s mind Characterized by neural response (L, M, S) of human cones Cannot distinguish light stimuli with the same cone excitation response (L, M, S) Color Description Need three values that are related the neural response (L, M, S) Basis for today’s color description: XYZ color space (experimentally determined) Quality of color can be described by two values (chromaticity coordinates x and y) Color Spaces of Cameras and Displays Use linear color spaces (linearly related to LMS and XYZ color spaces) Typically specified by chromaticity coordinates of primaries and white point Displays cannot represent all colors: Color Gamut (set of representable colors) Need conversion from camera to display color space (typically includes white balancing)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 44 / 58

Visual Acuity

Visual Acuity

Ability of human visual system to resolve fine details is determined by Resolution of human optics Sampling of projected image by photoreceptor cells Neural processing of photoreceptor signals Ophthalmologists use Snellen chart Identify letters in a certain distance Normal vision: Can resolve spatial frequencies of at least 30 cpd Contrast sensitivity functions Specify contrast threshold between visible and invisible Measured for spatio-temporal sinusoidal stimuli I(α, t) = ¯ I ·

• 1 + C · cos(2πu α) · cos(2πv t)
• with

C = Imax − Imin Imax + Imin with ¯ I = (Imax + Imin)/2 being the average luminance and C the constrast

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 45 / 58

Visual Acuity / Contrast Sensitivity

Spatial Contrast Sensitivity – Campbell-Robson Chart

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 46 / 58

Visual Acuity / Contrast Sensitivity

Spatial Contrast Sensitivity – Isoluminant Stimuli

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 47 / 58

Visual Acuity / Contrast Sensitivity

Spatial Contrast Sensitivity

3 30 300 1 10 100 1000 0.03 0.3 3 30 0.1 1 10 100 1000 ca/m2 100 ca/m2 10 ca/m2 1 ca/m2 0.1 ca/m2 contrast sensitivity sc(u) spatial frequency u [cpd] Weber-Fechner law is valid in this range 3 30 300 1 10 100 0.03 0.3 3 30 0.1 1 10 100 isochromatic red-green blue-yellow contrast sensitivity sc(u) spatial frequency u [cpd]

Spatial contrast sensitivity functions Perceptual equivalent of modulation transfer function Sensitivity depends on average luminance (except for low frequency range) Human beings are much more sensitive to fine details in luminance than to fine details in color difference signals (with constant luminance)

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 48 / 58

Visual Acuity / Contrast Sensitivity

Spatio-Temporal Contrast Sensitivity

3 30 300 1 10 100 0.3 3 30 0.1 1 10 100 v = 1 Hz v = 6 Hz v = 16 Hz v = 22 Hz contrast sensitivity sc(u,v) spatial frequency u [cpd] 3 30 300 1 10 100 0.3 3 30 0.1 1 10 100 u = 0.5 cpd u = 4 cpd u = 16 cpd u = 22 cpd contrast sensitivity sc(u,v) temporal frequency v [Hz]

Spatio-temporal contrast sensitivity functions Spatial and temporal aspects are not independent of each other Critical flicker frequency (steady appearance): 50-60 Hz Critical flicker frequency for color differences: 25-30 Hz

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 49 / 58

Visual Acuity / Pattern Sensitivity

Pattern Sensitivity

Human vision Human visual system is non-linear Analysis of responses to harmonic stimuli (contrast sensitivity functions) is not sufficient for describing resolving capabilities There are several complicated neural aspects that influence the way we see and discriminate patterns track the motion of objects over time . . . The human brain always interprets the visual information

Seeing is actively interpreting the data

• btained by the photoreceptor cells

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 50 / 58

Visual Acuity / Pattern Sensitivity

Brightness Perception [E. H. Adelson]

The squares labeled as “A” and “B” have exactly the same gray value

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 51 / 58

Visual Acuity / Pattern Sensitivity

Color Perception [R. B. Lotto]

look at blue squares look at yellow squares

same color

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 52 / 58

Visual Acuity / Pattern Sensitivity

Color Perception [R. B. Lotto]

look at blue squares look at yellow squares

same color

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 52 / 58

Visual Acuity / Pattern Sensitivity

Geometric Illusions

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 53 / 58

Visual Acuity / Pattern Sensitivity

Turning the Tables [R. N. Shepard]

Both table plates have exactly the same shape

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 54 / 58

Visual Acuity / Pattern Sensitivity

Spiral or Circles? [A. Kitaoka]

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 55 / 58

Visual Acuity / Pattern Sensitivity

Illusion of Movement [A. Kitaoka]

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 56 / 58

Visual Acuity / Pattern Sensitivity

Illusion of Depth

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 57 / 58

Summary

Summary of Lecture

Color Perception Light stimuli are converted into 3D neural responses (L, M, S) Cannot distinguish light stimuli that produce the same neural response (L, M, S) Metamers Color Description Need three values that are related to cone excitation response (L, M, S) Common color system: CIE 1931 XYZ color space (based on experimental data) Linear color spaces (typically RGB): Linearly related to XYZ and LMS Color Reproduction Mixture of three primary lights (typically red, green, and blue) Not all colors can be reproduced Color gamut depends on choice of primaries Visual Acuity More sensitive to details in luminance than to details in color differences (see opponent processes) Several neural processes influence the visual perception

Heiko Schwarz (Freie Universität Berlin) — Image and Video Coding: Human Visual Perception 58 / 58