Human Visual System Models in Computer Graphics Tun O. Aydn MPI - - PowerPoint PPT Presentation

Human Visual System Models in Computer Graphics

Tunç O. Aydın

MPI Informatik Computer Graphics Department HDR and Visual Perception Group

Outline

• Reality vs. Perception

– Why even bother modeling visual perception

• The Human Visual System (HVS)

– How the “wetware” affects our perception

• HVS models in Computer Graphics

– Visual Significance of contrast – Contrast Detection

• Our contributions

– Key challenges

High Low

Difference Image (Color coded)

Invisible Bits & Bytes

Reference (bmp, 616K) Compressed (jpg, 48K)

Variations of Perception

No one-to-one correspondence between visual perception and reality ! “Perceived Visual Data” instead of luminance or arbitrary pixel values

The Human Visual System (HVS)

• Experimental

Methods of Vision Science

– Micro-electrode – Radioactive Marker – Vivisection – Psychophysical Experimentation

HVS effects (1): Glare

Video Courtesy of Tobias Ritschel

• Disability Glare

(blooming)

Disability Glare

• Model of Light

Scattering

– Point Spread Function in spatial domain – Optical Transfer Function in Fourier Domain [Deeley et al. 1991]

Spatial Frequency [cy/deg] Modulation

(2): Light Adaptation

Time

Adaptation Level:

10-4 cd/m2

Adaptation Level:

17 cd/m2

Perceptually Uniform Space

• Transfer function:

Maps Luminance to Just Noticeable Differences (JNDs) in

• Luminance. [Mantiuk

et al. 2004, Aydın et

• al. 2008]

Response [JND] Luminance [cd/m2]

(3): Contrast Sensitivity

CSF(spatial frequency, adaptation level, temporal freq., viewing dist, …)

Contrast Spatial Frequency

November 6, 2011

• Steady-state CSFS:

Returns the Sensitivity (1/Threshold contrast), given the adaptation luminance and spatial frequency [Daly 1993].

Contrast Sensitivity Function (CSF)

(4): Visual Channels

Cortex Transform

(5): Visual Masking

Loss of sensitivity to a signal in the presence of a “similar frequency” signal “nearby”.

Modeling Visual Masking

• Example: JPEG’s

pointwise extended masking:

C’: Normalized Contrast

HVS Models in Graphics/Vision

HDR LDR Tone Mapping Compression

Rate the Quality

Quality Assessment Panorama Stitching

Visual Significance Pipeline

 

k N n k ref k tst

R R R

/ 1 1

ˆ 



 

Contrast Detection Pipeline

Log Threshold Elevation

Log Contrast Contrast Difference

Probability of Detection

 





  

N n n

P P

1

1 1 ˆ

CONTRIBUTIONS: VISUAL SIGNIFICANCE

Visually Significant Edges

• Key Idea: Use the

magnitude of the HVS model’s response as the measure of edge strength, instead of gradient magnitude.

• Result (1): Significant

improvement in application results, especially for HDR images

• Result (2): Only minor

improvements observed in LDR retargeting and panorama stitching.

[Aydın, Čadík, Myszkowski, Seidel. 2010 ACM TAP]

Calibration Procedure

• CSF from the Visible Differences

Predictor [Daly’93]

• JPEG’s pointwise extended masking
• Calibration: CSF derived for

sinusoidal stimuli, not for edges.

• Perceptual experiment for

measuring edge thresholds

Calibration Function

Calibration function: R: Visual Significance for sinusoidal stimulus, R’: Visual Significance for edges. Subjective Measurements Polynomial Fit Metric Predictions Polynomial Fit Ideal Metric Response Calibrated Metric Response

Image Retargeting

Visual Significance Maps

High Low

Display Visibility under Dynamically Changing Illumination

• Key Idea: Extending

steady-state HVS models with temporal adaptation model

• Result: A visibility

class estimator integrated into a software that simulates illumination inside an automobile.

[Aydın, Myszkowski, Seidel. 2009 EuroGraphics]

L

Background Luminance

cvi for Steady-State Adaptation

• Contrast vs. Intensity

(cvi): function assumes perfect adaptation

• Contrast vs. Intensity

and adaptation (cvia) accounts for maladaptation

L + ∆L

Threshold Luminance

C L cvi   :

C L L cvia

a

  , :

cvia for Maladaptation

cvia cvi

Adaptation over time

Low High

Visual Significance t = 0 t = 0.2s t = 0.4s t = 0.8s

Rendering Adaptation

[Pająk, Čadík, Aydın, Myszkowski, Seidel. 2010 Electronic Imaging] Dark Adaptation Bright Adaptation

CONTRIBUTIONS: CONTRAST DETECTION

Quality Assessment (IQA, VQA)

Rate the Quality

+ Reliable - High cost

• Key Idea: Find a

transformation from Luminance to pixel values, such that:

– An increment of 1 pixel value corresponds to 1 JND Luminance in both HDR and LDR domains. – The pixel values in LDR domain should be close to sRGB pixel values

• Result: Common LDR

Quality metrics (SSIM, PSNR) extended to HDR through the PU space transformation

Perceptually Uniform Space

[Aydın, Mantiuk, Seidel. 2008 Electronic Imaging]

Perceptually Uniform Space

• Derivation:

for i = 2 to N Li = Li-1 + tvi(Li-1); end for

• Fit the absolute value and

subject sensitivity to sRGB within CRT luminance range

• Key Idea: Instead of the

traditional contrast difference, use distortion measures agnostic to dynamic range difference.

• Result: An IQA that can

meaningfully compare an LDR test image with an HDR reference image, and vice versa. Enables

• bjective evaluation of

tone mapping operators.

Dynamic Range Independent IQA

[Aydın, Mantiuk, Myszkowski, Seidel. 2008 SIGGRAPH]

5x LDR HDR Luminance Luminance LDR HDR

HDR vs. LDR

Problem with Visible Differences

HDR Reference LDR Test HDR-VDP 95% 75% 50% 25% Detection Probability Contrast Loss Local Gaussian Blur

Distortion Measures

Reference Test Contrast Loss Reference Test Contrast Amplification Reference Test Contrast Reversal

Novel Applications

Tone Mapping Inverse Tone Mapping

Video Quality Assessment

DRIVQM [Aydin et al. 2010] DRIVDP [Aydin et al. 2008] (frame-by-frame) HDR Video

(tone mapped for presentation)

• Key Idea: Extend the

Dynamic Range Independent pipeline with temporal aspects to evaluate video sequences.

• Result: An objective

VQM that evaluates rendering quality, temporal tone mapping and HDR compression.

Dynamic Range Independent VQA

[Aydın, Čadík, Myszkowski, Seidel. 2010 SIGGRAPH Asia]

• CSF: ω,ρ,La → S

– ω: temporal frequency, – ρ: spatial frequency, – La: adaptation level, – S: sensitivity.

Contrast Sensitivity Function

• CSF: ω,ρ,La → S

– ω: temporal frequency, – ρ: spatial frequency, – La: adaptation level, – S: sensitivity.

Contrast Sensitivity Function

Spatio-temporal CSFT

• CSF: ω,ρ,La → S

– ω: temporal frequency, – ρ: spatial frequency, – La: adaptation level, – S: sensitivity.

Contrast Sensitivity Function

Steady-state CSFS

( )

Contrast Sensitivity Function

=

CSF(ω,ρ,La = L) CSFT(ω,ρ,La = 100 cd/m2) f(ρ,La)

x ÷ f =

CSFS(ρ,La) CSFS(ρ,100 cd/m2)

La = 100 cd/m2

CSFT(ω,ρ)

Extended Cortex Transform

Sustained and Transient Temporal Channels [Winkler 2005] Spatial

Evaluation of Rendering Methods

No temporal filtering With temporal filtering [Herzog et al. 2010] Predicted distortion map

Evaluation of Rendering Qualities

High quality Low quality Predicted distortion map

Evaluation of HDR Compression

Medium Compression High Compression

Validation Study

• Noise, HDR video compression, tone mapping
• “2.5D videos”
• HDR-HDR, HDR-LDR, LDR-LDR

Psychophysical Validation

(1) Show videos side-by-side

• n a HDR Display

(2) Subjects mark regions where they detect differences [Čadík, Aydın, Myszkowski, Seidel. 2011 Electronic Imaging]

Validation Study Results

Stimulus DRIVQM PDM HDRVDP DRIVDP 1

0.765

• 0.0147

0.591 0.488

2

0.883 0.686 0.673 0.859

3

0.843 0.886 0.0769 0.865

4

0.815 0.0205 0.211

• 0.0654

5

0.844 0.565 0.803 0.689

6

0.761

• 0.462

0.709 0.299

7

0.879 0.155 0.882 0.924

8

0.733 0.109 0.339 0.393

9

0.753 0.368 0.473 0.617

Average 0.809 0.257 0.528 0.563

Conclusion

• Starting Intuition: Working on “perceived” visual data,

instead of “physical” visual data.

Limitations and Future Work

• What about the rest of

the brain?

– Visual Attention – Prior Knowledge – Gestalt Properties – Free will – …

• User interaction?
• Depth perception

Acknowledgements

• Advisors

– Karol Myszkowski, Hans Peter Seidel

• Collaborators

– Martin Čadík, Rafał Mantiuk, Dawid Pająk, Makoto Okabe

• AG4 Members

– Current and past

• AG4 Staff

– Sabine Budde, Ellen Fries, Conny Liegl, Svetlana Borodina, Sonja Lienard.

• Thesis Committee

– Phillip Slusallek, Jan Kautz, Thorsten Thormählen.

• Family

– Süheyla and Vahit Aydın, Irem Dumlupınar

THANK YOU.

Tunç O. Aydın <tunc@mpii.de>