Administritivia Assignment 1 due tomorrow - Please bring your - - PowerPoint PPT Presentation
C OMPUTATIONAL A SPECTS OF D IGITAL P HOTOGRAPHY Light & Color Wojciech Jarosz wojciech.k.jarosz@dartmouth.edu Administritivia Assignment 1 due tomorrow - Please bring your pinhole cameras to class on Thursday for show and tell :-)
COMPUTATIONAL ASPECTS OF DIGITAL PHOTOGRAPHY
Light & Color
Wojciech Jarosz
wojciech.k.jarosz@dartmouth.edu
Assignment 1 due tomorrow
tell” :-)
Assignment 2 available soon
Fill out poll to select paper for presentations (before class Thursday)
CS 89/189: Computational Photography, Fall 2015 2
Light & Color
CS 89/189: Computational Photography, Fall 2015 3
A form of electromagnetic (EM) radiation
We perceive a limited section of the spectrum as “visible” light
CS 89/189: Computational Photography, Fall 2015 4
CS 89/189: Computational Photography, Fall 2015
5
Wavelength 1nm=10-9 meters,
speed of light = wavelength * frequency
Simplified model
Roughly speaking
refracted
Can model most effects that are important for our daily experience
CS 89/189: Computational Photography, Fall 2015 6
Rays carry a spectrum of electromagnetic energy
CS 89/189: Computational Photography, Fall 2015 7
Light can be a mixture of many wavelengths
a function of wavelength over enter spectrum
We perceive these distributions as colors
CS 89/189: Computational Photography, Fall 2015 8
Where spectra come from:
There are different physical processes that explain this multiplication e.g. absorption, interferences
× =
CS 89/189: Computational Photography, Fall 2015 9 Foundations of Vision, by Wandell After a slide by Frédo Durand
Colors are the sensations that arise from light energy with different wavelength distributions Color is a phenomenon of human perception; it is not a universal property of light Roughly speaking, things appear “colored” when they depend on wavelength and “gray” when they do not
CS 89/189: Computational Photography, Fall 2015 10 After a slide by Steve Marschner
Build a model for human color perception That is, map a physical light description to a perceptual color sensation
CS 89/189: Computational Photography, Fall 2015 11 After a slide by Steve Marschner
Physical Perceptual
Light & Color
CS 89/189: Computational Photography, Fall 2015 12
We can model the low-level behavior of the eye by thinking of it as a light-measuring machine
like a camera
Lens focuses light on retina
CS 89/189: Computational Photography, Fall 2015 13 After a slide by Steve Marschner
Retinal composition: two kinds of cells
Cones are concentrated in fovea
Rods concentrated outside fovea
CS 89/189: Computational Photography, Fall 2015 14
near fovea away from fovea
Produces a scalar value (a number) when photons land on it
wavelength
This is a reasonable model for many detectors:
CS 89/189: Computational Photography, Fall 2015 15 After a slide by Steve Marschner
CS 89/189: Computational Photography, Fall 2015
16
n(λ) p(λ) X =
Z
n(λ)p(λ) dλ
After a slide by Steve Marschner
Same math carries over to spectral distributions
distribution (SPD), s(λ) ¡
CS 89/189: Computational Photography, Fall 2015 17
X =
Z
s(λ) r(λ) dλ
measured signal input spectrum detector’s sensitivity
After a slide by Steve Marschner
Three types of cones with broadband spectral sensitivity
S,M,L neural response is integrated w.r.t. λ ¡
Results in a trichromatic visual system S, M, and L are tristimulus values
CS 89/189: Computational Photography, Fall 2015 18
CS 89/189: Computational Photography, Fall 2015
Cone responses to a spectrum s (Math)
19
M =
Z
rM(λ) s(λ) dλ S =
Z
rS(λ) s(λ) dλ L =
Z
rL(λ) s(λ) dλ
After a slide by Steve Marschner
Stimulus (arbitrary spectrum) Response curves Multiply Integrate
1 number 1 number 1 number
Start with infinite number of values (one per wavelength) End up with 3 values (one per cone type)
Discrete representation of cones and input spectrum as vectors
CS 89/189: Computational Photography, Fall 2015 21
Input L M S
After a slide by Matthias Zwicker
CS 89/189: Computational Photography, Fall 2015 22
cone sensitivities Input L M S
After a slide by Matthias Zwicker
CS 89/189: Computational Photography, Fall 2015 23
*
cone sensitivities input spectrum Input L M S
After a slide by Matthias Zwicker
Tristimulus response is a matrix-vector multiplication Integration is now summation
CS 89/189: Computational Photography, Fall 2015 24
*
cone sensitivities input spectrum tristimulus response
=
Input L M S
After a slide by Matthias Zwicker
rS, rM and rL are N-dimensional vectors, where N ¡= ¡∞
CS 89/189: Computational Photography, Fall 2015 25
M =
Z
rM(λ) s(λ) dλ = rM · s S =
Z
rS(λ) s(λ) dλ = rS · s L =
Z
rL(λ) s(λ) dλ = rL · s S M L = rS rM rL
|
s
|
Integral notation: Matrix notation:
Colorimetry: an answer to the problem
Wanted to map a physical light description to a perceptual color sensation Basic solution was known and standardized by 1930
CS 89/189: Computational Photography, Fall 2015 26 After a slide by Steve Marschner
Physical Perceptual S = rS · s M = rM · s L = rL · s
Luminance
(independent of its color)
Vλ that describes the eye’s overall ability to detect light at each wavelength
efficiency (tungsten vs. fluorescent vs. sodium vapor)
CS 89/189: Computational Photography, Fall 2015 27 After a slide by Steve Marschner
Y: just another response curve (like S, M, and L)
Vλ is a linear combination of S, M, and L
CS 89/189: Computational Photography, Fall 2015 28
Y = rY · s
After a slide by Steve Marschner
Chromaticity
regard for overall brightness)
CS 89/189: Computational Photography, Fall 2015 29 After a slide by Steve Marschner
Different wavelength, different intensity Same response
CS 89/189: Computational Photography, Fall 2015 30
M
After a slide by Frédo Durand
Different wavelength, different intensity But different response for different cones
CS 89/189: Computational Photography, Fall 2015 31
M S L
After a slide by Frédo Durand
Classical case: 1 type of cone is missing (e.g. L) Makes it impossible to distinguish some spectra
CS 89/189: Computational Photography, Fall 2015 33
differentiated same responses
After a slide by Frédo Durand
8% male, 0.6% female Genetic Dichromate (strong color blind) — 2% male
Anomalous trichromat (weak color blind)
More at, e.g. http://en.wikipedia.org/wiki/Color_blindness
CS 89/189: Computational Photography, Fall 2015 34 After a slide by Frédo Durand
CS 89/189: Computational Photography, Fall 2015
35 After a slide by Frédo Durand
CS 89/189: Computational Photography, Fall 2015 36 After a slide by Frédo Durand
Maze in subtle intensity contrast Visible only to color blinds Color contrast overrides intensity otherwise
CS 89/189: Computational Photography, Fall 2015 37 After a slide by Frédo Durand
Links:
who is colorblind
CS 89/189: Computational Photography, Fall 2015 39 After a slide by Frédo Durand
We are all color blind! These two different spectra elicit the same cone responses Called metamers
CS 89/189: Computational Photography, Fall 2015 41 After a slide by Frédo Durand
Take a spectrum (which is a function) Eye produces three numbers This throws away a lot of information!
same S, M, L tristimulus values
CS 89/189: Computational Photography, Fall 2015 42 After a slide by Steve Marschner
A dot product is a projection Humans project an infinite dimensional vector (the SPD) onto a 3-D subspace
For intuition, we can imagine a 3D analog
CS 89/189: Computational Photography, Fall 2015 43 After a slide by Steve Marschner
CS 89/189: Computational Photography, Fall 2015 44
The information available to the visual system about a spectrum is just 3 numbers! Two spectra that project to the same response are metamers
After a slide by Steve Marschner
Which stimuli are metamers?
CS 89/189: Computational Photography, Fall 2015
sensitivity λ λ wavelength wavelength wavelength wavelength wavelength λ λ λ Energy Energy Energy Energy
Response curve Stimuli
A B C D
45 After a slide by Matthias Zwicker
CS 89/189: Computational Photography, Fall 2015
46 After a slide by Frédo Durand
3 primaries are (to a first order) enough to reproduce all colors!
CS 89/189: Computational Photography, Fall 2015 47 After a slide by Frédo Durand
Metamers under a given light source May not be metamers under a different lamp
CS 89/189: Computational Photography, Fall 2015 48 After a slide by Frédo Durand
Two grey patches in Billmeyer & Saltzman’s book look the same under daylight but different under neon or halogen
CS 89/189: Computational Photography, Fall 2015 49
Daylight Scan (neon) Hallogen
After a slide by Frédo Durand
Clothes appear to match in store (e.g. under fluorescent) Don’t match outdoors
CS 89/189: Computational Photography, Fall 2015 50 After a slide by Frédo Durand
CS 89/189: Computational Photography, Fall 2015
51
CS 89/189: Computational Photography, Fall 2015
infrared ultraviolet visible
500 1000 1500 2000 2500 3000 5.0 ¥1012 1.0 ¥1013 1.5 ¥1013 2.0 ¥1013 2.5 ¥1013 3.0 ¥1013 Wavelength HnmL Spectral Radiance
52
T = 3000 K T = 4000 K T = 4000 K T = 5000 K
CS 89/189: Computational Photography, Fall 2015
53
Emission spectrum of Hydrogen Emission spectrum of Iron
CS 89/189: Computational Photography, Fall 2015
54
Light emitted at 589nm and 589.6nm
Spectrum is an infinity of numbers Projected to 3D cone-response space
Metamerism: infinite-D points projected to the same 3D point (different spectrum, same perceived color)
CS 89/189: Computational Photography, Fall 2015 55 After a slide by Frédo Durand
Color perception in the animal kingdom
Humans project s(λ) into a 3D subspace
Most mammals have 2 types of cones (2D subspace) Many birds have UV receptors, some can see magnetic fields Some animals have even more:
CS 89/189: Computational Photography, Fall 2015 56
Light & Color
CS 89/189: Computational Photography, Fall 2015 58
We want to measure & reproduce color as seen by humans No need for full spectrum! Only need to match up to metamerism
CS 89/189: Computational Photography, Fall 2015 59 After a slide by Frédo Durand
We will focus on additive color
CS 89/189: Computational Photography, Fall 2015 60
CS 89/189: Computational Photography, Fall 2015 61
We’ll use 3 primaries (e.g. red, green, and blue) to match all colors
reproduce a given target color?
After a slide by Frédo Durand
Take a given stimulus and the corresponding responses S, M, L (here 0.5, 0, 0)
CS 89/189: Computational Photography, Fall 2015 62
M S L
After a slide by Frédo Durand
Use it to scale the cone spectra (here 0.5 * S) You don’t get the same cone response! (here 0.5, 0.1, 0.1)
CS 89/189: Computational Photography, Fall 2015 63
M S L
After a slide by Frédo Durand
The three cone responses are not orthogonal i.e. they overlap and “pollute” each other
CS 89/189: Computational Photography, Fall 2015 64
M S L
After a slide by Frédo Durand
Non-orthogonal bases are harder to handle Can’t use dot product on same vector to infer coordinates
components pollute each other
CS 89/189: Computational Photography, Fall 2015 65
j i x ¡≠ ¡(x·√i) ¡i ¡+ ¡(x·√j) ¡j x (x·√i) ¡i ¡+ ¡(x·√j) ¡j
After a slide by Frédo Durand
Non-orthogonal bases are harder to handle Can’t use dot product on same vector to infer coordinates Need a so-called dual basis
analysis/synthesis
CS 89/189: Computational Photography, Fall 2015 66
^ ^ (x·√i) ¡i ¡+ ¡(x·√j) ¡j ^ ^ j i x Note that i has negative coordinates ^
After a slide by Frédo Durand
Spectrum for the stimulus / synthesis
Response curve for receptor / analysis
Usually not the same Because cone responses are not orthogonal
CS 89/189: Computational Photography, Fall 2015 67 After a slide by Frédo Durand
Have a spectrum s; want to match on RGB monitor
color space is a good reproduction
So, we want to find a spectrum that the monitor can produce that matches s
CS 89/189: Computational Photography, Fall 2015 68 After a slide by Steve Marschner
CS 89/189: Computational Photography, Fall 2015
69
Curves determined by (fluorescent or LED) backlight and filters
After a slide by Steve Marschner
CS 89/189: Computational Photography, Fall 2015
70 After a slide by Steve Marschner
We want to compute the combination of R, G, B that will project to the same visual response as s
CS 89/189: Computational Photography, Fall 2015 71 After a slide by Steve Marschner
The projection onto the three response functions can be written in matrix form:
E = MSML s
CS 89/189: Computational Photography, Fall 2015 72
S M L = rS rM rL
|
s
|
After a slide by Steve Marschner
The spectrum that is produced by the monitor for the color signals R, G, and B is: Again, the discrete form can be written as a matrix:
|
sa
|
=
| | |
sR sG sB
| | |
R G B sa = MRGB C
CS 89/189: Computational Photography, Fall 2015 73
sa(λ) = R sR(λ) + G sG(λ) + B sB(λ)
After a slide by Steve Marschner
What color do we see when we look at the display?
CS 89/189: Computational Photography, Fall 2015 74
B C
After a slide by Steve Marschner
What color do we see when we look at the display?
CS 89/189: Computational Photography, Fall 2015 75
L MRGB C
After a slide by Steve Marschner
What color do we see when we look at the display?
CS 89/189: Computational Photography, Fall 2015 76
S M L = rS · sR rS · sG rS · sB rM · sR rM · sG rM · sB rL · sR rL · sG rL · sB R G B E = MSML MRGB C
After a slide by Steve Marschner
Goal of reproduction: visual response to s and sa is the same: Substitute in expression for sa ¡,
CS 89/189: Computational Photography, Fall 2015 77
MSML s = MSML sa MSML s = MSML MRGB C C = (MSML MRGB)−1MSML s
color matching matrix for RGB
After a slide by Steve Marschner
We now know how to match any color from the real world on a display We don’t need to know the whole spectrum, only the projections onto S, M, and L response functions There is then a simple linear procedure to work out the combination of any 3 primaries to match the color But there is a catch. More on that later.
CS 89/189: Computational Photography, Fall 2015 78
Physical color
spectrum
Perceptual color
same responses
reproduction with 3 primaries
Fundamental difficulty
(full function)
non-orthogonal
analysis and synthesis
CS 89/189: Computational Photography, Fall 2015 79
Light & Color
CS 89/189: Computational Photography, Fall 2015 80
How can we quantitatively represent, reproduce color? Brute force: store, reproduce full spectral energy distribution
CS 89/189: Computational Photography, Fall 2015 81 After a slide by Matthias Zwicker
Representation should be complete, but as compact as possible
should have two different representations
have the same representation
CS 89/189: Computational Photography, Fall 2015 82 After a slide by Matthias Zwicker
We need a principled color space Three types of cones, so expect three parameters to be sufficient Many possible definitions
Good news: color vision is linear and 3-dimensional, so any new color space based on color matching can be obtained using 3x3 matrix
CS 89/189: Computational Photography, Fall 2015 83 After a slide by Frédo Durand
Most standard color space: CIE XYZ SML and the various flavors of RGB are just linear transformations of the XYZ basis
CS 89/189: Computational Photography, Fall 2015 84 After a slide by Frédo Durand
Less directly measurable
Most directly available measurement:
given an input color, how to reproduce it with 3 primary colors?
(International Lighting Commission)
CS 89/189: Computational Photography, Fall 2015 85
CS 89/189: Computational Photography, Fall 2015 86
Separating plane
Given an input color, how to reproduce it with 3 primary colors? (Idea by Maxwell)
After a slide by Frédo Durand
Primaries (synthesis) at 435.8, 546.1 and 700nm
primaries (Linear algebra to the rescue!)
Resulting 3 weights for each primary are called tristimulus values
CS 89/189: Computational Photography, Fall 2015 87 After a slide by Frédo Durand
http://graphics.stanford.edu/courses/cs178-10/applets/colormatching.html
CS 89/189: Computational Photography, Fall 2015 88
Meaning of these curves: a monochromatic wavelength λ can be reproduced with: b(λ) amount of the 435.8nm primary, +g(λ) amount of the 546.1 primary, +r(λ) amount of the 700 nm primary This fully specifies the color perceived by a human
CS 89/189: Computational Photography, Fall 2015 89
What is this!?
Negative light doesn’t exist, so what do these mean? Some spectral colors could not be matched by primaries in the experiment The “Trick”:
considered negative
But negative light is…inconvenient
CS 89/189: Computational Photography, Fall 2015 90
CIE was not satisfied with range of RGB values for visible colors
Defined CIE XYZ color space via simple mathematical transformation
CIE_1931_color_space#Definition_of_the_CIE_XYZ_color_space
Most common color space still today
CS 89/189: Computational Photography, Fall 2015 91
Infinitely many ways to obtain non-negative matching functions! Let’s call ours XYZ
Linear transformation of CIE RGB
CS 89/189: Computational Photography, Fall 2015 92
CIE RGB CIE XYZ
sRGB to XYZ Adobe RGB to XYZ NTSC RGB to XYZ XYZ to sRGB XYZ to Adobe RGB XYZ to NTSC RGB
CS 89/189: Computational Photography, Fall 2015 93
¡0.412424 ¡ ¡ ¡0.212656 ¡ ¡ ¡ ¡ ¡0.0193324 ¡ ¡ ¡ ¡0.357579 ¡ ¡ ¡0.715158 ¡ ¡ ¡ ¡ ¡0.119193 ¡ ¡ ¡ ¡ ¡0.180464 ¡ ¡ ¡0.0721856 ¡ ¡ ¡0.950444 3.24071 ¡ ¡ ¡ ¡-‐‒0.969258 ¡ ¡ ¡ ¡0.0556352 ¡ ¡ ¡
0.576700 ¡ ¡ ¡ ¡0.297361 ¡ ¡ ¡ ¡0.0270328 ¡ ¡ ¡ ¡0.185556 ¡ ¡ ¡ ¡0.627355 ¡ ¡ ¡ ¡0.0706879 ¡ ¡ ¡ ¡0.188212 ¡ ¡ ¡ ¡0.0752847 ¡ ¡ ¡0.991248 2.04148 ¡ ¡ ¡ ¡-‐‒0.969258 ¡ ¡ ¡ ¡0.0134455 ¡ ¡ ¡
0.606734 ¡ ¡ ¡ ¡0.298839 ¡ ¡ ¡ ¡0.000000 ¡ ¡ ¡ ¡ ¡0.173564 ¡ ¡ ¡ ¡0.586811 ¡ ¡ ¡ ¡0.0661196 ¡ ¡ ¡ ¡0.200112 ¡ ¡ ¡ ¡0.114350 ¡ ¡ ¡ ¡1.11491 ¡ 1.91049 ¡ ¡ ¡ ¡-‐‒0.984310 ¡ ¡ ¡ ¡0.0583744 ¡ ¡ ¡
http://www.brucelindbloom.com/index.html?Eqn_RGB_XYZ_Matrix.html
CIE performed color matching experiments
Gives us XYZ color space Linear transform to/from LMS, RGB
CS 89/189: Computational Photography, Fall 2015 94
The most widely recognized color space Y corresponds to brightness (1924 CIE standard photometric observer) No negative values in matching curves But no physically-realizable primary (negative values in primary rather than in matching curve)
CS 89/189: Computational Photography, Fall 2015 95
3D spaces can be hard to visualize Chrominance is our notion of color, as opposed to brightness/luminance Recall that our eyes correct for multiplicative scale factors
CS 89/189: Computational Photography, Fall 2015 97
Chromaticity (x,y) can be derived by normalizing the XYZ color components:
Combining xy with Y allows us to represent any color Plotting on xy plane allows us to see all colors of a single brightness
CS 89/189: Computational Photography, Fall 2015 98
x = X X + Y + Z y = Y X + Y + Z
Spectral colors along curved boundary Linear combination of two colors: line connecting two points Linear combination of 3 colors span a triangle (Color Gamut)
CS 89/189: Computational Photography, Fall 2015 99
Color primaries at: 435.8, 546.1, 700.0 nm
CS 89/189: Computational Photography, Fall 2015 100
CS 89/189: Computational Photography, Fall 2015 101
values ce ut
White Point Dominant wavelength Inverse color
CS 89/189: Computational Photography, Fall 2015 102
A A’ B
All these color spaces so far are perceptually non- uniform:
visually similar
Measuring “perceptual distance” in color spaces important for many industries Experiments by MacAdams
CS 89/189: Computational Photography, Fall 2015 103
CS 89/189: Computational Photography, Fall 2015 104
Test patches
Two attempts to make a perceptually-uniform color space MacAdams ellipses become nearly (but not perfectly) circular
CS 89/189: Computational Photography, Fall 2015 105
Color perception is much more complicated than response of SML cones… Visual pathway
CS 89/189: Computational Photography, Fall 2015 107
Also known as chromatic adaptation Color of object is perceived as the same even under varying illumination For example:
perceived as white, even though the reflected light is green! The human brain infers the white color from the context, which is “green-ish“ too because of the green illumination.
CS 89/189: Computational Photography, Fall 2015 108
CS 89/189: Computational Photography, Fall 2015
109
blue and black?
white and gold?
Color constancy failure
http://xkcd.com/1492/
Hering’s opponent process theory (1874)
After sensing by cones, colors are encoded as red versus green, blue versus yellow, and black versus white Physiological evidence found in the 1950s
CS 89/189: Computational Photography, Fall 2015 112
+
Receptors Blue/Yellow Receptors Black/White Receptors
+
Input is LMS Output has a different parameterization:
CS 89/189: Computational Photography, Fall 2015 113 Bk G Y Wh B R S L M
Trichromatic Stage Opponent-Process Stage
CS 89/189: Computational Photography, Fall 2015 114
Sums for brightness Differences for color
At the end, it’s just a 3x3 matrix compared to LMS
Image Afterimage
Luminance, red-green, blue-yellow CIELab YUV YcrCb
CS 89/189: Computational Photography, Fall 2015 120
YUV YCrCb
Frédo Durand Steve Marschner Matthias Zwicker
CS 89/189: Computational Photography, Fall 2015 121