C OMPUTATIONAL A SPECTS OF C OMPUTATIONAL D IGITAL P HOTOGRAPHY P - - PowerPoint PPT Presentation

▶
c omputational a spects of
SMART_READER_LITE
LIVE PREVIEW

C OMPUTATIONAL A SPECTS OF C OMPUTATIONAL D IGITAL P HOTOGRAPHY P - - PowerPoint PPT Presentation

Aug 21, 2022 342 likes •881 views

CS 89.15/189.5, Fall 2015 C OMPUTATIONAL A SPECTS OF C OMPUTATIONAL D IGITAL P HOTOGRAPHY P HOTOGRAPHY Image Processing Basics Wojciech Jarosz wojciech.k.jarosz@dartmouth.edu Domain, range Domain vs. range 2D plane: domain of images color

slide-1
SLIDE 1

Image Processing Basics

COMPUTATIONAL ASPECTS OF DIGITAL PHOTOGRAPHY

Wojciech Jarosz wojciech.k.jarosz@dartmouth.edu

COMPUTATIONAL PHOTOGRAPHY

CS 89.15/189.5, Fall 2015

slide-2
SLIDE 2

Domain, range

slide-3
SLIDE 3

Domain vs. range

2D plane: domain of images color value: range (R3 for us)

  • red, green and blue components stored in


im(x, y, 0), im(x, y, 1), im(x, y, 2), respectively

CS 89/189: Computational Photography, Fall 2015 3 After a slide by Frédo Durand

slide-4
SLIDE 4

CS 89/189: Computational Photography, Fall 2015

Basic types of operations

4

Point operations: 
 range only Assignment 2

After a slide by Frédo Durand

  • utput(x,y) = f(image(x,y))

image(x,y)

  • utput(x,y)
slide-5
SLIDE 5

CS 89/189: Computational Photography, Fall 2015

Basic types of operations

5

Point operations: 
 range only Assignment 2

After a slide by Frédo Durand

Domain 


  • perations

Assignment 6

image(x,y)

  • utput(x,y)
  • utput(x,y) = image(f(x,y))
  • utput(x,y) = f(image(x,y))
slide-6
SLIDE 6

CS 89/189: Computational Photography, Fall 2015

Basic types of operations

6

Point operations: 
 range only Assignment 2

After a slide by Frédo Durand

Domain 


  • perations

Assignment 6 Neighborhood operations: 
 domain and range Assignments 3, 4, 5

image(x,y)

  • utput(x,y)
  • utput(x,y) = image(f(x,y))
  • utput(x,y) = f(image(x,y))
slide-7
SLIDE 7

Light & perception

slide-8
SLIDE 8

Light from sources is reflected by objects and reaches the eye The amount of light from the source gets multiplied by the object reflectance

  • on a per-wavelength basis

Light matter, eyes

CS 89/189: Computational Photography, Fall 2015 8 After a slide by Frédo Durand

slide-9
SLIDE 9

Human perception

Our eyes have an uncanny ability to discount the illumination

  • Only objects really matter for survival
  • Light is only useful to understand if you’re a photographer
  • r to choose your sun lotion

CS 89/189: Computational Photography, Fall 2015 9 After a slide by Frédo Durand

slide-10
SLIDE 10

Illusion by Adelson

A & B have exactly the same tone

CS 89/189: Computational Photography, Fall 2015 10 After a slide by Frédo Durand

slide-11
SLIDE 11

Illusion by Adelson

A & B have exactly the same tone

CS 89/189: Computational Photography, Fall 2015 11 After a slide by Frédo Durand

slide-12
SLIDE 12

Mechanism to discount light

Light adaptation

  • We re-center our neural response around the current average

brightness

  • neural + chemical + pupil

Chromatic adaptation

  • eliminate color cast due to light sources


e.g. Daylight is white but tungsten is yellowish

  • Related to white balance - more soon
  • and Spanish Castle illusion

CS 89/189: Computational Photography, Fall 2015 12 After a slide by Frédo Durand

slide-13
SLIDE 13

Contrast is about ratios

Contrast between 1 & 2 is the same as between 100 & 200 Useful to discount the multiplicative effect of light

CS 89/189: Computational Photography, Fall 2015 13

0.1 to 0.2 0.4 to 0.8

After a slide by Frédo Durand

slide-14
SLIDE 14

Exposure

On cameras, exposure (shutter speed, aperture, ISO) has a multiplicative effect on the values recorded by the sensor. Changes the “brightness”, not contrast

CS 89/189: Computational Photography, Fall 2015 14 http://photographystepbystep.com/exposure-2/auto-bracketing/ After a slide by Frédo Durand

slide-15
SLIDE 15

White balance

slide-16
SLIDE 16

White balance & Chromatic adaptation

Different illuminants have different color temperature Our eyes adapt: chromatic adaptation

  • We actually adapt better in brighter scenes
  • This is why candlelit scenes still look yellow

CS 89/189: Computational Photography, Fall 2015 16

www.shortcourses.com/guide/guide2-27.html

After a slide by Frédo Durand

slide-17
SLIDE 17

White balance problem

When watching a picture on screen or print, we adapt to the illuminant of the room, not that of the scene in the picture The eye cares more about objects’ intrinsic color, not the color of the light leaving the objects We need to discount the color of the light source

CS 89/189: Computational Photography, Fall 2015 17

Same object, 
 different illuminants

After a slide by Frédo Durand

slide-18
SLIDE 18

White balance & Film

Different types of film for fluorescent, tungsten, daylight Need to change film! Electronic & Digital imaging are more flexible

CS 89/189: Computational Photography, Fall 2015 18 After a slide by Frédo Durand

slide-19
SLIDE 19

Von Kries adaptation

Multiply each channel by a gain factor

  • R’=R*kr
  • G’=G*kg
  • B’=B*kb

CS 89/189: Computational Photography, Fall 2015 19 http://www.cambridgeincolour.com/tutorials/white-balance.htm After a slide by Frédo Durand

slide-20
SLIDE 20

Von Kries adaptation

Multiply each channel by a gain factor Note that the light source could have a more complex effect

  • Arbitrary 3⨉3 matrix
  • More complex spectrum transformation

CS 89/189: Computational Photography, Fall 2015 20 http://www.cambridgeincolour.com/tutorials/white-balance.htm After a slide by Frédo Durand

slide-21
SLIDE 21

White balance challenge

How do we find the scaling factors for r, g, and b?

CS 89/189: Computational Photography, Fall 2015 21 After a slide by Frédo Durand

slide-22
SLIDE 22

Best way to do white balance

Grey card: Take a picture of a neutral object 
 (white or gray) Deduce the weight of each channel If the object is recoded as rw, gw, bw 
 use weights k/rw, k/gw, k/bw
 where k controls the exposure

CS 89/189: Computational Photography, Fall 2015 22 After a slide by Frédo Durand

slide-23
SLIDE 23

Lightroom demo

Most photo editing software lets you click

  • n a neutral object to achieve white

balance

  • In “Levels” in Photoshop
  • In “Basic” in Lightroom
  • You also often have presets such as

daylight, tungsten

CS 89/189: Computational Photography, Fall 2015 23 After a slide by Frédo Durand

slide-24
SLIDE 24

Party name tags

Provide excellent white references!

CS 89/189: Computational Photography, Fall 2015 24

write(im/im(300, 214))

After a slide by Frédo Durand

slide-25
SLIDE 25

Without grey cards

We need to “guess” which pixels correspond to white

  • bjects

CS 89/189: Computational Photography, Fall 2015 25 After a slide by Frédo Durand

slide-26
SLIDE 26

Grey world assumption

Assume average color in the image is grey Use weights proportional to Usually assumes 18% grey to set exposure

CS 89/189: Computational Photography, Fall 2015 26 After a slide by Frédo Durand

slide-27
SLIDE 27

Brightest pixel assumption

Highlights usually have the color of the light source

  • At least for dielectric materials

White balance by using the brightest pixels

  • Plus potentially a bunch of heuristics
  • In particular use a pixel that is not saturated/clipped

CS 89/189: Computational Photography, Fall 2015 27 After a slide by Frédo Durand

slide-28
SLIDE 28

Refs

Recent work on color constancy

  • http://gvi.seas.harvard.edu/paper/perceptionbased-color-

space-illuminationinvariant-image-processing

  • http://gvi.seas.harvard.edu/paper/color-subspaces-

photometric-invariants

  • http://people.csail.mit.edu/billf/papers/BayesJOSA.pdf

Still an open problem!

CS 89/189: Computational Photography, Fall 2015 28 After a slide by Frédo Durand

slide-29
SLIDE 29

CS 89/189: Computational Photography, Fall 2015

Questions?

29

from xkcd

After a slide by Frédo Durand

slide-30
SLIDE 30

Take home messages

Discounting the illumination is useful Ratios matter Optical illusions are not optical but fun

CS 89/189: Computational Photography, Fall 2015 30 After a slide by Frédo Durand

slide-31
SLIDE 31

Gamma

slide-32
SLIDE 32

Images are usually gamma encoded Instead of storing the light intensity x, they store x𝛿 to get more precision in dark areas for 8-bit encoding

Linearity and gamma

CS 89/189: Computational Photography, Fall 2015 32 After a slide by Frédo Durand

gamma compression curve

real value stored value

slide-33
SLIDE 33

Linearity and gamma

Images are usually gamma encoded Instead of storing the light intensity x, they store x𝛿 to get more precision in dark areas for 8-bit encoding

CS 89/189: Computational Photography, Fall 2015 33

6 bit encoding for emphasis:

After a slide by Frédo Durand

slide-34
SLIDE 34

Gamma demo

http://web.mit.edu/lilis/www/gammavis.html

CS 89/189: Computational Photography, Fall 2015 34 After a slide by Frédo Durand

slide-35
SLIDE 35

Linearity and gamma

Images are usually gamma encoded Instead of storing the light intensity x, they store x𝛿 Half of image processing algorithms work better in linear space

  • If linearity is important
  • To deal with ratios and multiplicative factors better

Half work better in gamma space

  • closer to logarithmic scale

CS 89/189: Computational Photography, Fall 2015 35 After a slide by Frédo Durand

slide-36
SLIDE 36

How to capture linear images

http://www.mit.edu/~kimo/blog/linear.html

CS 89/189: Computational Photography, Fall 2015 36 After a slide by Frédo Durand

slide-37
SLIDE 37

Take home message

Images are usually gamma-encoded gamma ~2.2 provides better quantization sometimes good for algorithms sometimes bad

  • convert to linear values!

CS 89/189: Computational Photography, Fall 2015 37 After a slide by Frédo Durand

slide-38
SLIDE 38

Histograms

slide-39
SLIDE 39

Histogram

Histogram:

  • For each value (e.g. 0-255), how many pixels have

this value?

Cumulative histogram:

  • for each value x, how many pixels have a value

smaller than x?

Normalized: divide value of each bin by total number of pixels

  • histogram = discrete PDF
  • cumulative histogram = discrete CDF

CS 89/189: Computational Photography, Fall 2015 39

(wikipedia)

After a slide by Frédo Durand

pixel value #pixels

slide-40
SLIDE 40

Very useful on camera

Allows you to tell if you use the dynamic range entirely

CS 89/189: Computational Photography, Fall 2015 40 After a slide by Frédo Durand

slide-41
SLIDE 41

Bad: bright values under-used
 (underexposure) Bad: bright values saturate
 (overexposure)

http://www.luminous-landscape.com/tutorials/understanding-series/understanding-histograms.shtml

slide-42
SLIDE 42

Histogram equalization

slide-43
SLIDE 43

Histogram equalization

Given an image with a certain histogram, monotonic remapping to get a flat histogram

CS 89/189: Computational Photography, Fall 2015 43 pixel value #pixels pixel value #pixels

  • ld pixel value

new pixel value http://en.wikipedia.org/wiki/Histogram_equalization

slide-44
SLIDE 44

Histogram equalization

Ideal flat histogram: y% of pixels have a value less than y%

  • assuming everything is normalized to [0,1]

Flip it: a pixel with value larger than y% of all pixels should have value y% For an old value x%, we know the number of pixels that have value < x%: cumulative histogram (also called CDF) Therefore, we want x to be mapped to its cumulative histogram value.

CS 89/189: Computational Photography, Fall 2015 44

slide-45
SLIDE 45

Histogram matching

Histogram matching

  • Given a desired histogram
  • Map each value of an image channel to a new value, such

that the new histogram matches the desired histogram

Histogram equalization

  • The desired histogram is simply constant
  • What shape is the cumulative histogram?

CS 89/189: Computational Photography, Fall 2015 45

slide-46
SLIDE 46

target

Histogram matching

CS 89/189: Computational Photography, Fall 2015 46

% of pixels intensity bin

255 100 G1

Histogram matching is done by adjusting the cumulative distribution function (cdf)

  • Cumulative histogram of input

input

slide-47
SLIDE 47

Histogram matching is done by adjusting the cumulative distribution function (cdf)

  • Cumulative histogram of input
  • Followed by inverse cumulative

histogram of target

Histogram matching

CS 89/189: Computational Photography, Fall 2015 47

% of pixels intensity bin

255 100 G1 target G2 target input

slide-48
SLIDE 48

Histogram matching is done by adjusting the cumulative distribution function (cdf)

  • Cumulative histogram of input
  • Followed by inverse cumulative

histogram of target (linear)

Histogram equalization

CS 89/189: Computational Photography, Fall 2015 48

% of pixels intensity bin

255 100 G1 target G2 input

slide-49
SLIDE 49

Cumulative
 distribution
 function (CDF)

CS 89/189: Computational Photography, Fall 2015

Histogram equalization

49

Input Equalized

slide-50
SLIDE 50

Debugging

slide-51
SLIDE 51

Debugging

Doubt everything Debug pieces in isolation

  • Binary search/divide and conquer

Display/print everything

  • In particular intermediate results

Create simple inputs

  • where you can easily manually compute

the result

  • e.g. constant image, edge image, etc.
  • use small images (e.g. 3x3)
  • including (especially) inputs

to intermediate stages

  • use input thats isolate

different failure modes

Change one thing at a time

  • e.g. to verify that a given

command has the effect you want, modify it to break it

CS 89/189: Computational Photography, Fall 2015 51

slide-52
SLIDE 52

Slide credits

Frédo Durand

CS 89/189: Computational Photography, Fall 2015 52