Image formation Subhransu Maji CMPSCI 670: Computer Vision - - PowerPoint PPT Presentation

▶
image formation
SMART_READER_LITE
LIVE PREVIEW

Image formation Subhransu Maji CMPSCI 670: Computer Vision - - PowerPoint PPT Presentation

Sep 24, 2023 266 likes •1.13k views

Image formation Subhransu Maji CMPSCI 670: Computer Vision September 13, 2016 Administrivia and survey results Topics: deep learning, CNNs, machine learning, AI Applications: self driving cars, face detection/recognition, etc

slide-1
SLIDE 1

Subhransu Maji

CMPSCI 670: Computer Vision

Image formation

September 13, 2016

slide-2
SLIDE 2

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Topics:

  • deep learning, CNNs, machine learning, AI
  • Applications: self driving cars, face detection/recognition, etc
  • robotics, calibration, structure from motion
  • graphics, text/natural language processing, speech,

Goals:

  • Learn fundamentals of CV/ML/image processing
  • Do a supercool project
  • Get an awesome industry job (e.g., space exploration @ NASA)

Programming: 7.5 - 8.5, Math: 6.5 - 7.5 Spire: waitlisted students? there are a few more open slots Resources for vector algebra and probability added to the webpage

Administrivia and survey results

2

slide-3
SLIDE 3

Subhransu Maji (UMass, Fall 16) CMPSCI 670

The pinhole projection model

  • qualitative properties

Cameras with lenses

  • Depth of focus
  • Field of view
  • Lens aberrations

Digital cameras

  • Sensors
  • Colors
  • Artifacts

Computational photography

  • Novel sensors and cameras

Overview of the next two lectures

3

slide-4
SLIDE 4

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Cameras

4

Albrecht Dürer early 1500s Brunelleschi, early 1400s

slide-5
SLIDE 5

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Lets design a camera

5

Object Film Idea 1: Lets put a film in front of an object Do we get a reasonable image? A B

slide-6
SLIDE 6

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Pinhole camera

6

Object Film Add a barrier to block of most rays Barrier

slide-7
SLIDE 7

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Pinhole camera

7

Object Film Barrier

  • Captures pencil of rays - all rays through a single point: aperture,

center of projection, focal point, camera center

  • The image is formed on the image plane
slide-8
SLIDE 8

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Basic principle known to Mozi (470-390 BCE), Aristotle (384-322 BCE) Drawing aids for artists: described by Leonardo Da Vinci (1452-1519 AD)

Camera obscura

8

Gemma Frisius, 1558 “Camera obscure” Latin for “darkened room”

slide-9
SLIDE 9

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Pinhole cameras are everywhere

9

Tree shadow during a solar eclipse photo credit: Nils van der Burg http://www.physicstogo.org/index.cfm

Slide by Steve Seitz

slide-10
SLIDE 10

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Accidental pinhole cameras

10

  • A. Torralba and W. Freeman, Accidental Pinhole and Pinspeck Cameras, CVPR 2012
slide-11
SLIDE 11

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Home-made pinhole camera

11

http://www.pauldebevec.com/Pinhole

slide-12
SLIDE 12

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Dimensionality reduction: 3D to 2D

12

Point of observation

3D world 2D image

  • What is preserved?
  • Straight lines, incidence
  • What is not preserved?
  • Angles, lengths

Slide by A. Efros

slide-13
SLIDE 13

Subhransu Maji (UMass, Fall 16) CMPSCI 670

x y z

f To compute the projection P’ of a scene point P, form a visual ray connection P to the camera center O and find where it intersects the image plane

  • All scene points that lie on this visual ray have the same projection
  • n the image
  • Are there points for which this projection is not defined?

Modeling projection

13 Slide by Steve Seitz

slide-14
SLIDE 14

Subhransu Maji (UMass, Fall 16) CMPSCI 670

The coordinate system

  • The optical center (O) is at the origin
  • The image plane is parallel to the xy-plane (perpendicular to the z

axis)

Projection equations

  • Derive using similar triangles

Modeling projection

14

x y z

f

) , ( ) , , ( z y f z x f z y x →

Slide by Steve Seitz

slide-15
SLIDE 15

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Projection of a line

15

image plane camera center line in the scene

  • What if we add another line parallel to the first one?

vanishing point

Slide by Steve Seitz

slide-16
SLIDE 16

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Each direction in space has its own vanishing point

  • All lines going in the that direction converge at that point

Vanishing points

16

  • Exception: directions that are parallel to the image plane

Slide by Steve Seitz

slide-17
SLIDE 17

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Each direction in space has its own vanishing point

  • All lines going in the that direction converge at that point

Vanishing points

17

  • Exception: directions that are parallel to the image plane
  • What about the vanishing point of a plane?
slide-18
SLIDE 18

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Vanishing line of the ground plane

  • All points at the same height of the camera project to the horizon
  • Points above the camera project above the horizon
  • Provides a way of comparing heights of objects

The horizon

18

camera center ground plane

slide-19
SLIDE 19

Subhransu Maji (UMass, Fall 16) CMPSCI 670

The horizon

19

Is the person above or below the viewer?

slide-20
SLIDE 20

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Perspective cues

20

slide-21
SLIDE 21

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Perspective cues

21

slide-22
SLIDE 22

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Perspective cues

22

slide-23
SLIDE 23

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Comparing heights

23

vanishing point

slide-24
SLIDE 24

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Measuring heights

24

1 2 3 4 5 What is the height of the camera? 3.7 2.5 5.4 camera height

slide-25
SLIDE 25

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Perspective in art

25

Masaccio, Trinity, Santa Maria Novella, Florence, 1425-28 One of the first consistent uses of perspective in Western art

slide-26
SLIDE 26

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Perspective in art

26

(At least partial) Perspective projections in art well before the Renaissance

From ottobwiersma.nl

Also some Greek examples, So apparently pre-renaissance…

slide-27
SLIDE 27

Subhransu Maji (UMass, Fall 16) CMPSCI 670

What does a sphere project to?

Perspective distortion

27

  • M. H. Pirenne

Slide by Steve Seitz

slide-28
SLIDE 28

Subhransu Maji (UMass, Fall 16) CMPSCI 670

What does a sphere project to?

Perspective distortion

28

slide-29
SLIDE 29

Subhransu Maji (UMass, Fall 16) CMPSCI 670

The exterior looks bigger The distortion is not due to lens flaws Problem pointed out by Da Vinci

Perspective distortion

29 Slide by F. Durand

slide-30
SLIDE 30

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Special case of perspective projection

  • Distance of the object from the image plane is infinite
  • Also called the “parallel projection”

Orthographic projection

30

Image World

Slide by Steve Seitz

slide-31
SLIDE 31

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Special case of perspective projection

  • Distance of the object from the image plane is infinite
  • Also called the “parallel projection”

Orthographic projection

31

slide-32
SLIDE 32

Subhransu Maji (UMass, Fall 16) CMPSCI 670

The pinhole projection model

  • Qualitative properties

Cameras with lenses

  • Depth of focus
  • Field of view
  • Lens aberrations

Digital cameras

  • Sensors
  • Colors
  • Artifacts

Novel cameras

  • Computational photography

Overview of the next two lectures

32

slide-33
SLIDE 33

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Pinhole camera

33

Object Film Barrier

  • Captures pencil of rays - all rays through a single point:

aperture, center of projection, focal point, camera center

  • The image is formed on the image plane

aperture image

slide-34
SLIDE 34

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Why not make the aperture as small as possible?

  • Less light gets through
  • Diffraction effects

Shrinking the aperture

34 Slide by Steve Seitz

slide-35
SLIDE 35

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Shrinking the aperture

35 Slide by Steve Seitz

slide-36
SLIDE 36

Subhransu Maji (UMass, Fall 16) CMPSCI 670

A lens focuses light on to the film

  • Thin lens model:

➡ Rays passing through the center are not deviated (pinhole

projection model still holds)

Adding a lens

36

Object Film Lens

Slide by F. Durand

slide-37
SLIDE 37

Subhransu Maji (UMass, Fall 16) CMPSCI 670

A lens focuses light on to the film

  • Thin lens model:

➡ Rays passing through the center are not deviated (pinhole

projection model still holds)

➡ All parallel rays converge to one point on a plane located at

the focal length f

Adding a lens

37

Object Film Lens f

Slide by F. Durand

slide-38
SLIDE 38

Subhransu Maji (UMass, Fall 16) CMPSCI 670

A lens focuses light on to the film

  • There is a specific distance at which objects are “in focus”

➡other points project on to a “circle of confusion” in the image

Adding a lens

38

Object Film Lens

circle of confusion

Slide by F. Durand

slide-39
SLIDE 39

Subhransu Maji (UMass, Fall 16) CMPSCI 670

What is the relation between the focal length (f), the distance of the

  • bject from the optical center (D) and the distance at which the object

will be in focus (D’)?

Thin lens formula

39

f D D′

image plane lens

  • bject

Slide by F. Durand

slide-40
SLIDE 40

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Similar triangles everywhere!

Thin lens formula

40

image plane lens

  • bject

f D D′ y′ y

y′/y = D′/D

Slide by F. Durand

slide-41
SLIDE 41

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Similar triangles everywhere!

Thin lens formula

41

image plane lens

  • bject

y′/y = D′/D y′/y = (D′−f )/f

f D D′ y′ y

Slide by F. Durand

slide-42
SLIDE 42

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Thin lens formula

42

image plane lens

  • bject

f D D′ y′ y 1 D′ D 1 1 f + =

Any point satisfying the thin lens equation is in focus

Slide by F. Durand

slide-43
SLIDE 43

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Depth of field

43

http://www.cambridgeincolour.com/tutorials/depth-of-field.htm

DOF is the distance between the nearest and farthest objects in a scene that appear acceptably sharp in an image

Slide by A.Efros

slide-44
SLIDE 44

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Varying the aperture

44

Large aperture = small DOF Small aperture = large DOF

Slide by A.Efros

slide-45
SLIDE 45

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Changing the aperture size affects the depth of field

  • A smaller aperture increases the range in which the object is

approximately in focus

  • But small aperture reduces the amount of light — need to increase

the exposure for contrast

  • Pinhole camera has an infinite depth of field

Controlling depth of field

45

image credit Wikipedia

slide-46
SLIDE 46

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Field of view

46 Slide by A.Efros

slide-47
SLIDE 47

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Field of view

47 Slide by A.Efros

slide-48
SLIDE 48

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Field of view (FOV) depends on the focal length and the size of the camera retina

Field of view

48

Larger focal length = smaller FOV φ = tan−1 ✓ d 2f ◆

φ

f

Slide by A.Efros

slide-49
SLIDE 49

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Field of view, focal length

49

Large FOV, small f — Camera close to the car Small FOV, large f — Camera far from the car

tan(φ) × 2f = d ∼ (φ) × 2f = d

Slide by A.Efros, F.Durand

slide-50
SLIDE 50

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Same effect for faces

50

wide-angle (short focus) telephoto (long focus) standard

Slide by F.Durand

slide-51
SLIDE 51

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Approximating an orthographic camera

51 Source: Hartley & Zisserman

slide-52
SLIDE 52

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Continuously adjusting the camera focal length while the camera moves away from (or towards) the subject

The dolly zoom

52

http://en.wikipedia.org/wiki/Dolly_zoom

slide-53
SLIDE 53

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Continuously adjusting the camera focal length while the camera moves away from (or towards) the subject Also called as “Vertigo shot” or the “Hitchcock shot”

The dolly zoom

53

Example of dolly zoom from Goodfellas Example of dolly zoom from La Haine

slide-54
SLIDE 54

Subhransu Maji

CMPSCI 670: Computer Vision

Image formation …

September 13, 2016

slide-55
SLIDE 55

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Homework 01 posted

  • Due Sept 15, 1pm (That’s this Thursday before class)
  • Submissions as pdf via Moodle only

➡ Any combination of Latex, Word, print + scan, etc.

Mini-project 1 posted

  • Due Sept 29

Sign up on Piazza for announcements

  • I’ll use this as the primary place for announcements

Lecture slides and materials are posted on webpage TA office hours:

  • Wednesday 3-4PM, Location: CS 245

Waitlisted students?

  • Definitely talk to me after class (OH: Today, 2:30 - 3:30pm, CS 274)

Administrivia

55

slide-56
SLIDE 56

Subhransu Maji (UMass, Fall 16) CMPSCI 670

The pinhole projection model

  • Qualitative properties

Cameras with lenses

  • Depth of focus
  • Field of view
  • Lens aberrations

Digital cameras

  • Sensors
  • Colors
  • Artifacts

Computational photography

  • Novel sensors and cameras

Recap of the last lecture

56

x y z

f

pinhole camera f D D′ lens

slide-57
SLIDE 57

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Lens have different refractive indices (Snell’s law) for different wavelengths: causes color fringing

Lens flaws: Chromatic aberration

57

near lens center near lens outer

Slide by S.Seitz

slide-58
SLIDE 58

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Spherical lenses don’t focus light perfectly (thin lens model)

  • Rays farther from the optical axis are focussed closer

Lens flaws: Spherical aberration

58

  • bjects lack sharpness

Slide by S.Seitz

slide-59
SLIDE 59

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Reduction of image brightness in the periphery

Lens flaws: Vignetting

59

Not all rays reach the sensor

Slide by S.Seitz

slide-60
SLIDE 60

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Caused by asymmetry of lenses Deviations are most noticeable near the periphery

Lens flaws: Radial distortion

60

barrel distortion pincushion distortion mustache distortion

http://parkingandyou.com http://clanegesselphotography.blogspot.com/

slide-61
SLIDE 61

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Many uses: cameras, telescopes, microscopes, etc

Real photographic lens

61

Example of a prime lens - Carl Zeiss Tessar Nikkor 28-200 mm zoom lens, extended to 200 mm at left and collapsed to 28 mm focal length at right.

fixed focal length adjustable zoom

http://en.wikipedia.org/wiki/Zoom_lens

slide-62
SLIDE 62

Subhransu Maji (UMass, Fall 16) CMPSCI 670

The pinhole projection model

  • qualitative properties

Cameras with lenses

  • Depth of focus
  • Field of view
  • Lens aberrations

Digital cameras

  • Sensors
  • Colors
  • Artifacts

Novel cameras

  • Computational photography

Overview

62

slide-63
SLIDE 63

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Photographic film — strip of transparent plastic film base coated on

  • ne side with a gelatin emulsion containing light-sensitive materials

Creates a latent image when exposed to light for short duration Films are then chemically developed to form a photograph Early films/photographic plates could only capture intensity

Measuring light

63

slide-64
SLIDE 64

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Sergey Prokudin-Gorskii (1863-1944) Photographs of the Russian empire (1909-1916)

Early color photography

64

slide-65
SLIDE 65

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Only problem!

65

Homework 1: fix this by aligning the channels

slide-66
SLIDE 66

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Color photographic film — many layers of dyes and light sensitive materials to capture light of different frequencies simultaneously

  • Kodak pioneered color films for making paper prints

Simultaneous measurement solves the alignment problem

  • But needs complex film design and development process

Measuring light: color films

66

slide-67
SLIDE 67

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Color images are commonly represented using 3 channels [R, G, B]

  • The color of each pixel is given by the (r,g,b) value

Digital images

67

red green blue

slide-68
SLIDE 68

Subhransu Maji (UMass, Fall 16) CMPSCI 670

A digital camera replaces the film with a sensor array

  • Each cell in the array is a light-sensitive diode that converts

photons to electrons

  • Two common types of sensor arrays

➡ Charge Coupled Device (CCD) ➡ Complementary Metal Oxide Semiconductor (CMOS)

Digital camera

68

http://electronics.howstuffworks.com/digital-camera.htm

Slide by S.Seitz

slide-69
SLIDE 69

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Color sensing in the camera

69

Bayer grid

Estimate missing components from neighboring values (demosiacing)

Why more green?

Human luminance sensitivity function

Color filter array

Slide by S.Seitz

slide-70
SLIDE 70

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Estimate missing values

Demosaicing

70

Red Green Blue

slide-71
SLIDE 71

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Problem: guess the values of ? in each of the three channels Why is this even possible?

Demosaicing

71

slide-72
SLIDE 72

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Interpolation

72

gt gl ? gr gb gt gl ? gr gb gt gl ? gr gb

nearest neighbor copy one of your neighbors ? ←gl linear interpolation average values of your neighbors ? ←(gt+gl+gr+gb)/4 adaptive gradient average based on

  • nbhd. structure

if |gt-gb| > |gl-gr| ? ← (gl+gr)/2 else ? ← (gt+gb)/2 Similarly for the blue and red channels Homework 1: implement this

slide-73
SLIDE 73

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Problem with demosaicing: color moiré

73 Slide by F.Durand

slide-74
SLIDE 74

Subhransu Maji (UMass, Fall 16) CMPSCI 670

The cause of color moiré

74

detector Fine black and white detail in the image scene is misinterpreted as color information

Slide by F.Durand

slide-75
SLIDE 75

Subhransu Maji (UMass, Fall 16) CMPSCI 670

White or “panchromatic” cells allow lights across all wavelengths

  • Better light efficiency

How would you go about picking the best one?

Alternatives to Bayer filter

75

Three new Kodak RGBW filter patterns Fujifilm "X-Trans" filter Source: https://en.wikipedia.org/wiki/Bayer_filter

slide-76
SLIDE 76

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Computational cameras

76

S.K. Nayar http://www1.cs.columbia.edu/CAVE/projects/what_is/

slide-77
SLIDE 77

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Goal: Design a sampling pattern + interpolation algorithm that archives the best color reconstruction Sampling patterns

  • Given a nxn filter array we have 4(nxn) possible choices

➡ More choices if we allow different color filters

  • Some patterns are obviously bad for reconstruction

Interpolation algorithms

  • Can’t easily enumerate this space
  • Non trivial algorithms for interpolation

Computational color photography

77

slide-78
SLIDE 78

Subhransu Maji (UMass, Fall 16) CMPSCI 670 78

slide-79
SLIDE 79

Subhransu Maji (UMass, Fall 16) CMPSCI 670 79

linearly interpolate color value using intensity-based affinities

slide-80
SLIDE 80

Subhransu Maji (UMass, Fall 16) CMPSCI 670 80

increasing noise level

slide-81
SLIDE 81

Subhransu Maji (UMass, Fall 16) CMPSCI 670 81

http://ttic.uchicago.edu/~ayanc/learncfa/

To appear at NIPS’16

slide-82
SLIDE 82

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Sample over time, lighting, viewing direction, pose

Light Stage 6

82

Paul Debevec’s group at USC-ICT http://ict.usc.edu/prototypes/light-stages/

slide-83
SLIDE 83

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Light field camera: capture intensity along each direction of the light

  • Traditional cameras integrate light coming from all directions

A captured light field allows you re-render an image post-hoc

  • https://pictures.lytro.com/lytro/collections/41/pictures/1088670

Lytro camera

83

slide-84
SLIDE 84

Subhransu Maji (UMass, Fall 16) CMPSCI 670

History of optics, Wikipedia

  • A. Torralba and W. Freeman, Accidental Pinhole and Pinspeck

Cameras, CVPR 2012 DIY http://www.pauldebevec.com/Pinhole In MATLAB, compute the projection of a sphere using the perspective model and visualize the distortions Light stages over time http://gl.ict.usc.edu/LightStages Sergey Prokudin-Gorskii photographic collection at the Library of Congress http://www.loc.gov/exhibits/empire/index.html Richard Szeliski’s book, Sections 2.2.3 - 2.3.2

More readings and thoughts

84