Einfhrung in Visual Computing U it 26 C Unit 26: Computational - PowerPoint PPT Presentation

Einführung in Visual Computing U it 26 C Unit 26: Computational Photography t ti l Ph t h http:// www.caa.tuwien.ac.at/cvl/teaching/sommersemester/evc Content:  Introduction to Computational Photography Introduction to Computational Photography  Examples  Image Warping  Image Mosaic  Image Morphing (several slides inspired/borrowed from Jack Tumblin, Northwestern University, Marc Pollefeys, ETHZ and Fredo Durand, MIT) 1 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

Focus, Click, Print: ‘Film ‐ Like Photography’ , , g p y Light + 3D Scene: 2D Image: Rays Rays Illumination, , ‘Instantaneous’ Instantaneous shape, movement, Rays Rays Intensity Map surface BRDF,… gle(  ,  ) on(x,y) Ang Positio ‘Center of Projection’ j (P 3 or P 2 Origin) 2 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

Perfect Copy : Perfect Photograph? py g p Scene Light Intensities ‘Pixel values’ ‘Pixel values’ scene scene (scene intensity? display intensity? perceived intensity? ‘blackness/whiteness’ ?) Display Display Light Intensities Intensities display display display display display display display display 3 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

‘Film ‐ Like’ Photography g p y Ideals, Design Goals:  ‘Instantaneous’ light measurement  Instantaneous light measurement…  Of focal plane image behind a lens.  Reproduce those amounts of light. R d th t f li ht Implied: “What we see is  What we see is  focal ‐ plane intensities.” well, no…we see much more! (seeing is deeply cognitive) 4 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

Definitions  ‘Film ‐ like’ Photography: Displayed image  sensor image i l d i i  ‘Computational’ Photography: ‘C t ti l’ Ph t h Displayed image  sensor image  visually meaningful scene contents A more expressive & controllable displayed result, transformed merged decoded data from transformed, merged, decoded data from compute ‐ assisted sensors, lights, optics, displays 5 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

What is Photography? g p y  Safe answer:  Safe answer: A wholly new, expressive medium (ca. 1830s)  Manipulated display of what we think, feel, want, …  Capture a memory, a visual experience in tangible form Capture a memory, a visual experience in tangible form  ‘painting with light’; express the subject’s visual essence “Exactitude is not the truth ” Exactitude is not the truth. – Henri Matisse Henri Matisse  6 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

What is Photography? g p y  A ‘bucket’ word: a neat container for messy notions A ‘b k t’ d t t i f ti (e.g. aviation, music, comprehension)  A record of what we see, or would like to see, in tangible form.  Does ‘film’ photography always capture it? Um, no...  What do we see? 7 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

What is Photography? g p y PHYSICAL PHYSICAL PERCEIVED PERCEIVED Exposure Light & Control, , 3D Scene 3D Scene Scene Scene Optics tone map light sources, light sources, BRDFs, BRDFs, shapes, shapes, Image Display Display sion positions, positions, movements, movements movements, movements I(x,y, λ ,t) I( λ ) RGB( RGB( RGB(x,y,t RGB( x,y,t n ) t ) Vis … … Eyepoint Eyepoint Eyepoint Eyepoint position, position, movement, movement, projection, projection, … Photo: A Tangible Record … Editable storable as Editable, storable as Film or Pixels 8 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

Ultimate Photographic Goals g p PERCEIVED PHYSICAL PHYSICAL or UNDERSTOOD UNDERSTOOD Scene Light & 3D Scene 3D Scene light sources, l h Optics light sources, BRDFs, BRDFs, shapes, shapes, g puting sor(s) shapes, positions, sion Visual Visual positions, movements, Comp Sens movements, movements Stimulus Stimulus Stimulus Stimulus Vis … … Eyepoint Eyepoint Eyepoint position, iti position, movement, movement, p projection, j , Photo: A Tangible Record projection, … Scene estimates we can … Meaning capture, edit, store, display capture, edit, store, display 9 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

Photographic Signal: Pixels Rays g p g y  Core ideas are ancient, simple, seem obvious:  Lighting: ray sources g g y  Optics: ray bending/ folding devices folding devices  Sensor: measure light  Processing: assess it Processing: assess it  Display: reproduce it  Ancient Greeks: ‘ ‘eye rays’ wipe the world ’ h ld to feel its contents… 10 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

Light Field 11

The Photographic Signal Path g p g  Claim: Computing can improve every step Light Sources Sensors Data Types, Processing Optics Optics Display Rays Rays Scene Scene Eyes Eyes 12 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

Review: How many Rays in a 3 ‐ D Scene? y y A 4 ‐ D set of infinitesimal members.  Imagine:  Convex Enclosure of a 3D scene  Inward ‐ facing ray camera at every surface point Inward facing ray camera at every surface point  Pick the rays you need for ANY camera outside. 2D surface of cameras, 2D f f + 2D ray set for each camera,   4D set of rays. f 13 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

4 ‐ D Light Field / Lumigraph g / g p  Measure all the outgoing outgoing light rays. 14 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

4 ‐ D Illumination Field  Same Idea: Measure all the incoming d ll h incoming light rays l h 15 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

object bj 16 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

17 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

18 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

enclose object within a convex surface (sphere) ( u i ,v i ) indicate the position on the surface where the light enters, i i (  i ,  i ) indicate the direction in which it enters. R i ( u i ,v i ,  i ,  i ) R i ( u i ,v i ,  i ,  i ) incident light field incident light field 19 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

( u r ,v r ) indicate the position on the surface where the light leaves, (  r ,  r ) indicate the direction in which it leaves. r  r R i ( u i ,v i ,  i ,  i ) R i ( u i ,v i ,  i ,  i ) R r ( u r ,v r ,  r ,  r ) R r ( u r ,v r ,  r ,  r ) incident light field incident light field radiant light field radiant light field 20 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

( u i ,v i ) indicate the position on the surface where the light enters, (  i ,  i ) indicate the direction in which it enters. i  i The Reflectance Field h fl ld ( u r ,v r ) indicate the position on the surface where the light leaves, (  r ,  r ) indicate the direction in which it leaves. R ( u i ,v i ,  i ,  i ; u r ,v r ,  r ,  r ) R ( u i ,v i ,  i ,  i ; u r ,v r ,  r ,  r ) 8D 8D 8D reflectance field 8D reflectance field fl fl t t fi ld fi ld Since it is linear, we can represent as a matrix Since it is linear, we can represent as a matrix 21 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

Reflectance Field: Storage Requirements g q R ( u i , v i ,  i ,  i ; u r , v r ,  r ,  r ) R ( u i , v i ,  i ,  i ; u r , v r ,  r ,  r ) 360 x 180 x 180 x 180 x 360 x 180 x 180 x 180   = 4.4e18 measurements x 6 bytes/pixel (in RGB 16 ‐ bit) x 6 bytes/pixel (in RGB 16 bit)  = 26 exabytes (billion GB)   = 82 million 300GB hard drives 22 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

Because Ray Changes Convey Appearance y g y pp  These rays + all these rays give me…  MANY more useful details one can examine… details one can examine… 23 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

Digital Refocusing using Light Field Camera Light Field Camera https://www.lytro.com/living ‐ pictures#living ‐ pictures/ 125 μ square ‐ sided microlenses 125 μ square ‐ sided microlenses 24 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

Missing: Expressive Time Manipulations g p p  What other ways better reveal appearance pp to human viewers? (Without direct shape ( p measurement? ) Can you understand this shape better? 25 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

Missing: Viewpoint Freedom g p 26 Robert Sablatnig, Computer Vision Lab, EVC ‐ 26: Computational Photography

Missing: Interaction… g Adjust everything: lighting pose viewpoint focus FOV Adjust everything: lighting, pose, viewpoint, focus, FOV,… 27