Page 1 Cameras Removing Camera Shake [Fergus 06] sensor optics - PDF document

Organization � lectures � discussion of research papers � student projects Computational Photography � (1-2 student(s) per group) � list of possible ideas � presentation of ideas Sommersemester 2007 � project proposal (2 pages) � implementation Hendrik Lensch � presentation of results lensch@mpi-inf.mpg.de � report (like a conference paper 6-8 pages) Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Computational Photography Film-like Outline Photography with bits Computational Camera Smart Light Computational Computational Computational Digital Computational Imaging/Optics Sensor Illumination Photography Processing � digital camera technology � computational sensing Capture of optically Detectors that Adapting and Image processing Processing of a set applied to captured of captured images coded images and combine sensing Controlling � multi-dimensional imaging images to produce to create “new” computational and processing to Illumination to decoding to produce create “smart” Create ‘revealing’ “better” images. images. � computational imaging in other fields “new?” images. pixels. image Examples: Examples: Examples: Examples: Examples: Interpolation, Filtering, Mosaicing, Matting, Coded Aperture, Artificial Retina, Flash/no flash, Enhancement, Dynamic Super-Resolution, Optical Tomography, Retinex Sensors, Lighting domes, Range Compression, Multi-Exposure HDR, Diaphanography, Adaptive Dynamic Multi-flash Color Management, Light Field from SA Microscopy, Range Sensors, for depth edges, Morphing, Hole Filling, Mutiple View, Integral Imaging, Artistic Image Effects, Structure from Motion, Assorted Pixels, Edge Detect Chips, Dual Photos, Image Compression, Shape from X. Catadioptric Imaging, Focus of Expansion Polynomial texture Watermarking. Holographic Imaging. Chips, Motion Maps, 4D light Sensors. source [Tumblin 2005] Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Cameras � sensor � optics Conventional Cameras � processing and Advanced Processing Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Page 1

Cameras Removing Camera Shake [Fergus 06] � sensor � optics � processing � weird stuff [Georgiev 06] Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Removing Camera Shake Hallucinating Resolution [Fergus 06] applications � high-resolution pre-image + normal-resolution snapshot = high-resolution snapshot [Sawhney01] & → left eye right eye left eye Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Hallucinating Resolution Hallucinating Resolution applications applications � high-resolution pre-image + � high-resolution pre-image + normal-resolution snapshot = normal-resolution snapshot = high-resolution snapshot high-resolution snapshot � normal-resolution full-frame snapshot + high-resolution partial-frame patches = high-resolution snapshot � normal-resolution normal-frame-rate → video + high-resolution low-frame-rate video = high-resolution HDTV Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Page 2

[Wei01] Image Stacks [Agarwala SIGGRAPH 2004] extended depth of field original extrapolated low-resolution samples of original reconstruction Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Image Stacks Image Stacks [Agarwala SIGGRAPH 2004] [Agarwala SIGGRAPH 2004] foreground removal multi-shot � translate the camera images � align images shoot until � choose minimum contrast pixels everybody has smiled at least once Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Image Stacks Image Stacks [Agarwala SIGGRAPH 2004] [Agarwala SIGGRAPH 2004] � segment � assemble Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Page 3

Circles of Cameras Fun with Multiple Cameras Manex’s bullet time array [Rademacher 1998] [Seitz 2001] and Videos multi-perspective panorama peripheral photograph Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Stanford Multi-Camera Array Light Fields [Levoy & Hanrahan, 1996] widely spaced light field capture tightly packed high-performance imaging intermediate spacing synthetic aperture photography Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 View Interpolation with Image-based Priors View Interpolation with Image-based Priors [Fitzgibbon et al. 2003] [Fitzgibbon et al. 2003] Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Page 4

Synthetic Aperture Synthetic Aperture [Wilburn 05] [Wilburn 05] Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Synthetic Aperture Tiled Camera Array [Wilburn 05] Can we match the image quality of a cinema camera? world’s largest video camera no parallax for distant objects poor lenses limit image quality seamless mosaicing isn’t hard Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Tiled panoramic image Tiled panoramic image (before geometric or color calibration) (after calibration and blending) Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Page 5

Tiled Camera Array same exposure in all cameras Can we match the image quality of a cinema camera? world’s largest video camera no parallax for distant objects individually metered poor lenses limit image quality seamless mosaicing isn’t hard per-camera exposure metering checkerboard HDR within and between tiles of exposures Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 High-performance photography as multi- Spacetime Aperture Shaping dimensional sampling [Wilburn 05] spatial resolution shorten exposure time to freeze motion → dark field of view stretch contrast to restore level frame rate → noisy dynamic range increase (synthetic) aperture to capture more light → bits of precision decreases depth of field depth of field focus setting color sensitivity Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Spacetime Aperture Shaping Spacetime Aperture Shaping few cameras, long exposure → center of aperture: high depth of field, low noise, but action is blurred periphery of aperture: many cameras, short exposure → freezes action, low noise, but low depth of field Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Page 6

Spacetime Aperture Shaping Low Dynamic Range Video Enhancement [Bennett & McMillan 2005] Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Stanford CityBlock Project Stanford CityBlock Project capture video while driving extract middle column from each frame stack them to create a panorama Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Stanford CityBlock Project Stanford CityBlock Project ? ? Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Page 7

Abnormal Illumination [Román & Lensch 06] Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Light Sources Flash/No-Flash Photography � enhance image quality flash no-flash combined � point light sources � monitors � projectors � natural lighting [Petschnigg et al. 2004] Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 More Flash-noflash Algorithms Multi-Flash Images [Agrawal SIGGRAPH 2005] � extract edge information remove features that don’t appear in both (as determined from image gradients) Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Page 8

Multi-Flash Images 3D Range Scanning � extract edge information [Levoy et al. 2000] Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Shader Lamps Non-photorealistic Illumination (NPI) [Raskar et al. 2001] (Blois) what are interesting “light field bulbs”, and what are they good for? � recoloring Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Non-photorealistic illumination (NPI) Non-photorealistic illumination (NPI) [Grossberg 2004] [Akers 2003] what are interesting “light field bulbs”, what are interesting “light field bulbs”, and what are they good for? and what are they good for? � recoloring � recoloring ? � cloaking � cloaking � everywhere-grazing light [Tachi 2004] Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Page 9

Measuring the BRDF Digitizing Actors � opaque surfaces [Steve Marschner 1999] The Matrix Reloaded [ESC Entertainment 2003] Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Light Stage I Light Stage III � capturing a reflectance field � relighting real objects with virtual light sources [Debevec et al. 2000] [Debevec et al. 2002] Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Reflectance Fields Environment Matting [Fuchs 07] � capture refracting objects [Zongker et al. 1999] Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Page 10

Environment Matte Acquisition Natural Lighting � outdoor environment mattes [Zongker et al. 1999] [Matusik 2004] Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Dual Photography Relighting with 4D incident light fields photograph dual image [Masselus et al. 2003] Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 6D Reflectance Field 6D Reflectance Field Hendrik Lensch, Summer 2007 Hendrik Lensch, Summer 2007 Page 11