Passive 3D Photography Passive 3D Photography Steve Seitz Steve - PDF document

SIGGRAPH 2000 Course on SIGGRAPH 2000 Course on 3D Photography 3D Photography Passive 3D Photography Passive 3D Photography Steve Seitz Steve Seitz Carnegie Mellon University Carnegie Mellon University University of Washington University of Washington http://www.cs http://www. cs. .cmu cmu. .edu edu/~ /~seitz seitz Visual Cues Visual Cues Shading Shading Merle Norman Cosmetics, Los Angeles 1

Visual Cues Visual Cues Shading Shading Texture Texture The Visual Cliff , by William Vandivert, 1960 Visual Cues Visual Cues Shading Shading Texture Texture Focus Focus From The Art of Photography , Canon 2

Visual Cues Visual Cues Shading Shading Texture Texture Focus Focus Motion Motion Visual Cues Visual Cues Shading Shading Others: Others: • Highlights Highlights Texture Texture • • Shadows Shadows • • Silhouettes Silhouettes • • Inter Inter- -reflections reflections Focus Focus • • Symmetry Symmetry • • Light Polarization Light Polarization • Motion Motion • ... ... • Shape From X Shape From X X = shading, texture, focus, motion, ... X = shading, texture, focus, motion, ... • • 3

Talk Outline Talk Outline Overview Leading Approaches Overview Leading Approaches 1. Single view modeling 1. Single view modeling 2. Stereo reconstruction 2. Stereo reconstruction 3. Structure from motion 3. Structure from motion Single View Modeling Single View Modeling 4

How Do Humans Do This? How Do Humans Do This? Good Guesswork Based on Priors Good Guesswork Based on Priors • “these lines “these lines look look parallel” parallel” • • “this “this looks looks like a cube” like a cube” • • “this “this looks looks like a shadow” like a shadow” • Computers Can Do This Too Computers Can Do This Too • Shape from shading Shape from shading [Horn 89] [Horn 89] • • User User- -aided modeling aided modeling • > “Tour into the Picture” [ > “Tour into the Picture” [Horry Horry 97] 97] > “Facade” [ > “Facade” [Debevec Debevec 96] 96] > “Single View Metrology” [ “Single View Metrology” [Criminisi Criminisi 99] 99] > • Learning approaches Learning approaches • > “Morphable Models” [ > “Morphable Models” [Blanz Blanz 99] 99] Perspective Cues Perspective Cues 5

Perspective Cues Perspective Cues Perspective Cues Perspective Cues 6

Vanishing Points Vanishing Points Vanishing Vanishing Point Point Measuring Height Measuring Height 5.4 5 4 3.3 3 2.8 2 1 Same Concepts Enable Same Concepts Enable • Reconstructing X, Y, and Z Reconstructing X, Y, and Z • • Computing camera projection matrix Computing camera projection matrix • • Eliminating the ruler Eliminating the ruler • 7

“Single View Metrology” [Criminisi Criminisi 99] 99] “Single View Metrology” [ “Single View Metrology” [Criminisi “Single View Metrology” [ Criminisi 99] 99] The Music Lesson , Jan Vermeer, 1662-65 Royal Collection of Her Majesty Queen Elizabeth II 8

“Morphable Models” [Blanz Blanz 99] 99] “Morphable Models” [ Video Video Stereo Reconstruction Stereo Reconstruction The Stereo Problem The Stereo Problem • Shape from two (or more) images Shape from two (or more) images • • Biological motivation Biological motivation • known known camera camera viewpoints viewpoints 9

Stereo Stereo scene point scene point image plane image plane focal point focal point Stereo Stereo Basic Principle: Triangulation Basic Principle: Triangulation • Gives reconstruction as intersection of two rays Gives reconstruction as intersection of two rays • • Requires Requires point correspondence point correspondence • 10

Stereo Correspondence Stereo Correspondence Determine Pixel Correspondence Determine Pixel Correspondence • Pairs of points that correspond to same scene point Pairs of points that correspond to same scene point • epipolar line line epipolar epipolar line epipolar line epipolar plane epipolar plane Epipolar Epipolar Constraint Constraint • Reduces correspondence problem to 1D search along Reduces correspondence problem to 1D search along • conjugate epipolar conjugate epipolar lines lines Stereo Matching Algorithms Stereo Matching Algorithms Match Pixels in Conjugate Match Pixels in Conjugate Epipolar Epipolar Lines Lines • Assume color of point does not change Assume color of point does not change • • Pitfalls Pitfalls • > specularities > specularities > low low- -contrast regions contrast regions > > occlusions occlusions > > image error > image error > camera calibration error > camera calibration error • Numerous approaches Numerous approaches • > dynamic programming [Baker 81, dynamic programming [Baker 81,Ohta Ohta 85] 85] > > smoothness > smoothness functionals functionals > more images ( > more images (trinocular trinocular, N , N- -ocular) [ ocular) [Okutomi Okutomi 93] 93] > graph cuts [ > graph cuts [Boykov Boykov 00] 00] 11

Structure from Motion Structure from Motion Unknown Unknown camera camera viewpoints viewpoints Reconstruct Reconstruct • Scene Scene geometry geometry • • Camera Camera motion motion • Structure from Motion Structure from Motion The SFM Problem The SFM Problem • Reconstruct scene Reconstruct scene geometry geometry and camera and camera motion motion from from • two or more images two or more images Track 2D Features Estimate 3D Optimize Fit Surfaces 12

Structure from Motion Structure from Motion Step 1: Track Features Step 1: Track Features • Detect good features Detect good features • > corners, line segments > corners, line segments • Find correspondences between frames Find correspondences between frames • > window window- -based correlation based correlation > Structure from Motion Structure from Motion I     1 1     I [ ]     2 2 X X X = L 1 2 n     M M Structure     I     f f Images Motion Step 2: Estimate Motion and Structure Step 2: Estimate Motion and Structure • Orthographic projection, e.g., Orthographic projection, e.g., [ [Tomasi Tomasi 92] 92] • • 2 or 3 views at a time 2 or 3 views at a time [Hartley 00] • [Hartley 00] 13

Structure from Motion Structure from Motion Step 3: Refine Estimates Step 3: Refine Estimates • Nonlinear optimization over cameras and points Nonlinear optimization over cameras and points • > [Hartley 94] > [Hartley 94] • “Bundle adjustment” in “Bundle adjustment” in photogrammetry photogrammetry • Structure from Motion Structure from Motion Poor mesh Poor mesh Good mesh Good mesh Morris and Kanade Morris and Kanade, 2000 , 2000 Step 4: Recover Surfaces Step 4: Recover Surfaces • Image Image- -based triangulation based triangulation [Morris 00, [Morris 00, Baillard Baillard 99] 99] • • Silhouettes Silhouettes [Fitzgibbon 98] • [Fitzgibbon 98] • Stereo Stereo [ [Pollefeys Pollefeys 99] 99] • 14