Passive 3D Photography Passive 3D Photography Steve Seitz Steve - - PDF document

▶
passive 3d photography passive 3d photography
SMART_READER_LITE
LIVE PREVIEW

Passive 3D Photography Passive 3D Photography Steve Seitz Steve - - PDF document

Sep 14, 2022 311 likes •493 views

SIGGRAPH 2000 Course on SIGGRAPH 2000 Course on 3D Photography 3D Photography Passive 3D Photography Passive 3D Photography Steve Seitz Steve Seitz Carnegie Mellon University Carnegie Mellon University University of Washington University

slide-1
SLIDE 1

1

Steve Seitz Steve Seitz Carnegie Mellon University Carnegie Mellon University University of Washington University of Washington

http://www. http://www.cs cs. .cmu cmu. .edu edu/~ /~seitz seitz

Passive 3D Photography Passive 3D Photography

SIGGRAPH 2000 Course on SIGGRAPH 2000 Course on 3D Photography 3D Photography

Shading Shading

Visual Cues Visual Cues

Merle Norman Cosmetics, Los Angeles

slide-2
SLIDE 2

2

Visual Cues Visual Cues

Shading Shading Texture Texture

The Visual Cliff, by William Vandivert, 1960

Visual Cues Visual Cues

From The Art of Photography, Canon

Shading Shading Texture Texture Focus Focus

slide-3
SLIDE 3

3

Visual Cues Visual Cues

Shading Shading Texture Texture Focus Focus Motion Motion

Visual Cues Visual Cues

Others: Others:

  • Highlights

Highlights

  • Shadows

Shadows

  • Silhouettes

Silhouettes

  • Inter

Inter-

  • reflections

reflections

  • Symmetry

Symmetry

  • Light Polarization

Light Polarization

  • ...

...

Shading Shading Texture Texture Focus Focus Motion Motion Shape From X Shape From X

  • X = shading, texture, focus, motion, ...

X = shading, texture, focus, motion, ...

slide-4
SLIDE 4

4

Talk Outline Talk Outline

Overview Leading Approaches Overview Leading Approaches

1.

  • 1. Single view modeling

Single view modeling 2.

  • 2. Stereo reconstruction

Stereo reconstruction 3.

  • 3. Structure from motion

Structure from motion

Single View Modeling Single View Modeling

slide-5
SLIDE 5

5

How Do Humans Do This? How Do Humans Do This?

Good Guesswork Based on Priors Good Guesswork Based on Priors

  • “these lines

“these lines look look parallel” parallel”

  • “this

“this looks looks like a cube” like a cube”

  • “this

“this looks looks like a shadow” like a shadow”

Computers Can Do This Too Computers Can Do This Too

  • Shape from shading

Shape from shading [Horn 89] [Horn 89]

  • User

User-

  • aided modeling

aided modeling > > “Tour into the Picture” [ “Tour into the Picture” [Horry Horry 97] 97] > > “Facade” [ “Facade” [Debevec Debevec 96] 96] > > “Single View Metrology” [ “Single View Metrology” [Criminisi Criminisi 99] 99]

  • Learning approaches

Learning approaches > > “Morphable Models” [ “Morphable Models” [Blanz Blanz 99] 99]

Perspective Cues Perspective Cues

slide-6
SLIDE 6

6

Perspective Cues Perspective Cues Perspective Cues Perspective Cues

slide-7
SLIDE 7

7

Vanishing Points Vanishing Points

Vanishing Vanishing Point Point

Measuring Height Measuring Height

1 2 3 4 5 5.4 2.8 3.3

Same Concepts Enable Same Concepts Enable

  • Reconstructing X, Y, and Z

Reconstructing X, Y, and Z

  • Computing camera projection matrix

Computing camera projection matrix

  • Eliminating the ruler

Eliminating the ruler

slide-8
SLIDE 8

8

“Single View Metrology” [ “Single View Metrology” [Criminisi Criminisi 99] 99] “Single View Metrology” [ “Single View Metrology” [Criminisi Criminisi 99] 99]

The Music Lesson, Jan Vermeer, 1662-65

Royal Collection of Her Majesty Queen Elizabeth II

slide-9
SLIDE 9

9

“Morphable Models” [ “Morphable Models” [Blanz Blanz 99] 99]

Video Video

Stereo Reconstruction Stereo Reconstruction

The Stereo Problem The Stereo Problem

  • Shape from two (or more) images

Shape from two (or more) images

  • Biological motivation

Biological motivation

known known camera camera viewpoints viewpoints

slide-10
SLIDE 10

10

Stereo Stereo

scene point scene point focal point focal point image plane image plane

Stereo Stereo

Basic Principle: Triangulation Basic Principle: Triangulation

  • Gives reconstruction as intersection of two rays

Gives reconstruction as intersection of two rays

  • Requires

Requires point correspondence point correspondence

slide-11
SLIDE 11

11

Stereo Correspondence Stereo Correspondence

Determine Pixel Correspondence Determine Pixel Correspondence

  • Pairs of points that correspond to same scene point

Pairs of points that correspond to same scene point

Epipolar Epipolar Constraint Constraint

  • Reduces correspondence problem to 1D search along

Reduces correspondence problem to 1D search along conjugate conjugate epipolar epipolar lines lines

epipolar epipolar plane plane

epipolar epipolar line line epipolar epipolar line line

Stereo Matching Algorithms Stereo Matching Algorithms

Match Pixels in Conjugate Match Pixels in Conjugate Epipolar Epipolar Lines Lines

  • Assume color of point does not change

Assume color of point does not change

  • Pitfalls

Pitfalls > > specularities specularities > > low low-

  • contrast regions

contrast regions > > occlusions

  • cclusions

> > image error image error > > camera calibration error camera calibration error

  • Numerous approaches

Numerous approaches > > dynamic programming [Baker 81, dynamic programming [Baker 81,Ohta Ohta 85] 85] > > smoothness smoothness functionals functionals > > more images ( more images (trinocular trinocular, N , N-

  • ocular) [
  • cular) [Okutomi

Okutomi 93] 93] > > graph cuts [ graph cuts [Boykov Boykov 00] 00]

slide-12
SLIDE 12

12

Structure from Motion Structure from Motion

Reconstruct Reconstruct

  • Scene

Scene geometry geometry

  • Camera

Camera motion motion

Unknown Unknown camera camera viewpoints viewpoints

Structure from Motion Structure from Motion

The SFM Problem The SFM Problem

  • Reconstruct scene

Reconstruct scene geometry geometry and camera and camera motion motion from from two or more images two or more images

Track 2D Features Estimate 3D Optimize Fit Surfaces

slide-13
SLIDE 13

13

Structure from Motion Structure from Motion

Step 1: Track Features Step 1: Track Features

  • Detect good features

Detect good features > > corners, line segments corners, line segments

  • Find correspondences between frames

Find correspondences between frames > > window window-

  • based correlation

based correlation

Structure from Motion Structure from Motion

Step 2: Estimate Motion and Structure Step 2: Estimate Motion and Structure

  • Orthographic projection, e.g.,

Orthographic projection, e.g., [ [Tomasi Tomasi 92] 92]

  • 2 or 3 views at a time

2 or 3 views at a time [Hartley 00] [Hartley 00]

[ ]

n 2 1 f 2 1 f 2 1

X X X I I I L M M             =            

Images Motion Structure

slide-14
SLIDE 14

14

Structure from Motion Structure from Motion

Step 3: Refine Estimates Step 3: Refine Estimates

  • Nonlinear optimization over cameras and points

Nonlinear optimization over cameras and points > > [Hartley 94] [Hartley 94]

  • “Bundle adjustment” in

“Bundle adjustment” in photogrammetry photogrammetry

Structure from Motion Structure from Motion

Step 4: Recover Surfaces Step 4: Recover Surfaces

  • Image

Image-

  • based triangulation

based triangulation [Morris 00, [Morris 00, Baillard Baillard 99] 99]

  • Silhouettes

Silhouettes [Fitzgibbon 98] [Fitzgibbon 98]

  • Stereo

Stereo [ [Pollefeys Pollefeys 99] 99]

Poor mesh Poor mesh Good mesh Good mesh

Morris and Morris and Kanade Kanade, 2000 , 2000

slide-15
SLIDE 15

15

Resources Resources

Computer Vision Home Page Computer Vision Home Page

  • http://www.

http://www.cs cs. .cmu cmu. .edu edu/ /afs afs/ /cs cs/project/ /project/cil cil/ftp/html/vision.html /ftp/html/vision.html

Computer Vision Textbooks Computer Vision Textbooks

  • O. Faugeras,
  • O. Faugeras, Three

Three-

  • Dimensional Computer Vision

Dimensional Computer Vision, MIT Press, 1993. , MIT Press, 1993.

  • E.
  • E. Trucco

Trucco and A. and A. Verri Verri, , Introductory Techniques for 3 Introductory Techniques for 3-

  • D Computer Vision

D Computer Vision, Prentice , Prentice-

  • Hall,

Hall, 1998. 1998.

  • V. S.
  • V. S. Nalwa

Nalwa, , A Guided Tour of Computer Vision A Guided Tour of Computer Vision, Addison , Addison-

  • Wesley, 1993.

Wesley, 1993.

  • R.
  • R. Jain

Jain, R. , R. Kasturi Kasturi and B. G. and B. G. Schunck Schunck, , Machine Vision Machine Vision, McGraw , McGraw-

  • Hill, 1995.

Hill, 1995.

  • R.
  • R. Klette

Klette, K. , K. Schluns Schluns and A. and A. Koschan Koschan, , Computer Vision: Three Computer Vision: Three-

  • Dimensional Data from

Dimensional Data from Images Images, Springer , Springer-

  • Verlag

Verlag, 1998. , 1998.

  • M.
  • M. Sonka

Sonka, V. , V. Hlavac Hlavac and R. Boyle, and R. Boyle, Image Processing, Analysis, and Machine Vision Image Processing, Analysis, and Machine Vision, , Brooks/Cole Publishing, 1999. Brooks/Cole Publishing, 1999.

  • D. H. Ballard and C. M. Brown,
  • D. H. Ballard and C. M. Brown, Computer Vision

Computer Vision, Prentice , Prentice-

  • Hall, 1982.

Hall, 1982.

  • B. K. P. Horn,
  • B. K. P. Horn, Robot Vision

Robot Vision, McGraw , McGraw-

  • Hill, 1986.

Hill, 1986.

  • J.
  • J. Koenderink

Koenderink, , Solid Shape Solid Shape, MIT Press, 1990. , MIT Press, 1990.

  • D. Marr,
  • D. Marr, Vision

Vision, Freeman, 1982. , Freeman, 1982.

Single View Modeling Single View Modeling

  • V.
  • V. Blanz

Blanz & T. Vetter, “A Morphable Model for the Synthesis of 3D Faces”, & T. Vetter, “A Morphable Model for the Synthesis of 3D Faces”, SIGGRAPH 99, SIGGRAPH 99,

  • pp. 187
  • pp. 187-
  • 194.

194.

  • A.
  • A. Criminisi

Criminisi, I. Reid, & A. , I. Reid, & A. Zisserman Zisserman, “Single View Metrology”, ICCV 2000, pp. 434 , “Single View Metrology”, ICCV 2000, pp. 434-

  • 441.

441.

  • B. K. P. Horn & M. Brooks, “Shape from Shading”, 1989, MIT Press
  • B. K. P. Horn & M. Brooks, “Shape from Shading”, 1989, MIT Press, Cambridge, M.A.

, Cambridge, M.A.

  • Y.
  • Y. Horry

Horry, K. , K. Anjyo Anjyo, & K. Arai, “Tour into the Picture”, SIGGRAPH 97, pp. 225 , & K. Arai, “Tour into the Picture”, SIGGRAPH 97, pp. 225-

  • 232.

232.

  • R. Zhang, P
  • R. Zhang, P-
  • S. Tsai, J.
  • S. Tsai, J. Cryer

Cryer, & M. Shah, “Shape from Shading: A Survey”, IEEE Trans. , & M. Shah, “Shape from Shading: A Survey”, IEEE Trans.

  • n PAMI, 21(8), 1999.
  • n PAMI, 21(8), 1999.

Stereo Stereo

  • Y.
  • Y. Boykov

Boykov, O. , O. Veksler Veksler, & R. , & R. Zabih Zabih, “Fast Approximate Energy Minimization via Graph , “Fast Approximate Energy Minimization via Graph Cuts”, ICCV, 1999. Cuts”, ICCV, 1999.

  • Y.
  • Y. Ohta

Ohta & T. & T. Kanade Kanade, "Stereo by Intra , "Stereo by Intra-

  • and Inter

and Inter-

  • Scanline

Scanline Search Using Dynamic Search Using Dynamic Programming", IEEE Trans. on PAMI, 7(2), 1985, pp. 129 Programming", IEEE Trans. on PAMI, 7(2), 1985, pp. 129-

  • 154.

154.

  • M.
  • M. Okutomi

Okutomi & T. & T. Kanade Kanade, ”A Multiple , ”A Multiple-

  • Baseline Stereo", IEEE Trans. on Pattern Analysis

Baseline Stereo", IEEE Trans. on Pattern Analysis and Machine Intelligence", 15(4), 1993, 353 and Machine Intelligence", 15(4), 1993, 353-

  • 363.

363.

Bibliography Bibliography

slide-16
SLIDE 16

16

Structure from Motion Structure from Motion

  • C.
  • C. Baillard

Baillard & A. & A. Zisserman Zisserman, “Automatic Reconstruction of Planar Models from Multiple , “Automatic Reconstruction of Planar Models from Multiple Views”, CVPR 99, pp. 559 Views”, CVPR 99, pp. 559-

  • 565.

565.

  • A.W. Fitzgibbon, G. Cross, & A.

A.W. Fitzgibbon, G. Cross, & A. Zisserman Zisserman, “ , “Automatic 3D Model Construction for Turn Automatic 3D Model Construction for Turn-

  • Table Sequences”,

Table Sequences”, SMILE Workshop, 1998. SMILE Workshop, 1998.

  • R. Hartley & A.
  • R. Hartley & A. Zisserman

Zisserman, “Multiple View Geometry”, Cambridge Univ. Press, 2000. , “Multiple View Geometry”, Cambridge Univ. Press, 2000.

  • R. Hartley, “Euclidean Reconstruction from Uncalibrated Views”,
  • R. Hartley, “Euclidean Reconstruction from Uncalibrated Views”, In Applications of

In Applications of Invariance in Computer Vision, Springer Invariance in Computer Vision, Springer-

  • Verlag

Verlag, 1994, pp. 237 , 1994, pp. 237-

  • 256.

256.

  • D. Morris & T.
  • D. Morris & T. Kanade

Kanade, “Image , “Image-

  • Consistent Surface Triangulation”, CVPR 00, pp. 332

Consistent Surface Triangulation”, CVPR 00, pp. 332-

  • 338.

338.

  • M.
  • M. Pollefeys

Pollefeys, R. Koch & L. Van , R. Koch & L. Van Gool Gool, “Self , “Self-

  • Calibration and Metric Reconstruction in spite

Calibration and Metric Reconstruction in spite

  • f Varying and Unknown Internal Camera Parameters”, Int. J. of C
  • f Varying and Unknown Internal Camera Parameters”, Int. J. of Computer Vision, 32(1),
  • mputer Vision, 32(1),

1999, pp. 7 1999, pp. 7-

  • 25.

25.

  • C.
  • C. Tomasi

Tomasi & T. & T. Kanade Kanade, ”Shape and Motion from Image Streams Under Orthography: A , ”Shape and Motion from Image Streams Under Orthography: A Factorization Method", Int. Journal of Computer Vision, 9(2), 19 Factorization Method", Int. Journal of Computer Vision, 9(2), 1992, pp. 137 92, pp. 137-

  • 154.

154.

Bibliography Bibliography