Passive 3D Photography Passive 3D Photography Steve Seitz Steve - - PDF document

▶
passive 3d photography passive 3d photography
SMART_READER_LITE
LIVE PREVIEW

Passive 3D Photography Passive 3D Photography Steve Seitz Steve - - PDF document

Feb 27, 2024 414 likes •557 views

SIGGRAPH 99 Course on SIGGRAPH 99 Course on 3D Photography 3D Photography Passive 3D Photography Passive 3D Photography Steve Seitz Steve Seitz Carnegie Mellon University University Carnegie Mellon http http:// ://www www. .cs cs.

slide-1
SLIDE 1

1

Steve Steve Seitz Seitz Carnegie Mellon Carnegie Mellon University University

http http:// ://www www. .cs cs. .cmu cmu. .edu edu/~ /~seitz seitz

Passive 3D Photography Passive 3D Photography

SIGGRAPH 99 Course on SIGGRAPH 99 Course on 3D Photography 3D Photography

Talk Outline Talk Outline

  • 1. Visual Cues
  • 1. Visual Cues
  • 2. Classical Vision Algorithms
  • 2. Classical Vision Algorithms
  • 3. State of the Art (video)
  • 3. State of the Art (video)
slide-2
SLIDE 2

2

Motion Motion

Visual Cues Visual Cues

Motion Motion Shading Shading

Visual Cues Visual Cues

Merle Norman Cosmetics, Los Angeles

slide-3
SLIDE 3

3

Visual Cues Visual Cues

Motion Motion Shading Shading Texture Texture

The Visual Cliff, by William Vandivert, 1960

Visual Cues Visual Cues

Motion Motion Shading Shading Texture Texture Focus Focus

From The Art of Photography, Canon

slide-4
SLIDE 4

4

Visual Cues Visual Cues

Motion Motion Shading Shading Texture Texture Focus Focus Others: Others:

  • Highlights

Highlights

  • Shadows

Shadows

  • Silhouettes

Silhouettes

  • Inter-reflections

Inter-reflections

  • Symmetry

Symmetry

  • Light Polarization

Light Polarization

  • ...

...

Reconstruction Algorithms Reconstruction Algorithms

Shape From X Shape From X

  • Stereo (shape from parallax)

Stereo (shape from parallax)

  • Structure from motion

Structure from motion

  • Shape from shading

Shape from shading

  • Photometric stereo

Photometric stereo

  • Shape from texture

Shape from texture

  • Shape from focus/defocus

Shape from focus/defocus

  • Shape from silhouettes, ...

Shape from silhouettes, ...

✔ ✔

slide-5
SLIDE 5

5

Stereo Stereo

The Stereo Problem The Stereo Problem

  • Reconstruct scene geometry from two or more

Reconstruct scene geometry from two or more calibrated calibrated images images

scene point focal point image plane

Stereo Stereo

The Stereo Problem The Stereo Problem

  • Reconstruct scene geometry from two or more

Reconstruct scene geometry from two or more calibrated calibrated images images

Basic Principle: Triangulation Basic Principle: Triangulation

  • Gives reconstruction as intersection of two rays

Gives reconstruction as intersection of two rays

  • Requires

Requires point correspondence point correspondence

slide-6
SLIDE 6

6

Stereo Correspondence Stereo Correspondence

Determine Pixel Correspondence Determine Pixel Correspondence

  • Pairs of points that correspond to same scene point

Pairs of points that correspond to same scene point

Epipolar Epipolar Constraint Constraint

  • Reduces correspondence problem to 1D search along

Reduces correspondence problem to 1D search along conjugate conjugate epipolar epipolar lines lines

epipolar plane

epipolar line epipolar line

Stereo Matching Algorithms Stereo Matching Algorithms

Match Pixels in Conjugate Match Pixels in Conjugate Epipolar Epipolar Lines Lines

  • Assume color of point does not change

Assume color of point does not change

  • Pitfalls

Pitfalls > > specularities specularities (non- (non-Lambertian Lambertian surfaces) surfaces) > > ambiguity (low-contrast regions) ambiguity (low-contrast regions) > > missing data (occlusions) missing data (occlusions) > > intensity error ( intensity error (quantization quantization, sensor error) , sensor error) > > position error (camera calibration) position error (camera calibration)

  • Numerous approaches

Numerous approaches > > winner-take all winner-take all > > dynamic programming [ dynamic programming [Ohta Ohta 85] 85] > > smoothness smoothness functionals functionals > > more images ( more images (trinocular trinocular, N-ocular) [ , N-ocular) [Okutomi Okutomi 93] 93]

slide-7
SLIDE 7

7

Structure from Motion Structure from Motion

The SFM Problem The SFM Problem

  • Reconstruct scene

Reconstruct scene geometry geometry and camera and camera motion motion from from two or more images two or more images

Assume Assume

  • Pixel correspondence

Pixel correspondence > > via tracking via tracking

  • Projection model

Projection model > > classic methods are orthographic classic methods are orthographic

Orthographic Projection Orthographic Projection

1 2 1 2 1 2

t X u

× × × ×

+ =

3 3

image point projection matrix scene point image

  • ffset

Trick Trick

  • Choose scene origin to be

Choose scene origin to be centroid centroid of 3D points

  • f 3D points
  • Choose image origins to be

Choose image origins to be centroid centroid of 2D points

  • f 2D points
  • Allows us to drop the camera translation:

Allows us to drop the camera translation:

1 2 1 2

X u

× × × = 3 3

slide-8
SLIDE 8

8

Shape by Factorization Shape by Factorization [

[Tomasi Tomasi & & Kanade Kanade, 92] , 92]

[ ]

[ ]

n 3 n 2 n 2 × × = ×

Π

n 2 1 n 2 1

X X X u u u

  • projection of n features in one image:

[ ]

n 3 3 2f n 2f × ×               = ×              

n 2 1 f 2 1 f n f 2 f 1 2 n 2 2 2 1 1 n 1 2 1 1

X X X u u u u u u u u u

  • projection of n features in f images

W measurement M motion S shape

n 3 3 2f n 2f

S M W

× × × =

’ ’

Factorization Technique Factorization Technique

  • W

W is at most rank 3 (assuming no noise) is at most rank 3 (assuming no noise)

  • We can use

We can use singular value decomposition singular value decomposition to factor W: to factor W:

Shape by Factorization Shape by Factorization [

[Tomasi Tomasi & & Kanade Kanade, 92] , 92]

  • S’

S’ differs from differs from S S by a linear transformation by a linear transformation A A: :

  • Solve for

Solve for A A by enforcing constraints on by enforcing constraints on M M

) )( ( ’ ’ AS MA S M W

1 −

= =

n 3 3 2f n 2f

S M W

× × × = known solve for

slide-9
SLIDE 9

9

Shape from Shading Shape from Shading

Classical Approach Classical Approach

  • Suppose reflected light depends only on

Suppose reflected light depends only on α α

Shape from Shading [Horn, 1970] Shape from Shading [Horn, 1970]

α cos k radiance =

α α N N L L

slide-10
SLIDE 10

10

The Reflectance Map The Reflectance Map

Image Image α α N N L L Reflectance Map: R Reflectance Map: R

[ ]

1 q p

N

− =

The Reflectance Map The Reflectance Map

Reflectance Map Reflectance Map Image Image

slide-11
SLIDE 11

11

Finding a Unique Solution Finding a Unique Solution

Three Approaches Three Approaches

  • Characteristic Strip Method [Horn, 77]

Characteristic Strip Method [Horn, 77] > > select a few points where normal is known select a few points where normal is known > > grow solution by moving direction of grow solution by moving direction of ∇ ∇R R

  • Variational

Variational Method [ Method [Ikeuchi Ikeuchi & Horn, 81] & Horn, 81] > > start with an initial guess of surface shape start with an initial guess of surface shape > > define energy function define energy function > > refine to minimize energy function refine to minimize energy function

  • Photometric Stereo [

Photometric Stereo [Woodham Woodham 80] 80] > > use more images use more images

Photometric Stereo Photometric Stereo

Two Images Under Different Lighting Two Images Under Different Lighting Need Three Images for Unique Solution Need Three Images for Unique Solution

slide-12
SLIDE 12

12

Photometric Stereo: Matrix Formulation Photometric Stereo: Matrix Formulation

Write Equations in Matrix Form Write Equations in Matrix Form

N L N L N L

T 3 T 2 T 1

ˆ ˆ ˆ ˆ ˆ ˆ

3 2

k I k I k I1

  • =
  • =
  • =

1 3

I ×

3 × 3

L

1

~

× 3

N N I L N

1 1

~ ~ = =

−

k

Advantage: Advantage:

  • Can solve for variable reflectance

Can solve for variable reflectance k k

Resources Resources

Computer Vision Home Page Computer Vision Home Page

  • http://www.

http://www.cs cs. .cmu cmu. .edu edu/ /afs afs/ /cs cs/project/ /project/cil cil/ftp/html/vision.html /ftp/html/vision.html

Computer Vision Textbooks Computer Vision Textbooks

  • D. H. Ballard and C. M. Brown,
  • D. H. Ballard and C. M. Brown, Computer Vision

Computer Vision, Prentice-Hall, 1982. , Prentice-Hall, 1982.

  • O.
  • O. Faugeras

Faugeras, , Three-Dimensional Computer Vision Three-Dimensional Computer Vision, MIT Press, 1993. , MIT Press, 1993.

  • B. K. P. Horn,
  • B. K. P. Horn, Robot Vision

Robot Vision, McGraw-Hill, 1986. , McGraw-Hill, 1986.

  • R.
  • R. Jain

Jain, R. , R. Kasturi Kasturi and B. G. and B. G. Schunck Schunck, , Machine Vision Machine Vision, McGraw-Hill, 1995. , McGraw-Hill, 1995.

  • R.
  • R. Klette

Klette, K. , K. Schluns Schluns and A. and A. Koschan Koschan, , Computer Vision: Three-Dimensional Data from Computer Vision: Three-Dimensional Data from Images Images, Springer- , Springer-Verlag Verlag, 1998. , 1998.

  • V. S.
  • V. S. Nalwa

Nalwa, , A Guided Tour of Computer Vision A Guided Tour of Computer Vision, Addison-Wesley, 1993. , Addison-Wesley, 1993.

  • M.
  • M. Sonka

Sonka, V. , V. Hlavac Hlavac and R. Boyle, and R. Boyle, Image Processing, Analysis, and Machine Vision Image Processing, Analysis, and Machine Vision, , Brooks/Cole Publishing, 1999. Brooks/Cole Publishing, 1999.

  • E.
  • E. Trucco

Trucco and A. and A. Verri Verri, , Introductory Techniques for 3-D Computer Vision Introductory Techniques for 3-D Computer Vision, Prentice-Hall, , Prentice-Hall, 1998. 1998.

  • D.
  • D. Marr

Marr, , Vision Vision, Freeman, 1982. , Freeman, 1982.

  • J.
  • J. Koenderink

Koenderink, , Solid Shape Solid Shape, MIT Press, 1990. , MIT Press, 1990.

slide-13
SLIDE 13

13

Stereo Stereo

  • Yuichi

Yuichi Ohta Ohta & & Takeo Kanade Takeo Kanade, "Stereo by Intra- and Inter- , "Stereo by Intra- and Inter-Scanline Scanline Search Using Search Using Dynamic Programming", IEEE Trans. on Pattern Analysis and Machine Intelligence, Dynamic Programming", IEEE Trans. on Pattern Analysis and Machine Intelligence, 7(2), 1985, pp. 129-154. 7(2), 1985, pp. 129-154.

  • Masatoshi

Masatoshi Okutomi Okutomi & & Takeo Kanade Takeo Kanade, ”A Multiple-Baseline Stereo", IEEE Trans. on , ”A Multiple-Baseline Stereo", IEEE Trans. on Pattern Analysis and Machine Intelligence", 15(4), 353-363, 1985. Pattern Analysis and Machine Intelligence", 15(4), 353-363, 1985.

Structure-from-Motion Structure-from-Motion

  • Carlo

Carlo Tomasi Tomasi & & Takeo Kanade Takeo Kanade, ”Shape and Motion from Image Streams Under , ”Shape and Motion from Image Streams Under Orthography: A Factorization Method", Int. Journal of Computer Vision, 9(2), 1992, Orthography: A Factorization Method", Int. Journal of Computer Vision, 9(2), 1992,

  • pp. 137-154.
  • pp. 137-154.

Shape from Shading Shape from Shading

  • B. Horn and M. Brooks, “Shape from Shading”, 1989, MIT Press.
  • B. Horn and M. Brooks, “Shape from Shading”, 1989, MIT Press.
  • L.
  • L. Wolff

Wolff, S. , S. Shafer Shafer, and G. E. , and G. E. Healey Healey, “Physics-Based Vision: Shape Recovery”, 1992, , “Physics-Based Vision: Shape Recovery”, 1992, Jones and Bartlett. Jones and Bartlett.

  • R. J.
  • R. J. Woodham

Woodham, “ , “Photometric Photometric Method for Determining Surface Orientation from Method for Determining Surface Orientation from Multiple Images”, Optical Engineering, 1980, Multiple Images”, Optical Engineering, 1980, pp

  • pp. 139-144.

. 139-144.

Bibliography Bibliography Video Video