From Images to Voxels From Images to Voxels Steve Steve Seitz - - PDF document

SLIDE 1

1

From Images to Voxels From Images to Voxels

Steve Steve Seitz Seitz Carnegie Mellon Carnegie Mellon University University

http http:// ://www www. .cs cs. .cmu cmu. .edu edu/~ /~seitz seitz

SIGGRAPH 99 Course on SIGGRAPH 99 Course on 3D Photography 3D Photography

3D Reconstruction from Calibrated Images 3D Reconstruction from Calibrated Images

• Goal:

Goal: Determine transparency, radiance of points in V

Determine transparency, radiance of points in V

SLIDE 2

2

Discrete Formulation: Discrete Formulation: Voxel Voxel Coloring Coloring

• Goal:

Goal: Assign RGBA values to voxels in V

Assign RGBA values to voxels in V photo-consistent photo-consistent with images with images

• S

S ∈ ∈ ℵ ℵ ⊂ ⊂ G G

• G

G = space of all colorings (C ) = space of all colorings (C ) ℵ ℵ = space of all photo-consistent colorings (computable?) = space of all photo-consistent colorings (computable?) S = true scene (not computable) S = true scene (not computable)

N N3

3

Complexity and Computability Complexity and Computability

SLIDE 3

3

• 1. C=2 (silhouettes)
• 1. C=2 (silhouettes)
• Volume intersection [Martin 81,

Volume intersection [Martin 81, Szeliski Szeliski 93] 93]

• 2. C unconstrained, viewpoint constraints
• 2. C unconstrained, viewpoint constraints
• Voxel

Voxel coloring algorithm [Seitz & Dyer 97] coloring algorithm [Seitz & Dyer 97]

• 3. General Case
• 3. General Case
• Space carving [Kutulakos & Seitz 98]

Space carving [Kutulakos & Seitz 98]

Voxel Voxel Coloring Solutions Coloring Solutions Reconstruction from Silhouettes (C = 2) Reconstruction from Silhouettes (C = 2)

• Approach:

Approach:

• Backproject

Backproject each silhouette each silhouette

• Intersect backprojected volumes

Intersect backprojected volumes

SLIDE 4

4

Volume Intersection Volume Intersection

Reconstruction Contains the True Scene Reconstruction Contains the True Scene

• But is generally not the same (no concavities)

But is generally not the same (no concavities)

• In the limit (all views) get

In the limit (all views) get visual hull visual hull or

• r line hull

line hull > > Complement of all lines that don’t intersect S Complement of all lines that don’t intersect S

Voxel Voxel Algorithm for Volume Intersection Algorithm for Volume Intersection

Color Color voxel voxel black if on silhouette in every image black if on silhouette in every image

• O(MN

O(MN3

3), for M images, N

), for M images, N3

3 voxels

voxels

• Don’t have to search 2

Don’t have to search 2N

N3 3 possible scenes!

possible scenes!

SLIDE 5

5

Properties of Volume Intersection Properties of Volume Intersection

Pros Pros

• Easy to implement, fast

Easy to implement, fast

• Accelerated via

Accelerated via octrees

• ctrees [

[Szeliski Szeliski 1993] 1993]

Cons Cons

• No concavities

No concavities

• Reconstruction is not photo-consistent

Reconstruction is not photo-consistent

• Requires identification of silhouettes

Requires identification of silhouettes

• 1. C=2 (silhouettes)
• 1. C=2 (silhouettes)
• Volume intersection [Martin 81,

Volume intersection [Martin 81, Szeliski Szeliski 93] 93]

• 2. C unconstrained, viewpoint constraints
• 2. C unconstrained, viewpoint constraints
• Voxel

Voxel coloring algorithm [Seitz & Dyer 97] coloring algorithm [Seitz & Dyer 97]

• 3. General Case
• 3. General Case
• Space carving [Kutulakos & Seitz 98]

Space carving [Kutulakos & Seitz 98]

Voxel Voxel Coloring Solutions Coloring Solutions

SLIDE 6

6

• Voxel Coloring Approach

Voxel Coloring Approach

Visibility Problem: Visibility Problem: in which images is each voxel visible?

in which images is each voxel visible?

The Global Visibility Problem The Global Visibility Problem

Inverse Visibility Inverse Visibility

known images known images

• Which points are visible in which images?

Which points are visible in which images?

• Forward Visibility

Forward Visibility

known scene known scene

SLIDE 7

7

• Depth Ordering: visit

Depth Ordering: visit occluders

• ccluders first!

first!

• Condition:

Condition: depth order is

depth order is view-independent view-independent

What is A What is A View-Independent View-Independent Depth Order? Depth Order?

A function A function f f over a scene S and a camera space C

• ver a scene S and a camera space C
• For example:

For example: f = distance from separating plane

f = distance from separating plane

• Such that

Such that for all p and q in S, v in C

for all p and q in S, v in C p occludes q from v p occludes q from v only if

• nly if

f(p) < f(q) f(p) < f(q)

SLIDE 8

8

Panoramic Depth Ordering Panoramic Depth Ordering

• Cameras oriented in many different directions

Cameras oriented in many different directions

• Planar depth ordering does not apply

Planar depth ordering does not apply

Panoramic Depth Ordering Panoramic Depth Ordering

Layers radiate outwards from cameras Layers radiate outwards from cameras

SLIDE 9

9

Panoramic Layering Panoramic Layering

Layers radiate outwards from cameras Layers radiate outwards from cameras

Panoramic Layering Panoramic Layering

Layers radiate outwards from cameras Layers radiate outwards from cameras

SLIDE 10

10

Compatible Camera Configurations Compatible Camera Configurations

Depth-Order Constraint Depth-Order Constraint

• Scene outside convex hull of camera centers

Scene outside convex hull of camera centers

Outward-Looking Outward-Looking

cameras inside scene cameras inside scene

Inward-Looking Inward-Looking

cameras above scene cameras above scene

Calibrated Image Acquisition Calibrated Image Acquisition

Calibrated Turntable Calibrated Turntable

360° rotation (21 images) 360° rotation (21 images) Selected Dinosaur Images Selected Dinosaur Images Selected Flower Images Selected Flower Images

SLIDE 11

11

Voxel Voxel Coloring Results (Video) Coloring Results (Video)

• Limitations of Depth Ordering

Limitations of Depth Ordering

A view-independent depth order may not exist A view-independent depth order may not exist

• Need more powerful general-case algorithms

Need more powerful general-case algorithms

• Unconstrained camera positions

Unconstrained camera positions

• Unconstrained scene geometry/topology

Unconstrained scene geometry/topology

SLIDE 12

12

A More Difficult Problem: Walkthrough A More Difficult Problem: Walkthrough

Input: calibrated images from arbitrary positions Input: calibrated images from arbitrary positions Output: 3D model photo-consistent with all images Output: 3D model photo-consistent with all images

• 1. C=2 (silhouettes)
• 1. C=2 (silhouettes)
• Volume intersection [Martin 81,

Volume intersection [Martin 81, Szeliski Szeliski 93] 93]

• 2. C unconstrained, viewpoint constraints
• 2. C unconstrained, viewpoint constraints
• Voxel

Voxel coloring algorithm [Seitz & Dyer 97] coloring algorithm [Seitz & Dyer 97]

• 3. General Case
• 3. General Case
• Space carving [Kutulakos & Seitz 98]

Space carving [Kutulakos & Seitz 98]

Voxel Voxel Coloring Solutions Coloring Solutions

SLIDE 13

13

Space Carving Algorithm Space Carving Algorithm

• Step 1:

Step 1: Initialize V to volume containing true scene Initialize V to volume containing true scene

• Step 2:

Step 2: For every For every voxel voxel on surface of V

• n surface of V

> > test test photo-consistency photo-consistency of

• f voxel

voxel > > if if voxel voxel is inconsistent, carve it is inconsistent, carve it

• Step 3:

Step 3: Repeat Step 2 until all Repeat Step 2 until all voxels voxels consistent consistent

Convergence: Convergence:

• Always

Always converges to a photo-consistent model converges to a photo-consistent model (when all assumptions are met) (when all assumptions are met)

• Good results on difficult real-world scenes

Good results on difficult real-world scenes

Visibility Property Visibility Property

p p ∈ ∈ S consistent S consistent ⇒ ⇒ p p ∈ ∈ consistent consistent p p ∈ ∈ inconsistent inconsistent ⇒ ⇒ p p ∈ ∈ S inconsistent S inconsistent This property ensures that carving converges This property ensures that carving converges

• 1

2 3 4

• 1

2 3 4

SLIDE 14

14

Space Carving Convergence Properties Space Carving Convergence Properties

Properties Properties

• Guaranteed convergence to photo-consistent

Guaranteed convergence to photo-consistent reconstruction (M) called the reconstruction (M) called the photo hull photo hull > > M = M = ∪ℵ ∪ℵ --- union of all photo-consistent scenes

• -- union of all photo-consistent scenes
• Tightest possible bound on true scene

Tightest possible bound on true scene

• Worst case #

Worst case # consistency checks: consistency checks: (# cameras)2(# voxels)

True Scene True Scene

• Reconstruction

Reconstruction

Multi-Pass Plane Sweep Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions

Sweep plane in each of 6 principle directions

• Consider cameras on only one side of plane

Consider cameras on only one side of plane

• Repeat until convergence

Repeat until convergence

True Scene Reconstruction

SLIDE 15

15

Multi-Pass Plane Sweep Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions

Sweep plane in each of 6 principle directions

• Consider cameras on only one side of plane

Consider cameras on only one side of plane

• Repeat until convergence

Repeat until convergence

Multi-Pass Plane Sweep Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions

Sweep plane in each of 6 principle directions

• Consider cameras on only one side of plane

Consider cameras on only one side of plane

• Repeat until convergence

Repeat until convergence

SLIDE 16

16

Multi-Pass Plane Sweep Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions

Sweep plane in each of 6 principle directions

• Consider cameras on only one side of plane

Consider cameras on only one side of plane

• Repeat until convergence

Repeat until convergence

Multi-Pass Plane Sweep Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions

Sweep plane in each of 6 principle directions

• Consider cameras on only one side of plane

Consider cameras on only one side of plane

• Repeat until convergence

Repeat until convergence

SLIDE 17

17

Space Carving Algorithm Space Carving Algorithm

Optimal algorithm is unwieldy Optimal algorithm is unwieldy

• Complex visibility update procedure

Complex visibility update procedure

Alternative: multi-pass plane sweep Alternative: multi-pass plane sweep

• Efficient, can use texture-mapping hardware

Efficient, can use texture-mapping hardware

• Converges quickly in practice

Converges quickly in practice

• Easy to implement

Easy to implement

Space Carving Results: African Violet Space Carving Results: African Violet

SLIDE 18

18

Space Carving Results: Hand Space Carving Results: Hand

• House

House Walkthrough Walkthrough

24 rendered input views from inside 24 rendered input views from inside and and outside

• utside

SLIDE 19

19

Space Carving Results: House Space Carving Results: House

• Space Carving Results: House

Space Carving Results: House

SLIDE 20

20

• Space Carving Results: House

Space Carving Results: House

• Other Features

Other Features

Coarse-to-fine Reconstruction Coarse-to-fine Reconstruction

• Represent scene as

Represent scene as octree

• ctree
• Reconstruct low-

Reconstruct low-res res model first, then refine model first, then refine

Hardware-Acceleration Hardware-Acceleration

• Use texture-mapping to compute

Use texture-mapping to compute voxel voxel projections projections

• Process

Process voxels voxels an entire plane at a time an entire plane at a time

Limitations Limitations

• Need to acquire calibrated images

Need to acquire calibrated images

• Restriction to simple radiance models

Restriction to simple radiance models

• Bias toward maximal (fat)

Bias toward maximal (fat) reconstructions reconstructions

• Transparency not supported

Transparency not supported

SLIDE 21

21

Other Approaches Other Approaches

Level-Set Methods Level-Set Methods [

[Faugeras Faugeras & & Keriven Keriven 1998] 1998]

• Evolve implicit function by solving

Evolve implicit function by solving PDE’s PDE’s

Transparency and Matting Transparency and Matting [

[Szeliski Szeliski & & Golland Golland 1998] 1998]

• Compute

Compute voxels voxels with alpha-channel with alpha-channel

Max Flow/Min Cut Max Flow/Min Cut [Roy &

[Roy & Cox Cox 1998] 1998]

• Graph theoretic formulation

Graph theoretic formulation

Mesh-Based Stereo Mesh-Based Stereo [

[Fua Fua & & Leclerc Leclerc 95] 95]

• Mesh-based but similar consistency formulation

Mesh-based but similar consistency formulation

Virtualized Virtualized Reality Reality [

[Narayan Narayan, , Rander Rander, , Kanade Kanade 1998] 1998]

• Perform stereo 3 images at a time, merge results

Perform stereo 3 images at a time, merge results

Advantages of Advantages of Voxels Voxels

• Non-

Non-parametric parametric > > can model arbitrary geometry can model arbitrary geometry > > can model arbitrary topology can model arbitrary topology

• Good reconstruction algorithms

Good reconstruction algorithms

• Good rendering algorithms (

Good rendering algorithms (splatting splatting, LDI) , LDI)

Disadvantages Disadvantages

• Expensive to process hi-

Expensive to process hi-res voxel res voxel grids grids

• Large number of parameters

Large number of parameters > > Simple scenes (e.g., planes) require lots of Simple scenes (e.g., planes) require lots of voxels voxels

• Meshes simplify better

Meshes simplify better

Conclusions Conclusions

SLIDE 22

22

Volume Intersection Volume Intersection

• Martin &

Martin & Aggarwal Aggarwal, “Volumetric description of objects from multiple , “Volumetric description of objects from multiple views”, views”, Trans

• Trans. Pattern Analysis and Machine Intelligence, 5(2), 1991,

. Pattern Analysis and Machine Intelligence, 5(2), 1991, pp

• pp. 150-158.

. 150-158.

• Szeliski

Szeliski, “Rapid , “Rapid Octree Octree Construction from Image Sequences”, Construction from Image Sequences”, Computer Vision, Graphics, and Image Processing: Image Computer Vision, Graphics, and Image Processing: Image Understanding, 58(1), 1993, Understanding, 58(1), 1993, pp

• pp. 23-32.

. 23-32.

Voxel Voxel Coloring and Space Carving Coloring and Space Carving

• Seitz & Dyer, “

Seitz & Dyer, “Photorealistic Photorealistic Scene Reconstruction by Scene Reconstruction by Voxel Voxel Coloring”, Coloring”, Proc

• Proc. Computer Vision and Pattern Recognition (CVPR),

. Computer Vision and Pattern Recognition (CVPR), 1997, 1997, pp

• pp. 1067-1073.

. 1067-1073.

• Seitz & Kutulakos, “

Seitz & Kutulakos, “Plenoptic Plenoptic Image Editing”, Image Editing”, Proc Proc. . Int Int. . Conf

• Conf. on

. on Computer Vision (ICCV), 1998, Computer Vision (ICCV), 1998, pp

• pp. 17-24.

. 17-24.

• Kutulakos & Seitz, “A Theory of Shape by Space Carving”, U.

Kutulakos & Seitz, “A Theory of Shape by Space Carving”, U. Rochester C.S. Rochester C.S. Dept

• Dept. TR #692, May 1998, to appear in

. TR #692, May 1998, to appear in Proc

• Proc. ICCV 99.

. ICCV 99.

Bibliography Bibliography

Related References Related References

• Faugeras

Faugeras & & Keriven Keriven, “ , “Variational Variational principles, surface evolution, principles, surface evolution, PDE's PDE's, , level set methods and the stereo problem", IEEE level set methods and the stereo problem", IEEE Trans

• Trans. on Image

. on Image Processing, 7(3), 1998, Processing, 7(3), 1998, pp

• pp. 336-344.

. 336-344.

• Szeliski

Szeliski & & Golland Golland, “Stereo Matching with Transparency and Matting”, , “Stereo Matching with Transparency and Matting”, Proc Proc. . Int Int. . Conf

• Conf. on Computer Vision (ICCV), 1998, 517-524.

. on Computer Vision (ICCV), 1998, 517-524.

• Roy &

Roy & Cox Cox, “A Maximum-Flow Formulation of the N-camera Stereo , “A Maximum-Flow Formulation of the N-camera Stereo Correspondence Problem”, Correspondence Problem”, Proc

• Proc. ICCV, 1998,

. ICCV, 1998, pp

• pp. 492-499.

. 492-499.

• Fua

Fua & & Leclerc Leclerc, “Object-centered surface reconstruction: Combining , “Object-centered surface reconstruction: Combining multi multi-image stereo and shading",

• image stereo and shading", Int
• Int. Journal of Computer Vision, 16,

. Journal of Computer Vision, 16, 1995, 1995, pp

• pp. 35-56.

. 35-56.

• Narayanan

Narayanan, , Rander Rander, & , & Kanade Kanade, “Constructing Virtual Worlds Using , “Constructing Virtual Worlds Using Dense Stereo”, Dense Stereo”, Proc

• Proc. ICCV, 1998,

. ICCV, 1998, pp

• pp. 3-10.

. 3-10.

Bibliography Bibliography