From Images to Voxels From Images to Voxels Steve Seitz Steve - - PDF document

▶
from images to voxels from images to voxels
SMART_READER_LITE
LIVE PREVIEW

From Images to Voxels From Images to Voxels Steve Seitz Steve - - PDF document

Jun 07, 2023 339 likes •582 views

SIGGRAPH 2000 Course on SIGGRAPH 2000 Course on 3D Photography 3D Photography From Images to Voxels From Images to Voxels Steve Seitz Steve Seitz Carnegie Mellon University Carnegie Mellon University University of Washington University of

slide-1
SLIDE 1

1

From Images to Voxels From Images to Voxels

Steve Seitz Steve Seitz Carnegie Mellon University Carnegie Mellon University University of Washington University of Washington

http://www. http://www.cs cs. .cmu cmu. .edu edu/~ /~seitz seitz

SIGGRAPH 2000 Course on SIGGRAPH 2000 Course on 3D Photography 3D Photography

3D Reconstruction from Calibrated Images 3D Reconstruction from Calibrated Images

6FHQH9ROXPH 6FHQH9ROXPH 9 9 ,QSXW,PDJHV ,QSXW,PDJHV &DOLEUDWHG &DOLEUDWHG

Goal: Goal: Determine transparency, radiance of points in V

Determine transparency, radiance of points in V

slide-2
SLIDE 2

2

Discrete Formulation: Voxel Coloring Discrete Formulation: Voxel Coloring

'LVFUHWL]HG 'LVFUHWL]HG 6FHQH9ROXPH 6FHQH9ROXPH ,QSXW,PDJHV ,QSXW,PDJHV &DOLEUDWHG &DOLEUDWHG

Goal: Goal: Assign RGBA values to voxels in V

Assign RGBA values to voxels in V photo photo-

  • consistent

consistent with images with images

Complexity and Computability Complexity and Computability

'LVFUHWL]HG 'LVFUHWL]HG 6FHQH9ROXPH 6FHQH9ROXPH N voxels N voxels C colors C colors

3 3

$OO6FHQHVCN3 3KRWR&RQVLVWHQW 6FHQHV 7UXH 6FHQH

slide-3
SLIDE 3

3

Issues Issues

Theoretical Questions Theoretical Questions

  • Identify class of

Identify class of all all photo photo-

  • consistent scenes

consistent scenes

Practical Questions Practical Questions

  • How do we compute photo

How do we compute photo-

  • consistent models?

consistent models?

  • 1. C=2 (silhouettes)
  • 1. C=2 (silhouettes)
  • Volume intersection [Martin 81,

Volume intersection [Martin 81, Szeliski Szeliski 93] 93]

  • 2. C unconstrained, viewpoint constraints
  • 2. C unconstrained, viewpoint constraints
  • Voxel coloring algorithm [Seitz & Dyer 97]

Voxel coloring algorithm [Seitz & Dyer 97]

  • 3. General Case
  • 3. General Case
  • Space carving [Kutulakos & Seitz 98]

Space carving [Kutulakos & Seitz 98]

Voxel Coloring Solutions Voxel Coloring Solutions

slide-4
SLIDE 4

4

Reconstruction from Silhouettes (C = 2) Reconstruction from Silhouettes (C = 2)

%LQDU\ %LQDU\ ,PDJHV ,PDJHV

Approach: Approach:

  • Backproject

Backproject each silhouette each silhouette

  • Intersect backprojected volumes

Intersect backprojected volumes

Volume Intersection Volume Intersection

Reconstruction Contains the True Scene Reconstruction Contains the True Scene

  • But is generally not the same

But is generally not the same

  • No concavities

No concavities

  • In the limit get

In the limit get visual hull visual hull

slide-5
SLIDE 5

5

Voxel Algorithm for Volume Intersection Voxel Algorithm for Volume Intersection

Color voxel black if on silhouette in every image Color voxel black if on silhouette in every image

  • O(MN

O(MN3

3), for M images, N

), for M images, N3

3 voxels

voxels

  • Don’t have to search 2

Don’t have to search 2N

N3 3 possible scenes!

possible scenes!

Properties of Volume Intersection Properties of Volume Intersection

Pros Pros

  • Easy to implement, fast

Easy to implement, fast

  • Accelerated via

Accelerated via octrees

  • ctrees [

[Szeliski Szeliski 1993] 1993]

Cons Cons

  • No concavities

No concavities

  • Reconstruction is not photo

Reconstruction is not photo-

  • consistent

consistent

  • Requires identification of silhouettes

Requires identification of silhouettes

slide-6
SLIDE 6

6

  • 1. C=2 (silhouettes)
  • 1. C=2 (silhouettes)
  • Volume intersection [Martin 81,

Volume intersection [Martin 81, Szeliski Szeliski 93] 93]

  • 2. C unconstrained, viewpoint constraints
  • 2. C unconstrained, viewpoint constraints
  • Voxel coloring algorithm [Seitz & Dyer 97]

Voxel coloring algorithm [Seitz & Dyer 97]

  • 3. General Case
  • 3. General Case
  • Space carving [Kutulakos & Seitz 98]

Space carving [Kutulakos & Seitz 98]

Voxel Coloring Solutions Voxel Coloring Solutions

&KRRVHYR[HO &KRRVHYR[HO 3URMHFWDQGFRUUHODWH 3URMHFWDQGFRUUHODWH &RORULIFRQVLVWHQW &RORULIFRQVLVWHQW

Voxel Coloring Approach Voxel Coloring Approach

Visibility Problem: Visibility Problem: in which images is each voxel visible?

in which images is each voxel visible?

slide-7
SLIDE 7

7

The Global Visibility Problem The Global Visibility Problem

Inverse Visibility Inverse Visibility

known images known images 8QNQRZQ6FHQH 8QNQRZQ6FHQH

Which Points are Visible in Which Images? Which Points are Visible in Which Images?

.QRZQ6FHQH .QRZQ6FHQH

Forward Visibility Forward Visibility

known scene known scene /D\HUV /D\HUV

Depth Ordering: Visit Depth Ordering: Visit Occluders Occluders First! First!

6FHQH 6FHQH 7UDYHUVDO 7UDYHUVDO

Condition: Condition: depth order is

depth order is view view-

  • independent

independent

slide-8
SLIDE 8

8

Panoramic Depth Ordering Panoramic Depth Ordering

  • Cameras oriented in many different directions

Cameras oriented in many different directions

  • Planar depth ordering does not apply

Planar depth ordering does not apply

Panoramic Depth Ordering Panoramic Depth Ordering

Layers radiate outwards from cameras Layers radiate outwards from cameras

slide-9
SLIDE 9

9

Panoramic Layering Panoramic Layering

Layers radiate outwards from cameras Layers radiate outwards from cameras

Panoramic Layering Panoramic Layering

Layers radiate outwards from cameras Layers radiate outwards from cameras

slide-10
SLIDE 10

10

Compatible Camera Configurations Compatible Camera Configurations

Depth Depth-

  • Order Constraint

Order Constraint

  • Scene outside convex hull of camera centers

Scene outside convex hull of camera centers

Outward Outward-

  • Looking

Looking

cameras inside scene cameras inside scene

Inward Inward-

  • Looking

Looking

cameras above scene cameras above scene

Calibrated Image Acquisition Calibrated Image Acquisition

Calibrated Turntable Calibrated Turntable

360° rotation (21 images) 360° rotation (21 images) Selected Dinosaur Images Selected Dinosaur Images Selected Flower Images Selected Flower Images

slide-11
SLIDE 11

11

Voxel Coloring Results (Video) Voxel Coloring Results (Video)

'LQRVDXU5HFRQVWUXFWLRQ 'LQRVDXU5HFRQVWUXFWLRQ

. .YR[HOVFRORUHG YR[HOVFRORUHG 0YR[HOVWHVWHG YR[HOVWHVWHG PLQ PLQWRFRPSXWH WRFRPSXWH RQD0+]6*, RQD0+]6*,

)ORZHU5HFRQVWUXFWLRQ )ORZHU5HFRQVWUXFWLRQ

. .YR[HOVFRORUHG YR[HOVFRORUHG 0YR[HOVWHVWHG YR[HOVWHVWHG PLQ PLQWRFRPSXWH WRFRPSXWH RQD0+]6*, RQD0+]6*,

Limitations of Depth Ordering Limitations of Depth Ordering

A View A View-

  • Independent Depth Order May Not Exist

Independent Depth Order May Not Exist

S T

Need More Powerful General Need More Powerful General-

  • Case Algorithms

Case Algorithms

  • Unconstrained camera positions

Unconstrained camera positions

  • Unconstrained scene geometry/topology

Unconstrained scene geometry/topology

slide-12
SLIDE 12

12

A More Difficult Problem: Walkthrough A More Difficult Problem: Walkthrough

Input: calibrated images from arbitrary positions Input: calibrated images from arbitrary positions Output: 3D model photo Output: 3D model photo-

  • consistent with all images

consistent with all images WUHH ZLQGRZ

  • 1. C=2 (silhouettes)
  • 1. C=2 (silhouettes)
  • Volume intersection [Martin 81,

Volume intersection [Martin 81, Szeliski Szeliski 93] 93]

  • 2. C unconstrained, viewpoint constraints
  • 2. C unconstrained, viewpoint constraints
  • Voxel coloring algorithm [Seitz & Dyer 97]

Voxel coloring algorithm [Seitz & Dyer 97]

  • 3. General Case
  • 3. General Case
  • Space carving [Kutulakos & Seitz 98]

Space carving [Kutulakos & Seitz 98]

Voxel Coloring Solutions Voxel Coloring Solutions

slide-13
SLIDE 13

13

Space Carving Algorithm Space Carving Algorithm

Space Carving Algorithm Space Carving Algorithm

Image 1 Image N

…...

  • Initialize to a volume V containing the true scene

Initialize to a volume V containing the true scene

  • Repeat until convergence

Repeat until convergence

  • Choose a voxel on the current surface

Choose a voxel on the current surface

  • Carve if not photo

Carve if not photo-

  • consistent

consistent

  • Project to visible input images

Project to visible input images

Convergence Convergence

Consistency Property Consistency Property

  • The resulting shape is photo

The resulting shape is photo-

  • consistent

consistent > > all inconsistent voxels are removed all inconsistent voxels are removed

Convergence Property Convergence Property

  • Carving converges to a non

Carving converges to a non-

  • empty shape

empty shape > > a point on the true scene is a point on the true scene is never never removed removed

V’ V

p

slide-14
SLIDE 14

14

Output of Space Carving: Photo Hull Output of Space Carving: Photo Hull

The The Photo Hull Photo Hull is the UNION of all photo is the UNION of all photo-

  • consistent scenes in V

consistent scenes in V

  • Tightest possible bound on the true scene

Tightest possible bound on the true scene

True Scene True Scene V V Photo Hull Photo Hull V V

Space Carving Algorithm Space Carving Algorithm

The Basic Algorithm is Unwieldy The Basic Algorithm is Unwieldy

  • Complex update procedure

Complex update procedure

Alternative: Multi Alternative: Multi-

  • Pass Plane Sweep

Pass Plane Sweep

  • Efficient, can use texture

Efficient, can use texture-

  • mapping hardware

mapping hardware

  • Converges quickly in practice

Converges quickly in practice

  • Easy to implement

Easy to implement

Results Related Methods

slide-15
SLIDE 15

15

Multi Multi-

  • Pass Plane Sweep

Pass Plane Sweep

  • Sweep plane in each of 6 principle directions

Sweep plane in each of 6 principle directions

  • Consider cameras on only one side of plane

Consider cameras on only one side of plane

  • Repeat until convergence

Repeat until convergence

True Scene Reconstruction

Multi Multi-

  • Pass Plane Sweep

Pass Plane Sweep

  • Sweep plane in each of 6 principle directions

Sweep plane in each of 6 principle directions

  • Consider cameras on only one side of plane

Consider cameras on only one side of plane

  • Repeat until convergence

Repeat until convergence

slide-16
SLIDE 16

16

Multi Multi-

  • Pass Plane Sweep

Pass Plane Sweep

  • Sweep plane in each of 6 principle directions

Sweep plane in each of 6 principle directions

  • Consider cameras on only one side of plane

Consider cameras on only one side of plane

  • Repeat until convergence

Repeat until convergence

Multi Multi-

  • Pass Plane Sweep

Pass Plane Sweep

  • Sweep plane in each of 6 principle directions

Sweep plane in each of 6 principle directions

  • Consider cameras on only one side of plane

Consider cameras on only one side of plane

  • Repeat until convergence

Repeat until convergence

slide-17
SLIDE 17

17

Multi Multi-

  • Pass Plane Sweep

Pass Plane Sweep

  • Sweep plane in each of 6 principle directions

Sweep plane in each of 6 principle directions

  • Consider cameras on only one side of plane

Consider cameras on only one side of plane

  • Repeat until convergence

Repeat until convergence

Multi Multi-

  • Pass Plane Sweep

Pass Plane Sweep

  • Sweep plane in each of 6 principle directions

Sweep plane in each of 6 principle directions

  • Consider cameras on only one side of plane

Consider cameras on only one side of plane

  • Repeat until convergence

Repeat until convergence

slide-18
SLIDE 18

18

Multi Multi-

  • Pass Plane Sweep

Pass Plane Sweep

  • Sweep plane in each of 6 principle directions

Sweep plane in each of 6 principle directions

  • Consider cameras on only one side of plane

Consider cameras on only one side of plane

  • Repeat until convergence

Repeat until convergence

Space Carving Results: African Violet Space Carving Results: African Violet

,QSXW,PDJHRI ,QSXW,PDJHRI 5HFRQVWUXFWLRQ 5HFRQVWUXFWLRQ 5HFRQVWUXFWLRQ 5HFRQVWUXFWLRQ 5HFRQVWUXFWLRQ 5HFRQVWUXFWLRQ

slide-19
SLIDE 19

19

Space Carving Results: Hand Space Carving Results: Hand

,QSXW,PDJH ,QSXW,PDJH RI RI 9LHZVRI5HFRQVWUXFWLRQ 9LHZVRI5HFRQVWUXFWLRQ

House Walkthrough House Walkthrough

24 rendered input views from inside 24 rendered input views from inside and and outside

  • utside
slide-20
SLIDE 20

20

Space Carving Results: House Space Carving Results: House

,QSXW,PDJH ,QSXW,PDJH

WUXHVFHQH WUXHVFHQH

5HFRQVWUXFWLRQ 5HFRQVWUXFWLRQ

YR[HOV YR[HOV

Space Carving Results: House Space Carving Results: House

,QSXW,PDJH ,QSXW,PDJH

WUXHVFHQH WUXHVFHQH

5HFRQVWUXFWLRQ 5HFRQVWUXFWLRQ

YR[HOV YR[HOV

slide-21
SLIDE 21

21

1HZ9LHZWUXHVFHQH 1HZ9LHZWUXHVFHQH

Space Carving Results: House Space Carving Results: House

5HFRQVWUXFWLRQ 5HFRQVWUXFWLRQ 1HZ9LHZ 1HZ9LHZ

WUXHVFHQH WUXHVFHQH

5HFRQVWUXFWLRQ 5HFRQVWUXFWLRQ 5HFRQVWUXFWLRQ 5HFRQVWUXFWLRQ

ZLWKQHZLQSXWYLHZ ZLWKQHZLQSXWYLHZ

Other Approaches Other Approaches

Level Level-

  • Set Methods

Set Methods [Faugeras &

[Faugeras & Keriven Keriven 1998] 1998]

  • Evolve implicit function by solving

Evolve implicit function by solving PDE’s PDE’s

Transparency and Matting Transparency and Matting [

[Szeliski Szeliski & & Golland Golland 1998] 1998]

  • Compute voxels with alpha

Compute voxels with alpha-

  • channel

channel

Max Flow/Min Cut Max Flow/Min Cut [Roy & Cox 1998]

[Roy & Cox 1998]

  • Graph theoretic formulation

Graph theoretic formulation

Mesh Mesh-

  • Based Stereo

Based Stereo [Fua &

[Fua & Leclerc Leclerc 95] 95]

  • Mesh

Mesh-

  • based but similar consistency formulation

based but similar consistency formulation

Virtualized Reality Virtualized Reality [

[Narayan Narayan, , Rander Rander, , Kanade Kanade 1998] 1998]

  • Perform stereo 3 images at a time, merge results

Perform stereo 3 images at a time, merge results

slide-22
SLIDE 22

22

Advantages of Voxels Advantages of Voxels

  • Non

Non-

  • parametric

parametric > > can model arbitrary geometry can model arbitrary geometry > > can model arbitrary topology can model arbitrary topology

  • Good reconstruction algorithms

Good reconstruction algorithms

  • Good rendering algorithms (

Good rendering algorithms (splatting splatting, LDI) , LDI)

Disadvantages Disadvantages

  • Expensive to process hi

Expensive to process hi-

  • res

res voxel grids voxel grids

  • Large number of parameters

Large number of parameters > > Simple scenes (e.g., planes) require lots of voxels Simple scenes (e.g., planes) require lots of voxels

Conclusions Conclusions

Volume Intersection Volume Intersection

  • Martin &

Martin & Aggarwal Aggarwal, “Volumetric description of objects from multiple views”, , “Volumetric description of objects from multiple views”,

  • Trans. Pattern Analysis and Machine Intelligence, 5(2), 1991, p
  • Trans. Pattern Analysis and Machine Intelligence, 5(2), 1991, pp. 150
  • p. 150-
  • 158.

158.

  • Szeliski

Szeliski, “Rapid , “Rapid Octree Octree Construction from Image Sequences”, Computer Vision, Construction from Image Sequences”, Computer Vision, Graphics, and Image Processing: Image Understanding, 58(1), 1993 Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23 , pp. 23-

  • 32.

32.

Voxel Coloring and Space Carving Voxel Coloring and Space Carving

  • Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Colo

Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Proc. ring”, Proc. Computer Vision and Pattern Recognition (CVPR), 1997, pp. 1067 Computer Vision and Pattern Recognition (CVPR), 1997, pp. 1067-

  • 1073.

1073.

  • Seitz & Kutulakos, “Plenoptic Image Editing”, Proc. Int. Conf.

Seitz & Kutulakos, “Plenoptic Image Editing”, Proc. Int. Conf. on Computer

  • n Computer

Vision (ICCV), 1998, pp. 17 Vision (ICCV), 1998, pp. 17-

  • 24.

24.

  • Kutulakos & Seitz, “A Theory of Shape by Space Carving”, Proc.

Kutulakos & Seitz, “A Theory of Shape by Space Carving”, Proc. ICCV, 1998, pp. ICCV, 1998, pp. 307 307-

  • 314.

314.

Bibliography Bibliography

slide-23
SLIDE 23

23

Related References Related References

  • Faugeras &

Faugeras & Keriven Keriven, “ , “Variational Variational principles, surface evolution, principles, surface evolution, PDE's PDE's, level set , level set methods and the stereo problem", IEEE Trans. on Image Processing methods and the stereo problem", IEEE Trans. on Image Processing, 7(3), 1998, , 7(3), 1998,

  • pp. 336
  • pp. 336-
  • 344.

344.

  • Szeliski

Szeliski & & Golland Golland, “Stereo Matching with Transparency and Matting”, Proc. Int. , “Stereo Matching with Transparency and Matting”, Proc. Int.

  • Conf. on Computer Vision (ICCV), 1998, 517
  • Conf. on Computer Vision (ICCV), 1998, 517-
  • 524.

524.

  • Roy & Cox, “A Maximum

Roy & Cox, “A Maximum-

  • Flow Formulation of the N

Flow Formulation of the N-

  • camera Stereo

camera Stereo Correspondence Problem”, Proc. ICCV, 1998, pp. 492 Correspondence Problem”, Proc. ICCV, 1998, pp. 492-

  • 499.

499.

  • Fua &

Fua & Leclerc Leclerc, “Object , “Object-

  • centered surface reconstruction: Combining multi

centered surface reconstruction: Combining multi-

  • image

image stereo and shading", Int. Journal of Computer Vision, 16, 1995, stereo and shading", Int. Journal of Computer Vision, 16, 1995, pp. 35

  • pp. 35-
  • 56.

56.

  • Narayanan,

Narayanan, Rander Rander, & , & Kanade Kanade, “Constructing Virtual Worlds Using Dense , “Constructing Virtual Worlds Using Dense Stereo”, Proc. ICCV, 1998, pp. 3 Stereo”, Proc. ICCV, 1998, pp. 3-

  • 10.

10.

Bibliography Bibliography