Stereo Geometry of Non-Central Cameras Tom a s Pajdla Center for - - PowerPoint PPT Presentation
Stereo Geometry of Non-Central Cameras Tom a s Pajdla Center for Machine Perception Department of Cybernetics Faculty of Electrical Engineering Czech Technical University in Prague Karlovo n am. 13, 12135 Prague
Stereo Geometry of Non-Central Cameras
Tom´ aˇ s Pajdla Center for Machine Perception Department of Cybernetics Faculty of Electrical Engineering Czech Technical University in Prague Karlovo n´
pajdla@cmp.felk.cvut.cz
2/47
Classical Cameras
3/47
Classical camera = Pinhole camera = central camera
Pinhole camera
Pinhole
Geometrical model
X C
There is a projection center
4/47
3D Scene Reconstruction from Images
an important task in Computer Vision
3D Model
4/47
3D Scene Reconstruction from Images
an important task in Computer Vision
3D Model After 10 years of research . . . developed and understood for central cameras
4/47
3D Scene Reconstruction from Images
an important task in Computer Vision
3D Model After 10 years of research . . . developed and understood for central cameras Conclusion − →
4/47
3D Scene Reconstruction from Images
an important task in Computer Vision
3D Model After 10 years of research . . . developed and understood for central cameras Conclusion − → geometry of cameras needs to be understood
5/47
Interesting Problem Geometry of non-central cameras and how to use them for the reconstruction
6/47
Outline
(a) Examples (b) Previous work
(a) Stereo geometry of non-central cameras (b) Oblique camera
7/47
8/47
Non-central cameras
Space is projected to images along more general arrangements of lines called non-central cameras Central camera Non-central camera a set of rays (just) a set of rays incident with one point
9/47
Example: Circular panoramas
T.Pajdla, H.Bakstein, D.Veˇ cerka, ‘Botanical garden’ 2002
10/47
Example: Circular stereo panoramas
use red/cyan glasses for left/right eye
10/47
Example: Circular stereo panoramas
use red/cyan glasses for left/right eye
Non-central cameras can be used for reconstruction Advantages: , , . . .
10/47
Example: Circular stereo panoramas
use red/cyan glasses for left/right eye
Non-central cameras can be used for reconstruction Advantages: large view field, , . . .
10/47
Example: Circular stereo panoramas
use red/cyan glasses for left/right eye
Non-central cameras can be used for reconstruction Advantages: large view field, higher precision, . . .
10/47
Example: Circular stereo panoramas
use red/cyan glasses for left/right eye
Non-central cameras can be used for reconstruction Advantages: large view field, higher precision, interesting . . .
11/47
Other examples
Peripheral image Pushbroom image
Shmuel Peleg, 2001
12/47
13/47
Previous work: Non-perspective, maybe non-central
Non-perspective, maybe even non-central images, appeared in art at the same time with perspective images.
Perspective painting Non-perspective (non-central?) by Massacio, 1426 painting by Jan Van Eyck, 1434
14/47
Previous work on non-central cameras
1434 Jan Van Eyck painted ‘Trinita’. 1843 Joseph Puchberger patented the ‘slit camera’ (similar to pushbroom camera).
14/47
Previous work on non-central cameras
1434 Jan Van Eyck painted ‘Trinita’. 1843 Joseph Puchberger patented the ‘slit camera’ (similar to pushbroom camera). 1990 – ... Non-central cameras used in mosaicing (Ishiguro et. al 1992, Peleg et. al 1999, Shum et. al 1999, Huang et. al 2000, Nayar & Karmarkar 2000), reconstruction (Gupta & Hartley 1997), visualization (McMillan et. al 1995, Gortler et. al 1996, Levoy et. al 1996, Rademacher et. al 1998, Weinshall et. al 2002), ...
14/47
Previous work on non-central cameras
1434 Jan Van Eyck painted ‘Trinita’. 1843 Joseph Puchberger patented the ‘slit camera’ (similar to pushbroom camera). 1990 – ... Non-central cameras used in mosaicing (Ishiguro et. al 1992, Peleg et. al 1999, Shum et. al 1999, Huang et. al 2000, Nayar & Karmarkar 2000), reconstruction (Gupta & Hartley 1997), visualization (McMillan et. al 1995, Gortler et. al 1996, Levoy et. al 1996, Rademacher et. al 1998, Weinshall et. al 2002), ... Isolated case studies, no general theory.
14/47
Previous work on non-central cameras
1434 Jan Van Eyck painted ‘Trinita’. 1843 Joseph Puchberger patented the ‘slit camera’ (similar to pushbroom camera). 1990 – ... Non-central cameras used in mosaicing (Ishiguro et. al 1992, Peleg et. al 1999, Shum et. al 1999, Huang et. al 2000, Nayar & Karmarkar 2000), reconstruction (Gupta & Hartley 1997), visualization (McMillan et. al 1995, Gortler et. al 1996, Levoy et. al 1996, Rademacher et. al 1998, Weinshall et. al 2002), ... Isolated case studies, no general theory. Feb 2001 T.Pajdla presented the generalization of epipolar planes to epipolar surfaces and their characterization at the Computer Winter Workshop’2001.
14/47
Previous work on non-central cameras
1434 Jan Van Eyck painted ‘Trinita’. 1843 Joseph Puchberger patented the ‘slit camera’ (similar to pushbroom camera). 1990 – ... Non-central cameras used in mosaicing (Ishiguro et. al 1992, Peleg et. al 1999, Shum et. al 1999, Huang et. al 2000, Nayar & Karmarkar 2000), reconstruction (Gupta & Hartley 1997), visualization (McMillan et. al 1995, Gortler et. al 1996, Levoy et. al 1996, Rademacher et. al 1998, Weinshall et. al 2002), ... Isolated case studies, no general theory. Feb 2001 T.Pajdla presented the generalization of epipolar planes to epipolar surfaces and their characterization at the Computer Winter Workshop’2001. Jun 2001 S.Seitz received the Marr prize for the same idea and its applications at ICCV’2001.
15/47
Main Results
16/47
Central cameras − → epipolar geometry
epipolar gemetry by O.Chum and J.Matas
→ constraints
→ easier search
17/47
Non-central cameras
?
− → stereo geometry
17/47
Non-central cameras
?
− → stereo geometry
Epipolar lines − → stereo correspondence curves left image right image Every point in space that projects on the curve in the left image projects
17/47
Non-central cameras
?
− → stereo geometry
Epipolar lines − → stereo correspondence curves left image right image Every point in space that projects on the curve in the left image projects
Stereo correspondence curves = ⇒ stereo reconstruction with non-central cameras similar to stereo reconstruction with central cameras
18/47
Stereo geometry of non-central cameras
to have corresponding epipolar curves Epipolar planes − → Stereo correspondence surfaces
18/47
Stereo geometry of non-central cameras
to have corresponding epipolar curves Epipolar planes − → Stereo correspondence surfaces Result: Interesting stereo correcpondence surfaces are double ruled quadrics epipolar plane hyperbolic paraboloid hyperboloid of one sheet
19/47
There are many stereo geometries
Double ruled quadrics can be arranged in space in many different ways Examples Pushbroom camera Stereo panorama
(Gupta & Hartley 1997) (Shum et. al 1999, Nayar & Karmarkar 2000)
l2 l1
l c
two intersecting lines circle the situation can be somewhat complicated in general − → current research
19/47
There are many stereo geometries
Double ruled quadrics can be arranged in space in many different ways Examples Pushbroom camera Stereo panorama
(Gupta & Hartley 1997) (Shum et. al 1999, Nayar & Karmarkar 2000)
l2 l1
l c
two intersecting lines circle the situation can be somewhat complicated in general − → current research Interesting: Epilinear geometries
20/47
Epilinear geometries
21/47
X-Slits Cameras
X-Slits camera (Weinshall et al. ECCV 2002) projects space onto a plane along along the set of all lines that meet two skew focal lines (i.e. along a hyperbolic congruence)
l2 l1 X
22/47
X-Slits Cameras by Sampling Image Volumes
Moving central camera Virtual slits Image volume
One sampling function → ones slit X-Slits Image
23/47
X-Slits Cameras generalize Pushbroom Cameras
π
8
π
8
Pushbroom ≡ one slit at π∞ X-Slits ≡ general slits
24/47
X-Slits Stereo Panoramas Posses Epilinear Stereo Geometry
One slit is shared
25/47
X-Slits Stereo Panorama
Anaglyph of a stereo panorama taken by an X-Slits (generalized pushbroom) camera. (D. Feldman, D. Weinshall, T. Pajdla) Stereo correspondence curves are horizontal lines ⇒ correct perception of stereo
26/47
Application: Visualization with X-Slits Cameras
Original sequence acquired by a perspective camera along a circle
27/47
Application: Visualization with X-Slits Cameras
Synthesized sequence as if taken by a (non-central) camera inside the circle
28/47
Application: Visualization with X-Slits Cameras
Original sequence acquired by a central omni-camera along a circle
29/47
Application: Visualization with X-Slits Cameras
Synthesized sequence as if taken by a (non-central) camera inside the circle
30/47
Application: Visualization with X-Slits Cameras
Synthesized sequence as if taken by a (non-central) camera inside the circle
31/47
Main Results
32/47
Hierarchy of cameras
central camera all other cameras ? all rays intersect at C − → some rays intersect ← − no rays intersect
X C
l c
X
l2 l1
33/47
Oblique camera
X
Definition An oblique camera is a collection of lines such that every point in space is contained in exactly one line.
33/47
Oblique camera
X
Definition An oblique camera is a collection of lines such that every point in space is contained in exactly one line. Observation Rays of an oblique camera do not intersect.
33/47
Oblique camera
X
Definition An oblique camera is a collection of lines such that every point in space is contained in exactly one line. Observation Rays of an oblique camera do not intersect.
Do oblique cameras exist
34/47
Oblique cameras exist
picture by Rolf Riesinger
A set of lines generated by the linnear mapping σ point X line[X σ(X)] span x y z w − → span x −y y x z w w −z
picture by Hans Havlicek
Lines are reguli of pairwise non-intersecting rotational hyperboloids
XT s s s − 1 s − 1 X = 0, s ∈ [0, 1] Remark: OC are called spreads & wild spreads (not cospreads) exist!
35/47
Stereo geometry of oblique cameras
l k
A set of lines generated by the mappings: point C1 = red lines C2 = blue lines x y z w − → x −y y x z w w −z x −y y x z −w w z form two reguli of pairwise non-intersecting rotational hyperboloids + two lines l, k Remark: lines l (s = 1) and k (s = 0) are in both cameras C1 and C2.
36/47
Linear OC ≡ generated by collineations over P3
Oblique cameras can be generated by a linear mapping σ : P3 → P3 as { line [X σ(X)] | ∀X ∈ P3} for some mappings (Y) Y (X) σ σ σ σ X Result: linear mappings that generate OC
σ : Y = QX, Q ∈ R4×4 change
coordinates: ∃S ∈ R4×4, rank S = 4, ∃α ∈ R, α = 0 S−1QS = α 1 −1 α α 1 −1 α
36/47
Linear OC ≡ generated by collineations over P3
Oblique cameras can be generated by a linear mapping σ : P3 → P3 as { line [X σ(X)] | ∀X ∈ P3} for some mappings (Y) Y (X) σ σ σ σ X Result: linear mappings that generate OC
σ : Y = QX, Q ∈ R4×4 change
coordinates: ∃S ∈ R4×4, rank S = 4, ∃α ∈ R, α = 0 S−1QS = α 1 −1 α α 1 −1 α
Observation: Central, Pushbroom, X-Slits, and some Oblique cameras are linearly generated
37/47
Realization of oblique cameras
l r C
l1 l4 l3 l2 s
38/47
Realization of oblique cameras
l
l2 l1 c k d l
e
C b a
l r C
l1 l4 l3 l2 s
By storing two curves of pixels from each image, rays passing through the volume swept by the mirror are generated.
39/47
Summary
Non-central cameras and their stereo geometries were introduced. In particular, I have:
non-central cameras are practically important and that there is a need to understand their stereo geometries.
cameras consisting of quadratic stereo correspondence surfaces and shown that the generalization supports an efficient scene reconstruction.
explains stereo geometry of symmetric concentric panoramas.
geometry.
40/47
Publications
[Paj99]
[HPa00] F. Huang and T. Pajdla. Epipolar geometry in concentric panoramas. Research Report CTU–CMP–2000–07, March 2000. [Paj01a] T. Pajdla. Characterization of epipolar geometries of non-classical cameras. Research Report CTU–CMP–2001–05, February 2001. [Paj01b] T. Pajdla. Epipolar geometry of some non-classical cameras. In B Likar, editor, Computer Vision Winter Workshop, pages 223–233, Ljubljana, Slovenia, February 2001. [Paj01c] T. Pajdla. Oblique cameras generated by collineations. Research Report CTU–CMP–2001–14, April 2001. [Paj01d] T. Pajdla. Rotational hyperboloids as a class of oblique cameras with double ruled quadric visibility
[Paj01e] T. Pajdla. Stereo with oblique cameras. In Bradski G.R and T.E Boult, editors, IEEE Workshop on Stereo and Multi-Baseline Vision, pages 85–91. IEEE Computer Society Press, December 2001. [Paj02a] T. Pajdla. Geometry of two-slit camera. Research Report CTU–CMP–2002–02, March 2002. [Paj02b] T. Pajdla. Stereo with oblique cameras. International Journal of Computer Vision, 47(1-3):161–170, May 2002.
41/47
The END
Anaglyph of a stereo panorama taken by an X-Slits (generalized pushbroom) camera. (D. Feldman, D. Weinshall, T. Pajdla) Stereo correspondence curves are horizontal lines.
42/47
43/47
Examples of stereo geometries
Examples of stereo geometries of central cameras, mosaics, and panoramas, were reviewed and it has been shown that some of them posses a generalization of epipolar geometry, e.g. the following concentric symmetric stereo panorama Stereo correspondence lines in a concentric symmetric stereo panorama (Courtesy of J. ˇ Sivic) and the corresponding stereo correspondence surfaces. while others, e.g. concentric non-symmetric stereo panoramas, do not.
44/47
Precision of reconstruction
A higher precision of reconstruction can be achieved with mosaics than with perspective images in many situations. For instance,
0.5 1 1.5 2 2.5 3 −1.5 −1 −0.5 0.5 1 1.5 2 2.5 y log10(r)
The logarithm of the radius r = 1/δ of the uncertainty circle for a stereo pair of central cameras (blue) as well as for a 360× 360 stereo mosaic (red) as a function of the distance from the respective camera.
shows that 360 × 360 stereo mosaics have smaller reconstruction error, compared to a pair
45/47
Generalization of epipolar planes
Definition 1. Let there hold for two sets U1, U2 of lines in P3:
Theorem 1. Lines in U1, U2 are only in one of the following configurations
a b c d e f g h i j k l
46/47
O Π C
O C O C O C O C
O C
(a-1) (b-1) (c-1) (d-1) (e-1) (f-1) (a-2) (b-2) (c-2) (d-2) (e-2) (f-2) (a-3) (b-3) (c-3) (d-3) (e-3) (f-3) All stereo geometries of concentric stereo panoramas. (*–1) Central camera orientation w.r.t. the circle of motion, (*–2) the resulting stereo correspondence surfaces, and (*–3) the corresponding stereo correspondence lines in images.
Moving central camera Virtual slits Image volume
8
8
l2 l1 c k d l
C b a
0.5 1 1.5 2 2.5 3 −1.5 −1 −0.5 0.5 1 1.5 2 2.5 y log10(r)
a b c d e f g h i j k l