Consistent Multi-View Reconstruction from Epipolar Geometries with - - PowerPoint PPT Presentation

SLIDE 1

1/22

Consistent Multi-View Reconstruction from Epipolar Geometries with Outliers

Daniel Martinec and Tom´ aˇ s Pajdla Center for Machine Perception Department of Cybernetics, Faculty Electrical Engineering Czech Technical University, Prague

SLIDE 2

. . .

2/22

What is it about. . .

SLIDE 3

Epipolar geometries between image pairs avialable (Matas & al BMVC’02)

2/22

What is it about. . .

SLIDE 4

Epipolar geometries between image pairs avialable (Matas & al BMVC’2002)

2/22

What is it about. . .

SLIDE 5

2/22

What is it about. . .

SLIDE 6

Goal: projective reconstruction consistent with all images constructed from pairwise epipolar geometries

2/22

What is it about. . .

SLIDE 7

3/22

Problems

• 1. occlusions

x1

1

x2

1

SLIDE 8

3/22

Problems

• 1. occlusions

x1

1

x2

1

• 2. outliers

(may satisfy the epipolar geometry)

SLIDE 9

4/22

Contribution

• Technique for construction of

consistent projective reconstruction from pairwise epipolar geometries

• perspective cameras
• occlusions
• outliers

SLIDE 10

• Martinec & Pajdla ECCV’2002

Martinec & Pajdla PCV’2002

4/22

Contribution

• Technique for construction of

consistent projective reconstruction from pairwise epipolar geometries

• perspective cameras
• occlusions
• outliers

SLIDE 11

5/22

Outline

INTRODUCTION

• Problem formulation, previous work

PART I: Perspective Cameras & Occlusions

• Projective depth estimation
• Filling the missing elements
• Experiments

PART II: Outlier Detection

• New idea & algorithm
• Experiments

SLIDE 12

6/22

Previous Work

[1] C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization method. In IJCV(9)2, 1992.

SLIDE 13

6/22

Previous Work

[1] C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization method. In IJCV(9)2, 1992. [2] D. Jacobs. Linear fitting with missing data: Applications to structure from motion and to characterizing intensity images. In CVPR, 1997.

SLIDE 14

6/22

Previous Work

[1] C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization method. In IJCV(9)2, 1992. [2] D. Jacobs. Linear fitting with missing data: Applications to structure from motion and to characterizing intensity images. In CVPR, 1997. [3] P. Sturm and B. Triggs. A factorization based algorithm for multi-image projective structure and motion. In ECCV, 1996.

SLIDE 15

6/22

Previous Work

[1] C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization method. In IJCV(9)2, 1992. [2] D. Jacobs. Linear fitting with missing data: Applications to structure from motion and to characterizing intensity images. In CVPR, 1997. [3] P. Sturm and B. Triggs. A factorization based algorithm for multi-image projective structure and motion. In ECCV, 1996. [4] D. Q. Huynh and A. Heyden. Outlier detection in video sequences under affine

• projection. In CVPR, 2001.

SLIDE 16

7/22

PART I: Perspective Cameras & Occlusions

Review of ECCV’2002

SLIDE 17

8/22

Problem Formulation & Solution

x1

1

x2

1

← −

X1

SLIDE 18

8/22

Problem Formulation & Solution

x1

1

x2

1

← −

X1

Perspective camera projection: λi

p xi p

• 3×1

= Pi

3×4

Xp

• 4×1

SLIDE 19

8/22

Problem Formulation & Solution

x1

1

x2

1

← −

X1

Perspective camera projection: λi

p xi p

• 3×1

= Pi

3×4

Xp

• 4×1

    λ1

1x1 1

λ1

2x1 2

. . . λ1

nx1 n

× λ2

2x2 2

× . . . ... . . . λm

1 xm 1

× . . . λm

n xm n

   

• R

rescaled measurement matrix =     P1 P2 . . . Pm    

• 3m × 4

motion [X1X2 . . . Xn]

• 4 × n

structure

SLIDE 20

8/22

Problem Formulation & Solution

x1

1

x2

1

← −

X1

Perspective camera projection: λi

p xi p

• 3×1

= Pi

3×4

Xp

• 4×1

    λ1

1x1 1

λ1

2x1 2

. . . λ1

nx1 n

× λ2

2x2 2

× . . . ... . . . λm

1 xm 1

× . . . λm

n xm n

   

• R

rescaled measurement matrix =     P1 P2 . . . Pm    

• 3m × 4

motion [X1X2 . . . Xn]

• 4 × n

structure

• 1. Estimate (some) λi

j using, e.g., the epipolar geometry.

• 2. Fill ×.

Repeat 1 and 2 until R is complete.

• 3. Factorize complete R.

SLIDE 21

9/22

Estimation of λi

p (Sturm & Triggs)

uses the epipolar geometry

✂✁ ☎✄ ✆ ✝ ✞ ✁ ✞ ✄

SLIDE 22

9/22

Estimation of λi

p (Sturm & Triggs)

uses the epipolar geometry

✂✁ ☎✄ ✆ ✝ ✞ ✁ ✞ ✄

λ1x1 = τe + λ2x2

SLIDE 23

9/22

Estimation of λi

p (Sturm & Triggs)

uses the epipolar geometry

✂✁ ☎✄ ✆ ✝ ✞ ✁ ✞ ✄

λ1x1 = τe + λ2x2

λi

p = (eij ∧ xi

p) · (Fij xj p)

eij ∧ xi

p 2

λj

p

• Alternatives:
• 1. central image

c

• 2. sequence

SLIDE 24

10/22

Filling of ×

• Jacobs’ algorithm: for affine cameras only (λi

p = 1)

SLIDE 25

10/22

Filling of ×

• Jacobs’ algorithm: for affine cameras only (λi

p = 1)

• ur algorithm:

for perspective cameras (various λi

p)

SLIDE 26

10/22

Filling of ×

• Jacobs’ algorithm: for affine cameras only (λi

p = 1)

• ur algorithm:

for perspective cameras (various λi

p)

• fill R so that rank R = 4

(rank R = 4 because R = P

• 3m×4

X

• 4×n

)

SLIDE 27

10/22

Filling of ×

• Jacobs’ algorithm: for affine cameras only (λi

p = 1)

• ur algorithm:

for perspective cameras (various λi

p)

• fill R so that rank R = 4

(rank R = 4 because R = P

• 3m×4

X

• 4×n

)

a 4-tuple of “LI” columns Bk =      ? x1

1

λ1

2x1 2

λ1

3x1 3

λ1

4x1 4

λ2

1x2 1

λ2

2x2 2

λ2

3x2 3

λ2

4x2 4

. . . λm

1 xm 1

λm

2 xm 2

λm

3 xm 3

×      =                 

? x1

1

? y1

1

? w1

1

λ1

2x1 2

λ1

2y1 2

λ1

2w1 2

λ1

3x1 3

λ1

3y1 3

λ1

3w1 3

λ1

4x1 4

λ1

4y1 4

λ1

4w1 4

λ2

1x2 1

λ2

1y2 1

λ2

1w2 1

λ2

2x2 2

λ2

2y2 2

λ2

2w2 2

λ2

3x2 3

λ2

3y2 3

λ2

3w2 3

λ2

4x2 4

λ2

4y2 4

λ2

4w2 4

. . .

λm

1 xm 1

λm

1 ym 1

λm

1 wm 1

λm

2 xm 2

λm

2 ym 2

λm

2 wm 2

λm

3 xm 3

λm

3 ym 3

λm

3 wm 3

× × ×

                 linear hull of all possible fillings Bk = Span(                  

• x1

1

y1

1

w1

1

λ1

2x1 2

λ1

2y1 2

λ1

2w1 2

λ1

3x1 3

λ1

3y1 3

λ1

3w1 3

• λ1

4x1 4

λ1

4y1 4

λ1

4w1 4

λ2

1x2 1

λ2

1y2 1

λ2

1w2 1

λ2

2x2 2

λ2

2y2 2

λ2

2w2 2

λ2

3x2 3

λ2

3y2 3

λ2

3w2 3

λ2

4x2 4

λ2

4y2 4

λ2

4w2 4

. . .

λm

1 xm 1

λm

1 ym 1

λm

1 wm 1

λm

2 xm 2

λm

2 ym 2

λm

2 wm 2

λm

3 xm 3

λm

3 ym 3

λm

3 wm 3

1 1 1

                  )

SLIDE 28

10/22

Filling of ×

• Jacobs’ algorithm: for affine cameras only (λi

p = 1)

• ur algorithm:

for perspective cameras (various λi

p)

• fill R so that rank R = 4

(rank R = 4 because R = P

• 3m×4

X

• 4×n

)

a 4-tuple of “LI” columns Bk =      ? x1

1

λ1

2x1 2

λ1

3x1 3

λ1

4x1 4

λ2

1x2 1

λ2

2x2 2

λ2

3x2 3

λ2

4x2 4

. . . λm

1 xm 1

λm

2 xm 2

λm

3 xm 3

×      =                 

? x1

1

? y1

1

? w1

1

λ1

2x1 2

λ1

2y1 2

λ1

2w1 2

λ1

3x1 3

λ1

3y1 3

λ1

3w1 3

λ1

4x1 4

λ1

4y1 4

λ1

4w1 4

λ2

1x2 1

λ2

1y2 1

λ2

1w2 1

λ2

2x2 2

λ2

2y2 2

λ2

2w2 2

λ2

3x2 3

λ2

3y2 3

λ2

3w2 3

λ2

4x2 4

λ2

4y2 4

λ2

4w2 4

. . .

λm

1 xm 1

λm

1 ym 1

λm

1 wm 1

λm

2 xm 2

λm

2 ym 2

λm

2 wm 2

λm

3 xm 3

λm

3 ym 3

λm

3 wm 3

× × ×

                 linear hull of all possible fillings Bk = Span(                  

• x1

1

y1

1

w1

1

λ1

2x1 2

λ1

2y1 2

λ1

2w1 2

λ1

3x1 3

λ1

3y1 3

λ1

3w1 3

• λ1

4x1 4

λ1

4y1 4

λ1

4w1 4

λ2

1x2 1

λ2

1y2 1

λ2

1w2 1

λ2

2x2 2

λ2

2y2 2

λ2

2w2 2

λ2

3x2 3

λ2

3y2 3

λ2

3w2 3

λ2

4x2 4

λ2

4y2 4

λ2

4w2 4

. . .

λm

1 xm 1

λm

1 ym 1

λm

1 wm 1

λm

2 xm 2

λm

2 ym 2

λm

2 wm 2

λm

3 xm 3

λm

3 ym 3

λm

3 wm 3

1 1 1

                  )

• Bk contains filled Bk (also filled R) −

→ constraint on R

• intersection of many Bk −

→ fill R

SLIDE 29

11/22

Wide Base-Line Stereo

LM = lin. method, BA = bundle adjustment

Scene House 10 images [2952×2003] Point detection manual, 203 points in space Depth estimation central image No. 1 Amount of missing data 47.83 % LM Mean error per image point [pxl] 3.91 LM + BA 1.44 10 images      203 points

• 

     ”•” λi

p ←

− F (75.7 %), ”◦” λi

p ←

− B (24.3 %), ” ” occlusion

SLIDE 30

12/22

Dense Sequence (Hannover)

Scene Dinosaur (Oxford) 36 images [720×576] Point detection Harris’ operator, 4983 points in space Depth estimation sequence Amount of missing data 90.84 % LM Mean error per image point [pxl] 1.76 LM + BA 0.64 36 images      4983 points

• 

     ”•” λi

p ←

− F (100.0 %), ” ” occlusion

SLIDE 31

13/22

PART II: Outlier Detection

Review of PCV’2002

SLIDE 32

14/22

Correspondences from the Epipolar Geometry

some outliers satisfy the epipolar geometry epipolar geometry from a dominant plane & outliers

SLIDE 33

15/22

Problem Formulation & Related Work

Perspective camera projection: λi

p xi p

• 3×1

= Pi

3×4

Xp

• 4×1

xi

p . . . outliers

    λ1

1x1 1

λ1

2x1 2

. . . λ1

nx1 n

× λ2

2x2 2

× . . . ... . . . λm

1 xm 1

× . . . λm

n xm n

   

• R

rescaled measurement matrix =     P1 P2 . . . Pm    

• 3m × 4

motion [X1X2 . . . Xn]

• 4 × n

structure

SLIDE 34

16/22

Outlier Detection — Main Idea

Pairwise epipolar geometry − → not many outliers

SLIDE 35

16/22

Outlier Detection — Main Idea

Pairwise epipolar geometry − → not many outliers Assumption One consistent structure & random independent outliers

SLIDE 36

16/22

Outlier Detection — Main Idea

Pairwise epipolar geometry − → not many outliers Assumption One consistent structure & random independent outliers Main Idea Minimal configurations of points in triples of images are sufficient to validate inliers reliably.

SLIDE 37

16/22

Outlier Detection — Main Idea

Pairwise epipolar geometry − → not many outliers Assumption One consistent structure & random independent outliers Main Idea Minimal configurations of points in triples of images are sufficient to validate inliers reliably. = ⇒

• 1. Choose a triple of images and six common points in them in random −

→ T .

• 2. If T consistent with enough correspondences −

→ validate points not used for estimation T as inliers. Repeat 1 and 2 until there is enough inliers.

SLIDE 38

16/22

Outlier Detection — Main Idea

Pairwise epipolar geometry − → not many outliers Assumption One consistent structure & random independent outliers Main Idea Minimal configurations of points in triples of images are sufficient to validate inliers reliably. = ⇒

• 1. Choose a triple of images and six common points in them in random −

→ T .

• 2. If T consistent with enough correspondences −

→ validate points not used for estimation T as inliers. Repeat 1 and 2 until there is enough inliers. − → estimate reconstruction using our method for occlusions [ECCV’2002] (xi

p → ×)

SLIDE 39

17/22

Consistent Multi-View Reconstruction — Algorithm

• 1. Obtain pairwise matches by estimating epipolar geometries on (all) image pairs.

SLIDE 40

17/22

Consistent Multi-View Reconstruction — Algorithm

• 1. Obtain pairwise matches by estimating epipolar geometries on (all) image pairs.
• 2. Build the measurement matrix by joining the overlapping matches and ignoring the

conflicting ones.

SLIDE 41

17/22

Consistent Multi-View Reconstruction — Algorithm

• 1. Obtain pairwise matches by estimating epipolar geometries on (all) image pairs.
• 2. Build the measurement matrix by joining the overlapping matches and ignoring the

conflicting ones.

• 3. Find the initial set of inliers using validation by T (previous slide).

SLIDE 42

17/22

Consistent Multi-View Reconstruction — Algorithm

• 1. Obtain pairwise matches by estimating epipolar geometries on (all) image pairs.
• 2. Build the measurement matrix by joining the overlapping matches and ignoring the

conflicting ones.

• 3. Find the initial set of inliers using validation by T (previous slide).
• 4. Reconstruction from inliers using [ECCV’2002].

SLIDE 43

17/22

Consistent Multi-View Reconstruction — Algorithm

• 1. Obtain pairwise matches by estimating epipolar geometries on (all) image pairs.
• 2. Build the measurement matrix by joining the overlapping matches and ignoring the

conflicting ones.

• 3. Find the initial set of inliers using validation by T (previous slide).
• 4. Reconstruction from inliers using [ECCV’2002].
• 5. Check all data whether it is consistent with the reconstruction.

SLIDE 44

17/22

Consistent Multi-View Reconstruction — Algorithm

• 1. Obtain pairwise matches by estimating epipolar geometries on (all) image pairs.
• 2. Build the measurement matrix by joining the overlapping matches and ignoring the

conflicting ones.

• 3. Find the initial set of inliers using validation by T (previous slide).
• 4. Reconstruction from inliers using [ECCV’2002].
• 5. Check all data whether it is consistent with the reconstruction.

In each column, p, of R: (a) Random triple of image points, P − → Xp (b) If repr. error small − → inliers Repeat (a) and (b) until the column is sufficiently sampled

SLIDE 45

17/22

Consistent Multi-View Reconstruction — Algorithm

• 1. Obtain pairwise matches by estimating epipolar geometries on (all) image pairs.
• 2. Build the measurement matrix by joining the overlapping matches and ignoring the

conflicting ones.

• 3. Find the initial set of inliers using validation by T (previous slide).
• 4. Reconstruction from inliers using [ECCV’2002].
• 5. Check all data whether it is consistent with the reconstruction.

In each column, p, of R: (a) Random triple of image points, P − → Xp (b) If repr. error small − → inliers Repeat (a) and (b) until the column is sufficiently sampled

• 6. Iteration of Steps 4 and 5 while any new inlier appeared

SLIDE 46

18/22

Experiments on WBS

SLIDE 47

18/22

Experiments on WBS

Scene Valbonne all pairs (Oxford) 14 images [768 × 512] Outliers 396 (28.14 % of 1407 image points)

• Rec. / not-reconstructed cameras

14 / 0

• Rec. / partially rec. / not-rec. corresp.

271 / 32 / 105 of 376 Mean / maximal reprojection error 0.45 / 3.66 pxl (from inliers) 14 images      376 correspondences

• 

    

SLIDE 48

19/22

SLIDE 49

20/22

Some Other WBS Experitments. . .

Scene Movi-house (CMP) 14 images [512 × 512] Outliers 207 (44.90 % of 461 image points)

• Rec. / not-reconstructed cameras

9 / 0

• Rec. / partially rec. / not-rec. corresp.

67 / 33 / 34 of 101 Mean / maximal reprojection error 0.75 / 5.27 pxl (from inliers) 9 images      101 correspondences

• 

    

SLIDE 50

21/22

Some Other WBS Experitments. . .

Scene Shelf (CMP) 12 images [1200 × 1600] Outliers 414 (6.46 % of 6411 image points)

• Rec. / not-reconstructed cameras

12 / 0

• Rec. / partially rec. / not-rec. corresp.

1839 / 72 / 114 of 1953 Mean / maximal reprojection error 0.51 / 4.90 pxl (from inliers) 12 images      1953 correspondences

• 

    

SLIDE 51

22/22

Conclusion

• A new algorithm for a general situation
• perspective

camera

• cclusions
• utliers
• Experiments
• real scenes: applicable in wide base-line as well as on sequences

SLIDE 52

SLIDE 53

SLIDE 54

SLIDE 55

SLIDE 56

SLIDE 57

SLIDE 58

SLIDE 59

SLIDE 60

SLIDE 61

SLIDE 62

SLIDE 63

SLIDE 64

SLIDE 65

SLIDE 66

SLIDE 67

SLIDE 68

SLIDE 69

SLIDE 70

SLIDE 71

SLIDE 72

SLIDE 73

SLIDE 74

SLIDE 75

SLIDE 76

x1

1

SLIDE 77

x2

1

SLIDE 78

SLIDE 79

SLIDE 80

SLIDE 81

x1

1

SLIDE 82

x2

1

SLIDE 83

SLIDE 84

X1

SLIDE 85 ✂✁ ☎✄ ✆ ✝ ✞ ✁ ✞ ✄

SLIDE 86

c

SLIDE 87

SLIDE 88

a 4-tuple of “LI” columns Bk =      ? x1

1

λ1

2x1 2

λ1

3x1 3

λ1

4x1 4

λ2

1x2 1

λ2

2x2 2

λ2

3x2 3

λ2

4x2 4

. . . λm

1 xm 1

λm

2 xm 2

λm

3 xm 3

×      =                 

? x1

1

? y1

1

? w1

1

λ1

2x1 2

λ1

2y1 2

λ1

2w1 2

λ1

3x1 3

λ1

3y1 3

λ1

3w1 3

λ1

4x1 4

λ1

4y1 4

λ1

4w1 4

λ2

1x2 1

λ2

1y2 1

λ2

1w2 1

λ2

2x2 2

λ2

2y2 2

λ2

2w2 2

λ2

3x2 3

λ2

3y2 3

λ2

3w2 3

λ2

4x2 4

λ2

4y2 4

λ2

4w2 4

. . .

λm

1 xm 1

λm

1 ym 1

λm

1 wm 1

λm

2 xm 2

λm

2 ym 2

λm

2 wm 2

λm

3 xm 3

λm

3 ym 3

λm

3 wm 3

× × ×

                 linear hull of all possible fillings Bk = Span(                  

• x1

1

y1

1

w1

1

λ1

2x1 2

λ1

2y1 2

λ1

2w1 2

λ1

3x1 3

λ1

3y1 3

λ1

3w1 3

• λ1

4x1 4

λ1

4y1 4

λ1

4w1 4

λ2

1x2 1

λ2

1y2 1

λ2

1w2 1

λ2

2x2 2

λ2

2y2 2

λ2

2w2 2

λ2

3x2 3

λ2

3y2 3

λ2

3w2 3

λ2

4x2 4

λ2

4y2 4

λ2

4w2 4

. . .

λm

1 xm 1

λm

1 ym 1

λm

1 wm 1

λm

2 xm 2

λm

2 ym 2

λm

2 wm 2

λm

3 xm 3

λm

3 ym 3

λm

3 wm 3

1 1 1

                  )

SLIDE 89

SLIDE 90

SLIDE 91

SLIDE 92

SLIDE 93

SLIDE 94

SLIDE 95

SLIDE 96

SLIDE 97

SLIDE 98

SLIDE 99

SLIDE 100

SLIDE 101

SLIDE 102

SLIDE 103

SLIDE 104

SLIDE 105

SLIDE 106

SLIDE 107

SLIDE 108

SLIDE 109

SLIDE 110

SLIDE 111