Towards Complete Free-Form Reconstruction of Complex 3D Scenes from - - PowerPoint PPT Presentation

▶
towards complete free form reconstruction of complex 3d
SMART_READER_LITE
LIVE PREVIEW

Towards Complete Free-Form Reconstruction of Complex 3D Scenes from - - PowerPoint PPT Presentation

Mar 09, 2023 181 likes •539 views

Towards Complete Free-Form Reconstruction of Complex 3D Scenes from an Unordered Set of Uncalibrated Images R. H. Cornelius 2 ara 1 D. Martinec 1 T. Pajdla 1 O. Chum 1 S J. Matas 1 1 Center for Machine Perception, Czech Technical

slide-1
SLIDE 1

Towards Complete Free-Form Reconstruction

  • f Complex 3D Scenes

from an Unordered Set of Uncalibrated Images

  • H. Cornelius2
  • R. ˇ

S´ ara1

  • D. Martinec1
  • T. Pajdla1
  • O. Chum1
  • J. Matas1

1Center for Machine Perception, Czech Technical University, Prague 2Computational Vision and Active Perception Laboratory, KTH, Stockholm

presentation given by Daniel Martinec

slide-2
SLIDE 2

2/17

Scenario

  • 1. Targeted imaging — dense high-resolution 3D models
slide-3
SLIDE 3

2/17

Scenario

  • 1. Targeted imaging — dense high-resolution 3D models
  • verviews

wide baseline calibration detailed views narrow baseline dense matching

slide-4
SLIDE 4

2/17

Scenario

  • 1. Targeted imaging — dense high-resolution 3D models
  • verviews

wide baseline calibration detailed views narrow baseline dense matching

  • 2. View planning difficult

gluing of detailed images difficult

slide-5
SLIDE 5

2/17

Scenario

  • 1. Targeted imaging — dense high-resolution 3D models
  • verviews

wide baseline calibration detailed views narrow baseline dense matching

  • 2. View planning difficult

gluing of detailed images difficult

  • 3. Work for standard imaging too
slide-6
SLIDE 6

3/17

Overview: Data Processing Pipeline

  • 1. Maximally stable region matching

[Matas, Chum, Urban, Pajdla, BMVC 2002]

  • 2. Estimation of a consistent system of cameras

[Martinec, Pajdla, ECCV 2002]

  • 3. Dense matching

[ˇ S´ ara, ECCV 2002]

  • 4. Local aggregation to fish-scales

[ˇ S´ ara, ICCV 1998]

slide-7
SLIDE 7

4/17

Step 1: Region Matching

  • stability
  • discriminability

Maximally Stable Extremal Regions (MSER) [Matas, Chum, Urban, Pajdla, BMVC 2002]

  • 1. Match regions across all images
  • 2. Estimate the epipolar geometry for all image pairs
slide-8
SLIDE 8

. . .

5/17

Step 2: Estimation of a Consistent System of Cameras

slide-9
SLIDE 9

Epipolar geometries between image pairs

5/17

Step 2: Estimation of a Consistent System of Cameras

slide-10
SLIDE 10

Epipolar geometries between image pairs

5/17

Step 2: Estimation of a Consistent System of Cameras

slide-11
SLIDE 11

Goal: reconstruction consistent with all images

5/17

Step 2: Estimation of a Consistent System of Cameras

slide-12
SLIDE 12

Goal: reconstruction consistent with all images Problems: 1. occlusions

  • 2. outliers

5/17

Step 2: Estimation of a Consistent System of Cameras

slide-13
SLIDE 13

6/17

Problem Formulation

x1

1

x2

1

← −

X1

Perspective camera projection: λi

p xi p

  • 3×1

= Pi

3×4

Xp

  • 4×1
slide-14
SLIDE 14

6/17

Problem Formulation

x1

1

x2

1

← −

X1

Perspective camera projection: λi

p xi p

  • 3×1

= Pi

3×4

Xp

  • 4×1

    λ1

1x1 1

λ1

2x1 2

. . . λ1

nx1 n

× λ2

2x2 2

× . . . ... . . . λm

1 xm 1

× . . . λm

n xm n

   

  • R

rescaled measurement matrix =     P1 P2 . . . Pm    

  • 3m × 4

motion [X1X2 . . . Xn]

  • 4 × n

structure ⇒

slide-15
SLIDE 15

6/17

Problem Formulation

x1

1

x2

1

← −

X1

Perspective camera projection: λi

p xi p

  • 3×1

= Pi

3×4

Xp

  • 4×1

    λ1

1x1 1

λ1

2x1 2

. . . λ1

nx1 n

× λ2

2x2 2

× . . . ... . . . λm

1 xm 1

× . . . λm

n xm n

   

  • R

rescaled measurement matrix =     P1 P2 . . . Pm    

  • 3m × 4

motion [X1X2 . . . Xn]

  • 4 × n

structure ⇒

  • 1. Estimate λi

p using the epipolar geometry.

⇒ [Sturm, Triggs, ECCV 1996]

slide-16
SLIDE 16

6/17

Problem Formulation

x1

1

x2

1

← −

X1

Perspective camera projection: λi

p xi p

  • 3×1

= Pi

3×4

Xp

  • 4×1

    λ1

1x1 1

λ1

2x1 2

. . . λ1

nx1 n

× λ2

2x2 2

× . . . ... . . . λm

1 xm 1

× . . . λm

n xm n

   

  • R

rescaled measurement matrix =     P1 P2 . . . Pm    

  • 3m × 4

motion [X1X2 . . . Xn]

  • 4 × n

structure ⇒

  • 1. Estimate λi

p using the epipolar geometry.

⇒ [Sturm, Triggs, ECCV 1996]

  • 2. Find basis of R.

[Jacobs, CVPR 1997], [Martinec, Pajdla, ECCV 2002]

— does not work for big camera movements and sparse data ⇒

slide-17
SLIDE 17

6/17

Problem Formulation

x1

1

x2

1

← −

X1

Perspective camera projection: λi

p xi p

  • 3×1

= Pi

3×4

Xp

  • 4×1

    λ1

1x1 1

λ1

2x1 2

. . . λ1

nx1 n

× λ2

2x2 2

× . . . ... . . . λm

1 xm 1

× . . . λm

n xm n

   

  • R

rescaled measurement matrix =     P1 P2 . . . Pm    

  • 3m × 4

motion [X1X2 . . . Xn]

  • 4 × n

structure ⇒

  • 1. Estimate λi

p using the epipolar geometry.

⇒ [Sturm, Triggs, ECCV 1996]

  • 2. Find basis of R.

[Jacobs, CVPR 1997], [Martinec, Pajdla, ECCV 2002]

— does not work for big camera movements and sparse data ⇒

  • 3. [ Fill × and factorize complete R by SVD. ]

— no improvement for very sparse data

slide-18
SLIDE 18

7/17

Optimization & Outlier Removal

  • 1. oriented-projective reconstruction

remove outliers

  • 2. projective BA

remove outliers [+iterate]

  • 3. metric update by DIAC
  • 4. metric BA

remove outliers [+iterate]

  • 5. metric BA with radial distortion

remove outliers [+iterate]

slide-19
SLIDE 19

8/17

Step 2: Results – Outlier Suppression

slide-20
SLIDE 20

9/17

Step 3: Search for dense correspondences

  • 1. suitable pair identification
  • epipole outside the image (not necessarily)
  • #(sparse correspondences)
  • 2. rectification

[Gluckman, Nayar, CVPR 2001]

  • 3. dense matching
  • few mismatches, strict (silent when model not fulfilled)
  • stable matching

[ˇ S´ ara, ECCV 2002]

slide-21
SLIDE 21

10/17

  • Each disparity map is equivalent to a point cloud −

→ merge them

(2% of points shown) <video>

slide-22
SLIDE 22

11/17

Step 4: Local Aggregation to Fish-scales

dense, redundant = ⇒ density reduction

  • piece-wise approximation by simple independent models
  • in fact density approximation: mixture of Gaussians
  • local orientation ∼ covariance of the points

Principal plane Normal vector Center

slide-23
SLIDE 23

12/17

Texturing

  • Use texture from the image contributing most points to the fish-scale
slide-24
SLIDE 24

13/17

Higher Resolution for Details

  • Close-up image pairs
slide-25
SLIDE 25

14/17

Head Scene

slide-26
SLIDE 26

15/17

Building the Measurement Matrix

  • matches of more reliable EGs are merged sooner
  • added matches are tested w.r.t. the already merged ones
slide-27
SLIDE 27

15/17

Building the Measurement Matrix

  • matches of more reliable EGs are merged sooner
  • added matches are tested w.r.t. the already merged ones

= ⇒works well even when some EGs are absolutely wrong natural joining of overview and detailed images ⇐

slide-28
SLIDE 28

16/17

Depth Estimation

  • [Sturm, Triggs, ECCV 1996]

λi

p = (eij ∧ xi

p) · (Fij xj p)

eij ∧ xi

p 2

λj

p

c

  • ptimal strategy: connect image pairs with highest #matches

⇐

slide-29
SLIDE 29

17/17

Valbonne Scene

14 images [768 × 512] Outliers 9.15% Mean / maximal reprojection error 0.23 / 0.94 pxl 14 images      382 correspondences

     ⇐

slide-30
SLIDE 30
slide-31
SLIDE 31
slide-32
SLIDE 32
slide-33
SLIDE 33
slide-34
SLIDE 34
slide-35
SLIDE 35