Refinement of Surface Mesh for Accurate Multi-View Reconstruction - - PowerPoint PPT Presentation

▶
refinement of surface mesh for accurate multi view
SMART_READER_LITE
LIVE PREVIEW

Refinement of Surface Mesh for Accurate Multi-View Reconstruction - - PowerPoint PPT Presentation

Aug 25, 2022 247 likes •544 views

Introduction Depth Map Fusion Mesh Refinement Experiments Refinement of Surface Mesh for Accurate Multi-View Reconstruction International Workshop on Representation and Modeling of Large-scale 3D Environments Asian Conference on Computer

slide-1
SLIDE 1

Introduction Depth Map Fusion Mesh Refinement Experiments

Refinement of Surface Mesh for Accurate Multi-View Reconstruction

International Workshop on Representation and Modeling

  • f Large-scale 3D Environments

Asian Conference on Computer Vision Xian, China, September 2009

Radim Tyleˇ cek, Radim ˇ S´ ara

{tylecr1|sara}@cmp.felk.cvut.cz

Center for Machine Perception, Czech Technical University, Prague

1 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-2
SLIDE 2

Introduction Depth Map Fusion Mesh Refinement Experiments Reconstruction Pipeline

Motivation

High resolution images available State-of-the-art MVS results still below accuracy of laser scanners Goal: elimination of sources of inaccuracy

imprecise camera calibration variable capture conditions suboptimal representation

3D Photography

2 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-3
SLIDE 3

Introduction Depth Map Fusion Mesh Refinement Experiments Reconstruction Pipeline

Surface Reconstruction Pipeline

Input images ⇒ Corresponding regions ⇒ Pair-wise disparity maps ⇓ Refined mesh ⇐ Surface mesh ⇐ Fused depth maps

3 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-4
SLIDE 4

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Representation Model Refinement

Depth Map Fusion [Tyl09]

fused depth map pairwise disparity map supporting camera supporting camera supporting camera camera reference

1

scene

3 2 4 2−3 1−2 2−4

Image-based representation with a set of reference cameras Global problem of joint estimation of depths and cameras

[Tyl09] R.Tylecek, R.Sara: Depth Map Fusion with Camera Calibration Refinement, CVWW 2009 4 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-5
SLIDE 5

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Representation Model Refinement

Why Pair-wise Stereo?

Mature methods developed and available [Cech07] Less vulnerable to calibration errors than traditional MVS

[Cech07] J.Cech, R.Sara: Efficient sampling of disparity space for fast and accurate matching. BenCOS CVPR 2007 5 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-6
SLIDE 6

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Representation Model Refinement

Depth Map Representation

Depth maps Visibility and discontinuity maps ⇒ Back- projection Registered depth maps

Effective representation natural to input data Complexity linear in the number of reference cameras

6 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-7
SLIDE 7

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Representation Model Refinement

Model Refinement

Model = depth, visibility and discontinuity maps + cameras

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . φ ¯ Xi p Xij pq λq j (Rj )⊤xj q Cj (Ri )⊤xi p Ci . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . . ¯ λi p . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

♣♣♣♣♣♣♣♣♣♣♣♣♣ ♣♣♣♣♣♣♣♣♣♣♣♣♣ ♣♣♣♣♣♣♣♣♣♣♣♣♣

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

RjCi − RjCj + ¯ λi

p Rj (Ri)⊤(Ki)−1xi p = λj q

Camera-depth constraint for each correspondence (K-means like) Second-order surface model (depth continuity assumption) One global optimization problem with depths ¯ λ and camera translations C as free parameters

7 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-8
SLIDE 8

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Photo-consistency Contour Matching Surface Evolution

Surface Reconstruction and Refinement

Change of representation to triangular mesh Depth maps merged with PSR [Kaz06] Good initial estimate of surface Use of camera calibration refined in previous step Refinement by combined stereo and contour matching for photo-consistency

[Kaz06] M. Kazhdan, M. Bolitho and H. Hoppe: Poisson surface reconstruction. Eurographics 2006. 8 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-9
SLIDE 9

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Photo-consistency Contour Matching Surface Evolution

Photo-consistency Measure

We define a stereo photo-consistency function (Normalized SSD) φI(X) =

  • i,j∈V (X), i=j

2Ii(πi(X)) − Ij(πj(X))2 σ2

i (πi(X)) + σ2 j (πj(X))

(1) Given world point X, set of images Ii, i = 1, . . . , N Images Ii = I 0

i − Ci are offset-corrected for overall color

balance estimated from projections on current surface V (X) is a set of images in which point X is visible πi(X) ≃ PiX is perspective projection function σi,j independently pre-computed image variances (normalizing factors)

9 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-10
SLIDE 10

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Photo-consistency Contour Matching Surface Evolution

Photo-consistency Measure

What is the effect of offset correction? with offset correction without offset correction

10 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-11
SLIDE 11

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Photo-consistency Contour Matching Surface Evolution

Contour Matching

Projection of contour generators on a smooth surface should match local maxima of image gradient ∇I (apparent contours) φC(X) = 1 |Ω(X)|

  • k∈Ω(X)
  • ∇I
  • πk(X)
  • , ̟k
  • N(X)
  • Ω(X) – set of cameras that see X as a

contour point Avoids explicit detection of contours in images Takes direction into account Requires robust detection of contour vertices (paths) Smooth vs. sharp contour generators

11 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-12
SLIDE 12

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Photo-consistency Contour Matching Surface Evolution

Contour Matching

Projection of contour generators on a smooth surface should match local maxima of image gradient ∇I (apparent contours) φC(X) = 1 |Ω(X)|

  • k∈Ω(X)
  • ∇I
  • πk(X)
  • , ̟k
  • N(X)
  • Ω(X) – set of cameras that see X as a

contour point Avoids explicit detection of contours in images Takes direction into account Requires robust detection of contour vertices (paths) Smooth vs. sharp contour generators

12 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-13
SLIDE 13

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Photo-consistency Contour Matching Surface Evolution

Contour Matching

Projection of contour generators on a smooth surface should match local maxima of image gradient ∇I (apparent contours) φC(X) = 1 |Ω(X)|

  • k∈Ω(X)
  • ∇I
  • πk(X)
  • , ̟k
  • N(X)
  • Ω(X) – set of cameras that see X as a

contour point Avoids explicit detection of contours in images Takes direction into account Requires robust detection of contour vertices (paths) Smooth vs. sharp contour generators

13 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-14
SLIDE 14

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Photo-consistency Contour Matching Surface Evolution

Photo-consistency Measure

What is the effect of contour matching? with contours without contours

14 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-15
SLIDE 15

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Photo-consistency Contour Matching Surface Evolution

Surface evolution

We define a surface energy EΩ(S) =

  • S
  • φI(X) − αφC(X)
  • dA =
  • S

φ(X)dA (2) combining stereo and contour matching and minimize it by iterative surface flow [2] ∂S ∂t (X) =

  • H(X)φ(X) − ∇φ(X), N
  • N,

(3) H(X) is the mean curvature of surface at point X implicit regularization

[2] H.Jin: Variational methods for shape reconstruction in computer vision. PhD thesis, Washington Univ. (2003) 15 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-16
SLIDE 16

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Photo-consistency Contour Matching Surface Evolution

Computation of the gradient ∇φ

Sampling of image points on projection of surface normal Second-order curve fitting for filtering Pixel-wide image sampling (needs adequate mesh resolution) Coarse-to-fine strategy (scale-space approach) Decreasing window size (by 5% in iteration down to 0.1 of original size)

−0.1 −0.05 0.05 0.1 −0.2 0.2 0.4 0.6 0.8 1 1.2 a φ(a) φ φ’ φ(0) at+1 grad φ min(φ’)

16 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-17
SLIDE 17

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Photo-consistency Contour Matching Surface Evolution

Computation of the gradient ∇φ

Sampling of image points on projection of surface normal Second-order curve fitting for filtering Pixel-wide image sampling (needs adequate mesh resolution) Coarse-to-fine strategy (scale-space approach) Decreasing window size (by 5% in iteration down to 0.1 of original size)

−0.1 −0.05 0.05 0.1 −0.2 0.2 0.4 0.6 0.8 1 1.2 a φ(a) φ φ’ φ(0) at+1 grad φ min(φ’)

17 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-18
SLIDE 18

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Photo-consistency Contour Matching Surface Evolution

Computation of the gradient ∇φ

Sampling of image points on projection of surface normal Second-order curve fitting for filtering Pixel-wide image sampling (needs adequate mesh resolution) Coarse-to-fine strategy (scale-space approach) Decreasing window size (by 5% in iteration down to 0.1 of original size)

−0.1 −0.05 0.05 0.1 −0.2 0.2 0.4 0.6 0.8 1 1.2 a φ(a) φ φ’ φ(0) at+1 grad φ min(φ’)

18 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-19
SLIDE 19

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Photo-consistency Contour Matching Surface Evolution

Computation of the gradient ∇φ

Sampling of image points on projection of surface normal Second-order curve fitting for filtering Pixel-wide image sampling (needs adequate mesh resolution) Coarse-to-fine strategy (scale-space approach) Decreasing window size (by 5% in iteration down to 0.1 of original size)

−0.1 −0.05 0.05 0.1 −0.2 0.2 0.4 0.6 0.8 1 1.2 a φ(a) φ φ’ φ(0) at+1 grad φ min(φ’)

19 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-20
SLIDE 20

Introduction Depth Map Fusion Mesh Refinement Experiments The Idea Photo-consistency Contour Matching Surface Evolution

Computation of the gradient ∇φ

What is the effect of scale-space approach? variable scale fixed scale

20 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-21
SLIDE 21

Introduction Depth Map Fusion Mesh Refinement Experiments Standard datasets Large outdoor datasets

Experiments on Standard datasets

fountain-P11 Increase of accuracy Edges emphasized Higher surface quality Flat surfaces smooth

21 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-22
SLIDE 22

Introduction Depth Map Fusion Mesh Refinement Experiments Standard datasets Large outdoor datasets

Experiments on Standard datasets

Fountain-P11 dataset detailed rendering.

a) input image b) ground truth c) depth map fusion d) mesh refinement e) result of FUR f) result of VU

22 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-23
SLIDE 23

Introduction Depth Map Fusion Mesh Refinement Experiments Standard datasets Large outdoor datasets

Evaluation on Standard datasets

2 4 6 8 10 12 5 10 15 20 25 30 error σ percentage fountain−P11 Fur07 Vu09 Tyl09 refined

http://cvlab.epfl.ch/˜strecha/multiview/denseMVS.html

Ground truth from laser scanners Surface projected to cameras Depth measurement error σ Most details below ground truth error σ Completeness vs. Accuracy

[Fur07] Y.Furukawa, J.Ponce: Accurate, dense, and robust multi-view stereopsis. CVPR 2007. [Vu09] H.Vu, R.Keriven, P.Labatut, J.P.Pons: Towards high-resolution large-scale multi-view stereo. CVPR 2009. 23 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-24
SLIDE 24

Introduction Depth Map Fusion Mesh Refinement Experiments Standard datasets Large outdoor datasets

Experiments on large outdoor dataset

Asia scene input images (238)

24 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-25
SLIDE 25

Introduction Depth Map Fusion Mesh Refinement Experiments Standard datasets Large outdoor datasets

Experiments on large outdoor dataset

Asia refined result textured

25 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-26
SLIDE 26

Introduction Depth Map Fusion Mesh Refinement Experiments Standard datasets Large outdoor datasets

Results on the Asia dataset.

a) depth map fusion b) elephant’s head c) refined detail

26 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-27
SLIDE 27

Introduction Depth Map Fusion Mesh Refinement Experiments Standard datasets Large outdoor datasets

Summary

Refinement of Surface Mesh for Accurate Multi-View Reconstruction Pipeline for accurate 3D reconstruction Surface reconstruction with Depth Map Fusion Camera calibration refinement Image offset correction Photometric mesh refinement

Combining stereo and contour matching Scale-space approach

27 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement

slide-28
SLIDE 28

Introduction Depth Map Fusion Mesh Refinement Experiments Standard datasets Large outdoor datasets

Thank you.

28 / 28 R.Tyleˇ cek, R.ˇ S´ ara, CMP CTU Prague Modeling-3D: Multi-view Mesh Refinement