Input Rectified stereo image pair All correspondences lie in - - PowerPoint PPT Presentation

▶
input
SMART_READER_LITE
LIVE PREVIEW

Input Rectified stereo image pair All correspondences lie in - - PowerPoint PPT Presentation

Apr 08, 2024 414 likes •790 views

Problem Definition 3 Input Rectified stereo image pair All correspondences lie in same scan lines Output Disparity map of the reference view Foreground: large disparity Background: small disparity Matching Cost

slide-1
SLIDE 1

3

Input

  • Rectified stereo image pair
  • All correspondences lie in same scan lines

Output

  • Disparity map of the reference view
  • Foreground: large disparity
  • Background: small disparity

Problem Definition

slide-2
SLIDE 2

2

Matching Cost Volume 𝐷 𝑦, 𝑧, 𝑒 denotes the matching cost of pixel (x,y) at different disparity level d

slide-3
SLIDE 3

3

WTA (Winner Takes All)

  • 𝐷 𝑦, 𝑧, 𝑒 denotes the matching cost of

pixel (x,y) at different disparity level d

  • Select d with lowest cost as final disparity
slide-4
SLIDE 4

4

Some Milestone Approaches

  • Graph Cut (Energy Minimization via Graph Cuts)

Boykov et al., ICCV 1999

  • ASW (Cost Aggregation by adaptive support weight)

Yoon and Kweon, CVPR 2005

  • SGM (Semi-Global Matching)

Hirschmuller, CVPR 2005, PAMI 2008

  • PatchMatch Stereo (Cost aggregation using slanted

support windows)

Bleyer et al., BMVC 2011

slide-5
SLIDE 5

5

Unary Cost

  • Photo consistency of each label

Pairwise Cost

  • Penalize disparity difference between

neighboring pixels

Graph Cut

  • Frame the problem as an energy minimization on a multi-

labeled MRF

  • Solve the MRF by Graph Cut
slide-6
SLIDE 6

6

Graph Cut

  • Frame the problem as an energy minimization on a multi-

labeled MRF

  • Solve the MRF by Graph Cut
slide-7
SLIDE 7

7

Cost aggregation Bilateral filter each 𝐷(𝑦, 𝑧, 𝑗)

𝐷 𝑦, 𝑧, 𝑒 𝐷

𝐵 𝑦, 𝑧, 𝑒

Traditional Local Methods

slide-8
SLIDE 8

8

ASW (Adaptive Support Weights)

  • Given an initial matching cost volume,
  • Refine the volume by aggregating cost locally

and adaptively

slide-9
SLIDE 9

9

ASW (Adaptive Support Weights)

  • Given an initial matching cost volume,
  • Refine the volume by aggregating cost locally

and adaptively

slide-10
SLIDE 10

10

SGM (Semi-Global Matching)

  • Instead of aggregating cost at a

local window,

  • SGM Aggregate cost in paths
slide-11
SLIDE 11

11

SGM (Semi-Global Matching)

  • Instead of aggregating cost at a

local window,

  • SGM Aggregate cost in paths
slide-12
SLIDE 12

12

PatchMatch Stereo

  • Parametrize each pixel as a disparity plane
  • Aggregate cost in the slanted window induced by the plane
  • Too many (infinite) possible states, solve by PatchMatch
slide-13
SLIDE 13

13

PatchMatch Stereo

  • Parametrize each pixel as a disparity plane
  • Aggregate cost in the slanted window induced by the plane
  • Too many (infinite) possible states, solve by PatchMatch
slide-14
SLIDE 14

MeshStereo:

A Global Stereo Model with Mesh Alignment Regularization for View Interpolation

Chi Zhang, Zhiwei Li, Yanhua Cheng, Rui Cai, Hongyang Chao, Yong Rui

Presented by Chi Zhang

  • Dec. 15th, 2015
slide-15
SLIDE 15

2

Goal

  • Output high-quality mesh for view interpolation

Motivation

  • Depth estimation and mesh generation are separated

in traditional approach, which is not optimal in terms of rendering

  • We aim at unifying such separation, and develop

an integrated stereo approach for view interpolation

Motivation

slide-16
SLIDE 16

3

Traditional Pipeline

  • IL, IR -> Point Clouds (Disparity Maps)
  • Point Clouds -> Mesh
  • Mesh -> New View Angles

Ours

  • IL, IR -> Mesh
  • Mesh -> New View Angles

Movitation

slide-17
SLIDE 17

17

Mesh Representation

  • Delauney triangulated SLIC Segmentation
  • Assign a depth value to each vertex
  • Lifting the 2D triangulation to 3D

naturally generate a mesh

Technical Difficulty

  • How to split vertices into

multiple copies at depth discontinuities

Formulation

Solution

  • Assign a splitting probability to each vertex
slide-18
SLIDE 18

18

Formulation

Parameterization

A Splitting probability for each 2D vertex denoted by α A depth value for each triangle’s barycenter and a normal for each triangle denoted by N,D

slide-19
SLIDE 19

19

Formulation

Objective Function

  • Objective function is a two-layered MRF

glued by an Alignment energy term

Gluer Upper Layer MRF Lower Layer MRF

slide-20
SLIDE 20

20

Formulation

The Lower Layer

  • The lower layer MRF is on the “dual”

grid of the 2D triangulation

slide-21
SLIDE 21

21

Formulation

The Lower Layer

  • Favorites photo-consistent triangles
slide-22
SLIDE 22

22

Formulation

The Lower Layer

  • Encourages normal smoothness.

Encouraged Discouraged

slide-23
SLIDE 23

23

Formulation

The Upper Layer

  • The upper layer MRF is on the original

grid of the 2D triangulation

slide-24
SLIDE 24

24

Formulation

The Upper Layer

  • Favorite non-split vertices on

homogeneous regions

slide-25
SLIDE 25

25

Formulation

The Upper Layer

  • Encourages similar splitting properties

when adjacent vertices share similar “visual complexity”

Similar visual complexity Non-similar visual complexity Similar visual complexity Similar visual complexity

slide-26
SLIDE 26

26

Formulation

The Upper Layer

  • Encourages similar splitting properties

when adjacent vertices share similar “visual complexity”

slide-27
SLIDE 27

27

Formulation

The Gluer

  • Enforce strong alignment or split a 2D

vertex to multiple copies in 3D according to corresponding splitting probability

slide-28
SLIDE 28

28

Optimization

Optimization

  • Iterative Gradient Descent in the blue part and the orange part
  • Fix N, D, minimize the orange part w.r.t. α in closed form
  • Fix α, minimize the blue part by PatchMatch with

iterative relaxation (detail at next page)

Objective Function

slide-29
SLIDE 29

29

Optimization

  • Fix α, minimize ELOWER by PatchMatch with iterative relaxation

The orange part sub-energy Optimization

  • When 𝜄 goes to ∞,

minimizing ERELAXED is equivalent to minizing ELOWER

slide-30
SLIDE 30

30

Optimization

  • Fix α, minimize ELOWER by PatchMatch with iterative relaxation

Minimize by PatchMatch Minimize in closed form

The orange part sub-energy Optimization

slide-31
SLIDE 31

31

Stereo Results on Herodion Dataset

Results

slide-32
SLIDE 32

32

Ranking on Midd3 benchmark

Results

slide-33
SLIDE 33

33

Examples of generated meshes

Results

slide-34
SLIDE 34

34

Some synthesized views

Results

slide-35
SLIDE 35

35

Some synthesized views

Results

slide-36
SLIDE 36

18

Conclusion

  • We proposed an integrated stereo model for view

interpolation

  • Take IL, IR as inputs, produce a mesh directly
  • It achieves state-of-the-arts performance on both stereo

quality and rendering

slide-37
SLIDE 37

Thank you!

The End