Simple but Effective Tree Structures for Dynamic Programming-Based - - PowerPoint PPT Presentation

▶
simple but effective tree structures for dynamic
SMART_READER_LITE
LIVE PREVIEW

Simple but Effective Tree Structures for Dynamic Programming-Based - - PowerPoint PPT Presentation

Dec 26, 2023 16 likes •731 views

Simple but Effective Tree Structures for Dynamic Programming-Based Stereo Matching Michael Bleyer and Margrit Gelautz Vienna University of Technology Dense Stereo Matching (Left Image) (Right Image) SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR

slide-1
SLIDE 1

Simple but Effective Tree Structures for Dynamic Programming-Based Stereo Matching

Michael Bleyer and Margrit Gelautz Vienna University of Technology

slide-2
SLIDE 2

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Dense Stereo Matching

(Left Image) (Right Image)

slide-3
SLIDE 3

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Dense Stereo Matching

(Left Image) (Right Image) (Disparity Map)

slide-4
SLIDE 4

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Structure

  • Introduction
  • Previous work
  • The Simple Tree Method
  • Energy function
  • Energy optimization
  • Results
  • Conclusions
slide-5
SLIDE 5

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

What stereo method to choose for a practical application?

  • Local methods
  • Computationally efficient
  • Results often too poor
  • Global methods
  • Good-quality results
  • Usually too slow
  • Goal
  • Develop a stereo algorithm that delivers maximum

accuracy at minimum computation time

slide-6
SLIDE 6

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Global Stereo Methods

  • Find a disparity map D that minimizes
slide-7
SLIDE 7

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Global Stereo Methods

  • Find a disparity map D that minimizes

Photo consistency assumption

slide-8
SLIDE 8

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Global Stereo Methods

  • Find a disparity map D that minimizes

Photo consistency assumption Smoothness assumption

slide-9
SLIDE 9

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Global Stereo Methods

  • Find a disparity map D that minimizes
  • Definition of smoothness neighbourhood defines complexity of
  • ptimization problem

Photo consistency assumption Smoothness assumption

slide-10
SLIDE 10

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Optimization on 4-Connected Grid

slide-11
SLIDE 11

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Optimization on 4-Connected Grid

slide-12
SLIDE 12

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Optimization on 4-Connected Grid

(4-Connected Grid)

  • Optimization NP-

complete (discontinuity preserving smoothness functions)

  • Approximation via

Graph-Cuts or Belief Propagation

  • Good results, but

computationally expansive

slide-13
SLIDE 13

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Disparity Map computed via Graph-Cuts

(taken from the Middlebury website)

(Ground Truth) (Graph-Cuts)

slide-14
SLIDE 14

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Dynamic Programming (DP)

(4-Connected Grid)

  • Discard vertical

smoothness edges

  • Exact optimization via

DP

  • Computationally fast,

but scanline streaking

slide-15
SLIDE 15

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Dynamic Programming (DP)

  • Discard vertical

smoothness edges

  • Exact optimization via

DP

  • Computationally fast,

but scanline streaking

(4-Connected Grid)

slide-16
SLIDE 16

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Dynamic Programming (DP)

(DP Neighbourhood Structure)

  • Discard vertical

smoothness edges

  • Exact optimization via

DP

  • Computationally fast,

but scanline streaking

slide-17
SLIDE 17

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Disparity Map computed using DP

(taken from the Middlebury website)

(Ground Truth) (Scanline Optimization)

slide-18
SLIDE 18

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

SemiGlobal Matching [Hirschmüller05]

(4-Connected Grid)

  • Individual disparity

computation at each pixel

  • Aggregate DP costs

computed from paths in various directions

  • Computationally fast,

almost no streaks, but poor performance in regions of low texture

p

slide-19
SLIDE 19

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

SemiGlobal Matching [Hirschmüller05]

(4-Connected Grid)

  • Individual disparity

computation at each pixel

  • Aggregate DP costs

computed from paths in various directions

  • Computationally fast,

almost no streaks, but poor performance in regions of low texture

p

slide-20
SLIDE 20

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

SemiGlobal Matching [Hirschmüller05]

(4-Connected Grid)

  • Individual disparity

computation at each pixel

  • Aggregate DP costs

computed from paths in various directions

  • Computationally fast,

almost no streaks, but poor performance in regions of low texture

p

slide-21
SLIDE 21

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

SemiGlobal Matching [Hirschmüller05]

(4-Connected Grid)

  • Individual disparity

computation at each pixel

  • Aggregate DP costs

computed from paths in various directions

  • Computationally fast,

almost no streaks, but poor performance in regions of low texture

p

slide-22
SLIDE 22

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

SemiGlobal Matching [Hirschmüller05]

(4-Connected Grid)

  • Individual disparity

computation at each pixel

  • Aggregate DP costs

computed from paths in various directions

  • Computationally fast,

almost no streaks, but poor performance in regions of low texture

p

slide-23
SLIDE 23

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

SemiGlobal Matching [Hirschmüller05]

(4-Connected Grid)

  • Individual disparity

computation at each pixel

  • Aggregate DP costs

computed from paths in various directions

  • Computationally fast,

almost no streaks, but poor performance in regions of low texture

p

slide-24
SLIDE 24

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

SemiGlobal Matching in Untextured Regions

(Left Image)

p

(Right Image)

slide-25
SLIDE 25

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

SemiGlobal Matching in Untextured Regions

(Left Image)

p

(Right Image)

slide-26
SLIDE 26

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

SemiGlobal Matching in Untextured Regions

(Left Image)

p

(Right Image)

  • None of the DP paths

captures texture at the correct disparity

  • Disparity selection

guided by noise

slide-27
SLIDE 27

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Reimplementation of SemiGlobal Matching

(Left Image) (Disparity Map)

slide-28
SLIDE 28

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Reimplementation of SemiGlobal Matching

(Left Image) (Disparity Map)

slide-29
SLIDE 29

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Reimplementation of SemiGlobal Matching

(Left Image) (Disparity Map)

slide-30
SLIDE 30

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Our Approach (Simple Tree Method)

(Simple Tree Structure)

  • Perform a separate

disparity computation for each pixel

  • Root a tree on the pixel
  • DP also works on trees
  • Compute exact energy

minimum on the tree

  • Assign p to the disparity

that lies on the energy minimum

p

slide-31
SLIDE 31

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Our Approach (Simple Tree Method)

(Simple Tree Structure)

p

  • Perform a separate

disparity computation for each pixel

  • Root a tree on the pixel
  • DP also works on trees
  • Compute exact energy

minimum on the tree

  • Assign p to the disparity

that lies on the energy minimum

slide-32
SLIDE 32

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Advantages of Simple Trees

(Simple Tree on the Previous Example)

  • Tree structure spans all

pixels (does not miss image features)

  • Vertical and horizontal

smoothness edges (against scanline streaks)

  • We include all smoothness

edges by using two different tree structures

p

slide-33
SLIDE 33

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Two Simple Tree Structures

Horizontal Tree

  • Allow for incremental computation of optima
  • Only 4 DP passes needed

p p

Vertical Tree

slide-34
SLIDE 34

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Energy Function

slide-35
SLIDE 35

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Energy Function

BT-measurement on RGB values

slide-36
SLIDE 36

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Energy Function

BT-measurement on RGB values Modified Potts model

slide-37
SLIDE 37

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Energy Function

BT-measurement on RGB values Modified Potts model

slide-38
SLIDE 38

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Energy Function

BT-measurement on RGB values Modified Potts model Weighted by intensity gradient

slide-39
SLIDE 39

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Energy Function

BT-measurement on RGB values Modified Potts model Weighted by intensity gradient

slide-40
SLIDE 40

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Energy Optimization on Simple Trees

  • Extremely large amount of different trees
  • Tree DP on every tree is extremely slow
slide-41
SLIDE 41

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

I ncremental Computation of Horizontal Trees

  • Optimize horizontal scanlines only
  • Compute DP path costs for reaching each pixel p at each disparity d

from left and right-most pixels of the scanline

slide-42
SLIDE 42

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

I ncremental Computation of Horizontal Trees

p (Forward pass)

  • Optimize horizontal scanlines only
  • Compute DP path costs for reaching each pixel p at each disparity d

from left and right-most pixels of the scanline

slide-43
SLIDE 43

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

I ncremental Computation of Horizontal Trees

p

p

(Forward pass) (Backward pass)

  • Optimize horizontal scanlines only
  • Compute DP path costs for reaching each pixel p at each disparity d

from left and right-most pixels of the scanline

slide-44
SLIDE 44

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

I ncremental Computation of Horizontal Trees

p

p p

(Forward pass) (Backward pass)

(Combining forward and backward passes)

  • Optimize horizontal scanlines only
  • Compute DP path costs for reaching each pixel p at each disparity d

from left and right-most pixels of the scanline

slide-45
SLIDE 45

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

I ncremental Computation of Horizontal Trees

p

  • Compute path costs for reaching pixel p at disparity d

from all leaf nodes

slide-46
SLIDE 46

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

I ncremental Computation of Horizontal Trees

p

  • Compute path costs for reaching pixel p at disparity d

from all leaf nodes

Precomputed

slide-47
SLIDE 47

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

I ncremental Computation of Horizontal Trees

p

  • Compute path costs for reaching pixel p at disparity d

from all leaf nodes

Precomputed

slide-48
SLIDE 48

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

I ncremental Computation of Horizontal Trees

p

  • Compute path costs for reaching pixel p at disparity d

from all leaf nodes

Precomputed

slide-49
SLIDE 49

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

I ncremental Computation of Horizontal Trees

p

  • Compute path costs for reaching pixel p at disparity d

from all leaf nodes

  • Forward Path
  • Backward Path
  • Combination gives

energy minima of all Horizontal Trees on this scanline

slide-50
SLIDE 50

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

What I do not discuss in this talk (but in the paper)

  • How are Horizontal and Vertical Trees combined

in the algorithm?

  • How are occlusions handeled?
slide-51
SLIDE 51

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Results on the Middlebury Evaluation Set

(Ground truth disparities) (Our results [Parameters kept constant])

slide-52
SLIDE 52

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Results on the Middlebury Evaluation Set

(Ground truth disparities) (Our results [Parameters kept constant]) (Tsukuba) 1.86% wrong unoccluded pixels

slide-53
SLIDE 53

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Results on the Middlebury Evaluation Set

(Ground truth disparities) (Our results [Parameters kept constant]) (Venus) 0.42% wrong unoccluded pixels

slide-54
SLIDE 54

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Results on the Middlebury Evaluation Set

(Ground truth disparities) (Our results [Parameters kept constant]) (Teddy) 7.31% wrong unoccluded pixels Large error due to poor texture and noise

slide-55
SLIDE 55

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Results on the Middlebury Evaluation Set

(Ground truth disparities) (Our results [Parameters kept constant]) (Cones) 4.00% wrong unoccluded pixels

slide-56
SLIDE 56

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Middlebury Ranking

  • Rank 8 of ~ 30 algorithms in online table
  • Computationally more efficient than better-ranked methods
  • Best-performing non-segmentation-based algorithm
slide-57
SLIDE 57

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Why we did not use Colour Segmentation

(Left image)

  • Segmentation is

computationally expensive

  • Works fine for

Middlebury set, but not in general

(Ground truth) (SegGlobVis [Bleyer04]) (SimpleTree)

slide-58
SLIDE 58

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Why we did not use Colour Segmentation

(Left image)

  • Segmentation is

computationally expensive

  • Works fine for

Middlebury set, but not in general

(Ground truth) (SegGlobVis [Bleyer04]) (SimpleTree) (Left image)

slide-59
SLIDE 59

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Why we did not use Colour Segmentation

(Left image)

  • Segmentation is

computationally expensive

  • Works fine for

Middlebury set, but not in general

(Ground truth) (SegGlobVis [Bleyer04]) (SimpleTree) (SegGlobVis [Bleyer04]) Errors due to wrong segmentation

slide-60
SLIDE 60

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Why we did not use Colour Segmentation

(Left image)

  • Segmentation is

computationally expensive

  • Works fine for

Middlebury set, but not in general

(Ground truth) (SegGlobVis [Bleyer04]) (SimpleTree) (SimpleTree)

slide-61
SLIDE 61

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Computational Performance [given in sec]

  • Potential for real-time performance
slide-62
SLIDE 62

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

The new Middlebury Data Sets

(30 ground truth image pairs)

slide-63
SLIDE 63

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Comparison against Graph-Cuts

  • Simple Tree outperforms Graph-Cuts on 20 of 30

stereo pairs

  • Simple Tree is significantly faster.
slide-64
SLIDE 64

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

The Dolls Test Set

(Left image) (Ground truth) (Disparity Graph-Cuts) (Disparity Simple Tree) (Error Graph-Cuts) (Error Simple Tree)

slide-65
SLIDE 65

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

The Dolls Test Set

(Left image) (Ground truth) (Disparity Graph-Cuts) (Disparity Simple Tree) (Error Graph-Cuts) (Error Simple Tree)

(Left image) (Ground truth)

slide-66
SLIDE 66

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

The Dolls Test Set

(Left image) (Ground truth) (Disparity Graph-Cuts) (Disparity Simple Tree) (Error Graph-Cuts) (Error Simple Tree)

(Disparity Graph-Cuts) (Disparity Simple Tree)

slide-67
SLIDE 67

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

The Dolls Test Set

(Left image) (Ground truth) (Disparity Graph-Cuts) (Disparity Simple Tree) (Error Graph-Cuts) (Error Simple Tree)

(Error Graph-Cuts) (Error Simple Tree)

slide-68
SLIDE 68

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

The Dolls Test Set

(Left image) (Ground truth) (Disparity Graph-Cuts) (Disparity Simple Tree) (Error Graph-Cuts) (Error Simple Tree)

(Error Graph-Cuts) (Error Simple Tree) Problems with slanted surfaces

slide-69
SLIDE 69

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

The Midd1 Test Set

  • Graph-Cuts perform better on images with extremely low

texture

(Left Image) (Ground Truth) (Simple Tree) (Graph-Cuts)

slide-70
SLIDE 70

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

Conclusions

  • Compute disparity map by solving an optimization problem

for each pixel

  • Approximation of 4-connected grid via a tree
  • Horizontal and Vertical Trees allow for fast computation
  • Results almost free of scanline streaks
  • Best-performing method in the Middlebury ranking that

does not use colour segmentation

  • Can represent a fast alternative to Graph-Cuts when speed

matters

slide-71
SLIDE 71

SIMPLE BUT EFFECTIVE TREE STRUCTURES FOR DYNAMIC PROGRAMMING-BASED STEREO MATCHING

The End Thank You