X Disparity Determination in Stereo Vision Lu Sang, Michael Haberl, - - PowerPoint PPT Presentation

▶
x disparity determination in stereo vision
SMART_READER_LITE
LIVE PREVIEW

X Disparity Determination in Stereo Vision Lu Sang, Michael Haberl, - - PowerPoint PPT Presentation

Mar 02, 2024 325 likes •1.07k views

X Disparity Determination in Stereo Vision Lu Sang, Michael Haberl, Raphael Ullmann 22.07.2017 Lu Sang, Michael Haberl, Raphael Ullmann Overview Problem Description Algorithms Results Lu Sang, Michael Haberl, Raphael Ullmann Overview

slide-1
SLIDE 1

X Disparity Determination in Stereo Vision

Lu Sang, Michael Haberl, Raphael Ullmann 22.07.2017

Lu Sang, Michael Haberl, Raphael Ullmann

slide-2
SLIDE 2

Overview

Problem Description Algorithms Results

Lu Sang, Michael Haberl, Raphael Ullmann

slide-3
SLIDE 3

Overview

Problem Description Algorithms Results

Lu Sang, Michael Haberl, Raphael Ullmann

slide-4
SLIDE 4

Problem Description

Lu Sang, Michael Haberl, Raphael Ullmann

slide-5
SLIDE 5

Problem Description

Lu Sang, Michael Haberl, Raphael Ullmann

slide-6
SLIDE 6

Problem Description

  • What is the distance to an object?
  • How to determine the distance by 2D images?

1https://upload.wikimedia.org/wikipedia/commons/4/49/Roboterhand.mit.Gluehbirne.png Lu Sang, Michael Haberl, Raphael Ullmann

slide-7
SLIDE 7

Problem Description

  • What is the distance to an object?
  • How to determine the distance by 2D images?

Applications

  • Autonomous driving
  • Robotics
  • Object recognition

1

1https://upload.wikimedia.org/wikipedia/commons/4/49/Roboterhand.mit.Gluehbirne.png Lu Sang, Michael Haberl, Raphael Ullmann

slide-8
SLIDE 8

Stereo Cameras

Lu Sang, Michael Haberl, Raphael Ullmann

slide-9
SLIDE 9

Corresponding Pixels

Figure: Left Picture [1] Figure: Right Picture

Lu Sang, Michael Haberl, Raphael Ullmann

slide-10
SLIDE 10

Corresponding Pixels

Figure: Left Picture [1] Figure: Right Picture

Lu Sang, Michael Haberl, Raphael Ullmann

slide-11
SLIDE 11

Corresponding Pixels

Figure: Left Picture [1] Figure: Right Picture

1 Calculate for the pixels in the left image costs in the right

image

Lu Sang, Michael Haberl, Raphael Ullmann

slide-12
SLIDE 12

Corresponding Pixels

Figure: Left Picture [1] Figure: Right Picture

1 Calculate for the pixels in the left image costs in the right

image

2 Pixel with minimal cost is the corresponding pixel

Lu Sang, Michael Haberl, Raphael Ullmann

slide-13
SLIDE 13

Disparity

Lu Sang, Michael Haberl, Raphael Ullmann

slide-14
SLIDE 14

Disparity

Disparity

Pixel distance of related pixels.

Lu Sang, Michael Haberl, Raphael Ullmann

slide-15
SLIDE 15

Getting distance from Disparity

Lu Sang, Michael Haberl, Raphael Ullmann

slide-16
SLIDE 16

Getting distance from Disparity

Lu Sang, Michael Haberl, Raphael Ullmann

slide-17
SLIDE 17

Getting distance from Disparity

Lu Sang, Michael Haberl, Raphael Ullmann

slide-18
SLIDE 18

Getting distance from Disparity

Lu Sang, Michael Haberl, Raphael Ullmann

slide-19
SLIDE 19

Distance

z ✏ f ☎ b d

  • Distance z
  • Focal length of the camera f
  • Disparity d

Lu Sang, Michael Haberl, Raphael Ullmann

slide-20
SLIDE 20

Overview

Problem Description Algorithms Results

Lu Sang, Michael Haberl, Raphael Ullmann

slide-21
SLIDE 21

Overview

Pre-processing Cost Calculation Pyramid Scheme Force Local Consistency Penalty Terms Median Filter

Lu Sang, Michael Haberl, Raphael Ullmann

slide-22
SLIDE 22

Overview

Pre-processing Cost Calculation Pyramid Scheme Force Local Consistency Penalty Terms Median Filter

Lu Sang, Michael Haberl, Raphael Ullmann

slide-23
SLIDE 23

Pre-processing: Undistortion

Original Picture Undistorted Picture

Lu Sang, Michael Haberl, Raphael Ullmann

slide-24
SLIDE 24

Pre-processing: Undistortion

Original Picture Undistorted Picture

Lu Sang, Michael Haberl, Raphael Ullmann

slide-25
SLIDE 25

Pre-processing: Rectification

Lu Sang, Michael Haberl, Raphael Ullmann

slide-26
SLIDE 26

Pre-processing: Rectification

Lu Sang, Michael Haberl, Raphael Ullmann

slide-27
SLIDE 27

Pre-processing: Rectification

Lu Sang, Michael Haberl, Raphael Ullmann

slide-28
SLIDE 28

Pre-processing: Rectification

Left Picture Right Picture

Lu Sang, Michael Haberl, Raphael Ullmann

slide-29
SLIDE 29

Pre-processing: Rectification

Left Picture Right Picture

Lu Sang, Michael Haberl, Raphael Ullmann

slide-30
SLIDE 30

Modelling: Energy Function

Energy Function

argmind C♣p, dq S♣p, dq

  • Cost C♣p, dq for every pixel p and disparities d ✏ 1, ..., D
  • Regularization S♣p, dq
  • E.g. penalty for deviation of neighbouring pixels

Lu Sang, Michael Haberl, Raphael Ullmann

slide-31
SLIDE 31

Overview

Pre-processing Cost Calculation Pyramid Scheme Force Local Consistency Penalty Terms Median Filter

Lu Sang, Michael Haberl, Raphael Ullmann

slide-32
SLIDE 32

Cost Calculation: Comparing Windows

Lu Sang, Michael Haberl, Raphael Ullmann

slide-33
SLIDE 33

Cost Calculation: Comparing Windows

Lu Sang, Michael Haberl, Raphael Ullmann

slide-34
SLIDE 34

Cost Calculation: Cross Correlation

❞ ✏ ñ ✏ ✁ ❞ ✏ ñ ✏ ✁ r✁ s

Lu Sang, Michael Haberl, Raphael Ullmann

slide-35
SLIDE 35

Cost Calculation: Cross Correlation

❞ ✏ ñ ✏ ✁ ❞ ✏ ñ ✏ ✁ r✁ s

Lu Sang, Michael Haberl, Raphael Ullmann

slide-36
SLIDE 36

Cost Calculation: Cross Correlation

❞ ✏ ñ ✏ ✁ ❞ ✏ ñ ✏ ✁ r✁ s

Lu Sang, Michael Haberl, Raphael Ullmann

slide-37
SLIDE 37

Cost Calculation: Cross Correlation

❞ ✏ Total sum = 2 ñ Costs ✏ ✁2 ❞ ✏ ñ ✏ ✁ r✁ s

Lu Sang, Michael Haberl, Raphael Ullmann

slide-38
SLIDE 38

Cost Calculation: Cross Correlation

❞ ✏ Total sum = 2 ñ Costs ✏ ✁2 ❞ ✏ ñ ✏ ✁ r✁ s

Lu Sang, Michael Haberl, Raphael Ullmann

slide-39
SLIDE 39

Cost Calculation: Cross Correlation

❞ ✏ Total sum = 2 ñ Costs ✏ ✁2 ❞ ✏ ñ ✏ ✁ r✁ s

Lu Sang, Michael Haberl, Raphael Ullmann

slide-40
SLIDE 40

Cost Calculation: Cross Correlation

❞ ✏ Total sum = 2 ñ Costs ✏ ✁2 ❞ ✏ Total sum = 4 ñ Costs ✏ ✁4 r✁ s

Lu Sang, Michael Haberl, Raphael Ullmann

slide-41
SLIDE 41

Cost Calculation: Cross Correlation

❞ ✏ Total sum = 2 ñ Costs ✏ ✁2 ❞ ✏ Total sum = 4 ñ Costs ✏ ✁4 Normalization and Zero Mean: Values in r✁1, 1s

Lu Sang, Michael Haberl, Raphael Ullmann

slide-42
SLIDE 42

Cost Calculation: Result

Lu Sang, Michael Haberl, Raphael Ullmann

slide-43
SLIDE 43

Cost Calculation: Result

Lu Sang, Michael Haberl, Raphael Ullmann

slide-44
SLIDE 44

Cost Calculation: Result

  • Error Rate of NCC: 33.09%

Lu Sang, Michael Haberl, Raphael Ullmann

slide-45
SLIDE 45

Test Data

Figure: Left Image Figure: Right Image

Lu Sang, Michael Haberl, Raphael Ullmann

slide-46
SLIDE 46

Test Data

  • Middlebury Dataset
  • Ground Truth
  • Leaderboard

Lu Sang, Michael Haberl, Raphael Ullmann

slide-47
SLIDE 47

Algorithm Defect

Lu Sang, Michael Haberl, Raphael Ullmann

slide-48
SLIDE 48

Overview

Pre-processing Cost Calculation Pyramid Scheme Force Local Consistency Penalty Terms Median Filter

Lu Sang, Michael Haberl, Raphael Ullmann

slide-49
SLIDE 49

Pyramid Scheme

Lu Sang, Michael Haberl, Raphael Ullmann

slide-50
SLIDE 50

Pyramid Scheme

Lu Sang, Michael Haberl, Raphael Ullmann

slide-51
SLIDE 51

Pyramid Scheme

Lu Sang, Michael Haberl, Raphael Ullmann

slide-52
SLIDE 52

Pyramid Scheme: Results

Figure: Results of NCC Figure: Results of Pyramid Scheme

  • Error Rate of NCC: 33.09%
  • Error Rate of Pyramid Scheme: 28.1%

Lu Sang, Michael Haberl, Raphael Ullmann

slide-53
SLIDE 53

Overview

Pre-processing Cost Calculation Pyramid Scheme Force Local Consistency Penalty Terms Median Filter

Lu Sang, Michael Haberl, Raphael Ullmann

slide-54
SLIDE 54

Force Local Consistency

  • Interpret Cross Correlation as confidence indicator
  • Use only pixels with high confidence
  • Replace low confidence by values with high confidence in the

window

Lu Sang, Michael Haberl, Raphael Ullmann

slide-55
SLIDE 55

Force Local Consistency: Confidence Map

Figure: Results of NCC Figure: Confidence Map

  • Black point: trustworthy pixel with correct disparity

(C♣pq ➙ T).

  • Red point: unreliable pixel with violated disparity (C♣pq ➔ T).

Lu Sang, Michael Haberl, Raphael Ullmann

slide-56
SLIDE 56

Force Local Consistency C♣pq

Correct Disparity ➙ T Violated Disparity C♣p✶q Closest p✶ not unique Unique p✶ unique maxp✶PW C♣p✶q Original Disparity C♣p✶q P W all violated ➔ T

1 Mark all violated pixels.

Lu Sang, Michael Haberl, Raphael Ullmann

slide-57
SLIDE 57

Force Local Consistency C♣pq

Correct Disparity ➙ T Violated Disparity C♣p✶q Closest p✶ not unique Unique p✶ unique maxp✶PW C♣p✶q Original Disparity C♣p✶q P W all violated ➔ T

1 Mark all violated pixels. 2 Find the pixel with max NCC coefficient.

Lu Sang, Michael Haberl, Raphael Ullmann

slide-58
SLIDE 58

Force Local Consistency C♣pq

Correct Disparity ➙ T Violated Disparity C♣p✶q Closest p✶ not unique Unique p✶ unique maxp✶PW C♣p✶q Original Disparity C♣p✶q P W all violated ➔ T

1 Mark all violated pixels. 2 Find the pixel with max NCC coefficient. 3 Replace the disparity value by using the disparity of new pixel.

Lu Sang, Michael Haberl, Raphael Ullmann

slide-59
SLIDE 59

Force Local Consistency: Results

Figure: Results of NCC Figure: Results of FLC

  • Error Rate of NCC: 33.09%
  • Error Rate of FLC: 26.90%

Lu Sang, Michael Haberl, Raphael Ullmann

slide-60
SLIDE 60

Overview

Pre-processing Cost Calculation Pyramid Scheme Force Local Consistency Penalty Terms Median Filter

Lu Sang, Michael Haberl, Raphael Ullmann

slide-61
SLIDE 61

Local Penalty

Local Penalty

1 Smoothness check 2 Search four directions 3 Punish on NCC coefficient 4 Iterate

Lu Sang, Michael Haberl, Raphael Ullmann

slide-62
SLIDE 62

Overview

Pre-processing Cost Calculation Pyramid Scheme Force Local Consistency Penalty Terms Median Filter

Lu Sang, Michael Haberl, Raphael Ullmann

slide-63
SLIDE 63

Post Processing: Median Filter

ñ

Lu Sang, Michael Haberl, Raphael Ullmann

slide-64
SLIDE 64

Final Results

Figure: Results of FLC Figure: After Post-processing

  • Error Rate of FLC: 26.90%
  • Error Rate of FLC + Median filter: 21.10%

Lu Sang, Michael Haberl, Raphael Ullmann

slide-65
SLIDE 65

Overview

Problem Description Algorithms Results

Lu Sang, Michael Haberl, Raphael Ullmann

slide-66
SLIDE 66

Results: Average Error Rates

38.78% 37.37% 32.55% 32.88% 38.78% 36.04% 31.12% 31.60% 0.0% 10.0% 20.0% 30.0% 40.0% NCC Pyramid Scheme +Median Filter +Local Penalty NCC FLC +Median Filter +Local Penalty

Lu Sang, Michael Haberl, Raphael Ullmann

slide-67
SLIDE 67

Own Pictures

Figure: Left Image Figure: Pyramid Scheme + Median Filter

Lu Sang, Michael Haberl, Raphael Ullmann

slide-68
SLIDE 68

Own Pictures

Figure: Left Image Figure: Pyramid Scheme + Median Filter

Lu Sang, Michael Haberl, Raphael Ullmann

slide-69
SLIDE 69

Summary

Pre-processing Cost Calculation Pyramid Scheme Force Local Consistency Penalty Terms Median Filter

Lu Sang, Michael Haberl, Raphael Ullmann

slide-70
SLIDE 70

Failed Approaches

  • Disparity Calculation:
  • Cost Aggregation
  • Gradient Guided Weighted Match
  • Neural Network
  • Disparity Refinement
  • Local Interpolation
  • Cubic Plane Interpolation
  • Segment Penalty
  • Post-processing
  • Bilateral Filter
  • Gradient Guided Mask Filter

Lu Sang, Michael Haberl, Raphael Ullmann

slide-71
SLIDE 71

Failed Approaches

  • Disparity Calculation:
  • Cost Aggregation
  • Gradient Guided Weighted Match
  • Neural Network
  • Disparity Refinement
  • Local Interpolation
  • Cubic Plane Interpolation
  • Segment Penalty
  • Post-processing
  • Bilateral Filter
  • Gradient Guided Mask Filter

Figure: Disparity Image using Neural Networks

Lu Sang, Michael Haberl, Raphael Ullmann

slide-72
SLIDE 72

Thanks!

Disparity Determination

Cross Correlation Force Con- sistency Pyramid Scheme Local Penalty Refinement Median Filter More to Think and Try Neural Network

Segmentation Based FC

Low Texture Surface Failed Approaches Local In- terpolation

Segmentation

Bilateral Filter

Lu Sang, Michael Haberl, Raphael Ullmann

slide-73
SLIDE 73

Sources

[1] D. Scharstein, H. Hirschmüller, Y. Kitajima, G. Krathwohl, N. Nesic, X. Wang, and P. Westling. High-resolution stereo datasets with subpixel-accurate ground truth. In German Conference on Pattern Recognition (GCPR 2014), Münster, Germany, September 2014

Lu Sang, Michael Haberl, Raphael Ullmann