Introduction to Computer Vision for Robotics AE640A Autonomous - - PowerPoint PPT Presentation

▶
introduction to computer vision for robotics
SMART_READER_LITE
LIVE PREVIEW

Introduction to Computer Vision for Robotics AE640A Autonomous - - PowerPoint PPT Presentation

Mar 23, 2023 276 likes •929 views

Introduction to Computer Vision for Robotics AE640A Autonomous Navigation 11 th March, 2019 Harsh Sinha 1 Introduction to Computer Vision 1 From the last class Actual image plane behind the pinhole Focal distance Focal distance Harsh

slide-1
SLIDE 1

Introduction to Computer Vision Harsh Sinha

Introduction to Computer Vision for Robotics

AE640A Autonomous Navigation

11th March, 2019

1 1

slide-2
SLIDE 2

Introduction to Computer Vision Harsh Sinha

… From the last class

2

Actual image plane behind the pinhole Focal distance Focal distance

slide-3
SLIDE 3

Introduction to Computer Vision Harsh Sinha

… From the last class

3

p’ K M = K [I 0] R T P H

slide-4
SLIDE 4

Introduction to Computer Vision Harsh Sinha

Lecture Outline

  • Stereo Vision

○ Introduction to Stereo Vision ○ Epipolar Geometry ○ The correspondence problem

  • Stereo Matching

○ Various methods for Stereo Matching ○ Stereo Block Matching ○ A look at SGBM

4

slide-5
SLIDE 5

Introduction to Computer Vision Harsh Sinha

Stereo Vision

5

slide-6
SLIDE 6

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

6 Credits: Kenji Hata, Silvio Savarese

slide-7
SLIDE 7

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

7 Credits: Fei Fei Li

slide-8
SLIDE 8

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

8 Credits: Fei Fei Li

slide-9
SLIDE 9

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

9 Credits: Fei Fei Li

How do humans figure out 3D in 2D images?

slide-10
SLIDE 10

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

10 Credits: Fei Fei Li

How do humans figure out 3D in 2D images? 1. Shading

slide-11
SLIDE 11

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

11 Credits: Fei Fei Li

How do humans figure out 3D in 2D images? 1. Shading 2. Texture

slide-12
SLIDE 12

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

12 Credits: Fei Fei Li

How do humans figure out 3D in 2D images? 1. Shading 2. Texture 3. Focus

slide-13
SLIDE 13

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

13 Credits: Gaurav Pandey, Ford

The stereo problem:

  • Nature Inspired approach to

vision, i.e, 3D with two sensors.

  • How to figure out the shape,

more specifically the depth, of

  • bjects from a set of two or

more images?

slide-14
SLIDE 14

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

So, How do we go we go from Stereo Images to Depth Information ?

14

slide-15
SLIDE 15

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

15 Credits: Gaurav Pandey, Ford

slide-16
SLIDE 16

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

16

slide-17
SLIDE 17

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

17

Note: We have the image planes parallel here. Creating such images from non parallel cameras is called rectification.

slide-18
SLIDE 18

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

18 Credits: Gaurav Pandey, Ford

slide-19
SLIDE 19

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

19 Credits: Fei Fei Li

slide-20
SLIDE 20

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

20 Credits: Fei Fei Li

Given this point how do you find the corresponding point on the other image?

slide-21
SLIDE 21

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

21 Credits: Fei Fei Li

Given this point how do you find the corresponding point on the other image? Search the whole image?

slide-22
SLIDE 22

Introduction to Computer Vision Harsh Sinha

Introduction to stereo vision

22

Difficult to solve accurately, very expensive without special methods

Given this point how do you find the corresponding point on the other image? Search the whole image?

slide-23
SLIDE 23

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry

23 Credits: Fei Fei Li

slide-24
SLIDE 24

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry

24 Credits: Fei Fei Li

Baseline Camera 1 Center Camera 2 Center Epipoles Epipolar Plane

slide-25
SLIDE 25

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry

25 Credits: Richard Hartley, Andrew Zisserman

Epipolar Pencil

slide-26
SLIDE 26

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry

26 Credits: Fei Fei Li

slide-27
SLIDE 27

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry

27 Credits: Fei Fei Li

slide-28
SLIDE 28

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry

28 Credits: Fei Fei Li

Easier to solve. Can use simple SSD or similar methods.

Search along this line for the closest point. Computationally way more efficient.

slide-29
SLIDE 29

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry

29 Credits: Fei Fei Li

slide-30
SLIDE 30

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry

30

Assumed to be canonical camera

slide-31
SLIDE 31

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry

31

Assumed to be canonical camera RTp’ - RTT is p’ in SO RTT also lies in plane => RTT x (RTp’ - RTT) is perpendicular to epipolar plane

slide-32
SLIDE 32

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry

32

Assumed to be canonical camera => RTT x (RTp’ - RTT) = RT(T x p’) is perpendicular to p => (RT(T x p’))Tp = 0 => (T x p’T)Rp = 0

slide-33
SLIDE 33

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry

33

From Linear Algebra, the cross product of two vectors can be written as :

slide-34
SLIDE 34

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry

34

From Linear Algebra, the cross product of two vectors can be written as : [ax] : skew symmetric

slide-35
SLIDE 35

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry

35 Credits: Fei Fei Li

slide-36
SLIDE 36

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry: Essential Matrix (E)

36 Credits: Fei Fei Li

slide-37
SLIDE 37

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry

37

Why?

slide-38
SLIDE 38

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry

38

ax + by + c = 0 i.e L = [a b c]T represents a line in homogeneous coordinates. => zTL = 0 where, z = [x, y, 1]T

slide-39
SLIDE 39

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry

39

slide-40
SLIDE 40

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry: Fundamental Matrix (F)

40

F: Fundamental Matrix

slide-41
SLIDE 41

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry: Properties of F

41 Credits: Richard Hartley, Andrew Zisserman

slide-42
SLIDE 42

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry: Estimating F

42 Credits: Robert Collins, Penn State

slide-43
SLIDE 43

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry: Estimating F

43 Credits: Robert Collins, Penn State

slide-44
SLIDE 44

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry: Estimating F

44 Credits: Robert Collins, Penn State

slide-45
SLIDE 45

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry: Estimating F

45 Credits: Robert Collins, Penn State

slide-46
SLIDE 46

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry: Estimating F

46 Credits: Robert Collins, Penn State

slide-47
SLIDE 47

Introduction to Computer Vision Harsh Sinha

Epipolar Geometry: Estimating F

47 Credits: Robert Collins, Penn State

slide-48
SLIDE 48

Introduction to Computer Vision Harsh Sinha

Stereo Matching

48

slide-49
SLIDE 49

Introduction to Computer Vision Harsh Sinha

Various Methods for Stereo Matching

49

slide-50
SLIDE 50

Introduction to Computer Vision Harsh Sinha

Various Methods for Stereo Matching

50

slide-51
SLIDE 51

Introduction to Computer Vision Harsh Sinha

Various Methods for Stereo Matching

51

slide-52
SLIDE 52

Introduction to Computer Vision Harsh Sinha

Various Methods for Stereo Matching

52

slide-53
SLIDE 53

Introduction to Computer Vision Harsh Sinha

Various Methods for Stereo Matching

53

slide-54
SLIDE 54

Introduction to Computer Vision Harsh Sinha

Various Methods for Stereo Matching

54

slide-55
SLIDE 55

Introduction to Computer Vision Harsh Sinha

Various Methods for Stereo Matching

55

slide-56
SLIDE 56

Introduction to Computer Vision Harsh Sinha

Various Methods for Stereo Matching

56

slide-57
SLIDE 57

Introduction to Computer Vision Harsh Sinha

Stereo Block Matching

  • Similar to what we just saw in window sizes example.
  • Idea is to instead of matching pixel values, match regions of image, this is

done in order to increase robustness in the depth prediction.

  • Sparse Stereo Matching: Use of key points or features to serve as

corresponding points on the two images.

  • Dense Stereo Matching: Match all pixels in a region along a scan line in pair
  • f stereo rectified images.

57

slide-58
SLIDE 58

Introduction to Computer Vision Harsh Sinha

Stereo Block Matching

58 Credits: Trym Vegard Haavardsholm

slide-59
SLIDE 59

Introduction to Computer Vision Harsh Sinha

Stereo Block Matching: Global Optimization

59 Credits: Trym Vegard Haavardsholm

slide-60
SLIDE 60

Introduction to Computer Vision Harsh Sinha

Stereo Block Matching: Global Optimization

60 Credits: HEIKO HIRSCHMÜLLER

Cost of pixel wise matching Penalty based on neighbours mismatches, I.e, penalty for neighbours having different disparity Minimize E

  • ver D to get

D*

slide-61
SLIDE 61

Introduction to Computer Vision Harsh Sinha

Stereo Block Matching: Global Optimization

61

Cost of pixel wise matching Penalty based on neighbours mismatches, I.e, penalty for neighbours having different disparity Minimize E

  • ver D to get

D*

Guess the Drawbacks!!

slide-62
SLIDE 62

Introduction to Computer Vision Harsh Sinha

Stereo Block Matching: Global Optimization

62

Cost of pixel wise matching Penalty based on neighbours mismatches, I.e, penalty for neighbours having different disparity Minimize E

  • ver D to get

D*

Guess the Drawbacks!!

  • Too Computationally Intensive
  • NP Complete Problem
slide-63
SLIDE 63

Introduction to Computer Vision Harsh Sinha

Stereo Block Matching: Semi Global Matching

63 Credits: HEIKO HIRSCHMÜLLER

slide-64
SLIDE 64

Introduction to Computer Vision Harsh Sinha

Stereo Block Matching: Semi Global Matching

64 Credits: HEIKO HIRSCHMÜLLER