Stereo Vision, Multi-View Object and Scene Reconstruction Veronica - - PDF document

29/05/2018 1

Institut Mines-Télécom

Stereo Vision, Multi-View Object and Scene Reconstruction

Veronica SCURTU

ARTEMIS Department

30/05/2018

1 IMA 4509 - Stereo vision Institut Mines-Télécom

3D Visual content

 Our goal:

• Estimate 3D world properties from images

30/05/2018

IMA 4509 - Stereo vision 2

Camera(s) Image processing 3D vision 3D properties

29/05/2018 2

Institut Mines-Télécom

3D Visual content – Related subject areas

 Image processing (low level) :

• Image enhancement (unknown degradation)
• Image restoration (known degradation)
• Compression (JPEG, MPEG, ...)
• Extraction of salient points (corners, edges, ...)

30/05/2018

IMA 4509 - Stereo vision 3 Institut Mines-Télécom

3D Visual content – Related subject areas

 Image recognition (high level)

• Requires the use of information external to the system

(e.g. road signs)

• Especially 2D
• Somewhat 3D (e.g. face recognition)
• 3D too difficult, requires 3D Vision

30/05/2018

IMA 4509 - Stereo vision 4

29/05/2018 3

Institut Mines-Télécom

3D Visual content

 Our goal:

• Estimate 3D world properties from images

30/05/2018

IMA 4509 - Stereo vision 5

Camera(s) Image processing 3D vision 3D properties

Institut Mines-Télécom

Inferring 3D from 2D

30/05/2018

IMA 4509 - Stereo vision 6

 Model based pose estimation

Known model We can determine the pose of the model

 Stereo vision

Single (calibrated) camera Two (calibrated) cameras Arbitrary scene We can determine the position of points in the scene

29/05/2018 4

Institut Mines-Télécom

Outline

 Introduction to stereo vision  Image formation and projective geometry  Epipolar geometry and the epipolar constraint  Correspondence problem  Camera calibration  Multi-view reconstruction

30/05/2018

IMA 4509 - Stereo vision 7 Institut Mines-Télécom

Outline

 Introduction to stereo vision  Image formation and projective geometry  Epipolar geometry and the epipolar constraint  Correspondence problem  Camera calibration  Multi-view reconstruction

30/05/2018

IMA 4509 - Stereo vision 8

29/05/2018 5

Institut Mines-Télécom

Why Multiple Views?

 Structure and depth are inherently ambiguous from single views.

30/05/2018

IMA 4509 - Stereo vision 9

Optical center

P1 P2 P1’=P2’

Institut Mines-Télécom

Why Multiple Views?

 Structure and depth are inherently ambiguous from single views.

30/05/2018

IMA 4509 - Stereo vision 10

29/05/2018 6

Institut Mines-Télécom

What is Stereo Vision?

 Inferring depth from 2D images taken at the same time by two or more cameras

30/05/2018

IMA 4509 - Stereo vision 11 Institut Mines-Télécom

What is Stereo Vision?

 Computational stereo vision

• Studied extensively in the last 30 years
• Difficult; still being researched
• Commercial systems become more and more available

 Can be used with 3D Pattern Matching and Object Tracking in applications such as:

• Surveillance,
• Robotics,
• Inspection of object surfaces, height, shape, etc.

30/05/2018

IMA 4509 - Stereo vision 12

29/05/2018 7

Institut Mines-Télécom

Example

30/05/2018

IMA 4509 - Stereo vision 13

Left image Right image Reconstructed surface

Institut Mines-Télécom

Video mechanism of the eye

 Same principle as that of the camera

30/05/2018

IMA 4509 - Stereo vision 14

29/05/2018 8

Institut Mines-Télécom

Human visual perception

 Eye

• Optical part of the visual perception
• Amount of light is controlled by iris
• Image is brought into focus by lens
• Retinal cells capture information

 Visual pathways

• Carry visual information from the retina to the brain

 Visual cortex

• Primary visual cortex responds to low level visual

information such as frequencies, color and direction

• Dorsal and ventral streams are dealing with motions

and objects

30/05/2018

IMA 4509 - Stereo vision 15 Institut Mines-Télécom

Recovering 3D from 2D

 How can we automatically computer 3D geometry from 2D images?

30/05/2018

16 IMA 4509 - Stereo vision

29/05/2018 9

Institut Mines-Télécom

Depth cues

 What cues help us to perceive 3D shape and depth?  Depth cues are sources of information from within

• ur body or from the environment, that help us to

perceive how far wavy objects are

30/05/2018

17 IMA 4509 - Stereo vision Institut Mines-Télécom

Oculomotor Accommodation (eye focus) Convergence (eye rotation angle) Myosis (pupil size) Visual Binocular (retinal disparity) Monocular Static (classic pictorial cues) Motion-based (motion parallax and dynamic

• cclusion)

Depth cues

30/05/2018

18 IMA 4509 - Stereo vision

29/05/2018 10

Institut Mines-Télécom

Oculomotor depth cues

 Accommodation

• Change to optical parameter of the lens to bring an
• bject into focus
• Only effective for distances < 2m

30/05/2018

19 IMA 4509 - Stereo vision Institut Mines-Télécom

Oculomotor depth cues

 Convergence

• Movement of the eyes to opposite directions to gaze at

an object

• Only effective for distances < 2m

30/05/2018

20 IMA 4509 - Stereo vision

29/05/2018 11

Institut Mines-Télécom

Oculomotor depth cues

 Myosis

• Size of the pupil determines both amount of light and

depth of filed (DOF)

• Very weak depth cue for short distances

30/05/2018

21 IMA 4509 - Stereo vision Institut Mines-Télécom

Visual depth cues

 Monocular cues

• Require the use of only one eye to perceive depth, but

also operates with both eyes

• Used for pictorial cues and over longer distances
• Some are primary some are secondary cues

 Binocular cues

• Require the use of both eye to distinguish depth.
• Used for close objects
• Primary internal cues

30/05/2018

22 IMA 4509 - Stereo vision

29/05/2018 12

Institut Mines-Télécom

Monocular depth cues

 Mainly experiential and learned over time

• Shadow
• Illumination and shadow
• Relative sizes differences
• Motion parallax
• Aerial perspective
• Linear perspective
• Interposition
• Texture gradient
• Intensity gradient

30/05/2018

23 IMA 4509 - Stereo vision Institut Mines-Télécom

Monocular depth cues

30/05/2018

24 IMA 4509 - Stereo vision

29/05/2018 13

Institut Mines-Télécom

Occlusion

 Near objects block visual access to far objects

30/05/2018

IMA 4509 - Stereo vision 25 Institut Mines-Télécom

Relative size

30/05/2018

IMA 4509 - Stereo vision 26

29/05/2018 14

Institut Mines-Télécom

Relative size

30/05/2018

IMA 4509 - Stereo vision 27 Institut Mines-Télécom

Relative height

30/05/2018

IMA 4509 - Stereo vision 28

29/05/2018 15

Institut Mines-Télécom

Shadows/Shading

[Figure from Prados & Faugeras 2006]

An image without shadows or shading Shading added Shadows added

30/05/2018

29 IMA 4509 - Stereo vision Institut Mines-Télécom

Known size or scale

30/05/2018

IMA 4509 - Stereo vision 30

29/05/2018 16

Institut Mines-Télécom

Texture gradient

30/05/2018

31 IMA 4509 - Stereo vision Institut Mines-Télécom

Texture gradient

Paris Street: A Rainy Day by Gustave Caillebotte

30/05/2018

32 IMA 4509 - Stereo vision

29/05/2018 17

Institut Mines-Télécom

Perspective effects

Image credit: S. Seitz

30/05/2018

33 IMA 4509 - Stereo vision Institut Mines-Télécom

Perspective effects

30/05/2018

IMA 4509 - Stereo vision 34

29/05/2018 18

Institut Mines-Télécom

Linear perspective

30/05/2018

IMA 4509 - Stereo vision 35 Institut Mines-Télécom

Motion parallax

 Motion cues are created when the viewer moves his eyes or head  Relative object motion around a fixation point serves as depth cue  Very important depth cue for a large range of scene depths

30/05/2018

IMA 4509 - Stereo vision 36

29/05/2018 19

Institut Mines-Télécom

Binocular depth cues

30/05/2018

37 IMA 4509 - Stereo vision

 Most important depth cue for medium viewing distances  Both eyes observe scene from two slightly different angles  Comparisons of these two views produce depth cues  Around 5% of the population have difficulties with binocular depth  Basic idea behind any stereoscopic display technology

Institut Mines-Télécom

Binocular depth cues

 They are:

• Stereopsis

─ Corresponding retinal points ─ Retinal Disparity

• Ocular convergence

30/05/2018

38 IMA 4509 - Stereo vision

29/05/2018 20

Institut Mines-Télécom

Human stereopsis

 Human eyes are separated horizontally by approx. 6.3 cm  Existence of different retinal images leads to binocular disparity  Disparity occurs when eyes fixate on one object; others appear at different visual angles

30/05/2018

IMA 4509 - Stereo vision 39 Institut Mines-Télécom

Binocular disparity

 Binocular disparity provides cues about the relative depth of objects and their environment  Very effective for large disparities at close distances

Disparity: d = r-l = D-F.

30/05/2018

40 IMA 4509 - Stereo vision

29/05/2018 21

Institut Mines-Télécom

Ocular convergence

 A binocular depth cue related to the tension in the eye muscles when the eyes track inward to focus

• n objects close to the viewer

 The more tension in the eye muscle, the closer the object is  Works best at close distances

30/05/2018

IMA 4509 - Stereo vision 41 Institut Mines-Télécom

Depth cues – Range of effectiveness

30/05/2018

IMA 4509 - Stereo vision 42

0-2 m 2-20 m 30+ m Type of cue Accomodation x Oculomotor Converngence x Oculomotor Occlusion x x x Pictorial Relative size x x x Pictorial Relative height x x Pictorial Familiar size x x x Pictorial Texture gradient x x Pictorial Shadows x x x Pictorial Motion parallax x x Motion

29/05/2018 22

Institut Mines-Télécom

History of stereo and 3D vision

30/05/2018

IMA 4509 - Stereo vision 43

2D Photography 1839 - Louis Daguerre, William Fox Talbot, John Herschel 2D (still or motion) picture technologies are well developed and well accepted

Institut Mines-Télécom

History of stereo and 3D vision

30/05/2018

IMA 4509 - Stereo vision 44

Stereoscope invented in 1838 by Sir Charles Wheatsone 179 years ago!

29/05/2018 23

Institut Mines-Télécom

History of stereo and 3D vision

30/05/2018

IMA 4509 - Stereo vision 45

 In 1849 Sir David Brewster invented a lens based stereoscope  Sterograms were popular in the early 1900’s  A special viewer was needed to display different images to the left and right eyes

Institut Mines-Télécom

History of stereo and 3D vision

30/05/2018

IMA 4509 - Stereo vision 46

29/05/2018 24

Institut Mines-Télécom

History of stereo and 3D vision

 The first 3D movies in the 1950’s

30/05/2018

IMA 4509 - Stereo vision 47 Institut Mines-Télécom

History of stereo and 3D vision

 Anaglyphs provide a stereoscopic 3D effect when viewed with 2-color glasses (each lens a chromatically opposite color, usually red and cyan)

30/05/2018

IMA 4509 - Stereo vision 48

29/05/2018 25

Institut Mines-Télécom

History of stereo and 3D vision

 Current technology for 3D movies and computer displays is to use polarized glasses  The viewer wears eyeglasses which contain circular polarizers of opposite handedness

30/05/2018

IMA 4509 - Stereo vision 49 Institut Mines-Télécom

History of stereo and 3D vision

 Active shutter 3D glasses

 Alternate frame sequencing

• Alternately displays different perspective for each eye
• Uses liquid crystal or active shutter glasses

30/05/2018

IMA 4509 - Stereo vision 50

29/05/2018 26

Institut Mines-Télécom

History of stereo and 3D vision

 Head mounted display  Helmet or glasses with two small LCD or LED displays with magnifying lenses, one for each eye

 Stereo films, images, games, maintenance of complex systems,…

30/05/2018

IMA 4509 - Stereo vision 51 Institut Mines-Télécom

History of stereo and 3D vision

 Autostereoscopic

• Eliminates the eyeglasses and presents the

depth as it is

• Initially developed by Sharp

 Provide multiple views of the same scene, rather than just two  Tow main methods providing autostereoscopic vision :

• Parallax barrier
• Lenticular lens

30/05/2018

IMA 4509 - Stereo vision 52

29/05/2018 27

Institut Mines-Télécom

History of stereo and 3D vision

 Parallax Barrier

30/05/2018

IMA 4509 - Stereo vision 53

 A mask is placed over the LCD display which directs light from alternate pixel columns to each eye  Instant switching between 2D and 3D modes

Institut Mines-Télécom

History of stereo and 3D vision

 Lenticular lens

30/05/2018

IMA 4509 - Stereo vision 54

 An array of cylindrical lenses directs light from alternate pixel columns to a defined viewing zone  Each eye to receives a different image at an optimum distance

29/05/2018 28

Institut Mines-Télécom

History of stereo and 3D vision

 Correct viewing position of an autosteroscopic display

30/05/2018

IMA 4509 - Stereo vision 55 Institut Mines-Télécom

3D displays

30/05/2018

IMA 4509 - Stereo vision 56

29/05/2018 29

Institut Mines-Télécom

3D displays

30/05/2018

IMA 4509 - Stereo vision 57 Institut Mines-Télécom

3D displays

30/05/2018

IMA 4509 - Stereo vision 58

29/05/2018 30

Institut Mines-Télécom

3D displays

30/05/2018

IMA 4509 - Stereo vision 59 Institut Mines-Télécom

Benefits of stereo vision

 Relative depth judgment  Spatial localization  Breaking camouflage  Surface material perception  Judgment of surface curvature

30/05/2018

IMA 4509 - Stereo vision 60

29/05/2018 31

Institut Mines-Télécom

Main issues of stereo

 Geometry:

• What information is available?
• How do the camera views relate?

 Correspondences:

• What feature in view 1 corresponds to what feature in

view 2?

 Triangulation, reconstruction

• Interference in presence of noise

30/05/2018

IMA 4509 - Stereo vision 61 Institut Mines-Télécom

Outline

 Introduction to stereo vision  Image formation and projective geometry  Epipolar geometry and epipolar constraints  Correspondence problem  Camera calibration  Multi-view reconstruction

30/05/2018

IMA 4509 - Stereo vision 62

29/05/2018 32

Institut Mines-Télécom

Perspective projection

30/05/2018

IMA 4509 - Stereo vision 63

The axis of the real image plane O is the center of the projection The axis of the front image plane

z x f

xi 

From similar triangles we can say:

z x f

xi 

Institut Mines-Télécom

Perspective projection to a 2D image

 In a camera we have a flat image  Points in the 3D world are projected onto the image plane.

30/05/2018

IMA 4509 - Stereo vision 64

29/05/2018 33

Institut Mines-Télécom

Perspective projection to a 2D image

30/05/2018

IMA 4509 - Stereo vision 65

z x x

c c i

f 

z y y

c c i

f 

Institut Mines-Télécom

Estimating depth with stereo

 If we have 2 images of a scene, and hence can define 2 viewing rays, we can find the 3D location

• f that point by finding the intersection of the two

viewing rays.

30/05/2018

IMA 4509 - Stereo vision 66

scene point

• ptical

center image plane

29/05/2018 34

Institut Mines-Télécom

2 cameras, 2 simultaneous views Single moving camera and static scene

Stereo vision

30/05/2018

67 IMA 4509 - Stereo vision Institut Mines-Télécom

Stereo principle

 If we know:

• Intrinsic parameters of each camera
• The relative pose between the cameras

 If we measure:

• An image point in the left camera
• The corresponding point in the right camera

 Each image point corresponds to a ray emanating from the camera  We can intersect these rays (triangulate) to find the absolute point position

30/05/2018

IMA 4509 - Stereo vision 68

29/05/2018 35

Institut Mines-Télécom

Camera frame 1

Intrinsic parameters: Image coordinates relative to camera  Pixel coordinates Extrinsic parameters: Camera frame 1  Camera frame 2

Camera frame 2

We’ll assume for now that these parameters are given and fixed.

Camera parameters

 Extrinsic parameters : rotation matrix and translation vector  Intrinsic parameters : focal length, pixel sizes (mm), image center point, radial distortion

30/05/2018

69 IMA 4509 - Stereo vision Institut Mines-Télécom

Stereo geometry – simple case

30/05/2018

IMA 4509 - Stereo vision 70

Disparity

x x

r l

d    Assuming parallel optical axes, known camera parameters (i.e., calibrated cameras)

29/05/2018 36

Institut Mines-Télécom

Stereo geometry – simple case

30/05/2018

IMA 4509 - Stereo vision 71

Disparity

x x

r l

d  

Institut Mines-Télécom

Stereo geometry – simple case

30/05/2018

IMA 4509 - Stereo vision 72

y y x x

r l r l

y y f z b x f z x f z     

Disparity

x x

r l

d  

29/05/2018 37

Institut Mines-Télécom

Stereo geometry – simple case

30/05/2018

IMA 4509 - Stereo vision 73

f z f z y f z b f z x d b f b f z

y y x x x x

r l r l r l

       

Disparity

x x

r l

d  

Disparity refers to the difference in the image location of the same 3D point when projected under perspective to 2 different cameras.

Institut Mines-Télécom

Stereo geometry – simple case

30/05/2018

IMA 4509 - Stereo vision 74

Disparity

x x

r l

d  

Depth disparity baseline

Important equation!!!

29/05/2018 38

Institut Mines-Télécom

Stereo geometry – simple case

30/05/2018

IMA 4509 - Stereo vision 75

Disparity

x x

r l

d  

Depth disparity

Triangulation = determining depth from disparity Depth and disparity are inversely proportional.

Institut Mines-Télécom

Stereo geometry – general case

 Cameras not aligned, but relative pose known  Assuming f=1, we have

30/05/2018

IMA 4509 - Stereo vision 76

 In principle, you can find P by intersecting the rays OLpL and ORpR  However, they may not intersect  Instead, find the midpoint of the segment perpendicular to the two rays

29/05/2018 39

Institut Mines-Télécom

Stereo geometry – general case

 The projection of P onto the left image is

30/05/2018

IMA 4509 - Stereo vision 77

 The projection of P onto the right image is  where

Institut Mines-Télécom

Stereo geometry – general case

 Note that pL and MLP are parallel, so their cross product should be zero  Similarly for pR and MRP

30/05/2018

IMA 4509 - Stereo vision 78

 Point P should satisfy both  This is a system of 4 equations  Can solve for the 3 unknowns (XL, YL, , ZL) using least squares  Method also works for more than 2 cameras

29/05/2018 40

Institut Mines-Télécom

Stereo geometry – general case

30/05/2018

IMA 4509 - Stereo vision 79

 The transformation of coordinate system, from left to right is described by a rotation matrix R and a translation vector T.  More precisely, a point P described as PL in the left frame will be described in the right frame as ) (

1

T P R P

l r

    



Institut Mines-Télécom

Stereo disparity

 Tie in with the intro: for our purposes Disparity = Parallax  Disparity/Parallax inversely proportional to depth  Near objects appear to move more faster than far away ones when the camera translates sideways

30/05/2018

IMA 4509 - Stereo vision 80

29/05/2018 41

Institut Mines-Télécom

Stereo process

 Extract features from the left and right images  Match the left and right image features, to get their disparity in position (the “correspondence problem”)  Use stereo disparity to compute depth (the “reconstruction problem”)  Need to know focal length f, baseline b

• use prior knowledge or camera calibration

30/05/2018

IMA 4509 - Stereo vision 81

 The correspondence problem is the most difficult

http://vision.middlebury.edu/stereo/data/scenes2003/ Institut Mines-Télécom

Correspondence problem

 For every point in the left image, there are many possible matches in the right image  Locally, main points look similar → matches are ambiguous

30/05/2018

IMA 4509 - Stereo vision 82

29/05/2018 42

Institut Mines-Télécom

Correspondence problem

 We have 2 images taken from cameras with different intrinsic and extrinsic parameters  How do we match a point in the first image to a point in the second? How can we constrain our search?

30/05/2018

IMA 4509 - Stereo vision 83 Institut Mines-Télécom

Given a point in the left image, do you need to search the entire right image for the corresponding point?

30/05/2018

IMA 4509 - Stereo vision 84

29/05/2018 43

Institut Mines-Télécom

Establishing correspondences

 We can use the (known) geometry of the cameras to help limit the search for matches - epipolar geometry  The most important constraint is the epipolar constraint

• We can limit the search for a match to be along a

certain line in the other image

• Reduces the search space to a one-dimensional line
• Makes search for correspondences quicker

30/05/2018

IMA 4509 - Stereo vision 85 Institut Mines-Télécom

Outline

 Introduction to stereo vision  Image formation and projective geometry  Epipolar geometry and the epipolar constraint  Correspondence problem  Camera calibration  Multi-view reconstruction

30/05/2018

IMA 4509 - Stereo vision 86

29/05/2018 44

Institut Mines-Télécom

Epipolar constraint : normal image pair

30/05/2018

IMA 4509 - Stereo vision 87

With aligned cameras, search for corresponding point is 1D along the corresponding row of the other camera

Institut Mines-Télécom

Epipolar constraint : normal image pair

30/05/2018

IMA 4509 - Stereo vision 88

The match for P1 in the other image, must lie on the same epipolar line . The epipolar plane cuts the through the image plane(s) forming 2 epipolar lines.

29/05/2018 45

Institut Mines-Télécom

Matching example using epipolar lines

30/05/2018

IMA 4509 - Stereo vision 89

• For a patch in the left image
• Compare with the patches along

the same line in the right image

• Select patch with highest match

score

Institut Mines-Télécom

Epipolar constraint : general image pair

30/05/2018

IMA 4509 - Stereo vision 90

If cameras are not aligned, a 1D search can still be determined for the corresponding point. P, C1, C2 determine a plane that cuts image I2 in a line: P2 will be on that line.

29/05/2018 46

Institut Mines-Télécom

Epipolar constraint : general image pair

30/05/2018

IMA 4509 - Stereo vision 91 Institut Mines-Télécom

Epipolar constraint : general image pair

 The optical centers of the 2 cameras, a point P, and the image points p0 and p1 of P all lie in the same plane : epipolar plane

30/05/2018

IMA 4509 - Stereo vision 92

Epipolar plane Image plane

 These vectors are co-planar:

Epipolar line

29/05/2018 47

Institut Mines-Télécom

Example: converging cameras

30/05/2018

93 IMA 4509 - Stereo vision Institut Mines-Télécom

Where are the epipoles?

Example: parallel cameras

30/05/2018

94 IMA 4509 - Stereo vision

29/05/2018 48

Institut Mines-Télécom

Stereo image rectification

 What happens when the cameras are not aligned?  Arbitrary arrangements of camera result in image planes that are not parallel  Complicated epipolar representation and correspondence search

30/05/2018

95 IMA 4509 - Stereo vision Institut Mines-Télécom

Stereo image rectification

 Re-project image planes

• nto a common plane

parallel to the line between camera centers  Pixel motion is horizontal after this transformation

30/05/2018

96 IMA 4509 - Stereo vision

29/05/2018 49

Institut Mines-Télécom

Rectification example

Original image pair

• verlaid with several

epipolar lines Images rectified so that epipolar lines are horizontal and in vertical correspondence

30/05/2018

97 IMA 4509 - Stereo vision Institut Mines-Télécom

From geometry to algebra

 So far, we have the explanation in terms of geometry.  Now, how to express the epipolar constraints algebraically?

30/05/2018

98 IMA 4509 - Stereo vision

29/05/2018 50

Institut Mines-Télécom

Fundamental matrix

 The fundamental matrix F is the algebraic representation of epipolar geometry  F is a 3×3 matrix which relates corresponding points in stereo images.

30/05/2018

IMA 4509 - Stereo vision 99 Institut Mines-Télécom

Fundamental matrix

 Let p be a point in left image, p’ in right image  Epipolar relation

• p maps to epipolar line l’
• p’ maps to epipolar line l

 Epipolar mapping described by a 3x3 matrix F  It follows that

l’ l p p’ P

30/05/2018

101 IMA 4509 - Stereo vision

29/05/2018 51

Institut Mines-Télécom

Main idea

Stereo geometry, with calibrated cameras

30/05/2018

102 IMA 4509 - Stereo vision Institut Mines-Télécom

If the stereo rig is calibrated, we know : how to rotate and translate camera reference frame 1 to get to camera reference frame 2.

Rotation: 3 x 3 matrix R; translation: 3 x 1 vector T.

Stereo geometry, with calibrated cameras

30/05/2018

103 IMA 4509 - Stereo vision

29/05/2018 52

Institut Mines-Télécom

T RX X  

c c

'

Stereo geometry, with calibrated cameras

If the stereo rig is calibrated, we know : how to rotate and translate camera reference frame 1 to get to camera reference frame 2.

Rotation: 3 x 3 matrix R; translation: 3 x 1 vector T.

30/05/2018

104 IMA 4509 - Stereo vision Institut Mines-Télécom

Reminder: cross product

 Vector cross product takes two vectors and returns a third vector that’s perpendicular to both inputs.  So here, c is perpendicular to both a and b, which means the dot product = 0.

30/05/2018

105 IMA 4509 - Stereo vision

29/05/2018 53

Institut Mines-Télécom

T RX X'  

T T RX T X T       RX T 

   

RX T X X T X        



Normal to the plane

From geometry to algebra

30/05/2018

106 IMA 4509 - Stereo vision Institut Mines-Télécom

Can be expressed as a matrix multiplication.

c b b b a a a a a a b a                             

3 2 1 1 2 1 3 2 3

 

             

1 2 1 3 2 3

a a a a a a ax

Reminder: Matrix form of cross product

30/05/2018

107 IMA 4509 - Stereo vision

29/05/2018 54

Institut Mines-Télécom

T RX X'  

T T RX T X T       RX T 

   

RX T X X T X        



Normal to the plane

From geometry to algebra

30/05/2018

108 IMA 4509 - Stereo vision Institut Mines-Télécom

 

    RX T X

 

] [T    RX X

x

E is called the essential matrix, and it relates corresponding image points between both cameras, given the rotation and translation. If we observe a point in one image, its position in other image is constrained to lie on line defined by above. Let

R E ] [T x 

  EX X T

Essential matrix

30/05/2018

109 IMA 4509 - Stereo vision

29/05/2018 55

Institut Mines-Télécom

Fundamental matrix

 This matrix E is called the “Essential Matrix”

• when image intrinsic parameters are known

 This matrix F is called the “Fundamental Matrix”

• more generally (uncalibrated case)

 Can solve for F from point correspondences

• Each (p, p’) pair gives one linear equation in entries of

F

• 8 points give enough to solve for F (8-point algorithm)

30/05/2018

110 IMA 4509 - Stereo vision Institut Mines-Télécom

Summary: Epipolar geometry

 Epipolar plane: plane containing baseline and world point  Epipole: point of intersection of baseline with image plane  Epipolar line: intersection of epipolar plane with the image plane  Baseline: line joining the camera centers

• Epipolar Plane

Epipole Epipolar Line Baseline Epipole

30/05/2018

112 IMA 4509 - Stereo vision

29/05/2018 56

Institut Mines-Télécom

Summary: Epipolar constraint

 All epipolar lines intersect at the epipole  An epipolar plane intersects the left and right image planes in epipolar lines

• Epipolar Plane

Epipole Epipolar Line Baseline Epipole

30/05/2018

113 IMA 4509 - Stereo vision Institut Mines-Télécom

Potential matches for x have to lie on the corresponding line l’. Potential matches for x’ have to lie on the corresponding line l.

Summary: Epipolar constraint

x x’ X x’ X x’ X

30/05/2018

114 IMA 4509 - Stereo vision

29/05/2018 57

Institut Mines-Télécom

Summary

 Epipolar geometry

• Fundamental matrix maps from a point in one image

to a line (its epipolar line) in the other

• Can solve for F given corresponding points (e.g.,

interest points)

 Stereo depth estimation

• Main idea is to triangulate from corresponding image

points.

• Estimate disparity by finding corresponding points

along scanlines

• Depth is inverse to disparity

30/05/2018

IMA 4509 - Stereo vision 115 Institut Mines-Télécom

Outline

 Introduction to stereo vision  Image formation and projective geometry  Epipolar geometry and the epipolar constraint  Correspondence problem  Camera calibration  Multi-view reconstruction

30/05/2018

IMA 4509 - Stereo vision 116

29/05/2018 58

Institut Mines-Télécom

Correspondence problem

 Epipolar geometry constrains our search, but we still have a difficult correspondence problem.  Worst case scenarios

• A white board (no features)
• A checkered wallpaper (ambiguous matches)

 The problem is under constrained  To solve, we need to impose assumptions about the real world:

• Disparity limits
• Appearance
• Uniqueness
• Ordering
• Smoothness

30/05/2018

IMA 4509 - Stereo vision 117 Institut Mines-Télécom

Disparity limits

 Assume that valid disparities are within certain limits

• Constrains search

 Why usually true?  When is it violated?

30/05/2018

IMA 4509 - Stereo vision 118

29/05/2018 59

Institut Mines-Télécom

Appearance

 Assume features should have similar appearance in the left and right images  Why usually true?  When is it violated?

30/05/2018

IMA 4509 - Stereo vision 119 Institut Mines-Télécom

Uniqueness

30/05/2018

IMA 4509 - Stereo vision 120

 Assume that a point in the left image can have at most one match in the right image  Why usually true?

29/05/2018 60

Institut Mines-Télécom

Uniqueness

30/05/2018

IMA 4509 - Stereo vision 121

 Assume that a point in the left image can have at most one match in the right image  Why usually true?  When is it violated?

Institut Mines-Télécom

Ordering

 Assume features should be in the same left to right

• rder in each image

 Why usually true?

30/05/2018

IMA 4509 - Stereo vision 122

29/05/2018 61

Institut Mines-Télécom

Ordering

 Assume features should be in the same left to right

• rder in each image

 Why usually true?  When is it violated?

30/05/2018

IMA 4509 - Stereo vision 123 Institut Mines-Télécom

Smoothness

 Assume objects have mostly smooth surfaces, meaning that disparities should vary smoothly  Why usually true?  When is it violated?

30/05/2018

IMA 4509 - Stereo vision 124

29/05/2018 62

Institut Mines-Télécom

Methods of correspondence

 Match points based on local similarity between images  Two general approaches

• Correlation-based approaches
• Feature-based approaches

30/05/2018

IMA 4509 - Stereo vision 125 Institut Mines-Télécom

Correlation approach

 Matches image patches using correlation  Assumes only a translational difference between the two local patches (no rotation, or differences in appearance due to perspective)  A good assumption if patch covers a single surface, and surface is far away compared to baseline between cameras  Works well for scenes with lots of texture

30/05/2018

IMA 4509 - Stereo vision 126

29/05/2018 63

Institut Mines-Télécom

Correlation approach

 Similarity measures

• CC (cross-correlation)
• SSD (sum of squared differences)
• SAD (sum of absolute differences)

30/05/2018

IMA 4509 - Stereo vision 127 Institut Mines-Télécom

Cross correlation approach

 Select a range of disparities to search  For each patch in the left image, compute cross correlation score for every point along the epipolar line  Find maximum correlation score along that line

30/05/2018

IMA 4509 - Stereo vision 128

29/05/2018 64

Institut Mines-Télécom

Matching cost disparity Left Right scanline

Correspondence search with similarity constraint

 Slide a window along the right epipolar line and compare contents of that window with the reference window in the left image  Matching cost: SSD or normalized correlation

30/05/2018

129 IMA 4509 - Stereo vision Institut Mines-Télécom

Left Right scanline

SSD

Correspondence search with similarity constraint

30/05/2018

130 IMA 4509 - Stereo vision

29/05/2018 65

Institut Mines-Télécom

Left Right scanline

Normalized correlation

Correspondence search with similarity constraint

30/05/2018

131 IMA 4509 - Stereo vision Institut Mines-Télécom

Effect on window size

 Larger windows:

+ Robust to noise

• Reduced precision, less detail

 Smaller windows:

+ Good precision, more detail

• Sensitive to noise

30/05/2018

IMA 4509 - Stereo vision 132

29/05/2018 66

Institut Mines-Télécom

Feature matching

 Matches edges, lines, or corners  Gives a sparse reconstruction  May be better for scenes with little texture

30/05/2018

IMA 4509 - Stereo vision 133 Institut Mines-Télécom

Correspondence error sources

 Low-contrast ; textureless image regions  Occlusions  Camera calibration errors  Violations of brightness constancy (e.g., specular reflections)  Large motions

30/05/2018

134 IMA 4509 - Stereo vision

29/05/2018 67

Institut Mines-Télécom

Main steps: stereo with calibrated cameras

 Given image pair, R, L  Detect some features  Compute essential matrix E  Match features using the epipolar and other constraints  Triangulate for 3D structure

30/05/2018

135 IMA 4509 - Stereo vision Institut Mines-Télécom

Outline

 Introduction to stereo vision  Image formation and projective geometry  Epipolar geometry and the epipolar constraint  Correspondence problem  Camera calibration  Multi-view reconstruction

30/05/2018

IMA 4509 - Stereo vision 136

29/05/2018 68

Institut Mines-Télécom

Uncalibrated case

 What if we don’t know the camera parameters?  We can still reconstruct the 3D structure, up to certain ambiguities, if we can find correspondences between points…

30/05/2018

137 IMA 4509 - Stereo vision Institut Mines-Télécom

Image, camera and world frames

 There are 3 coordinate systems involved:

• Image
• Camera
• World

30/05/2018

IMA 4509 - Stereo vision 138

29/05/2018 69

Institut Mines-Télécom

Intrinsic parameters

 Intrinsic parameters are the parameters necessary to link the image’s coordinate system (pixel coordinates) to the idealized coordinate system (camera reference frame):

• Focal length
• Pixel size
• Distortion coefficients
• Image center

30/05/2018

IMA 4509 - Stereo vision 139 Institut Mines-Télécom

Extrinsic camera parameters

 Extrinsic parameters

• Position
• Orientation (pose) of camera

30/05/2018

IMA 4509 - Stereo vision 140

29/05/2018 70

Institut Mines-Télécom

Camera calibration

 Calibration means estimating extrinsic (external) and intrinsic (internal) parameters using observed camera data  Key idea: write the projection equation linking the known coordinates of a set of 3D points and their projection onto the image, and solve for the camera parameters

30/05/2018

IMA 4509 - Stereo vision 141 Institut Mines-Télécom

Possible applications

 Use these parameters to:

• correct for lens distortion,
• measure the size of an object in world units,
• determine the location of the camera in the scene

 Used in applications such as:

• machine vision to detect and measure objects,
• robotics,
• navigation systems,
• 3D scene reconstruction

30/05/2018

IMA 4509 - Stereo vision 142

29/05/2018 71

Institut Mines-Télécom

Examples

30/05/2018

IMA 4509 - Stereo vision 143 Institut Mines-Télécom

Camera model : pinhole camera

 A pinhole camera is a simple camera without a lens and with a single small aperture.  Light rays pass through the aperture and project an inverted image on the opposite side of the camera.  Describes the mathematical relationship between the coordinates of a 3D point and its projection onto the image plane of an ideal pinhole camera.

30/05/2018

IMA 4509 - Stereo vision 144

29/05/2018 72

Institut Mines-Télécom

Pinhole camera parameters

30/05/2018

IMA 4509 - Stereo vision 145

The camera coordinates are mapped into the image plane using the intrinsics parameters. The world points are transformed to camera coordinates using the extrinsics parameters.

Institut Mines-Télécom

Pinhole camera parameters

 The calibration algorithm calculates the camera matrix using the extrinsic and intrinsic parameters.  The extrinsic parameters represent a rigid transformation from 3D world coordinate system to the 3D camera's coordinate system.  The intrinsic parameters represent a projective transformation from the 3D camera's coordinates into the 2D image coordinates.

30/05/2018

IMA 4509 - Stereo vision 146

29/05/2018 73

Institut Mines-Télécom

Two steps process

 Modeling:

• Determine the equation that approximates the camera

behavior

• Define the set of unknowns in the equation (camera

parameters)

• The camera model is an approximation of the physics

& optics of the camera

 Calibration:

• Get the numeric value of every camera parameter

30/05/2018

IMA 4509 - Stereo vision 147 Institut Mines-Télécom

Perspective projection

30/05/2018

IMA 4509 - Stereo vision 148

Z Y f y 

 3D world mapped to 2D projection in image plane

Z X f x  Z f Y y  Z f X x 

Scene point Image coordinates

) , ( ) , , ( Z Y f Z X f Z Y X 

Image plane Camera center

29/05/2018 74

Institut Mines-Télécom

Perspective projection

30/05/2018

IMA 4509 - Stereo vision 149

Z Y f y 

 3D world mapped to 2D projection in image plane

Z X f x  Z f Y y  Z f X x 

Scene point Image coordinates

) , ( ) , , ( Z Y f Z X f Z Y X 

Image plane Camera center

Institut Mines-Télécom

Perspective projection

30/05/2018

IMA 4509 - Stereo vision 150

 But “pixels” are in some arbitrary spatial units

Z Y f y  Z Y y   Z X f x  Z X x  

29/05/2018 75

Institut Mines-Télécom

Perspective projection

30/05/2018

IMA 4509 - Stereo vision 151

 If pixels are not square

Z Y y   Z Y y   Z X x   Z X x  

Institut Mines-Télécom

Perspective projection

30/05/2018

IMA 4509 - Stereo vision 152

 We don’t know the origin of our camera pixel coordinates

Z Y y  

y

Z Y y    Z X x  

x

Z X x  

29/05/2018 76

Institut Mines-Télécom

Perspective projection

30/05/2018

IMA 4509 - Stereo vision 153

 If we have skew between camera pixel axes

y

Z Y y   

y

Z Y y ) sin(    

x

Z X x  

x

Z Y Z X x ) cot(      

y x x’ y’ θ

 

x y x y y x x ) cot( ' ) cos( ' sin '        

Institut Mines-Télécom

Camera calibration matrix (K)

 α and β represent the focal lengths in units of physical pixels  x0, y0 represent the coordinates of the principle point

30/05/2018

IMA 4509 - Stereo vision 154

           1

y x

K  

29/05/2018 77

Institut Mines-Télécom

Camera calibration matrix (K)

 α and β represent the focal lengths in units of physical pixels  x0, y0 represent the coordinates of the principle point Intrinsic parameters

30/05/2018

IMA 4509 - Stereo vision 155

           1

y x

K  

Institut Mines-Télécom

Camera calibration matrix (K)

 α and β represent the focal lengths in units of physical pixels  x0, y0 represent the coordinates of the principle point  s represents the skew coefficient 5 Intrinsic parameters

30/05/2018

IMA 4509 - Stereo vision 156

           1

y x

s K  

29/05/2018 78

Institut Mines-Télécom

Intrinsic parameters : Homogeneous coordinates

 Using homogenous coordinates, we can write this as:

30/05/2018

IMA 4509 - Stereo vision 157

                                 1 1 1 Z Y X y x

y x

 

 In pixels:

x K x

C 



In camera-based coords

Institut Mines-Télécom

Lens distortion

 We have assumed that lines are imaged as lines  Not quite true for real lenses

• Significant error for cheap optics and for short focal

lengths

30/05/2018

IMA 4509 - Stereo vision 158

29/05/2018 79

Institut Mines-Télécom

Radial distortion

 In pixel coordinates the correction is written

30/05/2018

IMA 4509 - Stereo vision 159 Institut Mines-Télécom

Extrinsic parameters

 The extrinsic parameters consist of a rotation, R, and a translation, t.  The origin of the camera's coordinate system is at its

• ptical center and its x- and y-axis define the image

plane.

30/05/2018

IMA 4509 - Stereo vision 160

29/05/2018 80

Institut Mines-Télécom

Extrinsic parameters

30/05/2018

IMA 4509 - Stereo vision 161

                               

t t t Z Y X

z y x c c c

Z Y X R

3x3 rotation matrix 3x1 translation vector 3 angles Camera coordinate system World coordinate system

Institut Mines-Télécom

Extrinsic parameters

) ( T P R P  

w c

World reference frame Camera reference frame

 

T c

Z Y X , ,  P

30/05/2018

162 IMA 4509 - Stereo vision

29/05/2018 81

Institut Mines-Télécom

Camera matrix

30/05/2018

IMA 4509 - Stereo vision 163

                                           t Z Y X R K K y x

Z Y X

c c c

1

Institut Mines-Télécom

Camera matrix

30/05/2018

IMA 4509 - Stereo vision 164

                                           t Z Y X R K K y x

Z Y X

c c c

1

 

             1 Z Y X t R K

3x4 matrix

29/05/2018 82

Institut Mines-Télécom

Camera matrix

30/05/2018

IMA 4509 - Stereo vision 165

                                           t Z Y X R K K y x

Z Y X

c c c

1

 

             1 Z Y X t R K

P Camera matrix

Institut Mines-Télécom

Single camera calibration methods

 Calibration using calibration patterns

• taking multiple images of a pattern from different

viewpoints.

• estimating camera calibaration matrix using these

images

 Auto-calibration

• estimating camera calibration matrix directly from real

image sequences

30/05/2018

IMA 4509 - Stereo vision 166

29/05/2018 83

Institut Mines-Télécom

Single camera calibration methods

 Method of Hall

• Lineal method
• Transformation matrix

 Method of Faugeras-Toscani

• Lineal method
• Obtaining camera parameters

 Method of Faugeras-Toscani with distortion

• Iterative method
• Radial distortion

 Method of Tsai

• Iterative method
• Radial distortion
• Focal distance estimation

 Method of Weng

• Iterative method
• Radial and tangential distortion

 … and many more

30/05/2018

IMA 4509 - Stereo vision 167 Institut Mines-Télécom

Single camera calibration : resectioning

 Estimating the camera matrix P from known x, X

30/05/2018

IMA 4509 - Stereo vision 168

                                 1 1

34 33 32 31 24 23 22 21 14 13 12 11

Z Y X P P P P P P P P P P P P y x

i i i i i

...

14 13 12 11

P Z P Y P X P x

i i i i

   

29/05/2018 84

Institut Mines-Télécom

Single camera calibration : resectioning

 So for each feature point i, we have:

30/05/2018

IMA 4509 - Stereo vision 169

 p A

2n x 12 12x1 vector

 

t R K P 

n = the number of correspondences

Institut Mines-Télécom

Camera calibration with a calibration object

Main idea  Place “calibration object” with known geometry in the scene  Get correspondences  Solve for mapping from scene to image

30/05/2018

170 IMA 4509 - Stereo vision

29/05/2018 85

Institut Mines-Télécom

Camera calibration with a calibration object

30/05/2018

IMA 4509 - Stereo vision 171 Institut Mines-Télécom

Camera calibration with a calibration object

 The use of a calibration pattern is one of the more reliable ways to estimate a camera’s intrinsic parameters

• A planar target is often used along with multiple images

taken at different poses  You can move the target in a controlled or just move it in an uncontrolled way

• It is best if the calibration object spans as much of the

image as possible  The strategy is to first solve for all calibration parameters except lens distortion by assuming there is no distortion

• Then perform a final nonlinear optimization which

includes solving for lens distortion

30/05/2018

IMA 4509 - Stereo vision 172

29/05/2018 86

Institut Mines-Télécom

Calibration problem

 P1… Pn with known positions in [Ow,iw,jw,kw]

30/05/2018

IMA 4509 - Stereo vision 173 Institut Mines-Télécom

Calibration problem

 P1… Pn with known positions in [Ow,iw,jw,kw]  p1… pn with known positions in the image  Goal: compute intrinsic and extrinsic parameters

30/05/2018

IMA 4509 - Stereo vision 174

29/05/2018 87

Institut Mines-Télécom

Calibration problem

 How many correspondences do we need?

• Our camera matrix has 11 unknown
• We need 11 equations
• 6 correspondences would do it

30/05/2018

IMA 4509 - Stereo vision 175 Institut Mines-Télécom

Calibration problem

 In practice, using more than 6 correspondences enables more robust results

30/05/2018

IMA 4509 - Stereo vision 176

29/05/2018 88

Institut Mines-Télécom

Calibration problem

30/05/2018

IMA 4509 - Stereo vision 177

Changed notation M = projective matrix

Institut Mines-Télécom

Calibration problem

30/05/2018

IMA 4509 - Stereo vision 178

29/05/2018 89

Institut Mines-Télécom

Calibration problem

30/05/2018

IMA 4509 - Stereo vision 179 Institut Mines-Télécom

Calibration problem

30/05/2018

IMA 4509 - Stereo vision 180

29/05/2018 90

Institut Mines-Télécom

Calibration problem

30/05/2018

IMA 4509 - Stereo vision 181 Institut Mines-Télécom

Homogeneous M x N Linear Systems

 Rectangular system (M>N)

• 0 is always a solution
• To find non-zero solution

─ Minimize |P m|2 ─ under the constraint |m|2 =1

30/05/2018

IMA 4509 - Stereo vision 182

M = number of equations = 2n N = number of unknowns = 11

29/05/2018 91

Institut Mines-Télécom

Calibration problem

 How do we solve this homogenous linear system?  Via SVD (Singular Value Decomposition) decomposition!

30/05/2018

IMA 4509 - Stereo vision 183 Institut Mines-Télécom

Calibration problem

30/05/2018

IMA 4509 - Stereo vision 184

29/05/2018 92

Institut Mines-Télécom

Extracting camera parameters

30/05/2018

IMA 4509 - Stereo vision 185 Institut Mines-Télécom

Extracting camera parameters

30/05/2018

IMA 4509 - Stereo vision 186

29/05/2018 93

Institut Mines-Télécom

Extracting camera parameters

30/05/2018

IMA 4509 - Stereo vision 187 Institut Mines-Télécom

Extracting camera parameters

30/05/2018

IMA 4509 - Stereo vision 188

29/05/2018 94

Institut Mines-Télécom

Stereo camera calibration

 There are several tasks in developing a practical system for binocular stereo: 1. Calibrate the intrinsic parameters for each camera. 2. Solve the relative orientation problem. 3. Stereo image rectification. 4. Compute conjugate pairs by feature matching or correlation. 5. Solve the stereo intersection problem for each conjugate pair. 6. Determine baseline distance. 7. Solve the absolute orientation problem to transform point measurements from the coordinate system of the stereo cameras to an absolute coordinate system for the scene.

30/05/2018

IMA 4509 - Stereo vision 189 Institut Mines-Télécom

Stereo camera calibration

30/05/2018

IMA 4509 - Stereo vision 190

29/05/2018 95

Institut Mines-Télécom

Stereo camera calibration

 Solve the relative orientation problem and determine the baseline by other means, such as using the stereo cameras to measure points that are at a known distance apart.  Calibrates the rigid body transformation between the two cameras.  Since the baseline has been calibrated, the point measurements will be in real units and the stereo system can be used to measure the relationships between points on objects in the scene.  It is not necessary to solve the absolute orientation problem, unless the point measurements must be transformed into another coordinate system.

30/05/2018

IMA 4509 - Stereo vision 191 Institut Mines-Télécom

Stereo camera calibration

30/05/2018

IMA 4509 - Stereo vision 192

 Solve the relative orientation problem and

• btain point measurements in the arbitrary

system of measurement that results from assuming unit baseline distance.  The point measurements will be correct, except for the unknown scale factor.  Distance ratios and angles will be correct.  If the baseline distance is obtained later, then the point coordinates can be multiplied by the baseline distance to get point measurements in known units.

29/05/2018 96

Institut Mines-Télécom

Stereo camera calibration

30/05/2018

IMA 4509 - Stereo vision 193

 Solve the exterior orientation problem for each stereo camera.  This provides the transformation from the coordinate systems of the left and right camera into absolute coordinates.  The point measurements obtained by intersecting rays will automatically be in absolute coordinates with known units, and no further transformations are necessary.

Institut Mines-Télécom

Outline

 Introduction to stereo vision  Image formation and projective geometry  Epipolar geometry and the epipolar constraint  Correspondence problem  Camera calibration  Multi-view reconstruction

30/05/2018

IMA 4509 - Stereo vision 194

29/05/2018 97

Institut Mines-Télécom

Context

 A new trend in 3D Gfx: modeling by capturing the real world

30/05/2018

IMA 4509 - Stereo vision page 195 Institut Mines-Télécom

Acquisition of 3D data

 3D laser scanner

• Direct acquisition
• Renders in real-time the acquired points as coordinates
• Advantages: very accurate, fast results
• Disadvantages: quite expensive, difficulties with shiny,

shimmering or transparent objects

30/05/2018

IMA 4509 - Stereo vision 196

Structured light

29/05/2018 98

Institut Mines-Télécom

Acquisition of 3D data

 Photogrammetry

• Indirect acquisition
• Extracts spatial coordinates using different rendering

techniques

• Mono, stereo or multi images
• Disadvantages: slower, less accurate
• Advantages: cheaper, easier to use

30/05/2018

IMA 4509 - Stereo vision 197 Institut Mines-Télécom

3D reconstruction from images

 Goal

• Automatic construction of photo-realistic 3D models of

a scene from multiple images taken from a set of arbitrary viewpoints

• Image-based modeling; 3D photography

 Applications

• Interactive visualization of remote environments or
• bjects by a virtual video camera for fly-bys, mission

rehearsal and planning, site analysis

• Virtual modification of a real scene for augmented

reality tasks

30/05/2018

IMA 4509 - Stereo vision 198

29/05/2018 99

Institut Mines-Télécom

Dense reconstruction : motivation

 Accurate 3D models → cultural heritage  Reconstruction of houses, buildings, famous touristic sites

30/05/2018

IMA 4509 - Stereo vision 199 Institut Mines-Télécom

Dense reconstruction: challenges

 Scene elements do not always look the same in the images  Camera-related problems

• Image noise

30/05/2018

IMA 4509 - Stereo vision 200

29/05/2018 100

Institut Mines-Télécom

Dense reconstruction: challenges

 Scene elements do not always look the same in the images  Camera-related problems

• Image noise
• Lens distortion

30/05/2018

IMA 4509 - Stereo vision 201 Institut Mines-Télécom

Dense reconstruction: challenges

 Scene elements do not always look the same in the images  Camera-related problems

• Image noise
• Lens distortion
• Color/chromatic aberration

30/05/2018

IMA 4509 - Stereo vision 202

29/05/2018 101

Institut Mines-Télécom

Dense reconstruction: challenges

 Scene elements do not always look the same in the images  Camera-related problems

• Image noise
• Lens distortion
• Color/chromatic aberration

 Viewpoint‐related problems

30/05/2018

IMA 4509 - Stereo vision 203 Institut Mines-Télécom

Dense reconstruction: challenges

 Scene elements do not always look the same in the images  Camera-related problems

• Image noise
• Lens distortion
• Color/chromatic aberration

 Viewpoint‐related problems

• Perspective distortions

30/05/2018

IMA 4509 - Stereo vision 204

29/05/2018 102

Institut Mines-Télécom

Dense reconstruction: challenges

 Scene elements do not always look the same in the images  Camera-related problems

• Image noise
• Lens distortion
• Color/chromatic aberration

 Viewpoint‐related problems

• Perspective distortions
• Occlusions

30/05/2018

IMA 4509 - Stereo vision 205 Institut Mines-Télécom

Dense reconstruction: challenges

 Scene elements do not always look the same in the images  Camera-related problems

• Image noise
• Lens distortion
• Color/chromatic aberration

 Viewpoint‐related problems

• Perspective distortions
• Occlusions
• Specular reflections

30/05/2018

IMA 4509 - Stereo vision 206

29/05/2018 103

Institut Mines-Télécom

Dense reconstruction: challenges

 Scene elements do not always look the same in the images  Camera-related problems

• Image noise
• Lens distortion
• Color/chromatic aberration

 Viewpoint‐related problems

• Perspective distortions
• Occlusions
• Specular reflections

 Scene‐related problems

30/05/2018

IMA 4509 - Stereo vision 207 Institut Mines-Télécom

Dense reconstruction: challenges

 Scene elements do not always look the same in the images  Camera-related problems

• Image noise
• Lens distortion
• Color/chromatic aberration

 Viewpoint‐related problems

• Perspective distortions
• Occlusions
• Specular reflections

 Scene‐related problems

• Illumination changes

30/05/2018

IMA 4509 - Stereo vision 208

29/05/2018 104

Institut Mines-Télécom

Dense reconstruction: challenges

 Scene elements do not always look the same in the images  Camera-related problems

• Image noise
• Lens distortion
• Color/chromatic aberration

 Viewpoint‐related problems

• Perspective distortions
• Occlusions
• Specular reflections

 Scene‐related problems

• Illumination changes
• Moving objects

30/05/2018

IMA 4509 - Stereo vision 209 Institut Mines-Télécom

Multi-view reconstruction

 Main idea: reconstruction from multiple images

30/05/2018

IMA 4509 - Stereo vision 210

29/05/2018 105

Institut Mines-Télécom

Multi-view reconstruction

 Main idea: collect images from different views

30/05/2018

IMA 4509 - Stereo vision 211 Institut Mines-Télécom

Multi-view reconstruction

 Main idea: establish correspondences and triangulate

30/05/2018

IMA 4509 - Stereo vision 212

29/05/2018 106

Institut Mines-Télécom

3D from images methods

 Volumetric methods  Surface deformation methods  Patch-based methods  Depth map fusion methods

30/05/2018

IMA 4509 - Stereo vision 213 Institut Mines-Télécom

Volumetric methods

214

O1 O2 O3

• Calibrated cameras and object

30/05/2018

IMA 4509 - Stereo vision

29/05/2018 107

Institut Mines-Télécom

Volumetric methods

215

O1 O2 O3

• Set initial 3D volumetric region including object

30/05/2018

IMA 4509 - Stereo vision Institut Mines-Télécom

Volumetric methods

216

O1 O2 O3

• Back-project each silhouette along the ray

30/05/2018

IMA 4509 - Stereo vision

29/05/2018 108

Institut Mines-Télécom

Volumetric methods

217

O1 O2 O3

• Obtain 3D volumetric data from intersecting back-

projected volume

30/05/2018

IMA 4509 - Stereo vision Institut Mines-Télécom

Volumetric methods

 First the object space is split up into a 3D grid of voxels.  Each voxel is intersected with each silhouette volume.  Only voxels that lie inside all silhouette volumes remain part of the final shape.

30/05/2018

IMA 4509 - Stereo vision 218

29/05/2018 109

Institut Mines-Télécom

Volumetric methods

 Advantages:

• Simple to implement and fairly robust
• Complete closed surface
• Commonly used as the effective initial boundary

 Limitations:

• Only produced line hull

30/05/2018

IMA 4509 - Stereo vision 219 Institut Mines-Télécom

Volumetric methods

 Advantages:

• Simple to implement and fairly robust
• Complete closed surface
• Commonly used as the effective initial boundary

 Limitations:

• Only produced line hull
• Can’t detect non-convex region

30/05/2018

IMA 4509 - Stereo vision 220

29/05/2018 110

Institut Mines-Télécom

Volumetric methods

 Advantages:

• Simple to implement and fairly robust
• Complete closed surface
• Commonly used as the effective initial boundary

 Limitations:

• Only produced line hull
• Can’t detect non-convex region
• Specific color is used as the background

30/05/2018

IMA 4509 - Stereo vision 221 Institut Mines-Télécom

Volumetric methods : results

30/05/2018

IMA 4509 - Stereo vision 222

29/05/2018 111

Institut Mines-Télécom

Surface deformation methods

30/05/2018

IMA 4509 - Stereo vision 223 Institut Mines-Télécom

Surface deformation methods

30/05/2018

IMA 4509 - Stereo vision 224

29/05/2018 112

Institut Mines-Télécom

Surface deformation methods

 Challenges and issues:

• Large number of nodes needed to be able to wrap

really tightly around the object

• Maintaining the topology to stay a simple closed curve

 Solution: use a level set

30/05/2018

IMA 4509 - Stereo vision 225 Institut Mines-Télécom

Surface deformation methods : results

30/05/2018

IMA 4509 - Stereo vision 227

29/05/2018 113

Institut Mines-Télécom

Surface deformation methods : results

30/05/2018

IMA 4509 - Stereo vision 228 Institut Mines-Télécom

Patch-based methods

30/05/2018

IMA 4509 - Stereo vision 229

Extent Mesh Position Normal Patch

 Surface is locally approximated by a small rectangle → the patch  Patch consists of

• Position (x, y, z)
• Normal (nx, ny, nz)
• Extent (radius)

 Tangent plane approximation

29/05/2018 114

Institut Mines-Télécom

Extent

Patch definition

 Patch p is defined by

• Position c(p)
• Normal n(p)
• Visible images V(p)

 Extent is set so that p is roughly 9x9 pixels in V(p)

Position c(p) Normal n(p) Visible images V(p) 9x9 pixels

30/05/2018

230 IMA 4509 - Stereo vision Institut Mines-Télécom

Patch-based method

30/05/2018

IMA 4509 - Stereo vision 231

29/05/2018 115

Institut Mines-Télécom

Patch-based methods: Algorithm overview

• 1. Feature detection
• 2. Initial feature matching
• 3. Patch expansion
• 4. Patch filtering
• 5. Surface reconstruction

30/05/2018

232 IMA 4509 - Stereo vision Institut Mines-Télécom

Patch-based methods

 Advantages:

• Works well for various objects and scenes

 Disadvantages:

• Surfaces must be well-textured
• Problematic for architectural scenes

30/05/2018

IMA 4509 - Stereo vision 233

29/05/2018 116

Institut Mines-Télécom

Patch-based methods : results

30/05/2018

IMA 4509 - Stereo vision 234

Yasutaka Furukawa @ Washington University in St. Louis

https://www.youtube.com/watch?v=NdeD4cjLI0c

Institut Mines-Télécom

Depth map fusion methods

 Compute depth hypotheses form each view using neighboring views

• Compute completely un-regularized depth maps

independently for each viewpoint

 Merge depth maps intro single volume with the use of redundancy

• Compute one correlation curve per image, then find its

local maximum

• Utilize the pixel neighborhood in a 2D discrete MRF:

peaks as hypothesis and “unknown depth” label against spatially inconsistent peaks;

• Model sensor probabilistically as a Gaussian + Uniform

mixture, evolution of estimates in sequential inference.

 Extract 3D surface from this volume

• Regularize at final stage;
• Graph cut-based segmentation with photo consistency;

─ Surface extracted by marching cubes from binary map.

30/05/2018

IMA 4509 - Stereo vision 238

29/05/2018 117

Institut Mines-Télécom

Depth map fusion methods

 Fuses a set of depth maps computed using

• cclusion-robust photo-consistency

 Advantages:

• Elegant pipeline
• Plug-n-play blocks
• Easily parallelizable

 Disadvantages:

• Photo-consistency metric is simple, but not optimal
• The metric suffers when images are not well textured
• r at low resolution

30/05/2018

IMA 4509 - Stereo vision 240 Institut Mines-Télécom

Depth map fusion methods : results

30/05/2018

IMA 4509 - Stereo vision 241

Christopher Zach – “Fast and High Quality Fusion of Depth Maps” Simon Fuhrmann – “Fusion of Depth Maps with Multiple Scales”

29/05/2018 118

Institut Mines-Télécom

MVS (Multi-view Stereo) system

30/05/2018

IMA 4509 - Stereo vision 242 Institut Mines-Télécom

Platform Culture 3D Clouds

30/05/2018

IMA 4509 - Stereo vision 243

Image acquisition Automatic detection of correspondence between non-oriented images Automatic image calibration and orientation Dense 3D point cloud generation from oriented images

Multi-stereo correlation

29/05/2018 119

Institut Mines-Télécom

Tie points extraction

 Automatic detection of correspondence between non-

• riented images

30/05/2018

244 IMA 4509 - Stereo vision Institut Mines-Télécom

Orientation computation

30/05/2018

245 IMA 4509 - Stereo vision

29/05/2018 120

Institut Mines-Télécom Master image stereo images Depth map Point cloud

Depth maps

 Depth map computation and dense point cloud generation

30/05/2018

246 IMA 4509 - Stereo vision Institut Mines-Télécom

Point cloud generation

30/05/2018

247 IMA 4509 - Stereo vision

29/05/2018 121

Institut Mines-Télécom

Image acquisition

30/05/2018

IMA 4509 - Stereo vision 248 Institut Mines-Télécom

Point cloud result

30/05/2018

IMA 4509 - Stereo vision 249

29/05/2018 122

Institut Mines-Télécom

Point cloud result: zoom

30/05/2018

IMA 4509 - Stereo vision 250 Institut Mines-Télécom

Bibliography

 D. Scharstein, R. Szeliski and R. Zabih, "A taxonomy and evaluation of dense two-frame stereo correspondence algorithms," Proceedings IEEE Workshop on Stereo and Multi- Baseline Vision (SMBV 2001), Kauai, HI, 2001, pp. 131-140.  R. Hartley, A. Zisserman Multiple View Geometry in Computer Vision  Cambridge University Press, 2003 (2nd edition)  http://vision.middlebury.edu/stereo - extensive website with evaluations of algorithms, test data, code  C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.

30/05/2018

IMA 4509 - Stereo vision 251