Lecture 18: Multi-view reconstruction 1 Announcements If you - - PowerPoint PPT Presentation

▶
lecture 18 multi view reconstruction
SMART_READER_LITE
LIVE PREVIEW

Lecture 18: Multi-view reconstruction 1 Announcements If you - - PowerPoint PPT Presentation

Dec 11, 2022 340 likes •890 views

Lecture 18: Multi-view reconstruction 1 Announcements If you have a question: just ask it (or send message saying that you have a question) Please send us feedback! PS8 out: representation learning Final presentation

slide-1
SLIDE 1

1

Lecture 18: Multi-view reconstruction

slide-2
SLIDE 2
  • If you have a question: just ask it (or send message

saying that you have a question)

  • Please send us feedback!

  • PS8 out: representation learning
  • Final presentation will take place over video chat.
  • We’ll send a sign-up sheet next week

2

Announcements

slide-3
SLIDE 3

Today

  • Finding correspondences
  • RANSAC
  • Structure from motion

3

slide-4
SLIDE 4

Motivating example: panoramas

Source: N. Snavely

4

slide-5
SLIDE 5

Warping with a homography

Need correspondences!

5

slide-6
SLIDE 6

Local features: main components

1) Detection: Identify the interest points 2) Description: Extract vector feature

descriptor surrounding each interest point.

3) Matching: Determine correspondence

between descriptors in two views

] , , [

) 1 ( ) 1 ( 1 1 d

x x … = x

] , , [

) 2 ( ) 2 ( 1 2 d

x x … = x

Source: K. Grauman

6

slide-7
SLIDE 7

Which features should we match?

“flat” region:
 no change in all directions “edge”: no change along the edge direction “corner”:
 significant change in all directions

  • How does the window change when you shift it?
  • Shifting the window in any direction causes a big

change

Source: S. Seitz, D. Frolova, D. Simakov, N. Snavely 7

slide-8
SLIDE 8

Finding keypoints

Find local optima in space/ scale using pyramid Compute difference-of-Gaussians filter (approx. to Laplacian)

8

slide-9
SLIDE 9

Feature descriptors

We know how to detect good points Next question: How to match them? Answer: Come up with a descriptor for each point, find similar descriptors between the two images

?

Source: N. Snavely

9

slide-10
SLIDE 10

CSE 576: Computer Vision

Take 40x40 window around feature

  • Find dominant orientation
  • Rotate to horizontal
  • Sample 8x8 square window centered at

feature

  • Intensity normalize the window by

subtracting the mean, dividing by the standard deviation in the window

Simple idea: normalized image patch

8 pixels

40 pixels

Source: N. Snavely, M. Brown

10

slide-11
SLIDE 11

Basic idea: hand-crafted CNN

  • Take 16x16 square window around detected feature
  • Compute edge orientation for each pixel
  • Create histogram of edge orientations

Scale Invariant Feature Transform

Source: N. Snavely, D. Lowe

2π angle histogram

11

slide-12
SLIDE 12

Create the descriptor:

  • Rotation invariance: rotate by “dominant” orientation
  • Spatial invariance: spatial pool to 2x2
  • Compute an orientation histogram for each cell
  • 16 cells * 8 orientations = 128 dimensional descriptor

Scale Invariant Feature Transform

Source: N. Snavely, D. Lowe

12

slide-13
SLIDE 13

SIFT invariances

Source: N. Snavely

13

slide-14
SLIDE 14

Which features match?

Source: N. Snavely

14

slide-15
SLIDE 15

Finding matches

How do we know if two features match?

– Simple approach: are they the nearest neighbor in L2 distance, ||f1 - f2 || – Can give good scores to ambiguous (incorrect) matches

I1 I2

f1 f2

Source: N. Snavely

15

slide-16
SLIDE 16

f1 f2 f2'

Finding matches

Add extra tests:

  • Ratio distance = ||f1 - f2 || / || f1 - f2’ ||
  • f2 is best SSD match to f1 in I2
  • f2’ is 2nd best SSD match to f1 in I2
  • Forward-backward consistency: f1 should also be nearest neighbor of f2

I1 I2

Source: N. Snavely

16

slide-17
SLIDE 17

Feature matching example

51 feature matches after ratio test

Source: N. Snavely

17

slide-18
SLIDE 18

Feature matching example

58 feature matches after ratio test

Source: N. Snavely

18

slide-19
SLIDE 19

From matches to homography

x1’ y1’ w1 = x1 y1 1 a b c d e f g h i . (x1,y1) (x’1,y’1)

Source: Torralba, Isola, Freeman 19

slide-20
SLIDE 20

20

From matches to homography

  • Plug into nonlinear least squares solver and solve!
  • Can also use robust loss (e.g. L1)
  • Can be slow

Point in 1st image

J(H) = X

i

||fH(pi) − p0

i||2

fH(pi) = Hpi/(HT

3 pi)

Matched point in 2nd where applies homography minimize

slide-21
SLIDE 21

21

x1’ y1’ w1 = x1 y1 1 a b c d e f g h i .

x1’= ax1 + by1+c gx1 + hy1+i y1’= dx1 + ey1+f gx1 + hy1+i gx1x’1 + hy1x’1+ix1 = ax1 + by1+c gx1y’1 + hy1y’1+ix1 = dx1 + ey1+f

Going to heterogeneous coordinates: Re-arranging the terms:

Direct linear transform

Source: Torralba, Freeman, Isola

slide-22
SLIDE 22

22

gx1x’1 + hy1x’1+ix1 = ax1 + by1+c gx1y’1 + hy1y’1+ix1 = dx1 + ey1+f

Re-arranging the terms:

gx1x’1 + hy1x’1+ix’1 - ax1 - by1- c = 0 gx1y’1 + hy1y’1+iy’1 - dx1 - ey1- f = 0

  • x1 -y1 -1 0 0 0 x1x’1 y1x’1 x’1

a
 b
 c
 d
 e
 f
 g
 h
 i

In matrix form. Can solve using Singular Value Decomposition (SVD). 0 0 0 -x1 -y1 -1 x1y’1 y1y’1 y’1 0
 =

Direct linear transform

Fast to solve (but not using “right” loss function). Uses an algebraic trick. Often used in practice for initial solutions!

Source: Torralba, Freeman, Isola

slide-23
SLIDE 23

Outliers

  • utliers

inliers

23

Source: N. Snavely

slide-24
SLIDE 24

Robustness

  • Let’s consider the problem of linear regression
  • How can we fix this?

Problem: Fit a line to these data points Least squares fit

24

Source: N. Snavely

slide-25
SLIDE 25

Counting inliers

25

Source: N. Snavely

slide-26
SLIDE 26

Counting inliers

26

Inliers: 3

Source: N. Snavely

slide-27
SLIDE 27

Counting inliers

27

Inliers: 20

Source: N. Snavely

slide-28
SLIDE 28

28

  • M. A. Fischler, R. C.
  • Bolles. Random Sample

Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated

  • Cartography. Comm. of

the ACM, Vol 24, pp 381-395, 1981.

slide-29
SLIDE 29

RANSAC: random sample consensus

RANSAC loop (for N iterations):

  • Select four feature pairs (at random)
  • Compute homography H
  • Count inliers where ||pi’ - H pi|| < ε

Afterwards:

  • Choose largest set of inliers
  • Recompute H using only those inliers (often

using high-quality nonlinear least squares)

29 Source: Torralba, Freeman, Isola

slide-30
SLIDE 30

30

Simple example: fit a line

  • Rather than homography H (8 numbers) 


fit y=ax+b (2 numbers a, b) to 2D pairs

Source: Torralba, Freeman, Isola

slide-31
SLIDE 31

31

Simple example: fit a line

  • Pick 2 points
  • Fit line
  • Count inliers

3 inlier

Source: Torralba, Freeman, Isola

slide-32
SLIDE 32

32

Simple example: fit a line

  • Pick 2 points
  • Fit line
  • Count inliers

4 inlier

Source: Torralba, Freeman, Isola

slide-33
SLIDE 33

33

Simple example: fit a line

  • Pick 2 points
  • Fit line
  • Count inliers

9 inlier

Source: Torralba, Freeman, Isola

slide-34
SLIDE 34

34

Simple example: fit a line

  • Pick 2 points
  • Fit line
  • Count inliers

8 inlier

Source: Torralba, Freeman, Isola

slide-35
SLIDE 35

35

Simple example: fit a line

  • Use biggest set of inliers
  • Do least-square fit

Source: Torralba, Freeman, Isola

slide-36
SLIDE 36

Warping with a homography

  • 1. Compute features using SIFT
  • 2. Match features
  • 3. Compute homography using RANSAC

36 Source: N. Snavely

slide-37
SLIDE 37

Estimating 3D structure

  • Given many images, how can we

a) figure out where they were all taken from? b) build a 3D model of the scene? This is the structure from motion problem

37

Source: N. Snavely

slide-38
SLIDE 38

Structure from motion

  • Input: images with points in correspondence pi,j = (ui,j,vi,j)
  • Output
  • structure: 3D location xi for each point pi
  • motion: camera parameters Rj , tj possibly Kj
  • Objective function: minimize reprojection error

Reconstruction (side) (top)

38

Source: N. Snavely

slide-39
SLIDE 39

Camera calibration & triangulation

  • Suppose we know 3D points

– And have matches between these points and an image – Computing camera parameters similar to homography estimation

  • Suppose we have know camera parameters, each of

which observes a point

– How can we compute the 3D location of that point?

  • Seems like a chicken-and-egg problem, but in SfM we can

solve both at once

39

Source: N. Snavely

slide-40
SLIDE 40

Feature detection

Detect features using SIFT [Lowe, IJCV 2004]

40

Source: N. Snavely

slide-41
SLIDE 41

Feature detection

Detect features using SIFT [Lowe, IJCV 2004]

41

Source: N. Snavely

slide-42
SLIDE 42

Feature matching

Match features between each pair of images

42

Source: N. Snavely

slide-43
SLIDE 43

Feature matching

Refine matching using RANSAC to estimate fundamental matrix between each pair

43

Source: N. Snavely

slide-44
SLIDE 44

Correspondence estimation

  • Link up pairwise matches to form connected components of

matches across several images

Image 1 Image 2 Image 3 Image 4

44

Source: N. Snavely

slide-45
SLIDE 45

Image connectivity graph

45

Source: N. Snavely

slide-46
SLIDE 46

Structure from motion

Camera 1 Camera 2 Camera 3

R1,t1 R2,t2 R3,t3

X1 X4 X3 X2 X5 X6 X7

minimize

g(R, T, X)

p1,1 p1,2 p1,3 non-linear least squares

46

Source: N. Snavely

slide-47
SLIDE 47

Structure from motion

  • Minimize sum of squared reprojection errors:
  • Minimizing this function is called bundle adjustment

– Optimized using non-linear least squares, e.g. Levenberg-Marquardt

predicted image location

  • bserved

image location indicator variable: is point i visible in image j ?

47

Source: N. Snavely

slide-48
SLIDE 48

48

slide-49
SLIDE 49

49

slide-50
SLIDE 50

50

slide-51
SLIDE 51

Multi-view stereo

Source: N. Snavely

We have the camera pose. Estimate depth using stereo!

slide-52
SLIDE 52

Source: N. Snavely

slide-53
SLIDE 53

Source: N. Snavely