Image alignment What to do if we want to compare the appearance of - - PowerPoint PPT Presentation

Subhransu Maji

CMPSCI 670: Computer Vision

October 18, 2016

Image alignment

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Scaled and rotated versions of the same neighborhood will give rise to blobs that are related by the same transformation What to do if we want to compare the appearance of these image regions?

• Normalization: transform these regions into same-size

circles

• Problem: rotational ambiguity

Recap: from feature detection to description

2 Source: L. Lazebnik Subhransu Maji (UMass, Fall 16) CMPSCI 670

To assign a unique orientation to circular image windows:

• Create histogram of local gradient directions in the patch
• Assign canonical orientation at peak of smoothed histogram

Recap: Eliminating rotation ambiguity

3

2 π

Source: L. Lazebnik Subhransu Maji (UMass, Fall 16) CMPSCI 670

Detected features with characteristic scales and orientations

Recap: Local features

4

David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60 (2), pp. 91-110, 2004.

Source: L. Lazebnik

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Find regions and compare their feature descriptors

Feature descriptors

5 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Simplest descriptor: vector of raw intensity values How to compare two such vectors?

• Sum of squared differences (SSD) — this is a distance measure


➡ Not invariant to intensity change


• Normalized correlation — this is a similarity measure


➡ Invariant to affine (translation + scaling) intensity change

Feature descriptors

6

( )

∑

− =

i i i

v u

2

) SSD( v u, ! ! " # $ $ % & − ! ! " # $ $ % & − − − = − − ⋅ − − =

∑ ∑ ∑

j j j j i i i

v u v u

2 2

) ( ) ( ) )( ( || || ) ( || || ) ( ) ( v u v u v v v v u u u u v u, ρ

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Small deformations can affect the matching score a lot
 
 
 
 


Problem with intensity vectors

7 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Descriptor computation:

• Divide patch into 4x4 sub-patches
• Compute histogram of gradient orientations (8 reference angles) inside

each sub-patch

• Resulting descriptor: 4x4x8 = 128 dimensions

Feature descriptors: SIFT

8

David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60 (2),

• pp. 91-110, 2004.

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Descriptor computation:

• Divide patch into 4x4 sub-patches
• Compute histogram of gradient orientations (8 reference angles) inside

each sub-patch

• Resulting descriptor: 4x4x8 = 128 dimensions


Advantage over raw vectors of pixel values

• Gradients less sensitive to illumination change
• Pooling of gradients over the sub-patches achieves robustness to small

shifts, but still preserves some spatial information

Feature descriptors: SIFT

9

David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60 (2),

• pp. 91-110, 2004.

More on image representations later

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Problem: Ambiguous putative matches

10

Source: Y. Furukawa

Subhransu Maji (UMass, Fall 16) CMPSCI 670

How can we tell which putative matches are more reliable? Heuristic: compare distance of nearest neighbor to that of second nearest neighbor

• Ratio of closest distance to second-closest distance will be high 


for features that are not distinctive

Rejection of unreliable matches

11

David G. Lowe. "Distinctive image features from scale-invariant keypoints.” IJCV 60 (2),

• pp. 91-110, 2004.

Threshold of 0.8 provides good separation

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Matching local features

• Local information used, can contain outliers
• But hopefully enough of these matches are good

A framework for alignment

12

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Matching local features

• Local information used, can contain outliers
• But hopefully enough of these matches are good

Consensus building

• Aggregate the good matches and find a transformation that

explains these matches

A framework for alignment

13 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Families of transformation

14 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Think about moving coordinates, not pixels.

How do we move pixels?

15 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Points in 2D

16

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Translation

17 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Scaling

18 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Rotation

19 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Shear

20

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Arbitrary linear transformation

21 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Families of linear transforms

22 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Affine = linear + translation Approximates viewpoint changes for roughly planar objects and roughly orthographic cameras Can be used to initialize fitting for more complex models

Affine transformations

23

x → Mx + t

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Assume we know the correspondences, how do we get the transformation?

Fitting an affine transformation

24

) , (

i i y

x ! ! ) , (

i i y

x

! " # $ % & + ! " # $ % & ! " # $ % & = ! " # $ % & ' '

2 1 4 3 2 1

t t y x m m m m y x

i i i i

t Mx x + = !

i i

Want to find M, t to minimize

∑

=

− − #

n i i i 1 2

|| || t Mx x

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Assume we know the correspondences, how do we get the transformation?

Fitting an affine transformation

25

) , (

i i y

x ! ! ) , (

i i y

x

! " # $ % & + ! " # $ % & ! " # $ % & = ! " # $ % & ' '

2 1 4 3 2 1

t t y x m m m m y x

i i i i

! ! ! ! " # $ $ $ $ % & ' ' = ! ! ! ! ! ! ! ! " # $ $ $ $ $ $ $ $ % & ! ! ! ! " # $ $ $ $ % & ! ! ! !

i i i i i i

y x t t m m m m y x y x

2 1 4 3 2 1

1 1

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Linear system with six unknowns Each match gives us two linearly independent equations: need at least three to solve for the transformation parameters

Fitting an affine transformation

26

! ! ! ! " # $ $ $ $ % & ' ' = ! ! ! ! ! ! ! ! " # $ $ $ $ $ $ $ $ % & ! ! ! ! " # $ $ $ $ % & ! ! ! !

i i i i i i

y x t t m m m m y x y x

2 1 4 3 2 1

1 1

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Matching local features

• Local information used, can contain outliers
• But hopefully enough of these matches are good

Consensus building

• Aggregate the good matches and find a transformation that

explains these matches

A framework for alignment

27 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Random Sample Consensus Choose a small subset of points uniformly at random Fit a model to that subset Find all remaining points that are “close” to the model and reject the rest as outliers Do this many times and choose the best model For rigid transformation we can estimate the parameters of the transformation, e.g., rotation angle, scaling, translation, etc, from putative correspondence matches Lets see how RANSAC works for a simple example.

RANSAC

28

• M. A. Fischler, R. C. Bolles. Random Sample Consensus: A Paradigm for Model Fitting with

Applications to Image Analysis and Automated Cartography. Comm. of the ACM, Vol 24, pp 381-395, 1981.

Subhransu Maji (UMass, Fall 16) CMPSCI 670

RANSAC for line fitting example

29

Source: R. Raguram

Subhransu Maji (UMass, Fall 16) CMPSCI 670

RANSAC for line fitting example

30

Least-squares fit

Source: R. Raguram

min

a,b

X

i

(axi + b − yi)2

Subhransu Maji (UMass, Fall 16) CMPSCI 670

RANSAC for line fitting example

31

• 1. Randomly select

minimal subset of points

Source: R. Raguram

Subhransu Maji (UMass, Fall 16) CMPSCI 670

RANSAC for line fitting example

32

• 1. Randomly select

minimal subset of points

• 2. Hypothesize a

model

Source: R. Raguram

Subhransu Maji (UMass, Fall 16) CMPSCI 670

RANSAC for line fitting example

33

• 1. Randomly select

minimal subset of points

• 2. Hypothesize a

model

• 3. Compute error

function

Source: R. Raguram

Subhransu Maji (UMass, Fall 16) CMPSCI 670

RANSAC for line fitting example

34

• 1. Randomly select

minimal subset of points

• 2. Hypothesize a

model

• 3. Compute error

function

• 4. Select points

consistent with model

Source: R. Raguram

Subhransu Maji (UMass, Fall 16) CMPSCI 670

RANSAC for line fitting example

35

• 1. Randomly select

minimal subset of points

• 2. Hypothesize a

model

• 3. Compute error

function

• 4. Select points

consistent with model

• 5. Repeat hypothesize-

and-verify loop

Source: R. Raguram

Subhransu Maji (UMass, Fall 16) CMPSCI 670

RANSAC for line fitting example

36

• 1. Randomly select

minimal subset of points

• 2. Hypothesize a

model

• 3. Compute error

function

• 4. Select points

consistent with model

• 5. Repeat hypothesize-

and-verify loop

Source: R. Raguram

Subhransu Maji (UMass, Fall 16) CMPSCI 670

RANSAC for line fitting example

37

• 1. Randomly select

minimal subset of points

• 2. Hypothesize a

model

• 3. Compute error

function

• 4. Select points

consistent with model

• 5. Repeat hypothesize-

and-verify loop

Uncontaminated sample

Source: R. Raguram

Subhransu Maji (UMass, Fall 16) CMPSCI 670

RANSAC for line fitting example

38

• 1. Randomly select

minimal subset of points

• 2. Hypothesize a

model

• 3. Compute error

function

• 4. Select points

consistent with model

• 5. Repeat hypothesize-

and-verify loop

Source: R. Raguram

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Repeat N times:

• Draw s points uniformly at random
• Fit line to these s points
• Find inliers to this line among the remaining points (i.e., points

whose distance from the line is less than t)

• If there are d or more inliers and the number of inliers is higher

than the previous best, accept the line and refit using all inliers.

RANSAC for line fitting

39 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Approach

• Local feature matching
• RANSAC for alignment

Panoramic stitching

40

?

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Panoramic stitching

41 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Extract features

• corner detector

Panoramic stitching

42 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Extract features Compute putative matches

Panoramic stitching

43 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Extract features Compute putative matches Loop:

• Hypothesize transformation T

Panoramic stitching

44

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Extract features Compute putative matches Loop:

• Hypothesize transformation T
• Verify transformation (search for other matches consistent with T)

Panoramic stitching

45 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Mechanics of transformation

46 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Problems with this transformation

• Leaves gaps in the target image
• Interpolation less intuitive

Mechanics of transformation

47 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Rotation by 45 degrees

Example of forward warp

48

Subhransu Maji (UMass, Fall 16) CMPSCI 670

Mechanics of transformation

49 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Rotation by 45 degrees

Example of inverse warp

50 Subhransu Maji (UMass, Fall 16) CMPSCI 670

Panoramic stitching warping

51