[PPT] - Image Registration by Probabilistic Multi-Assignment Graph Matching PowerPoint Presentation

SLIDE 1

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Image Registration by Probabilistic Multi-Assignment Graph Matching

Uri Okun

Department of Electrical Engineering, Technion - Israel Institute of Technology

Supervised by Prof. Israel Cohen and Prof. Yosi Keller November 18, 2014

SLIDE 2

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Outline

1

Introduction

2

Common Matching Strategies Epipolar Geometry Robust Estimation Graph Matching Discussion

3

Our Contribution Overview of the Method Multi-Assignment Graph Matching Unified Probabilistic Stereo Matching MAGMA

4

Experimental Study Synthetic Datasets Real Image Sequences

5

Conclusion

SLIDE 3

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Wide-Baseline Image Registration

Image matching - Finding a (large) coherent set of tie points between two images depicting the same scene Wide-Baseline Stereo Pair Two photos taken from disparate positions and viewing angles Often causing for severe perspective deformations

SLIDE 4

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Motivation

Some applications: Three-dimensional scene reconstruction Image querying Object recognition Navigation and motion estimation Change detection Image-data fusion Object recognition Mapping ...

SLIDE 5

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Image Registration Challenges

The two images can be of: Different sensors Different acquisition times Different lighting conditions Occlusions Repetitive patterns Different viewpoints:

Rotations Scale changes Perspective transformations

SLIDE 6

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Image Matching Problem Statement

Image registration algorithms often consist of the following steps:

1 Detection of interest points (features) 2 Assigning each feature with a descriptor 3 Matching the features from one image to the other

The matching step is a difficult task - outliers may form due to false correspondences between repetitive image patterns or accidental feature similarity

SLIDE 7

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Points Matching by Descriptors Similarity

A naive implementation: Each feature may be assigned to its nearest neighbor (most similar descriptor)

SLIDE 8

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Common Matching Strategies

Two main approaches for solving the matching problem are based upon geometrical relations between feature points in both images:

1 Epipolar geometry robust estimation 2 Graph matching

SLIDE 9

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Epipolar Geometry

The epipolar constraint is the geometric structure which holds for all pairs of pinhole cameras, regardless of the captured scene [Hartley and Zisserman, 2004] The pinhole camera model:

SLIDE 10

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Epipolar Geometry (Cont.)

For stereo pair: The epipolar constraint is given by: xTFy = 0 x ∈ I1, y ∈ I2 are points in homogeneous coordinates F is the 3 × 3 Fundamental matrix The epipolar lines are given by: l′T = xTF, l = Fy

SLIDE 11

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Robust Model Estimation

The Fundamental matrix can be calculated using the linear Normalized 8-point algorithm, or non-linear 7-point Algorithm Epipolar robust estimation techniques relay on estimating the fundamental matrix coefficients from sets of putative matches while simultaneously removing gross outliers RANSAC (RANdom Sampling Consensus) [Fischler and Bolles, 1981] and RANSAC-based algorithms are the most popular choice

SLIDE 12

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

RANSAC [Fischler and Bolles, 1981]

SLIDE 13

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

RANSAC [Fischler and Bolles, 1981]

Select sample of s points at random Calculate model parameters that fit the data in the sample Calculate error function for each data point Select data that support current hypothesis Repeat sampling

SLIDE 14

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

RANSAC [Fischler and Bolles, 1981]

Select sample of s points at random Calculate model parameters that fit the data in the sample Calculate error function for each data point Select data that support current hypothesis Repeat sampling

SLIDE 15

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

RANSAC [Fischler and Bolles, 1981]

Select sample of s points at random Calculate model parameters that fit the data in the sample Calculate error function for each data point Select data that support current hypothesis Repeat sampling

SLIDE 16

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

RANSAC [Fischler and Bolles, 1981]

Select sample of s points at random Calculate model parameters that fit the data in the sample Calculate error function for each data point Select data that support current hypothesis Repeat sampling

SLIDE 17

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

RANSAC [Fischler and Bolles, 1981]

In order to ensure drawing at least one inlier-set with a certainty of c, the total number of samples is evaluated as following: Nsamples = log(1 − c) log(1 − pi s), where pi is the inlier rate, and s is the sample size Example: Epipolar geometry estimation using the linear algorithm, with a certainty of 95%: pi Nsamples 0.4 4.57e3 0.2 1.17e6 0.1 3e8

SLIDE 18

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

RANSAC-Based Algorithms

Improvements might... Use other variation of cost functions

MLESAC [Torr and Zisserman, 2000] MAPSAC [Torr, 2002]

Exploit each drawn sample to the fullest

LO-RANSAC [Chum et al., 2003]

Use guided sampling

NAPSAC [Myatt et al., 2002] PROSAC [Chum and Matas, 2005] BEEM [Goshen and Shimshoni, 2006] BLOGS [Brahmachari and Sarkar, 2013]

Decrease number of points per sample

BEEM [Goshen and Shimshoni, 2006]

SLIDE 19

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Graph Matching

Looking for an alignment which preserves geometrical consistency

f small groups of points.

Construct an affinity graph Extract prominent cluster within the graph The graph encodes spatial invariance properties: Pairwise affinities – encoding isometry (translation/rotation) Triplets affinities – encoding similarity (scale invariant)

SLIDE 20

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Pairwise Graph Matching

In pairwise GM we calculate the affinity between each pair of correspondences, φ2

cii′, cjj′

φ2

{xi, yi′} ,
xj, yj′

= exp

−1

ε

xi − xjL2 − yi − yjL2
.

The goal is to find C = {cii′}n

i=1 , n ≤ min {N1, N2} such that,

C ∗ = arg max

C

cii′ , cjj′∈C

φ2

cii′, cjj′

.

SLIDE 21

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Pairwise Graph Matching (Cont.)

The assignment matrix (N1 ×N2) represents the correspondences: Zi,i′ = 1 if cii′ ∈ C f cii′ / ∈ C The matching problem can be formulated as: z∗ = arg max

z

zTAz
s.t.

z ∈ {0, 1}N1N2,

i

zij ≤ 1 ∀j,

j

zij ≤ 1 ∀i where z is a row-stacked replica of Z, and A is the affinity matrix (A ∈ RN1·N2×N1·N2), defined as: A

i, i′; j, j′

A

i (N2 − 1) + i′, j (N2 − 1) + j′

= φ2

cii′, cjj′

.

SLIDE 22

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Pairwise Graph Matching (Cont.)

A =

                φ (c11, c11) φ (c11, c12) · · · φ (c11, c1N2) φ (c11, c21) φ (c11, c22) · · · φ (c11, cN1N2) φ (c12, c11) φ (c12, c12) · · · φ (c12, c1N2) · · · φ (c12, cN1N2) . . . . . . ... φ (c1N2, c11) φ (c1N2, c12) φ (c1N2, c1N2) φ (c21, c11) . . . φ (c22, c11) ... . . . φ (cN1N2, c11) · · · φ (cN1N2, cN1N2)                

z∗ = arg max

z

zTAz
s.t.

z ∈ {0, 1}N1N2,

i

zij ≤ 1 ∀j,

j

zij ≤ 1 ∀i

SLIDE 23

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Spectral Graph Matching

A spectral relaxation of the optimization problem by dropping the constraints [Leordeanu and Hebert, 2005]: w∗ = arg max

w

wT Aw

wT w

,

s.t. w ∈ RN1N2 Can be solved by computing the principal eigenvector of A, and applying a discretization to w∗. A probabilistic interpretation [Egozi et al., 2009]: The spectral relaxation approximates the assignment probabilities: p = w∗ = arg min

w

A − wwT
L2

By Perron-Frobenius theorem, p ≥ 0

SLIDE 24

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Points Matching by Triplets-Affinity Graph Matching

By marginalizing the triplets affinity graph and applying spectral relaxation to the result [Chertok and Keller, 2010],

SLIDE 25

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Discussion - Pros and Cons

Epipolar Geometry Robust Estimation

⊕ Rigorous and general - works for any pinhole-cameras stereo rig ⊖ Does not work for non-perspective deformations ⊖ Cluttered by repetitive patterns in a direction parallel to the baseline (e.g.

when shooting buildings)

⊖ Does not use any smoothness or rigidness assumptions ⊖ Unable to cope with more than just one candidate match for each point

in the source image Graph Matching

⊕ Efficient and robust cluster matching ⊕ Not so strict - insensitive to slight non-perspective deformations ⊖ Not so rigorous - based on a one-cluster heuristic assumption, which is

ften violated by perspective deformations

⊖ Sensitive to other erroneous alignments unrelated to camera

transformation, such as symmetries or repetitive objects

SLIDE 26

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Research Goal

Perspective transformations depend on the scene’s depth, which can be non smooth or discontinuous, and so can take any form compliant with the epipolar constraint Natural scenes tend to be piecewise rigid, and thus wide-baseline stereo pair might be related by multiple spatial transformations We would like to preserve GM strength of rigid cluster matching, without giving up the flexibility of complex perspective transformation

SLIDE 27

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Overview of the Method

Our method consists of the following steps:

1 Generalizing the graph matching concept to the case of

several unconnected components

2 Robustly inferring the epipolar geometry, to be used as a

global constraint binding these components together

3 Combining the two steps in a probabilistic fashion, by fusing

the epipolar information within the affinity graph, to form a stable iterative process

SLIDE 28

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Multiple Solution Graph Matching

The graph may consist of several unconnected components, or sub-graphs This gives rise to a clustering (or multi-assignment graph matching) formulation: Ω =

w(1)

w(2) ... w(M) A ≈ ΩΩT = w(1)w(1)T + w(2)w(2)T + ... + w(M)w(M)T

SLIDE 29

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Multi-Assignment Graph Matching - Spectral Approach

Multiple solutions can be obtained by performing spectral decomposition of the affinity matrix [Chertok and Keller, 2009]. Each subgraph component is linked to one of the few leading eigenvectors Generalization of SM [Leordeanu and Hebert, 2005] - Finds the Rank-M-Approximation under orthogonality constraint: Ω∗ = arg min

Ω

A − ΩΩT
L2

s.t. ΩTΩ = I

⊖ Nonnegativity is only guaranteed for the leading eigenvector

by Perron-Frobenius theorem - Problematic discretization and probabilistic interpretation

SLIDE 30

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Multi-Assignment Graph Matching - Spectral Approach

50 100 150 200 250 300 350 400 450 0.05 0.1 0.15 0.2 Eigenvector #1 50 100 150 200 250 300 350 400 450 −0.2 −0.1 0.1 0.2 Eigenvector #2 50 100 150 200 250 300 350 400 450 −0.2 −0.1 0.1 0.2 Eigenvector #3

SLIDE 31

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Multi-Assignment Graph Matching - NMF Approach

A more adequate way of finding unconnected components in the graph is by Symmetric Nonnegative Matrix Factorization (NMF) [Lee and Seung, 1999]. Each component is linked to one column in the factorizing matrix Generalization of SM [Leordeanu and Hebert, 2005] - Finds the Rank-M-Approximation under nonnegativity constraint: Ω∗ = arg min

Ω

A − ΩΩT
L2

s.t. Ω ≥ 0

⊕ Allows for an intuitive discretization and a meaningful

probabilistic interpretation

⊖ No stochastic normalization in the course of optimization

SLIDE 32

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Multi-Assignment Graph Matching - NMF Approach

50 100 150 200 250 300 350 400 450 0.2 0.4 0.6 0.8 NMF Component #1 50 100 150 200 250 300 350 400 450 0.2 0.4 0.6 0.8 NMF Component #2 50 100 150 200 250 300 350 400 450 0.2 0.4 0.6 0.8 NMF Component #3

SLIDE 33

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Multi-Assignment Graph Matching - SCMF Approach

We propose solving the semi-stochastic constrained problem: Ω∗ = arg min

Ω

A − ΩΩT
L2

s.t. Ω ∈ [0, 1] N×M,

i
W(m)

ij ≤ 1

∀m,j,

j
W(m)

ij ≤ 1

∀m, i An iterative projected least squares optimization scheme for Stochastic Constrained Matrix Factorization (SCMF) was developed SCMF is also equivalent to projected Newton algorithm

SLIDE 34

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Multi-Assignment Graph Matching - SCMF Approach

SCMF is best suited for our matching model It is shown to outperform Symmetric-NMF in finding several components in a graph created through projective transform

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.8 0.82 0.84 0.86 0.88 0.9 0.92 0.94 0.96 0.98 1

Recall Accuracy Symmetric−NMF SCMF

Simulation results based on 100 random ”scenes” composed

f several planes of points captured by two pinhole cameras

SLIDE 35

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Unified Probabilistic Stereo Matching

The GM probabilistic interpretation allows us to further improve the matching accuracy by incorporating additional knowledge. The graph edges approximate pairwise probabilities φ

cii′, cjj′

≈ P

cii′, cjj′

. A finer statistical model via conditioning the probabilities by unary independent probabilities, φ

cii′, cjj′

= P

cii′, cjj′|u(cii′), u(cjj′)
,

and following Bayes theorem, and the unary probabilities independence assumption: P

cii′, cjj′

= P

cii′, cjj′|u(cii′), u(cjj′)
· P
u(cii′), u(cjj′)
=

P

cii′, cjj′|u(cii′), u(cjj′)
· P (u(cii′)) · P
u(cjj′)

SLIDE 36

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Epipolar Probabilities

The epipolar band [Csurka et al, 1995] - The epipolar line’s uncertainty region, in which the probability of finding a match is less than a certain value.

data points reference point data points epipolar line p = 0.2 p = 0.4 p = 0.6 p = 0.7 p = 0.8 p = 0.85 p = 0.9 p = 0.95

We estimate the independent unary probabilities as the probabilities of each corresponding pair being compliant with the epipolar geometry, P(u (cii′)) = P ({xi, yi′} |F, Σf )

SLIDE 37

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

MAGMA - Multi Assignment Graph Matching Algorithm

SLIDE 38

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

MAGMA: Example I

∗ From BEEM dataset [Goshen and Shimshoni, 2006]

SLIDE 39

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

MAGMA: Example II

SLIDE 40

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

MAGMA: Example III

SLIDE 41

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

MAGMA: Example IV

SLIDE 42

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

MAGMA: Example V

SLIDE 43

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

MAGMA: Example VI

SLIDE 44

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

MAGMA: Example VII

SLIDE 45

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

MAGMA: Example VIII

SLIDE 46

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

MAGMA: Example IX

SLIDE 47

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

MAGMA: Example X

SLIDE 48

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

MAGMA: Example XI

SLIDE 49

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Experimental Study - Synthetic Experiments

Recall Number of plains RANSAC8 MLESAC MAPSAC NAPSAC SM GA IPFP RRWM HGM TGM THOA MAGMA

SLIDE 56

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Experimental Study - Image Sequence Experiments

The experimental setup: fountain-P11 dataset Herz-Jesu-K7 dataset

SLIDE 57

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Experimental Study - Image Sequences Results

fountain-P11 dataset

Accuracy =

TP TP+FP

, Recall =

TP TP+FN

1 2 3 4 5 6 7 8 9 10 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Accuracy Baseline diff RANSAC8 MLESAC MAPSAC NAPSAC BEEM BLOGS SM GA IPFP RRWM HGM TGM THOA MAGMA

1 2 3 4 5 6 7 8 9 10 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Recall Baseline diff RANSAC8 MLESAC MAPSAC NAPSAC BEEM BLOGS SM GA IPFP RRWM HGM TGM THOA MAGMA

SLIDE 58

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Experimental Study - Image Sequences Results

Herz-Jesu-K7 dataset

Accuracy =

TP TP+FP

, Recall =

TP TP+FN

1 2 3 4 5 6 7 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Accuracy Baseline diff RANSAC8 MLESAC MAPSAC NAPSAC BEEM BLOGS SM GA IPFP RRWM HGM TGM THOA MAGMA

1 2 3 4 5 6 7 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Recall Baseline diff RANSAC8 MLESAC MAPSAC NAPSAC BEEM BLOGS SM GA IPFP RRWM HGM TGM THOA MAGMA

SLIDE 59

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Summary

We have presented MAGMA - A graph based matching algorithm, which preserves the strengthes of other GM approaches while mitigating their shortcomings. Our main contributions: Generalization of graph matching paradigm that looks for several consistent components in the graph instead of just one New optimization method, SCMF, for solving the stochastic semi-normalized multi-assignment graph matching General probabilistic framework for fusing other sources of information into the affinity graph Iterative process for stereo matching - suitable for registering wide-baseline image pairs

SLIDE 60

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Future Research

Adaptively estimating the number of components in the graph from the data Assembling the graph in a higher dimension, allowing each component to have a more general transformation (e.g. affinity or homography) Higher-order matrix factorization Using different priors (other than epipolar geometry) to further improve the overall performance, or to generalize the approach beyond stereo matching, e.g. for matching non-rigid objects

SLIDE 61

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

Thank You!

SLIDE 62

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

References I

Hartley, R. I. and Zisserman, A. (2004) Multiple View Geometry in Computer Vision Cambridge University Press, ISBN: 0521540518. Fischler, M.A. and Bolles, R.C. (1981) Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography Communications of the ACM 24, 381-395. Torr, P. H. S. and Zisserman, A. (2000) MLESAC: A New Robust Estimator with Application to Estimating Image Geometry Computer Vision and Image Understanding 78(1), 138–156. Torr, P. H. S. (2002) Bayesian Model Estimation and Selection for Epipolar Geometry and Generic Manifold Fitting

Int. J. Comput. Vision 50(1), 35–61.

Myatt, D. R. and Torr, P. H. S. and Nasuto, S. J. and Bishop, J. M. and Craddock, R. (2002) NAPSAC: high noise, high dimensional robust estimation In BMVC02 2, 458–467.

SLIDE 63

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

References II

Chum, O. and Matas, J. (2005) Matching with PROSAC - progressive sample consensus Computer Vision and Pattern Recognition 1, 220–226. Chum, O. and Matas, J. and Kittler, J. (2003) Locally Optimized RANSAC Pattern Recognition, 236–243. Goshen, L. and Shimshoni, I. (2006) Balanced Exploration and Exploitation Model Search for Efficient Epipolar Geometry Estimation Computer Vision and Pattern Recognition 1, 151–164. Brahmachari, A. S. and Sarkar, S. (2013) Hop-Diffusion Monte Carlo for Epipolar Geometry Estimation Between Very Wide-Baseline Images IEEE Trans. Pattern Anal. Mach. Intell. 35(3), 755–762. Leordeanu, M. and Hebert, M. (2005) A Spectral Technique for Correspondence Problems using Pairwise Constraints International Conference of Computer Vision (ICCV) 2, 1482–1489.

SLIDE 64

Introduction Common Matching Strategies Our Contribution Experimental Study Conclusion References

References III

Egozi, A. and Keller, A. and Guterman, H. (2009) A probabilistic approach to spectral graph matching submited. Chertok, M. and Keller, Y. (2010) Efficient High Order Matching IEEE Trans. Pattern Anal. Mach. Intell. 32(12), 2205–2215. Chertok, M. and Keller, Y. (2009) Spectral Symmetry Analysis IEEE Transactions on Pattern Analysis and Machine Intelligence. Lee, D. D. and Seung, H. S. (1999) Learning the parts of objects by non-negative matrix factorization Nature 401(6755), 788–791. Csurka, G. and Faugeras, O. and Zhang, Z. and Zeller, C. (1995) Characterizing the Uncertainty of the Fundamental Matrix ROBOTVIS - INRIA Sophia Antipolis RR-2560.