Consistency-driven multiple graph matching Junchi - - PowerPoint PPT Presentation

▶
consistency driven multiple graph matching
SMART_READER_LITE
LIVE PREVIEW

Consistency-driven multiple graph matching Junchi - - PowerPoint PPT Presentation

Mar 06, 2023 329 likes •1.04k views

Consistency-driven multiple graph matching Junchi Yan IBM Research China (Shanghai) East China Normal University Outline Introduction on Graph Matching Reference graph based

slide-1
SLIDE 1

Consistency-driven multiple graph matching 一致性驱动的多图匹配模型和算法

Junchi Yan 严骏驰 IBM Research – China (Shanghai) East China Normal University

slide-2
SLIDE 2

Outline

Introduction on Graph Matching Reference graph based alternating approach, TIP’15

  • Consistency-driven Alternating Optimization for Multi-

graph Matching: a Unified Approach, IEEE Transactions

  • n Image Processing, 2015, 24 (3), 994-1009

More ‘distributed’ approach, TPAMI 2015

  • Multi-Graph Matching via Affinity Optimization with

Graduated Consistency Regularization, IEEE Transactions

  • n Pattern Analysis and Machine Intelligence, accepted
  • n Sep.1 2015, in press

Summary

slide-3
SLIDE 3

Outline

Introduction on Graph Matching

Reference graph based alternating approach, TIP’15

  • Consistency-driven Alternating Optimization for Multi-

graph Matching: a Unified Approach, IEEE Transactions

  • n Image Processing, 2015, 24 (3), 994-1009

More ‘distributed’ approach, TPAMI 2015

  • Multi-Graph Matching via Affinity Optimization with

Graduated Consistency Regularization, IEEE Transactions

  • n Pattern Analysis and Machine Intelligence, accepted
  • n Sep.1 2015, in press

Summary

slide-4
SLIDE 4

RPM / ICP Graph matching Point registration: Often use parametric transformation model between point sets, e.g. RPM, ICP Graph matching: non-parametric form, no prior transform model between graphs Node sets

RPM: Chui, H., Rangarajan, A.: A new point matching algorithm for non-rigid registration. Computer Vision and Image Understanding 89 (2003) 114–141 ICP: Z. Zhang. Iterative point matching for registration of free-form curves and surfaces. IJCV, 1994. Graph matching: Thirty years of graph matching in pattern recognition. IJPRAI, 2004.

Graph matching vs. registration

Transformation->correspondence

  • >Transformation->correspondence
slide-5
SLIDE 5

Node correspondence by an assignment matrix

Solution: assignment matrix, i.e. a partial permutation matrix

5 detected features 4 detected features

Assume one-to-one correspondence

b 4 Sift feature CNN feature …

slide-6
SLIDE 6

Node-wise linear assignment problem

Hungarian Algorithm (Kuhn & Munkres, 1955)

Global optimality is ensured by O(n^3) time complexity where n is the number of nodes

Linear Assignment Problem

T

Node-to-node affinity/cost matrix f4 fb

slide-7
SLIDE 7

Edge-wise graph matching

2nd-order Feature (eg, Edge Length) 1st-order Feature (eg. Local Texture) Graph Matching (quadratic) Feature Matching (linear)

Edge Similarity

Edge Similarity

5 3 c b Build graph by Delaunay Triangulation

slide-8
SLIDE 8

Graph matching: a combinatorial

  • ptimization formulation

Edge Similarity Node Similarity

Node Compatibility Edge Compatibility

Node-Edge Index Affinity maximization

  • S. Gold and A. Rangarajan, “A graduated assignment algorithm for graph matching,” IEEE Transaction on PAMI, 1996
  • A. Rangarajan

Steven Gold

slide-9
SLIDE 9

A more clean writing by an Affinity matrix

(Leordeanu & Hebert, 2005)

Edge-to-edge relations

  • M. Leordeanu
  • M. Hebert
  • M. Leordeanu and M. Hebert, “A spectral technique for correspondence problems using pairwise constraints,” in ICCV, 2005

Affinity matrix:

slide-10
SLIDE 10

Node-to-node affinity

(Diagonal) b 3 b 3

slide-11
SLIDE 11

Edge-to-edge affinity

(Off-Diagonal) c 5 b 3

slide-12
SLIDE 12

Quadratic Assignment Problem

NP-hard

Branch-Bound

Koopmans & Beckmann, 1955 Lawler, 1963 Loiola et al, 2007

slide-13
SLIDE 13

Spectral Approximation

Not Tight Not Discrete Spectral Method is Faster

  • M. Leordeanu
  • M. Hebert
  • T. Cour
  • J. Shi
  • M. Leordeanu and M. Hebert, “A spectral technique for correspondence problems using pairwise constraints,” in ICCV, 2005
  • P. S. T. Cour and J. Shi, “Balanced graph matching,” in NIPS, 2006

Faster

slide-14
SLIDE 14

Double-stochastic Approximation

Not Discrete Not Convex (K is Indefinite) Slower

  • S. Gold & Rangarajan, 1996
  • Cho et al, 2010
  • Leordeanu et al, 2009
  • …
  • S. Gold and A. Rangarajan, “A graduated assignment algorithm for graph matching,” IEEE Transaction on PAMI, 1996
  • M. Cho, J. Lee, and K. M. Lee, “Reweighted random walks for graph matching,” in ECCV, 2010
  • M. Leordeanu, M. Hebert, and R. Sukthankar, “An integer projected fixed point method for graph matching and map inference,” in NIPS, 2009

Gradient Method is More Accurate

  • A. Rangarajan
  • M. Cho
  • K. Lee
  • M. Leordeanu
  • M. Hebert
  • R. Sukthankar
  • S. Gold
slide-15
SLIDE 15

Beyond: Higher-order models

3-Order is Similarity Transformation Invariant 4-Order is Affine Transformation Invariant ?-Order is Non-rigid Invariant 2-Order is Rotation / Scale Invariant Pair-wise Matrix Triple-wise Tensor Complexity Memory for 100 nodes K (381 MB) (3.7 GB) Combinatorial Explosion

slide-16
SLIDE 16

Factorized model

Fernando De la Torre Feng Zhou

Node Similarity Edge Similarity

Sparse Block- Structured

?

  • F. Zhou and F. D. Torre, “Factorized graph matching,” in CVPR, 2012.
slide-17
SLIDE 17

Factorize the edge affinity

  • F. Zhou and F. D. Torre, “Factorized graph matching,” in CVPR, 2012.

Fernando De la Torre

  • F. Zhou
slide-18
SLIDE 18

Path-following optimization

Original objective New objective

Factorization

Convex relaxation Concave relaxation Optimal, Continuous

Assuming X is orthogonal Assuming X is binary Frank-Wolfe

Always discrete Interpolation Interpolation Interpolation

Initialize Frank-Wolfe

  • F. Zhou and F. D. Torre, “Factorized graph matching,” in CVPR, 2012.

Fernando De la Torre

  • F. Zhou
slide-19
SLIDE 19

Two paradigms for two-graph matching

Spectral / Gradient Discrete Rounding

Factorization

Convex Relaxation Concave Relaxation

Non-factorized paradigm Factorized paradigm

Fernando De la Torre Feng Zhou Minsu Cho

  • T. Cour
  • M. Leordeanu
slide-20
SLIDE 20

Matching more than two graphs

  • More practical problem, with more information to use

Graphical object query and indexing, shape analysis SIGGRAPH’12

Optical image Infra-red line- scan image Cartographic data

Info fusion PRL’97 3-D weak reconstruction ICCV’15 Exploring collections of 3D models using fuzzy correspondences, SIGGRAPH’12 Multiple Graph Matching with Bayesian Inference, Pattern recognition letters’97 Multi-Image Matching via Fast Alternating Minimization ICCV’15

slide-21
SLIDE 21

Motivating illustration

G1 G2 G3 G1->G2 G1->G3->G2

Interpolating graph (son) Father Mother

slide-22
SLIDE 22

Existing multiple GM methods

Main categories

 Designate one of the graphs as the reference, and match all the

  • thers to the reference graph
  • A. Sole-Ribalta, F. Serratosa, Models and algorithms for computing the

common labelling of a set of attributed graphs, CVIU 2011

 Compute pairwise matchings, based on which improve overall accuracy

  • D. Pachauriy, R. Kondorx, V. Singh, Solving the multi-way matching problem

by permutation synchronization, in NIPS 2013

  • Y. Chen, G. Leonidas, and Q. Huang. Matching partially similar objects via

matrix completion. In ICML, 2014

 One-shot multiple feature set (not graph) matching

  • Z. Zeng, T. H. Chan, K. Jia, and D. Xu. Finding correspondence from multiple

images via sparse and low-rank decomposition. In ECCV, 2012

  • X. Zhou, M. Zhu, and K. Daniilidis. Multi-image matching via fast alternating
  • minimization. In ICCV, 2015
slide-23
SLIDE 23

Outline

Introduction on Graph Matching

Reference graph based alternating approach, TIP’15

  • Consistency-driven Alternating Optimization for Multi-

graph Matching: a Unified Approach, IEEE Transactions

  • n Image Processing, 2015, 24 (3), 994-1009

More ‘distributed’ approach, TPAMI 2015

  • Multi-Graph Matching via Affinity Optimization with

Graduated Consistency Regularization, IEEE Transactions

  • n Pattern Analysis and Machine Intelligence, accepted
  • n Sep.1 2015, in press

Summary

slide-24
SLIDE 24

From two-graph to multi-graph (I)

  • Junchi Yan et al. 2013&2015
  • Journal extension
  • Consistency-driven Alternating Optimization for Multi-

graph Matching: a Unified Approach, IEEE Transactions

  • n Image Processing, 2015, 24 (3), 994-1009
  • Conference preliminary version
  • Joint optimization for consistent multiple graph

matching, in ICCV 2013

slide-25
SLIDE 25

Adding up pairwise affinities

Assumption: 1) All graphs are of equal size, can be realized by adding dummy nodes or outliers 2) We are matching a collection of related graphs with common structures

Redundancy Basis

slide-26
SLIDE 26

Use a basis set of pairwise matchings

Basis set: fixed updating Fix Xf1r, Xf2r, Xf3r, Xf4r Update Xur Iteration 1 Fix Xur, Xf2r, Xf3r, Xf4r Update Xf1r Iteration 2 Fix Xur, Xf1r, Xf3r, Xf4r Update Xf2r Iteration 3 Fix Xur, Xf2r, Xf1r, Xf4r Update Xf3r Iteration 4 Fix Xur, Xf2r, Xf3r, Xf1r Update Xf4r Iteration 5 Alternating updating new Graph: r,f1,f2,f3,f4,u

slide-27
SLIDE 27

Two challenges

  • How to select the reference graph?
  • How to decide the updating order?

 Consistency implies accuracy

Inconsistent matchings Consistent matchings G1 G2 G3 G1 G2 G3 1 1’ 1’’

slide-28
SLIDE 28

How to select the reference graph?

GK

G1 G2 G3 G4 G5 G6 i,j=1,2,… Find the reference graph by: Yan et al. 2015 Less consistent Consistent First compute all Xij Then have the basis set: X1u, X2u, X3u, …, XNu

slide-29
SLIDE 29

How to decide the updating order?

Gi Gj

G1 G2 K=1,2,… Decide the updating order of Xur in ascending order of Cp(Xij,X)

slide-30
SLIDE 30

Algorithm (Non-factorized model)

slide-31
SLIDE 31

How consistency helps alternating

  • ptimization?

Deform test Outlier test Density test Order gain Reference graph gain

slide-32
SLIDE 32

Factorized formulation

convex concave

Pass

slide-33
SLIDE 33

Alternating updating

Solve lver: r: Frank Wolfe’s algorithm (FW)

Direction Y Step Convex-concave relaxation

Pass

slide-34
SLIDE 34

Algorithm (Factorized model)

Pass

slide-35
SLIDE 35

Overview

Non-factorized model Factorized model

Convex-concave relaxation

Direction Y Step size λ Frank Wolfe’s algorithm (FW)

RRWM IPFP GAGM etc.

QAP based two- graph matching solvers (any)

Solve a two-graph matching problem in each iteration

Pass

slide-36
SLIDE 36

Pairwise GM solvers in two stages

Stage 1: compute the pairwise matchings Xij for initialization Stage 2: compute the graph matching problem each iteration

  • Re-weighted Random Walk Matching

(RRWM, Cho et al. ECCV10)

  • Integer Projected Fixed Point Method

(IPFP, Cho et al. NIPS09)

  • Graduated Assignment for Graph

Matching (GAGM, Gold et al. PAMI96)

  • Factorized Graph Matching (FGM,

Zhou et al. CVPR12)

  • Fast Bipartite Matching (FBP,

Serratosa, PRL 2014)

[1] GAGM: S. Gold and A. Rangarajan, A graduated assignment algorithm for graph matching, PAMI 1996. [2] RRWM: M. Cho, J. Lee, and K. M. Lee, Reweighted random walks for graph matching, ECCV 2010 [3] IPFP: M. Leordeanu and M. Herbert, An integer projected fixed point method for graph matching and map inference, NIPS 2009 [4] Zhou and F. D. Torre, Factorized graph matching, CVPR 2012 [5] F. Serratosa, Fast computation of bipartite graph matching, Pattern Recognition Letters 2014

Pairwise GM methods

slide-37
SLIDE 37

Impact of pairwise solvers – XX

Matching accuracy: RRWM~=FGM~=GAGM>IPFP>FBP By noise level Two stages use the same two-graph matching method

slide-38
SLIDE 38

Impact of pairwise solvers – XX

By graph count Matching accuracy: RRWM~=FGM~=GAGM>IPFP>FBP Two stages use the same two-graph matching method

slide-39
SLIDE 39

Influence of pairwise solvers - XY

Matching accuracy: RRWM~=FGM~=GAGM>IPFP>FBP Two stages use different two-graph matching method

slide-40
SLIDE 40

Results on real image tests

slide-41
SLIDE 41

Visual results

slide-42
SLIDE 42

Outline

Introduction on Graph Matching

Reference graph based alternating approach, TIP’15

  • Consistency-driven Alternating Optimization for Multi-

graph Matching: a Unified Approach, IEEE Transactions

  • n Image Processing, 2015, 24 (3), 994-1009

More ‘distributed’ approach, TPAMI 2015

  • Multi-Graph Matching via Affinity Optimization with

Graduated Consistency Regularization, IEEE Transactions

  • n Pattern Analysis and Machine Intelligence, accepted
  • n Sep.1 2015, in press

Summary

slide-43
SLIDE 43

From two-graph to multi-graph (II)

  • Junchi Yan et al. 2014&2015
  • Journal extension
  • Multi-Graph Matching via Affinity Optimization with

Graduated Consistency Regularization, IEEE Transactions

  • n Pattern Analysis and Machine Intelligence, accepted
  • n Sep.1 2015, in press
  • Conference preliminary version
  • Graduated consistency-regularized optimization for

multi-graph matching, in ECCV, 2014

slide-44
SLIDE 44

Motivation

Alternating optimization may get stuck; Error may accumulate over iterations; There is not always a good reference graph for all involved graphs for matching

slide-45
SLIDE 45

Composition based Affinity Optimization

Key idea: efficiently generate new candidate solutions by first-order composition

G1 G2 G3 G1->G2 G1->G3->G2

Find higher affinity score solution via composition Interpolating graph (son) Father Mother

slide-46
SLIDE 46

Extend the composition chain?

deform

  • utlier

Better accuracy, but not cost-effective First-order Second-order

slide-47
SLIDE 47

A simple & general algorithm

Post-processing For enforcing

  • verall consistency

Still only local two graphs are involved in evaluation function Jij Local noise Modeling error

slide-48
SLIDE 48

Recall over fitting in machine learning

  • Affinity score (noise, bias) <-> Empirical term

(noise, bias)

  • Consistency score <-> Regularization term
slide-49
SLIDE 49

Ablation experiment: Consistency

  • nly vs. Affinity only

Affinity score only Pairwise-consistency score only deform

  • utlier

Only consistency works worse than

slide-50
SLIDE 50

Gradually add consistency as the regularization

Consistency works when the pairwise matchings are in general good accuracy

  • Recall the consistency-driven alternating method

Hence, first use affinity to decide the composition path to lift accuracy, then gradually incorporate consistency

  • Re-weight between affinity and consistency

First meet affinity good, then meet consistency good! Local noise Modeling error

slide-51
SLIDE 51

Gradually consistency-weighed CAO

Affinity term Consistency score Consistency Somewhat like deep learning, first tune the network fitting with a small dataset, then try on a larger dataset!

slide-52
SLIDE 52

Inexact-consistency to speedup

Exact Inexact Unary-consistency Pairwise-consistency Affinity Why slow? Because need to compute Y from scratch!! See more details in the paper One-shot computing! See more details in the paper

slide-53
SLIDE 53

Two approximate variants

slide-54
SLIDE 54

Experiment settings

Key control parameters for random synthetic tests Dataset

  • CMU Motion Sequence

http://vasc.ri.cmu.edu//idb/html/motion/

  • POSE Sequence

http://www.cvl.isy.liu.se/research/objrec/posedb/

  • Willow-ObjectClass

http://www.di.ens.fr/willow/research/graphlearning/ Comparing methods

  • Re-weighted Random Walk

Matching (RRWM, Cho et al. ECCV10)

  • Max-Pooling Matching (MPM,

Cho et al. CVPR14)

  • MatchOpt (Yan et al. TIP15,

ICCV13)

  • MatchSync (Pachauri et al.

NIPS13)

  • MatchLift (Chen et al. ICML14)

NIPS13 ICML14 CVPR14

slide-55
SLIDE 55

Few outliers – synthetic tests (I)

CAO-C or CAO-C* works almost best!

slide-56
SLIDE 56

Few outliers – real image tests (I)

CAO-C or CAO-C* works almost best!

slide-57
SLIDE 57

General mechanism to handle

  • utliers

Inliers Outliers Evaluation function Inliers elicitation Node-wise consistency Node-wise affinity Higher node-wise consistency or affinity, higher confidence as inliers

slide-58
SLIDE 58

Assume inlier number ni is known

Can be estimated by certain means, or given a template graph with inliers Mask by node-wise consistency or affinity Elicited inliers Rejected outliers Inlier mask G1 G2 G3 G4

slide-59
SLIDE 59

Redefine the evaluation terms

Node-wise consistency mask Node-wise affinity mask Inlier elicitation

slide-60
SLIDE 60

Sensitivity of the pre-set ni

Synthetic test

Ground truth: ni is set to 10 Ground truth: ni is set to 6

slide-61
SLIDE 61

More outliers – synthetic tests

Inlier#=6 Outlier#=12 Inlier#=6 Outlier#=12

Inlier# = 6 Inlier# = 6

slide-62
SLIDE 62

More outliers – real image tests

Inlier# = 6 Inlier# = 6 Inlier# = 6 Inlier# = 6 Inlier# = 6 Inlier# = 6

slide-63
SLIDE 63

Outline

Introduction on Graph Matching

Reference graph based alternating approach, TIP’15

  • Consistency-driven Alternating Optimization for Multi-

graph Matching: a Unified Approach, IEEE Transactions

  • n Image Processing, 2015, 24 (3), 994-1009

More ‘distributed’ approach, TPAMI 2015

  • Multi-Graph Matching via Affinity Optimization with

Graduated Consistency Regularization, IEEE Transactions

  • n Pattern Analysis and Machine Intelligence, accepted
  • n Sep.1 2015, in press

Summary

slide-64
SLIDE 64

Uncovered topics in this slide

When & what relaxation is globally optimal?

  • On convex relaxation of graph isomorphism, PNAS, 2015

Many-to-many matching & progressive matching

  • Density maximization for improving graph matching with its application, T-IP, 2015

Learning for graph matching

  • Learning graphs to match In ICCV, 2013

Higher-order graph matching

  • A flexible tensor block coordinate ascent scheme for hypergraph matching, CVPR, 2015

… More can be found in my survey paper:

  • J. Yan et al. A Short Survey of Recent Advances in Graph Matching, submitted to ICMR

2016

slide-65
SLIDE 65

Conclusion & outlook

Obtains

  • Consistency is a simple and effective way to infuse

information across multiple graphs

  • Paradigms are in fact general to feature matching and graph

matching with different cost functions and solvers

For future work (limitation of current work)

  • More practical methods for graph construction
  • Partial matching across graphs
  • Scalable methods, parallelization
  • Methods to address many-to-one, many-to-many mappings

for real applications

  • Applications to practical computer vision problems, such as
  • bject discovery in the wild
slide-66
SLIDE 66

Code & paper can be found

  • http://www.cc.gatech.edu/~zha/papers.html
  • http://www.cc.gatech.edu/~zha/papers/demoCode.zip
  • Papers
  • Junchi Yan, M. Cho, H. Zha, X. Yang and S. Chu. Multi-Graph

Matching via Affinity Optimization with Graduated Consistency Regularization. To appear in IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016.

  • Junchi Yan, J. Wang, H. Zha, X.Yang and S. Chu. Consistency-

driven Alternating Optimization for Multi-Graph Matching: a Unified Approach. IEEE Transactions on Image Processing,

  • vol. 24: 994-1009, 2015.
slide-67
SLIDE 67

Question?

slide-68
SLIDE 68

Acknowledgement

Special thanks go to Dr. Feng Zhou, Dr. Minsu Cho

  • Some illustrations are originated/enlightened by their slides/papers
  • Their released code is used for evaluation

http://www.di.ens.fr/~mcho/ http://f-zhou.com/ Hongyuan Zha, ECNU Xiaokang Yang, SJTU Stephen Chu, IBM Wei Sun, IBM http://pages.cs.wisc.edu/~pachauri/ web.stanford.edu/~yxchen/ http://deim.urv.cat/ ~francesc.serratosa/

Minsu Cho Feng Zhou

slide-69
SLIDE 69

Graph isomorphism problem

https://www.sciencenews.org/article/new-algorithm-cracks-graph-problem Quasipolynomial algorithm for general graphs http://people.cs.uchicago.edu/~laci/quasipoly.html László Babai, Submitted on 11 Dec 2015, Graph Isomorphism in Quasipolynomial Time http://arxiv.org/abs/1512.03547v1

Thirty years