Incorporating Domain Knowledge in Matching Problems via Harmonic - - PowerPoint PPT Presentation

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Incorporating Domain Knowledge in Matching Problems via Harmonic Analysis

Deepti Pachauri (joint work with Maxwell Collins, Risi Kondor, Vikas Singh)

University of Wisconsin-Madison University of Chicago

International Conference on Machine Learning 2012

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Matching Problems are Ubiquitous

Photo Tourism

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Matching Problems are Ubiquitous

Shape Matching

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Matching Problems are Ubiquitous

Shape Matching General Strategy Write the functional form of the matching problem and then use an appropriate optimization engine to find a solution.

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Matching Problems are Ubiquitous

Shape Matching General Strategy Write the functional form of the matching problem and then use an appropriate optimization engine to find a solution. Use past knowledge to make future instances easier . . . ?

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Overview Motivation Problem Setup Graph Matching and QAPs Why learn QAPs? Algebraic Structure of Sn and Harmonic Analysis Learning in Fourier Space Evaluations

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

G =       1 1 1 1 1 1 1 1       G′ =       1 1 1 1 1 1 1 1      

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

G =       1 1 1 1 1 1 1 1       G′ =       1 1 1 1 1 1 1 1      

Solution of matching problem is a permutation matrix y

y =       1 1 1 1 1       σ := (51342) such that yGy⊤ = G′

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Quadratic Assignment Problem (QAP)

y∗ = arg max

y

• ii′

cii′yii′ +

• ii′jj′

dii′jj′yii′yjj′

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Quadratic Assignment Problem (QAP)

y∗ = arg max

y

• ii′

cii′yii′ +

• ii′jj′

dii′jj′yii′yjj′ Computationally expensive: n ≥ 40 infeasible in general.

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Supervised Learning

Given Training data : ((x1, y1), ..., (xm, ym)) f ω(xi) ≈ yi (x1, y1) : f ω(x1) ≈ y1 (x2, y2) : f ω(x2) ≈ y2 (x3, y3) : f ω(x3) ≈ y3 and so on ....... . . . and we will solve arg max f ω(x3) cheaply.

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Learning for QAPs?

Given Training data : ((x1, σ1), ..., (xm, σm)) arg max f ω(xi) ≈ σi (x1, σ1) : arg max f ω(x1) ≈ σ1 (x2, σ2) : arg max f ω(x2) ≈ σ2 (x3, σ3) : arg max f ω(x3) ≈ σ3 and so on .......

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Learning for QAPs?

Given Training data : ((x1, σ1), ..., (xm, σm)) arg max f ω(xi) ≈ σi (x1, σ1) : arg max f ω(x1) ≈ σ1 (x2, σ2) : arg max f ω(x2) ≈ σ2 (x3, σ3) : arg max f ω(x3) ≈ σ3 and so on ....... . . . and we want to solve arg max f ω(xi) cheaply.

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Inspired in part by

Caetano et al., PAMI 2009 Structure learning approach to find most violated constraints using linear assignment. Xu et al., JMLR 2009 Use disciminative learning to acquire a domain–specific heuristic for controlling beam–search. Stobbe et al., AISTATS 2012 Fourier space sparsity to recover a set function from very few samples.

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Structure of σ ∈ Sn

Harmonic Analysis Fourier transform of a function f : R → C ˆ f(λ) =

• x∈R

f(x)e2πixλ λ ∈ R,

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Structure of σ ∈ Sn

Harmonic Analysis on Symmetric Groups Sn ˆ f(ρλ) =

• σ∈Sn

f(σ)ρλ(σ) ρλ ∈ R λ is the integer partition of n, λ ⊢ n ρλ(σ) is the irreducible representation of Sn ρλ(σ) =   ρ1,1 · · ρ1,dλ · · · · ρdλ,1 · · ·  

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Properties Sn

Convolution (f ∗ g)(σ) =

τ∈Sn f(στ −1) g(τ)

• f ∗ g(λ) = ˆ

f(λ)ˆ g(λ) Correlation (f ⋆ g)(σ) =

τ∈Sn f(στ)g(τ)∗

• f ⋆ g(λ) = ˆ

f(λ)ˆ g(λ)†

Sn−1 is a subgroup of Sn

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Properties Sn

Convolution (f ∗ g)(σ) =

τ∈Sn f(στ −1) g(τ)

• f ∗ g(λ) = ˆ

f(λ)ˆ g(λ) Correlation (f ⋆ g)(σ) =

τ∈Sn f(στ)g(τ)∗

• f ⋆ g(λ) = ˆ

f(λ)ˆ g(λ)†

Sn−1 is a subgroup of Sn The set σSn−1 is called a left coset of σ Two left (right) cosets are either disjoint or the same

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Coset Tree

Cosets provide a partition of Sn:

213 S3 312 S2 S2 S2 132 123 321 231

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

f : Sn → C

Graph function of G fA(σ) = Aσ(n),σ(n−1) Properties: Sn−2-invariant function on adjacency matrix A (Kondor, 2010) Band-limited in Fourier domain (Rockmore, 2002) Under relabeling, fAπ = f π

A

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Graph Matching Problem

Standard QAP: Given a pair of graphs

max

σ∈Sn f(σ) = n

• i,j=1

Ai,jA′

σ(i),σ(j)

Graph Correlation: f(σ) = 1 (n − 2)!

• π∈Sn

fA(σπ)fA′(π) (A, A′) could be weighted or unweighted adjacency matrices.

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Learning Graph Matching

Given: A training set of related graph pairs with D encodings

• f adjacency matrices : (Gm, G′

m), m = {1, · · · , M}.

Goal: “Learn” parameters ω such that QAP procedure finds a good solution (quickly) for the test case (unseen graph pairs).

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Learning Graph Matching

Given: A training set of related graph pairs with D encodings

• f adjacency matrices : (Gm, G′

m), m = {1, · · · , M}.

Goal: “Learn” parameters ω such that QAP procedure finds a good solution (quickly) for the test case (unseen graph pairs). Define parameter vector ω ∈ RD

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Learning Graph Matching

Given: A training set of related graph pairs with D encodings

• f adjacency matrices : (Gm, G′

m), m = {1, · · · , M}.

Goal: “Learn” parameters ω such that QAP procedure finds a good solution (quickly) for the test case (unseen graph pairs). Define parameter vector ω ∈ RD

QAP Objective for Learning: f ω(σ) =

D

• d=1

ωdf d(σ)

where f d(σ) =

1 (n−2)!

• π∈Sn

fAd(σπ)fA′d(π) =

• i,j

Ad

ijA′d σ(i)σ(j)

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Learning Correct bounds on Coset Tree

213 S3 312 S2 S2 S2 132 123 321 231

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Learning Correct bounds on Coset Tree

213 S3 312 S2 S2 S2 132 123 321 231

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Learning Correct bounds on Coset Tree

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Learning Correct bounds on Coset Tree

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Fourier Domain QAP Solver

Fast Fourier Transform ˆ f ω(λ) =

n

• i=1

dλ ndµ ρλ([[i, n]])

• µ∈λ↓n−1

ˆ f ω

i (µ)

Fourier Space Bounds [Kondor et.al.] Bn→i =

• µ⊢n−1

ˆ f ω

i (µ)∗

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Risk Minimization

Loss Function

n

• k=1
• i∈children((n−k+1)∗)
• ˆ

f ω

i (µ)∗ − ˆ

f ω

i∗

n−k(µ)∗ + 1

+ i∗

n−k is the correct node at level n − k in coset tree.

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Risk Minimization

Jensen’s Inequality For parameterization: ˆ f ω

i (µ) = D d=1 ωdˆ

f d

i (µ)

ˆ f ω

i (µ)∗ = D

• d=1

ωdˆ f d

i (µ)∗ ≤ D

• d=1

ωdˆ f d

i (µ)∗

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Risk Minimization

Jensen’s Inequality For parameterization: ˆ f ω

i (µ) = D d=1 ωdˆ

f d

i (µ)

ˆ f ω

i (µ)∗ = D

• d=1

ωdˆ f d

i (µ)∗ ≤ D

• d=1

ωdˆ f d

i (µ)∗

Fourier space Stochastic Gradient Descent Solver Each update takes the form ωd ← ωd − η

• ˆ

f d

i (µ)∗ − ˆ

f d

i∗

n−k(µ)∗ +

ν MO(n2)ωd ν MO(n2)ωd

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Risk Minimization

Jensen’s Inequality For parameterization: ˆ f ω

i (µ) = D d=1 ωdˆ

f d

i (µ)

ˆ f ω

i (µ)∗ = D

• d=1

ωdˆ f d

i (µ)∗ ≤ D

• d=1

ωdˆ f d

i (µ)∗

Fourier space Stochastic Gradient Descent Solver Each update takes the form ωd ← ωd − η

• ˆ

f d

i (µ)∗ − ˆ

f d

i∗

n−k(µ)∗ +

ν MO(n2)ωd ν MO(n2)ωd

Convergence: emulate proof for D-dimensional Perceptron.

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Experimental Results

Setup Edge: Delaunay triangulation on interest points Distance: Euclidean distance between interest points Shape Context (60 in all): Similarities based on local shape-based appearance of interest points

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Experimental Results

Setup Edge: Delaunay triangulation on interest points Distance: Euclidean distance between interest points Shape Context (60 in all): Similarities based on local shape-based appearance of interest points Task Learn ω using training instances Solve the learnt problem “cheaply” (e.g., greedy or linear assignment) Evaluate compromise on accuracy? Evaluate improvements in running time?

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Experimental Results: CMU House

Figure: (Green) the ground truth and (red) the learnt correspondences.

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Experimental Results: CMU Hotel

Figure: (Green) the ground truth and (red) the learnt correspondences.

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Experimental Results: Silhouette

Figure: (Green) the ground truth and (red) the learnt correspondences.

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Accuracy vs. Offset: CMU House

Figure: Our method compared with no-learn baseline. (Red) learning and (blue) no-learning.

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Accuracy vs. Offset: CMU Hotel

Figure: Our method compared with no-learn baseline. (Red) learning and (blue) no-learning.

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Accuracy vs. Offset: Silhouette

Figure: Our method compared with no-learn baseline. (Red) learning and (blue) no-learning.

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Conclusions

Incorporating domain knowledge help solving hard problems. Harmonic analysis provide nice structure for matching problems. Other parameterization schemes might provide further insights. Please come to the poster session. Poster 15 in Informatics Forum.

Motivation Learning QAPs Algebra of Sn Algorithm Experiments

Thank You!