The Assignment Flow Christoph Schnrr Ruben Hhnerbein, Stefania - - PowerPoint PPT Presentation

The Assignment Flow

Christoph Schnörr Ruben Hühnerbein, Stefania Petra, Fabrizio Savarino, 
 Alexander Zeilmann, Artjom Zern, Matthias Zisler
 Image & Pattern Analysis Group
 Heidelberg University Mathematics of Imaging: W1
 Feb 4-8, Paris

Machine learning and …

• signal & image proc.
• computer vision
• math. imaging
• inverse problems
• statistics
• …

Machine learning and …

predictive accuracy descriptive accuracy

deep networks

• signal & image proc.
• computer vision
• math. imaging
• inverse problems
• statistics
• …

debugging heat maps …

Machine learning and …

predictive accuracy descriptive accuracy

deep networks debugging heat maps …

“interpretable ML !” “explainable AI !”

Machine learning and …

predictive accuracy descriptive accuracy

deep networks graphical models

Geman & Geman (T-PAMI 1984) … Kappes et al. (IJCV 2015)

Machine learning and …

predictive accuracy descriptive accuracy

deep networks graphical models

Machine learning and …

predictive accuracy descriptive accuracy

deep networks assignment flow

• smoothness
• hierarchical structure
• math. framework

graphical models

Machine learning and …

predictive accuracy descriptive accuracy

deep networks assignment flow

• smoothness
• hierarchical structure
• math. framework

graphical models

Outline

• set-up: assignment flow


supervised labeling

• unsupervised labeling

• label evolution

• label learning from scratch
• parameter estimation (control)
• outlook

JMIV’17 SIIMS’18

Set-up: assignment flow & supervised labeling

fi g1 V g2 gm

Di =

• d(fi, g1), . . . , d(fi, gm)
• feature

labels
 (prototypes) metric, distance vector (data term)

• fi ∈ (X, d)

features
 metric space

∈ X ∈ W Wi =

• Pr(g1|fi), . . . , Pr(gm|fi)
• assignment


vector

g1

g2 g3

Wi

Fisher-Rao metric

• n S

Set-up: assignment flow & supervised labeling

fi g1 V g2 gm

Di =

• d(fi, g1), . . . , d(fi, gm)
• labels


(prototypes) metric, distance vector (data term)

• fi ∈ (X, d)

features
 metric space

g1

g2 g3

Wi

˙ Wi = ΠWi

• Li(Wi)
• = ΠWi

⇣ expWi

• 1

ρDi ⌘

likelihood
 vector Fisher-Rao metric

• n S

feature

Set-up: assignment flow & supervised labeling

fi g1 V g2 gm

Di =

• d(fi, g1), . . . , d(fi, gm)
• labels


(prototypes) metric, distance vector (data term)

• fi ∈ (X, d)

features
 metric space

g1

g2 g3

Wi

Gw

i (W) = ExpWi

⇣ X

k2Ni

wik Exp1

Wi(Wk)

⌘

, Si(W) = Gw

i

• L(W)
• similarity


vector regularisation

control parameters

scale feature

Set-up: assignment flow & supervised labeling

fi g1 V g2 gm

Di =

• d(fi, g1), . . . , d(fi, gm)
• labels


(prototypes) metric, distance vector (data term)

• fi ∈ (X, d)

features
 metric space

˙ W = ΠW

• S(W)
• ,

W

∈ , W(0) = 1W.

W(t) 2 (W, gF R)

assignment
 manifold assignment
 flow

W = S ⇥ · · · ⇥ S

scale feature

Illustration

ρ

scale

Illustration

local 3 x 3 5 x 5

Assignment flow: geometric integration

ODEs on manifolds, Lie group methods (Iserles et al.’05, Hairer et al. ’06)

λ(v, p) = Λ(expG(v), p)

(λ∗v)p = d dtΛ(expG(tv), p)

• t=0

, y(0) = p. ˙ y =

• λ∗f(t, y)
• y,

y

y(t) = λ

• v(t), p
• ˙

v = (dexp−1

G )v

• f(t, λ(v, p))
• ,

v

, v(0) = 0,

Λ: G ⇥ M ! M

g X(M) G λ M Λ

λ∗ expG expM f

Assignment flow: geometric integration

nonlinear assignment flow linear assignment flow geometric RKMK implicit geometric Euler embedded RKMK adaptive RK exponential integrator numerical experiments

W(t) = expW0

• V (t)
• any W0 = W(0)

˙ V = ΠT0S

• expW0(V )
• ,

V , V (0) = 0.

(Zeilmann et al, arXiv’18)

Properties (more general viewpoint)

• elementary state space
• information geometry


• scale
• distances

statistical manifold dualistic structure 
 
 small / cooperative & large / competitive adaptive distances

e (g, r, r∗)

r Zg(X, Y ) = g(rZX, Y ) + g(X, r∗

ZY ),

(W, g)

D D

D D(W), W 2 W

(Amari & Chentsov)

• set-up: assignment flow


supervised labeling

• unsupervised labeling

• label evolution

• label learning from scratch
• parameter estimation (control)
• outlook

Outline

(Zern et al., GCPR’18) (submitted)

Label evolution

fi gj

data (feature)
 manifold

M

Di =

• d(fi, g1), . . . , d(fi, gm)
• metric, distance vector (data term)

fi gj

data labels

adapt online ! preprocessing

Label evolution

P

∈

˙ Wi(t) = ΠWi(t)

• Si
• W(t)
• ,

Wi(0) = 1S, i

, i ∈ I,

label flow (“m”: label as Riemannian means) assignment flow

coupling spatial regularisation time scale divergence measure

˙ mj(t) = α X

i∈I

νj|i

• M(t)
• b

g−1 djD(fi, mj(t))

• ,

mj(0) = mj0, α > 0 X

∈

• b
• νj|i(M) =

Lσ

ij(Wi; M)

P

k∈I Lσ kj(Wk; M),

Lσ

ij(Wi; M) =

Wije− 1

σ D(fi,mj)

P

l∈J Wile− 1

σ D(fi,ml) ,

σ > 0

Label evolution

SO(3)-valued data

Label evolution

Euclidean color space supervised: 200 labels unsupervised: few labels

Label evolution

positive-def. manifold (dim = 120) supervised: 200 labels

Pd 3 Fi = Z h(xi y) ⇥ (f Ei[f]) ⌦ (f Ei[f]) ⇤ (y) dy

1

Label evolution

supervised: 200 labels few labels few labels

• set-up: assignment flow


supervised labeling

• unsupervised labeling

• label evolution

• label learning from scratch
• parameter estimation (control)
• outlook

Outline

(Zern et al., GCPR’18) (submitted)

Label learning from scratch Di =

• d(fi, g1), . . . , d(fi, gm)
• metric, distance vector (data term)

D =

• d(fi, fk)
• i,k2I.

∈ X ∈ W Wi =

• Pr(g1|fi), . . . , Pr(gm|fi)
• ?

each datum is a label

?

Label learning from scratch Di =

• d(fi, g1), . . . , d(fi, gm)
• metric, distance vector (data term)

D =

• d(fi, fk)
• i,k2I.

∈ X ∈ W Wi =

• Pr(g1|fi), . . . , Pr(gm|fi)
• ?

each datum is a label

?

2 W Q(fi|fk) = P(fk|fi)P(fi) P

l2I P(fk|fl)P(fl),

P , P(fk) = 1 |I|, k 2 I

marginalize over “data labels’’

P

2

| | | Aji(W) = X

k2I

Q(fj|fk)P(fk|fi) =

• WC(W)1W >

ji

self-affinity matrix symmetric, non-negative, doubly stochastic parametrised by assigments

Label learning from scratch

• bjective: spatially regularised data self-assignment

min

W 2W E(W),

E , E(W) = hD, A(W)i

(generalizes )

hD, Wi

Label learning from scratch

• bjective: spatially regularised data self-assignment

min

W 2W E(W),

E , E(W) = hD, A(W)i

approach: redefine the likelihood vectors

L(W) = expW

• 1

ρrE(W)

• 2 W,

L

(generalizes )

hD, Wi

˙ W = ΠW

• S(t)
• unsupervised

self-assignment flow

scale

single parameter:

Label learning from scratch

• bjective: spatially regularised data self-assignment

min

W 2W E(W),

E , E(W) = hD, A(W)i

approach: redefine the likelihood vectors

L(W) = expW

• 1

ρrE(W)

• 2 W,

L

(generalizes )

hD, Wi

˙ W = ΠW

• S(t)
• unsupervised

self-assignment flow result: spatially regularised discrete optimal transport approaches a low-rank manifold labels and their number emerge from data as latent variables

D, A(W)

, gk

, gk fi

scale

single parameter:

Unsupervised self-assignment flow

S1 - valued data

• set-up: assignment flow


supervised labeling

• unsupervised labeling

• label evolution

• label learning from scratch
• parameter estimation (control)
• outlook

Outline

(submitted)

Recall: regularisation & control parameters

fi g1 V g2 gm

Di =

• d(fi, g1), . . . , d(fi, gm)
• labels


(prototypes) metric, distance vector (data term)

• fi ∈ (X, d)

features
 metric space

g1

g2 g3

Wi

Gw

i (W) = ExpWi

⇣ X

k2Ni

wik Exp1

Wi(Wk)

⌘

, Si(W) = Gw

i

• L(W)
• similarity


vector regularisation

control parameters

scale feature

Motivation

all patch-weights

uniform weights predicted weights after learning

• Ω = {wik : k 2 Ni, i 2 I} 2 P
• (= weight manifold)

Approach

all patch-weights linear assignment flow

W(t) = ExpW0

• V (t)
• ,

{ 2 N 2 } ˙ V = ΠW0

• S(W0) + dSW0V
• ,

, V (0) = 0,

linear

• Ω = {wik : k 2 Ni, i 2 I} 2 P

Approach

all patch-weights linear assignment flow

W(t) = ExpW0

• V (t)
• ,

{ 2 N 2 } ˙ V = ΠW0

• S(W0) + dSW0V
• ,

, V (0) = 0,

linear

• Ω = {wik : k 2 Ni, i 2 I} 2 P
• E
• V (T)
• = DKL
• W ⇤, exp1W
• V (T)
• bjective

ground-assignments (labelings)

Approach

all patch-weights linear assignment flow

W(t) = ExpW0

• V (t)
• ,

{ 2 N 2 } ˙ V = ΠW0

• S(W0) + dSW0V
• ,

, V (0) = 0,

linear

• Ω = {wik : k 2 Ni, i 2 I} 2 P
• E
• V (T)
• = DKL
• W ⇤, exp1W
• V (T)
• bjective

ground-assignments (labelings)

parameter estimation problem

min

Ω∈P

E

• V (T)
• s.t.

˙ V (t) = f(V (t), Ω), t 2 [0, T], V (0) = 0|I|×|J|.

training data for parameter prediction

Approach

all patch-weights linear assignment flow

W(t) = ExpW0

• V (t)
• ,

{ 2 N 2 } ˙ V = ΠW0

• S(W0) + dSW0V
• ,

, V (0) = 0,

linear

• Ω = {wik : k 2 Ni, i 2 I} 2 P
• E
• V (T)
• = DKL
• W ⇤, exp1W
• V (T)
• bjective

ground-assignments (labelings)

parameter estimation problem

min

Ω∈P

E

• V (T)
• s.t.

˙ V (t) = f(V (t), Ω), t 2 [0, T], V (0) = 0|I|×|J|.

parameter prediction

b w: Fi ! P,

(fk)k2Ni ! (wik)k2Ni, i

training data for parameter prediction novel data

definition & meaning of what the network learns!

Parameter estimation algorithm

• ˙

Ω = rPE

• V (T, Ω)
• = ΠΩ

⇣ d dΩE

• V (T, Ω)

⌘ , Ω(0) = 1P

geometric integration (parameter manifold)

recurring structure

• ptimize then discretise vs. discretise then optimise ?

Parameter estimation algorithm

• ˙

Ω = rPE

• V (T, Ω)
• = ΠΩ

⇣ d dΩE

• V (T, Ω)

⌘ , Ω(0) = 1P

E

• V (T, Ω)
• s.t. ˙

V = f(V, Ω) adjoint system nonlinear program sensitivity

diffentiate discretize discretize diffentiate

gradient flow (weight parameter manifold)

recurring structure

Either way yields the same solution ! Key aspect: symplectic integrator for the joint system

Adaptive regularization

image class: non-curvilinear letters image features: binary patches

Adaptive regularization

uniform uniform adaptive adaptive sanity check novel curvilinear structure

Adaptive regularization

noisy random voronoi images

training image test image

Ω* (sample of patches)

uniform weights adaptive weights weight deviation from uniform

Outlook

patch assignment pixel assignments unsupervised label learning patch dictionary based prior statistics model reduction & optimal control parameter estimation

̂ w (ℱ𝒪i) ̂ w (ℱ𝒪i, W𝒪i(t))

two-grid structure: data-driven model-driven dynamic adaptive parameter estimation

Publications

prior work: IPA group, Heidelberg https://ipa.math.uni-heidelberg.de geometric integration: arXiv:1810.06970, submitted to SI label learning from scratch: submitted to conference
 parameter estimation: full TRs: arXiv soon synopsis: handbook: variational methods for nonlinear geometric data
 and applications, ~ July’19

The Assignment Flow

Christoph Schnörr Ruben Hühnerbein, Stefania Petra, Fabrizio Savarino, 
 Alexander Zeilmann, Artjom Zern, Matthias Zisler
 Image & Pattern Analysis Group
 Heidelberg University Mathematics of Imaging: W1
 Feb 4-8, Paris