The Assignment Flow Christoph Schnrr Ruben Hhnerbein, Stefania - PowerPoint PPT Presentation

The Assignment Flow Christoph Schnörr Ruben Hühnerbein, Stefania Petra, Fabrizio Savarino, 
 Alexander Zeilmann, Artjom Zern, Matthias Zisler 
 Image & Pattern Analysis Group 
 Heidelberg University Mathematics of Imaging: W1 
 Feb 4-8, Paris

Machine learning and … • signal & image proc. • computer vision • math. imaging • inverse problems • statistics • …

Machine learning and … • signal & image proc. • computer vision • math. imaging predictive accuracy deep networks • inverse problems • statistics • … debugging heat maps … descriptive accuracy

Machine learning and … “interpretable ML !” “explainable AI !” predictive accuracy deep networks debugging heat maps … descriptive accuracy

Machine learning and … predictive accuracy deep networks Geman & Geman (T-PAMI 1984) … Kappes et al. (IJCV 2015) graphical models descriptive accuracy

Machine learning and … predictive accuracy deep networks graphical models descriptive accuracy

Machine learning and … predictive accuracy deep networks • smoothness • hierarchical structure • math. framework assignment flow graphical models descriptive accuracy

Machine learning and … predictive accuracy deep networks • smoothness • hierarchical structure • math. framework assignment flow graphical models descriptive accuracy

Outline • set-up: assignment flow 
 JMIV’17 supervised labeling SIIMS’18 • unsupervised labeling 
 - label evolution 
 - label learning from scratch • parameter estimation (control) • outlook

Set-up: assignment flow & supervised labeling � � D i = d ( f i , g 1 ) , . . . , d ( f i , g m ) metric, distance vector ( data term ) ∈ X ∈ W � � W i = Pr( g 1 | f i ) , . . . , Pr( g m | f i ) g m assignment 
 g 3 vector labels 
 (prototypes) V f i feature Fisher-Rao g 2 metric g 1 on S W i g 1 g 2 � features 
 f i ∈ ( X , d ) metric space

Set-up: assignment flow & supervised labeling � � D i = d ( f i , g 1 ) , . . . , d ( f i , g m ) metric, distance vector ( data term ) � 1 ⇣ �⌘ ˙ � � � W i = Π W i L i ( W i ) = Π W i exp W i ρ D i g m g 3 likelihood 
 labels 
 vector (prototypes) V f i feature Fisher-Rao g 2 metric g 1 on S W i g 1 g 2 � features 
 f i ∈ ( X , d ) metric space

Set-up: assignment flow & supervised labeling � � D i = d ( f i , g 1 ) , . . . , d ( f i , g m ) metric, distance vector ( data term ) ⇣ X ⌘ w ik Exp � 1 G w i ( W ) = Exp W i W i ( W k ) g m k 2 N i g 3 S i ( W ) = G w � � regularisation L ( W ) , i labels 
 (prototypes) V control f i feature similarity 
 parameters vector g 2 g 1 W i g 1 g 2 � features 
 f i ∈ ( X , d ) metric space scale

Set-up: assignment flow & supervised labeling � � D i = d ( f i , g 1 ) , . . . , d ( f i , g m ) metric, distance vector ( data term ) ∈ assignment 
 ˙ � � W (0) = 1 W . W = Π W S ( W ) , , W flow g m assignment 
 W ( t ) 2 ( W , g F R ) labels 
 manifold (prototypes) V f i feature W = S ⇥ · · · ⇥ S g 2 g 1 � features 
 f i ∈ ( X , d ) metric space scale

Illustration scale ρ

Illustration local 5 x 5 3 x 3

Assignment flow: geometric integration ODEs on manifolds, Lie group methods ( Iserles et al.’05, Hairer et al. ’06) ( λ ∗ v ) p = d λ ( v, p ) = Λ (exp G ( v ) , p ) � d t Λ (exp G ( tv ) , p ) � t =0 λ ∗ X ( M ) g � � y = ˙ λ ∗ f ( t, y ) y , y y (0) = p. , f exp G exp M G λ M � � y ( t ) = λ v ( t ) , p Λ v = (dexp − 1 � � ˙ G ) v f ( t, λ ( v, p )) , v (0) = 0 , v , Λ : G ⇥ M ! M

Assignment flow: geometric integration � � W ( t ) = exp W 0 V ( t ) any W 0 = W (0) nonlinear assignment flow ˙ � � V = Π T 0 S exp W 0 ( V ) , V V (0) = 0 . , (Zeilmann et al, arXiv’18) linear assignment flow geometric RKMK implicit geometric Euler exponential integrator adaptive RK embedded RKMK numerical experiments


 
 Properties (more general viewpoint) • elementary state space statistical manifold ( W , g ) e ( g, r , r ∗ ) • information geometry 
 dualistic structure 
 r (Amari & Chentsov) Zg ( X, Y ) = g ( r Z X, Y ) + g ( X, r ∗ Z Y ) , • scale small / cooperative & large / competitive • distances adaptive distances D D D ( W ) , W 2 W D

Outline • set-up: assignment flow 
 supervised labeling • unsupervised labeling 
 (Zern et al., GCPR’18) - label evolution 
 (submitted) - label learning from scratch • parameter estimation (control) • outlook

Label evolution � � D i = d ( f i , g 1 ) , . . . , d ( f i , g m ) metric, distance vector ( data term ) preprocessing labels adapt online ! g j g j f i data (feature) 
 manifold M data f i

time scale Label evolution divergence measure label flow ( “m” : label as Riemannian means) X X � � � � g − 1 � � � � m j ( t ) = � α ˙ M ( t ) b b d j D ( f i , m j ( t )) m j (0) = m j 0 , α > 0 ν j | i , i ∈ I ∈ W ij e − 1 σ D ( f i ,m j ) ij ( W i ; M ) L σ P ν j | i ( M ) = ij ( W i ; M ) = σ > 0 kj ( W k ; M ) , L σ σ D ( f i ,m l ) , P l ∈ J W il e − 1 k ∈ I L σ � � coupling P spatial regularisation assignment flow ∈ � � �� ˙ , i ∈ I, W i ( t ) = Π W i ( t ) W ( t ) W i (0) = 1 S , S i , i

Label evolution SO(3)-valued data

Label evolution supervised: 200 labels Euclidean color space unsupervised: few labels

Label evolution supervised: 200 labels positive-def. manifold (dim = 120) Z ⇥ ⇤ P d 3 F i = h ( x i � y ) ( f � E i [ f ]) ⌦ ( f � E i [ f ]) ( y ) d y 1

Label evolution supervised: 200 labels few labels few labels

Outline • set-up: assignment flow 
 supervised labeling • unsupervised labeling 
 (Zern et al., GCPR’18) - label evolution 
 (submitted) - label learning from scratch • parameter estimation (control) • outlook

Label learning from scratch � � D i = d ( f i , g 1 ) , . . . , d ( f i , g m ) metric, distance vector ( data term ) ∈ X ∈ W ? � � D = d ( f i , f k ) i,k 2 I . � � W i = Pr( g 1 | f i ) , . . . , Pr( g m | f i ) ? each datum is a label

Label learning from scratch � � D i = d ( f i , g 1 ) , . . . , d ( f i , g m ) metric, distance vector ( data term ) ∈ X ∈ W ? � � D = d ( f i , f k ) i,k 2 I . � � W i = Pr( g 1 | f i ) , . . . , Pr( g m | f i ) ? each datum is a label 2 W P ( f k | f i ) P ( f i ) P ( f k ) = 1 Q ( f i | f k ) = l 2 I P ( f k | f l ) P ( f l ) , , P | I | , k 2 I P marginalize over “data labels’’ self-a ffi nity matrix P | | | 2 X WC ( W ) � 1 W > � � A ji ( W ) = Q ( f j | f k ) P ( f k | f i ) = ji k 2 I symmetric, non-negative, doubly stochastic parametrised by assigments

Label learning from scratch objective: spatially regularised data self-assignment (generalizes ) h D, W i W 2 W E ( W ) , min E ( W ) = h D, A ( W ) i , E

Label learning from scratch objective: spatially regularised data self-assignment (generalizes ) h D, W i W 2 W E ( W ) , min E ( W ) = h D, A ( W ) i , E approach: redefine the likelihood vectors � 1 � � L ( W ) = exp W ρ r E ( W ) 2 W , L unsupervised � � self-assignment flow ˙ W = Π W S ( t ) single parameter: scale

Label learning from scratch objective: spatially regularised data self-assignment (generalizes ) h D, W i W 2 W E ( W ) , min E ( W ) = h D, A ( W ) i , E approach: redefine the likelihood vectors � 1 � � L ( W ) = exp W ρ r E ( W ) 2 W , L unsupervised � � self-assignment flow ˙ W = Π W S ( t ) single parameter: scale result: spatially regularised discrete optimal transport approaches a low-rank manifold D, A ( W ) , g k labels and their number emerge from data as latent variables , g k f i

Unsupervised self-assignment flow S 1 - valued data

Outline • set-up: assignment flow 
 supervised labeling • unsupervised labeling 
 - label evolution 
 - label learning from scratch • parameter estimation (control) (submitted) • outlook

Recall: regularisation & control parameters � � D i = d ( f i , g 1 ) , . . . , d ( f i , g m ) metric, distance vector ( data term ) ⇣ X ⌘ w ik Exp � 1 G w i ( W ) = Exp W i W i ( W k ) g m k 2 N i g 3 S i ( W ) = G w � � regularisation L ( W ) , i labels 
 (prototypes) V control f i feature similarity 
 parameters vector g 2 g 1 W i g 1 g 2 � features 
 f i ∈ ( X , d ) metric space scale

� � Motivation all patch-weights (= weight manifold) Ω = { w ik : k 2 N i , i 2 I } 2 P � � uniform weights predicted weights after learning

� � Approach all patch-weights Ω = { w ik : k 2 N i , i 2 I } 2 P � � linear linear assignment flow { 2 N 2 } � � ˙ � � W ( t ) = Exp W 0 V ( t ) , V = Π W 0 S ( W 0 ) + dS W 0 V , V (0) = 0 , ,