On Partial Optimality in Multi-label MRFs P. Kohli 1 A. Shekhovtsov 2 - PowerPoint PPT Presentation

Introduction Our Results Conclusion On Partial Optimality in Multi-label MRFs P. Kohli 1 A. Shekhovtsov 2 C. Rother 1 V. Kolmogorov 3 P. Torr 4 1 Microsoft Research Cambridge 2 Czech Technical University in Prague 3 University College London 4 Oxford Brookes University ICML, 2008 P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Energy Minimization Our Results Work We Build on Conclusion Outline Energy minimization min x E ( x | θ ) (MAP inference in MRF/CRF) E ( x | θ ) = � θ s ( x s ) + � θ st ( x s , x t ) s st � � � � variables x s ∈ L = { 1 . . . K } � � NP-hard in general Consider: conventional linear relaxation relaxation of a binarized problem Goal: study relations P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Energy Minimization Our Results Work We Build on Conclusion Linear Programming Relaxation Approach Relaxation LP-1 E ( x | θ ) = � θ s ( x s ) + � θ st ( x s , x t ) = � θ, µ ( x ) � , s st [ µ ( x )] s ( i ) = δ { x s = i } [ µ ( x )] st ( i , j ) = δ { x s = i } δ { x t = j } x ∈L V � θ, µ ( x ) � min = min � θ, µ � ≥ min � θ, µ � A µ = b A µ = b µ ∈{ 0 , 1 } n µ ∈ [0 , 1] n proposed many times independently [Schlesinger-76, Koster-98, Chekuri-00, Wainwright-03, Cooper-07] large-scale LP problem sub-optimal dual solvers [Koval-76, Wainwright-03, Kolmogorov-05] subgradient dual solvers [Schlesinger & Giginyak- 07, Komodakis et al .-07] P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Energy Minimization Our Results Work We Build on Conclusion Binary Problems L = { 0 , 1 } – pseudo-Boolean optimization [Boros, Hammer, ...] still NP-hard LP-relaxation (roof-dual) can be solved via network flow Can identify assignments which are persistent for all (some) optimal solutions Definition Relation ( e.g . x s = α ) is strongly persistent if it is satisfied for all minimizers x . P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Energy Minimization Our Results Work We Build on Conclusion Reduction to Binary Problem E ( x | θ ) = � θ s ( x s ) + � θ st ( x s , x t ) s st Introduce z ( s , i ) = δ { i ≤ x s } [Ishikawa-03, Kovtun-04, Schlesinger & Flach-06] � � �� �� � � �� �� � � �� �� � � � � P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Energy Minimization Our Results Work We Build on Conclusion Reduction to Binary Problem E ( x | θ ) = � θ s ( x s ) + � θ st ( x s , x t ) s st Introduce z ( s , i ) = δ { i ≤ x s } [Ishikawa-03, Kovtun-04, Schlesinger & Flach-06] � � �� �� � � �� �� � � �� �� � � � � P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Energy Minimization Our Results Work We Build on Conclusion Reduction to Binary Problem E ( x | θ ) = � θ s ( x s ) + � θ st ( x s , x t ) s st Introduce z ( s , i ) = δ { i ≤ x s } [Ishikawa-03, Kovtun-04, Schlesinger & Flach-06] � � �� �� � � �� �� � � � � � � �� �� P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Energy Minimization Our Results Work We Build on Conclusion Reduction to Binary Problem E ( x | θ ) = � θ s ( x s ) + � θ st ( x s , x t ) s st Introduce z ( s , i ) = δ { i ≤ x s } [Ishikawa-03, Kovtun-04, Schlesinger & Flach-06] � � �� �� � � � � � � �� �� � � �� �� P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Energy Minimization Our Results Work We Build on Conclusion Reduction to Binary Problem E ( x | θ ) = � θ s ( x s ) + � θ st ( x s , x t ) s st Introduce z ( s , i ) = δ { i ≤ x s } [Ishikawa-03, Kovtun-04, Schlesinger & Flach-06] � � �� �� � � � � � � �� �� � � �� �� E ( x | θ ) = E ( z | η ) = H ( z ) + � η u z u + � η uv z u z v + η const u uv P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Energy Minimization Our Results Work We Build on Conclusion Reduction to Binary Problem E ( x | θ ) = � θ s ( x s ) + � θ st ( x s , x t ) s st Introduce z ( s , i ) = δ { i ≤ x s } [Ishikawa-03, Kovtun-04, Schlesinger & Flach-06] � � �� �� � � �� �� � � � �� �� � � �� � E ( x | θ ) = E ( z | η ) = H ( z ) + � η u z u + � η uv z u z v + η const u ∈ V uv ∈ A P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Energy Minimization Our Results Work We Build on Conclusion Reduction to Binary Problem E ( x | θ ) = � θ s ( x s ) + � θ st ( x s , x t ) s st Introduce z ( s , i ) = δ { i ≤ x s } [Ishikawa-03, Kovtun-04, Schlesinger & Flach-06] Relaxation LP-2 (roof-dual) Apply conventional LP-relaxation to the binarized problem E ( z | η ) Yields relaxation of the original problem P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Persistencies Our Results Good Subclass Conclusion Equivalent Formulation Persistencies in Multi-Label ���������� Hard constraints imply that non-persistent labels form intervals problem restriction / part of optimal solution P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Persistencies Our Results Good Subclass Conclusion Equivalent Formulation Persistencies in LP-1 � ��� ��� � � ��� � ��� ���������� Theorem We show that persistency derived from LP-2 holds for LP-1 relaxation P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Persistencies Our Results Good Subclass Conclusion Equivalent Formulation Submodular Problems Definition Function f : L V → R is called submodular if ∀ x , y ∈ L V f ( x ∨ y ) + f ( x ∧ y ) ≤ f ( x ) + f ( y ) ( x ∨ y ) s = max( x s , y s ) ( x ∧ y ) s = min( x s , y s ) P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Persistencies Our Results Good Subclass Conclusion Equivalent Formulation Subclass on which LP-2 = LP-1 Consider E ( x | θ ) = � θ s ( x s ) + � θ st ( x s , x t ) s st Theorem If each θ st ( · , · ) is submodular or supermodular, then LP-2 = LP-1 LP-1 for this subclass can be solved using network flow model we have not found applications. P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Persistencies Our Results Good Subclass Conclusion Equivalent Formulation Submodular+Supermodular Decompose E ( x | θ ) = E ( x | θ sub ) + E ( x | θ sup ) 5 5 � � 0 0 −5 −5 x E ( x | θ sub ) + min x E ( x | θ sup ) – (computable LB for min x E ( x | θ ) ≥ min bipartite graphs) Statement Tightest bound = LP-2 c.f . [Wainwright et al .-03] decomposition with trees. P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Experiments Our Results Conclusion Conclusion Experiments Methods: derive restriction intervals [ x min , x max ] on the problem variables s s using network flow model for LP − 2 (MQPBO) some variables get determined exactly – use apply other methods on restricted problem (MQPBO+X) derive more persistent constraints by probing (MQPBO-P) P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Experiments Our Results Conclusion Conclusion Experiments For some instances global minimum can be found Original Noisy Image MQPBO-P (E=65382) BP (E=65424) TRW-S Expansion (E=65398) (E=65386) P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Experiments Our Results Conclusion Conclusion Experiments Random instance: how many variables are determined exactly? 50 × 50 variables, comparison with [Kovtun-03] P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs

Introduction Experiments Our Results Conclusion Conclusion Experiments Real Instance: combined methods Object segmentation and recognition model [Shotton et al .-05] P. Kohli, A. Shekhovtsov, C. Rother, V. Kolmogorov, P. Torr On Partial Optimality in Multi-label MRFs