G: Worst-case Complexity G: Integrality/Relaxations Determinism End Part 7: Structured Prediction and Energy Minimization (2/2) Sebastian Nowozin and Christoph H. Lampert Colorado Springs, 25th June 2011 Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Hard problem Worst-case Generality Optimality Integrality Determinism complexity Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Giving up Worst-case Complexity ◮ Worst-case complexity is an asymptotic behaviour ◮ Worst-case complexity quantifies worst case ◮ Practical case might be very different ◮ Issue: what is the distribution over inputs? Popular methods with bad or unknown worst-case complexity ◮ Simplex Method for Linear Programming ◮ Hash tables ◮ Branch-and-bound search Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Branch-and-bound Search ◮ Implicit enumeration : globally optimal ◮ Choose 1. Partitioning of solution space 2. Branching scheme 3. Upper and lower bounds over partitions Branch and bound ◮ is very flexible, many tuning possibilities in partitioning, branching schemes and bounding functions, ◮ can be very efficient in practise, ◮ typically has worst-case complexity equal to exhaustive enumeration Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Branch-and-Bound (cont) A 4 A 5 C 2 A 2 C 1 A 3 A 1 Y C 3 Work with partitioning of solution space Y ◮ Active nodes (white) ◮ Closed nodes (gray) Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Branch-and-Bound (cont) C A Y Y ◮ Initially: everything active ◮ Goal: everything closed Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Branch-and-Bound (cont) A 4 A 5 C 2 A 2 C 1 A 3 A 1 Y C 3 Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Branch-and-Bound (cont) A 2 ◮ Take an active element ( A 2 ) Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Branch-and-Bound (cont) A 2 ◮ Partition into two or more subsets of Y Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Branch-and-Bound (cont) C 4 A 6 C 5 ◮ Evaluate bounds and set node active or closed ◮ Closing possible if we can prove that no solution in a partition can be better than a known solution of value L ◮ g ( A ) ≤ L → close A Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Example 1: Efficient Subwindow Search Efficient Subwindow Search (Lampert and Blaschko, 2008) Find the bounding box that maximizes a linear scoring function y ∗ = argmax � w , φ ( y , x ) � y ∈Y Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Example 1: Efficient Subwindow Search Efficient Subwindow Search (Lampert and Blaschko, 2008) � g ( x , y ) = β + w ( x i ) x i within y Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Example 1: Efficient Subwindow Search Efficient Subwindow Search (Lampert and Blaschko, 2008) Subsets B of bounding boxes specified by interval coordinates, B ⊂ 2 Y Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Example 1: Efficient Subwindow Search Efficient Subwindow Search (Lampert and Blaschko, 2008) Upper bound: g ( x , B ) ≥ g ( x , y ) for all y ∈ B � � g ( x , B ) = β + max { w ( x i ) , 0 } + min { 0 , w ( x i ) } x i x i within within B max B min ≥ max y ∈ B g ( x , y ) Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Example 1: Efficient Subwindow Search Efficient Subwindow Search (Lampert and Blaschko, 2008) PASCAL VOC 2007 detection challenge bounding boxes found using ESS Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Example 2: Branch-and-Mincut Branch-and-Mincut (Lempitsky et al., 2008) Binary image segmentation with non-local interaction y 1 , y 2 ∈ Y Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Example 2: Branch-and-Mincut Branch-and-Mincut (Lempitsky et al., 2008) Binary image segmentation with non-local interaction � � � F p ( y ) z p + B p ( y )(1 − z p )+ P pq ( y ) | z p − z q | , E ( z , y ) = C ( y )+ p ∈ V p ∈ V { i , j }∈E g ( x , y ) = max z ∈ 2 V − E ( z , y ) ◮ Here: z ∈ { 0 , 1 } V is a binary pixel mask ◮ F p ( y ), B p ( y ) are foreground/background unary energies ◮ P pq ( y ) is a standard pairwise energy ◮ Global dependencies on y Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Worst-case Complexity Example 2: Branch-and-Mincut Branch-and-Mincut (Lempitsky et al., 2008) Upper bound for any subset A ⊆ Y : max y ∈ A g ( x , y ) = max y ∈ A max z ∈ 2 V − E ( z , y ) 2 X X F p ( y ) z p + B p ( y )(1 − z p ) = max y ∈ A max z ∈ 2 V − 4 C ( y ) + p ∈ V p ∈ V 3 X P pq ( y ) | z p − z q | + 5 { i , j }∈E 2 „ « „ « X y ∈ A − F p ( y ) ≤ max max y ∈ A − C ( y ) + max z p 4 z ∈ 2 V p ∈ V 3 „ « „ « X X y ∈ A − B p ( y ) y ∈ A − P pq ( y ) 5 . + max (1 − z p ) + max | z p − z q | p ∈ V { i , j }∈E Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Integrality/Relaxations Hard problem Worst-case Generality Optimality Integrality Determinism complexity Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Integrality/Relaxations Problem Relaxations ◮ Optimization problems (minimizing g : G → R over Y ⊆ G ) can become easier if ◮ feasible set is enlarged, and/or ◮ objective function is replaced with a bound. g ( x ) , h ( x ) h ( z ) Y g ( y ) Z ⊇ Y y, z Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
G: Worst-case Complexity G: Integrality/Relaxations Determinism End G: Integrality/Relaxations Problem Relaxations ◮ Optimization problems (minimizing g : G → R over Y ⊆ G ) can become easier if ◮ feasible set is enlarged, and/or ◮ objective function is replaced with a bound. Definition (Relaxation (Geoffrion, 1974)) Given two optimization problems ( g , Y , G ) and ( h , Z , G ), the problem ( h , Z , G ) is said to be a relaxation of ( g , Y , G ) if, 1. Z ⊇ Y , i.e. the feasible set of the relaxation contains the feasible set of the original problem, and 2. ∀ y ∈ Y : h ( y ) ≥ g ( y ), i.e. over the original feasible set the objective function h achieves no smaller values than the objective function g . Sebastian Nowozin and Christoph H. Lampert Part 7: Structured Prediction and Energy Minimization (2/2)
Recommend
More recommend
