Fast Semi-differential based Submodular Function Optimization - - PowerPoint PPT Presentation
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion Fast Semi-differential based Submodular Function Optimization Rishabh Iyer 1 Stefanie Jegelka 2 Jeff Bilmes 1 1 University of Washington, Seattle 2
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Fast Semi-differential based Submodular Function Optimization
Rishabh Iyer 1 Stefanie Jegelka 2 Jeff Bilmes 1
1University of Washington, Seattle 2University of California, Berkeley
ICML-2013
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 1 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Outline
1
Submodular Functions in Machine Learning
2
Convexity, Concavity & Submodular Semigradient Descent
3
Submodular Minimization
4
Submodular Maximization
5
Conclusion
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 2 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Set functions f : 2V → R
V is a finite “ground” set of objects.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 3 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Set functions f : 2V → R
A set function f : 2V → R produces a value for any subset A ⊆ V .
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 3 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Set functions f : 2V → R
A set function f : 2V → R produces a value for any subset A ⊆ V . For example, f (A) = 22,
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 3 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Set functions f : 2V → R
A set function f : 2V → R produces a value for any subset A ⊆ V . For example, f (A) = 22, General set function optimization can be really hard!
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 3 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Set Functions
Special class of set functions. f (A ∪ v) − f (A) ≥ f (B ∪ v) − f (B), if A ⊆ B (1)
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 4 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Set Functions
Special class of set functions. f (A ∪ v) − f (A) ≥ f (B ∪ v) − f (B), if A ⊆ B (1)
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 4 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Set Functions
Special class of set functions. f (A ∪ v) − f (A) ≥ f (B ∪ v) − f (B), if A ⊆ B (1) Gain = 1
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 4 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Set Functions
Special class of set functions. f (A ∪ v) − f (A) ≥ f (B ∪ v) − f (B), if A ⊆ B (1) Gain = 1 Gain = 0
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 4 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Set Functions
Special class of set functions. f (A ∪ v) − f (A) ≥ f (B ∪ v) − f (B), if A ⊆ B (1) Gain = 1 Gain = 0 Monotonicity: f (A) ≤ f (B), if A ⊆ B.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 4 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Set Functions
Special class of set functions. f (A ∪ v) − f (A) ≥ f (B ∪ v) − f (B), if A ⊆ B (1) Gain = 1 Gain = 0 Monotonicity: f (A) ≤ f (B), if A ⊆ B. Modular function f (X) =
i∈X f (i) analogous to linear functions.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 4 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Function Maximization
compute A∗ ∈ argmax
A∈C
f (A) where f is submodular, and where C is constraint set over which a modular function can be optimized efficiently.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 5 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Function Maximization
compute A∗ ∈ argmax
A∈C
f (A) where f is submodular, and where C is constraint set over which a modular function can be optimized efficiently.
Sensor Placement (Krause et al, 2008)
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 5 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Function Maximization
compute A∗ ∈ argmax
A∈C
f (A) where f is submodular, and where C is constraint set over which a modular function can be optimized efficiently.
Sensor Placement (Krause et al, 2008) Document Summarization (Lin & Bilmes, 2011)
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 5 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Function Maximization
compute A∗ ∈ argmax
A∈C
f (A) where f is submodular, and where C is constraint set over which a modular function can be optimized efficiently.
Sensor Placement (Krause et al, 2008) Document Summarization (Lin & Bilmes, 2011) Diversified Search (He et al 2012, Kulesza & Taskar, 2012)
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 5 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Function Minimization
compute A∗ ∈ argmin
A∈C
f (A) where f is submodular, and where C is constraint set over which a modular function can be optimized efficiently.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 6 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Function Minimization
compute A∗ ∈ argmin
A∈C
f (A) where f is submodular, and where C is constraint set over which a modular function can be optimized efficiently.
Image segmentation / MAP inference (Boykov & Jolly 2001, Jegelka & Bilmes 2011, Delong et al, 2012)
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 6 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Function Minimization
compute A∗ ∈ argmin
A∈C
f (A) where f is submodular, and where C is constraint set over which a modular function can be optimized efficiently.
Image segmentation / MAP inference (Boykov & Jolly 2001, Jegelka & Bilmes 2011, Delong et al, 2012) Clustering (Narasimhan & Bilmes 2011, Nagano et al, 2010)
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 6 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Function Minimization
compute A∗ ∈ argmin
A∈C
f (A) where f is submodular, and where C is constraint set over which a modular function can be optimized efficiently.
Image segmentation / MAP inference (Boykov & Jolly 2001, Jegelka & Bilmes 2011, Delong et al, 2012) Clustering (Narasimhan & Bilmes 2011, Nagano et al, 2010) Corpus Data Subset Selection (Lin & Bilmes, 2011)
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 6 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Current State of Affairs for Submodular Optimization
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 7 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Current State of Affairs for Submodular Optimization
Submodular Function Minimization Polynomial-time but too slow O(n5 × FuncEvalCost + n6). Constrained minimization is NP-hard. Algorithms differ depending
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 7 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Current State of Affairs for Submodular Optimization
Submodular Function Minimization Polynomial-time but too slow O(n5 × FuncEvalCost + n6). Constrained minimization is NP-hard. Algorithms differ depending
Submodular Function Maximization NP-hard but constant-factor approximable. Large class of algorithms – Local search, continuous greedy, bi-directional greedy, simulated annealing etc.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 7 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Current State of Affairs for Submodular Optimization
Submodular Function Minimization Polynomial-time but too slow O(n5 × FuncEvalCost + n6). Constrained minimization is NP-hard. Algorithms differ depending
Submodular Function Maximization NP-hard but constant-factor approximable. Large class of algorithms – Local search, continuous greedy, bi-directional greedy, simulated annealing etc. Algorithms look very different!
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 7 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Current State of Affairs for Submodular Optimization
Submodular Function Minimization Polynomial-time but too slow O(n5 × FuncEvalCost + n6). Constrained minimization is NP-hard. Algorithms differ depending
Submodular Function Maximization NP-hard but constant-factor approximable. Large class of algorithms – Local search, continuous greedy, bi-directional greedy, simulated annealing etc. Algorithms look very different! Which algorithm to use when?
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 7 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Current State of Affairs for Submodular Optimization
Submodular Function Minimization Polynomial-time but too slow O(n5 × FuncEvalCost + n6). Constrained minimization is NP-hard. Algorithms differ depending
Submodular Function Maximization NP-hard but constant-factor approximable. Large class of algorithms – Local search, continuous greedy, bi-directional greedy, simulated annealing etc. Algorithms look very different! Which algorithm to use when? Contribution: We present the first unifying framework for submodular min- imization & maximization. Our framework is scalable to large data.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 7 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Convex/Concave and Semigradients
A convex function φ has a subgradient hy and linear lower bound: φ(y) + hy, x − y ≤ φ(x), ∀x. A concave function ψ has a supergradient gy and linear upper bound: ψ(y) + gy, x − y ≥ ψ(x), ∀x.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 8 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Convex/Concave and Semigradients
A convex function φ has a subgradient hy and linear lower bound: φ(y) + hy, x − y ≤ φ(x), ∀x. A concave function ψ has a supergradient gy and linear upper bound: ψ(y) + gy, x − y ≥ ψ(x), ∀x. Submodular functions have properties analogous to convexity and concavity.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 8 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Subgradients (Fujishige 1984, 2005)
Like convex functions, submodular functions have sub-gradients. Defined at any Y ⊆ V .
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 9 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Subgradients (Fujishige 1984, 2005)
Like convex functions, submodular functions have sub-gradients. Defined at any Y ⊆ V . Permutation σ of the ground set.
Y
σ(1) σ(2) σ(3) σ(4) σ(5) σ(6) σ(7) σ(8)
Σ1 Σ2 Σ3
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 9 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Subgradients (Fujishige 1984, 2005)
Like convex functions, submodular functions have sub-gradients. Defined at any Y ⊆ V . Permutation σ of the ground set.
Y
σ(1) σ(2) σ(3) σ(4) σ(5) σ(6) σ(7) σ(8)
Σ1 Σ2 Σ3
Corresponding subgradient hσ
Y is:
hσ
Y (σ(i)) = f (Σi) − f (Σi−1)
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 9 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Subgradients (Fujishige 1984, 2005)
Like convex functions, submodular functions have sub-gradients. Defined at any Y ⊆ V . Permutation σ of the ground set.
Y
σ(1) σ(2) σ(3) σ(4) σ(5) σ(6) σ(7) σ(8)
Σ1 Σ2 Σ3
Corresponding subgradient hσ
Y is:
hσ
Y (σ(i)) = f (Σi) − f (Σi−1)
Modular lower bound: mhY (X) = f (Y ) + hY (X) − hY (Y ) ≤ f (X).
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 9 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Supergradients (Iyer et al, 2013)
Define gain of j in context of A: f (j|A) f (A ∪ j) − f (A)
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 10 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Supergradients (Iyer et al, 2013)
Define gain of j in context of A: f (j|A) f (A ∪ j) − f (A) Unlike convex functions, surprisingly, we show that submodular functions also have super-gradients. Defined at any Y ⊆ V .
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 10 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Supergradients (Iyer et al, 2013)
Define gain of j in context of A: f (j|A) f (A ∪ j) − f (A) Unlike convex functions, surprisingly, we show that submodular functions also have super-gradients. Defined at any Y ⊆ V . Three of these supergradients (which we call grow, shrink, and bar) are in fact easy to obtain.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 10 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Supergradients (Iyer et al, 2013)
Define gain of j in context of A: f (j|A) f (A ∪ j) − f (A) Unlike convex functions, surprisingly, we show that submodular functions also have super-gradients. Defined at any Y ⊆ V . Three of these supergradients (which we call grow, shrink, and bar) are in fact easy to obtain. Grow: ˆ gY (j) =
for j / ∈ Y f (j|V \{j}) for j ∈ Y
X Y V f(j|Y ) f(j|V \ j)
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 10 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Supergradients (Iyer et al, 2013)
Define gain of j in context of A: f (j|A) f (A ∪ j) − f (A) Unlike convex functions, surprisingly, we show that submodular functions also have super-gradients. Defined at any Y ⊆ V . Three of these supergradients (which we call grow, shrink, and bar) are in fact easy to obtain. Shrink: ˇ gY (j) =
for j / ∈ Y f (j|Y \{j}) for j ∈ Y
X Y V f(j|Y \ j) f(j|∅)
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 10 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Supergradients (Iyer et al, 2013)
Define gain of j in context of A: f (j|A) f (A ∪ j) − f (A) Unlike convex functions, surprisingly, we show that submodular functions also have super-gradients. Defined at any Y ⊆ V . Three of these supergradients (which we call grow, shrink, and bar) are in fact easy to obtain. Bar: ¯ gY (j) =
for j / ∈ Y f (j|V \{j}) for j ∈ Y
X Y V f(j|V \ j) f(j|∅)
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 10 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Submodular Supergradients (Iyer et al, 2013)
Define gain of j in context of A: f (j|A) f (A ∪ j) − f (A) Unlike convex functions, surprisingly, we show that submodular functions also have super-gradients. Defined at any Y ⊆ V . Three of these supergradients (which we call grow, shrink, and bar) are in fact easy to obtain. Bar: ¯ gY (j) =
for j / ∈ Y f (j|V \{j}) for j ∈ Y
X Y V f(j|V \ j) f(j|∅)
Modular upper bound: mgY (X) = f (Y ) + gY (X) − gY (Y ) ≤ f (X).
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 10 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Optimization Framework
Algorithm 1 Subgradient ascent [descent] algorithm for submodular max- imization [minimization].
1: Start with an arbitrary X 0.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 11 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Optimization Framework
Algorithm 1 Subgradient ascent [descent] algorithm for submodular max- imization [minimization].
1: Start with an arbitrary X 0. 2: repeat 6: until we have converged (X i−1 = X i) or i ≤ T
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 11 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Optimization Framework
Algorithm 1 Subgradient ascent [descent] algorithm for submodular max- imization [minimization].
1: Start with an arbitrary X 0. 2: repeat 3:
Pick a semigradient hX t [ gX t] at X t.
6: until we have converged (X i−1 = X i) or i ≤ T
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 11 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Optimization Framework
Algorithm 1 Subgradient ascent [descent] algorithm for submodular max- imization [minimization].
1: Start with an arbitrary X 0. 2: repeat 3:
Pick a semigradient hX t [ gX t] at X t.
4:
X t+1 ← argmaxX∈C mhXt (X) [ X t+1 ← argminX∈C mgXt (X)]
5:
t ← t + 1
6: until we have converged (X i−1 = X i) or i ≤ T
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 11 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Optimization Framework
Algorithm 1 Subgradient ascent [descent] algorithm for submodular max- imization [minimization].
1: Start with an arbitrary X 0. 2: repeat 3:
Pick a semigradient hX t [ gX t] at X t.
4:
X t+1 ← argmaxX∈C mhXt (X) [ X t+1 ← argminX∈C mgXt (X)]
5:
t ← t + 1
6: until we have converged (X i−1 = X i) or i ≤ T
Lemma: Algorithm 1 monotonically improves the objective function value for submodular maximization and minimization at every iteration.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 11 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Unconstrained Minimization
MMin-IIIa MMin-IIIb MMin-I MMin-II g ¯ g ¯ g ˆ g ˇ g X 0 ∅ V ∅ V X c A B A+ B+ MMin-IIIa and IIIb are first iterations of MMin-I and MMin-II.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 12 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Unconstrained Minimization
MMin-IIIa MMin-IIIb MMin-I MMin-II g ¯ g ¯ g ˆ g ˇ g X 0 ∅ V ∅ V X c A B A+ B+ MMin-IIIa and IIIb are first iterations of MMin-I and MMin-II. A and B obtainable in O(n) oracle calls.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 12 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Unconstrained Minimization
MMin-IIIa MMin-IIIb MMin-I MMin-II g ¯ g ¯ g ˆ g ˇ g X 0 ∅ V ∅ V X c A B A+ B+ MMin-IIIa and IIIb are first iterations of MMin-I and MMin-II. A and B obtainable in O(n) oracle calls. A+ and B+ are local minimizers obtainable in O(n2) calls.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 12 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Unconstrained Minimization
MMin-IIIa MMin-IIIb MMin-I MMin-II g ¯ g ¯ g ˆ g ˇ g X 0 ∅ V ∅ V X c A B A+ B+ MMin-IIIa and IIIb are first iterations of MMin-I and MMin-II. A and B obtainable in O(n) oracle calls. A+ and B+ are local minimizers obtainable in O(n2) calls. A ⊆ A+ ⊆ X ∗ ⊆ B+ ⊆ B
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 12 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Illustrating Unconstrained Minimization
V
MMin-I
V
MMin-II
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 13 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Illustrating Unconstrained Minimization
V A
MMin-I
V B
MMin-II
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 13 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Illustrating Unconstrained Minimization
V A
MMin-I
V B
MMin-II
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 13 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Illustrating Unconstrained Minimization
V A+ A
MMin-I
V B B+
MMin-II
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 13 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Illustrating Unconstrained Minimization
V B B+ A+ A X*
MMin-I
V B B+ A+ A X*
MMin-II
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 13 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Empirical Results: Submodular Minimization
Test function: concave over modular,
2 4 50 100
% Contraction/Relative Time
λ
MMin−I & II MMin−III
Lattice reduction (solid line), and runtime reduction (dotted line). Note: results for Bipartite Neighborhoods shown in paper.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 14 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Constrained Submodular Minimization
Curvature of a monotone submodular function: κf (X) 1 − min
j
f (j|X\j) f (j) . (2)
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 15 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Constrained Submodular Minimization
Curvature of a monotone submodular function: κf (X) 1 − min
j
f (j|X\j) f (j) . (2) Theorem The solution X returned by MMin-I satisfies: f ( X) ≤ |X ∗| 1 + (|X ∗| − 1)(1 − κf (X ∗))f (X ∗) ≤ 1 1 − κf (X ∗)f (X ∗)
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 15 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Constrained Submodular Minimization
Curvature of a monotone submodular function: κf (X) 1 − min
j
f (j|X\j) f (j) . (2) Theorem The solution X returned by MMin-I satisfies: f ( X) ≤ |X ∗| 1 + (|X ∗| − 1)(1 − κf (X ∗))f (X ∗) ≤ 1 1 − κf (X ∗)f (X ∗) Lower curvature ⇒ Better guarantees!
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 15 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Constrained Submodular Minimization
Curvature of a monotone submodular function: κf (X) 1 − min
j
f (j|X\j) f (j) . (2) Theorem The solution X returned by MMin-I satisfies: f ( X) ≤ |X ∗| 1 + (|X ∗| − 1)(1 − κf (X ∗))f (X ∗) ≤ 1 1 − κf (X ∗)f (X ∗) Lower curvature ⇒ Better guarantees! Improve the previous results when κf < 1.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 15 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Empirical Results: Constrained Submodular Minimization
CM CCM BS WC 1 2 3 Bipartite Matching
We compare MMin-I to two other algorithms.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 16 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Empirical Results: Constrained Submodular Minimization
CM CCM BS WC 1 2 3 Bipartite Matching
We compare MMin-I to two other algorithms.
1
Simple modular upper bound (MU) (i.e
j∈X f (j)).
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 16 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Empirical Results: Constrained Submodular Minimization
CM CCM BS WC 1 2 3 Bipartite Matching
We compare MMin-I to two other algorithms.
1
Simple modular upper bound (MU) (i.e
j∈X f (j)).
2
More complicated Ellipsoidal Approximation (EA) Algorithm.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 16 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Empirical Results: Constrained Submodular Minimization
CM CCM BS WC 1 2 3 Bipartite Matching
We compare MMin-I to two other algorithms.
1
Simple modular upper bound (MU) (i.e
j∈X f (j)).
2
More complicated Ellipsoidal Approximation (EA) Algorithm.
Performance of MMin-I:
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 16 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Empirical Results: Constrained Submodular Minimization
CM CCM BS WC 1 2 3 Bipartite Matching
We compare MMin-I to two other algorithms.
1
Simple modular upper bound (MU) (i.e
j∈X f (j)).
2
More complicated Ellipsoidal Approximation (EA) Algorithm.
Performance of MMin-I:
1
Much better than MU.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 16 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Empirical Results: Constrained Submodular Minimization
CM CCM BS WC 1 2 3 Bipartite Matching
We compare MMin-I to two other algorithms.
1
Simple modular upper bound (MU) (i.e
j∈X f (j)).
2
More complicated Ellipsoidal Approximation (EA) Algorithm.
Performance of MMin-I:
1
Much better than MU.
2
Comparable to EA.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 16 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Empirical Results: Constrained Submodular Minimization
CM CCM BS WC 1 2 3 Bipartite Matching
We compare MMin-I to two other algorithms.
1
Simple modular upper bound (MU) (i.e
j∈X f (j)).
2
More complicated Ellipsoidal Approximation (EA) Algorithm.
Performance of MMin-I:
1
Much better than MU.
2
Comparable to EA.
Submodular spanning tree & shortest path results given in paper.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 16 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Unconstrained Maximization
Our framework subsumes a number of state-of-the-art algorithms. For example, each of the below corresponds to subgradient ascent:
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 17 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Unconstrained Maximization
Our framework subsumes a number of state-of-the-art algorithms. For example, each of the below corresponds to subgradient ascent: Random Subgradient (RA/ RP): Random subgradients (permutations) at every iteration.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 17 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Unconstrained Maximization
Our framework subsumes a number of state-of-the-art algorithms. For example, each of the below corresponds to subgradient ascent: Random Subgradient (RA/ RP): Random subgradients (permutations) at every iteration. 1/4 Approximation in Expectation!
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 17 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Unconstrained Maximization
Our framework subsumes a number of state-of-the-art algorithms. For example, each of the below corresponds to subgradient ascent: Random Subgradient (RA/ RP): Random subgradients (permutations) at every iteration. 1/4 Approximation in Expectation! Randomized / Deterministic local search (RLS/DLS): Local search based techniques naturally define subgradients.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 17 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Unconstrained Maximization
Our framework subsumes a number of state-of-the-art algorithms. For example, each of the below corresponds to subgradient ascent: Random Subgradient (RA/ RP): Random subgradients (permutations) at every iteration. 1/4 Approximation in Expectation! Randomized / Deterministic local search (RLS/DLS): Local search based techniques naturally define subgradients. 1/3 Approximation (FMV’07)!
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 17 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Unconstrained Maximization
Our framework subsumes a number of state-of-the-art algorithms. For example, each of the below corresponds to subgradient ascent: Random Subgradient (RA/ RP): Random subgradients (permutations) at every iteration. 1/4 Approximation in Expectation! Randomized / Deterministic local search (RLS/DLS): Local search based techniques naturally define subgradients. 1/3 Approximation (FMV’07)! Bi-directional Greedy (BG): Bi-directional Greedy Subgradient (Buchbinder et al, 2012).
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 17 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Unconstrained Maximization
Our framework subsumes a number of state-of-the-art algorithms. For example, each of the below corresponds to subgradient ascent: Random Subgradient (RA/ RP): Random subgradients (permutations) at every iteration. 1/4 Approximation in Expectation! Randomized / Deterministic local search (RLS/DLS): Local search based techniques naturally define subgradients. 1/3 Approximation (FMV’07)! Bi-directional Greedy (BG): Bi-directional Greedy Subgradient (Buchbinder et al, 2012). 1/3 Approximation (BFNS’12)!
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 17 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Unconstrained Maximization
Our framework subsumes a number of state-of-the-art algorithms. For example, each of the below corresponds to subgradient ascent: Random Subgradient (RA/ RP): Random subgradients (permutations) at every iteration. 1/4 Approximation in Expectation! Randomized / Deterministic local search (RLS/DLS): Local search based techniques naturally define subgradients. 1/3 Approximation (FMV’07)! Bi-directional Greedy (BG): Bi-directional Greedy Subgradient (Buchbinder et al, 2012). 1/3 Approximation (BFNS’12)! Randomized Greedy (RG): Randomized variant of BG.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 17 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Unconstrained Maximization
Our framework subsumes a number of state-of-the-art algorithms. For example, each of the below corresponds to subgradient ascent: Random Subgradient (RA/ RP): Random subgradients (permutations) at every iteration. 1/4 Approximation in Expectation! Randomized / Deterministic local search (RLS/DLS): Local search based techniques naturally define subgradients. 1/3 Approximation (FMV’07)! Bi-directional Greedy (BG): Bi-directional Greedy Subgradient (Buchbinder et al, 2012). 1/3 Approximation (BFNS’12)! Randomized Greedy (RG): Randomized variant of BG. 1/2 Approximation in Expectation! (BFNS’12)!
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 17 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Constrained Maximization and Extensions
Y
σ(1) σ(2) σ(3) σ(4) σ(5) σ(6) σ(7) σ(8)
Σ1 Σ2 Σ3
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 18 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Constrained Maximization and Extensions
Y
σ(1) σ(2) σ(3) σ(4) σ(5) σ(6) σ(7) σ(8)
Σ1 Σ2 Σ3
Greedy subgradient for monotone submodular functions: σg(i) ∈ argmax
j / ∈Σσg
i−1 and Σσg i−1∪{j}∈C
f (j|Σσg
i−1).
(3)
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 18 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Constrained Maximization and Extensions
Y
σ(1) σ(2) σ(3) σ(4) σ(5) σ(6) σ(7) σ(8)
Σ1 Σ2 Σ3
Greedy subgradient for monotone submodular functions: σg(i) ∈ argmax
j / ∈Σσg
i−1 and Σσg i−1∪{j}∈C
f (j|Σσg
i−1).
(3) Algorithm 1 using the subgradient hσg exactly corresponds to the greedy algorithm.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 18 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Constrained Maximization and Extensions
Y
σ(1) σ(2) σ(3) σ(4) σ(5) σ(6) σ(7) σ(8)
Σ1 Σ2 Σ3
Greedy subgradient for monotone submodular functions: σg(i) ∈ argmax
j / ∈Σσg
i−1 and Σσg i−1∪{j}∈C
f (j|Σσg
i−1).
(3) Algorithm 1 using the subgradient hσg exactly corresponds to the greedy algorithm. ⇒ 1 − 1/e Approximation (NWF’78)!
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 18 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Constrained Maximization and Extensions
Y
σ(1) σ(2) σ(3) σ(4) σ(5) σ(6) σ(7) σ(8)
Σ1 Σ2 Σ3
Greedy subgradient for monotone submodular functions: σg(i) ∈ argmax
j / ∈Σσg
i−1 and Σσg i−1∪{j}∈C
f (j|Σσg
i−1).
(3) Algorithm 1 using the subgradient hσg exactly corresponds to the greedy algorithm. ⇒ 1 − 1/e Approximation (NWF’78)! Generality of Algorithm MMax: For every α-approximation algorithm, there exists a schedule of subgradients obtainable in poly-time, such that Algorithm 1 (MMax) achieves an approximation factor of at least α.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 18 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Summary
Submodular functions in machine learning. A generic sub-gradient ascent [super-gradient descent] framework for submodular maximization [minimization]. The first unifying framework for general submodular optimization. New theoretical results for unconstrained and constrained submodular minimization. A novel view as a framework for submodular maximization and subsuming number of existing algorithms. Empirical experimental validation.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 19 / 20
Background Submodular Semigradients Submodular Minimization Submodular Maximization Conclusion
Thank You
Thank You! Questions please.
Iyer et al, 2013 Fast Semi-differential based Submodular Function Optimization page 20 / 20