[PPT] - + + Concave Aspects of Submodular Functions International PowerPoint Presentation

SLIDE 1

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

+ + ≥ Concave Aspects of Submodular Functions

International Symposium on Information Theory 2020 Rishabh Iyer1 and Jeffrey A. Bilmes2

1 Department of CS, University of Texas at Dallas 2 Department of ECE, University of Washington

21st - 26th June, 2020

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 1 / 31

SLIDE 2

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Outline

1

Introduction

2

Two sides of Submodularity

3

Submodular Polyhedra

4

Submodular Sub-differentials and Super-differentials

5

Other Convex/Concave Aspects

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 2 / 31

SLIDE 3

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Outline

1

Introduction

2

Two sides of Submodularity

3

Submodular Polyhedra

4

Submodular Sub-differentials and Super-differentials

5

Other Convex/Concave Aspects

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 3 / 31

SLIDE 4

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Functions

Special class of set functions. f (A ∪ v) − f (A) ≥ f (B ∪ v) − f (B), if A ⊆ B (1) f = # of distinct colors of balls in the urn. Gain = 1 Gain = 0

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 4 / 31

SLIDE 5

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Alternate definition – Submodular Functions

A function f : 2V → R is submodular if:

+ +

+

f(A) f(B) f(A ∪ B)

= f(Ar) +f(C) + f(Br)

≥ ≥

= f(A ∩ B)

f(A ∩ B)

= f(Ar) + 2f(C) + f(Br)

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 5 / 31

SLIDE 6

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Alternate definition – Submodular Functions

A function f : 2V → R is submodular if:

+ +

+

f(A) f(B) f(A ∪ B)

= f(Ar) +f(C) + f(Br)

≥ ≥

= f(A ∩ B)

f(A ∩ B)

= f(Ar) + 2f(C) + f(Br)

Submodularity has been widely used: non-additive measure theory, economics, game theory, statistical physics and thermodynamics, electrical networks, operations research, information theory and machine learning.

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 5 / 31

SLIDE 7

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodularity and Information Theory

Given a set of random variables, X1, · · · , Xn, the Entropy of a Set

f random variables: H(XA) is a submodular function.
R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 6 / 31

SLIDE 8

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodularity and Information Theory

Given a set of random variables, X1, · · · , Xn, the Entropy of a Set

f random variables: H(XA) is a submodular function.

Similarly, the Mutual Information between a set of variables and its complement: I(XA; XV \A) is symmetric submodular.

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 6 / 31

SLIDE 9

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodularity and Information Theory

Given a set of random variables, X1, · · · , Xn, the Entropy of a Set

f random variables: H(XA) is a submodular function.

Similarly, the Mutual Information between a set of variables and its complement: I(XA; XV \A) is symmetric submodular. Mutual Information between a set of features XA and a class C: I(XA; C) is submodular for certain models (e.g. Naive Bayes)

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 6 / 31

SLIDE 10

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodularity and Information Theory

Given a set of random variables, X1, · · · , Xn, the Entropy of a Set

f random variables: H(XA) is a submodular function.

Similarly, the Mutual Information between a set of variables and its complement: I(XA; XV \A) is symmetric submodular. Mutual Information between a set of features XA and a class C: I(XA; C) is submodular for certain models (e.g. Naive Bayes) Many Mutual Information based Feature Selection algorithms are submodular!

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 6 / 31

SLIDE 11

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodularity and Information Theory

Given a set of random variables, X1, · · · , Xn, the Entropy of a Set

f random variables: H(XA) is a submodular function.

Similarly, the Mutual Information between a set of variables and its complement: I(XA; XV \A) is symmetric submodular. Mutual Information between a set of features XA and a class C: I(XA; C) is submodular for certain models (e.g. Naive Bayes) Many Mutual Information based Feature Selection algorithms are submodular! Submodular Functions strictly generalize Entropy as Combinatorial Information Functions and are becoming increasingly applicable in machine learning applications.

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 6 / 31

SLIDE 12

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Facets of Submodularity

Coverage ¡ Informa2on ¡ ¡

X1 Y X2 X4 X3

Diversity ¡

F(A) = H(XA)

F(A) = log det(LA)

F(A) = ∪s∈Aarea(s)

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 7 / 31

SLIDE 13

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Outline

1

Introduction

2

Two sides of Submodularity

3

Submodular Polyhedra

4

Submodular Sub-differentials and Super-differentials

5

Other Convex/Concave Aspects

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 8 / 31

SLIDE 14

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Convex Functions and Tight Subgradients

b fb(b) = f(b) fb(a) ≤ f(a) x f(x) fb(x)

A convex function f has a subgradient at any in-domain point b, namely there exists fb such that f (x) − f (b) ≥ fb, x − b, ∀x. (2) we have fb(x) = f (b) + fb, x − b We have that f (x) is convex, fb(x) is affine, and can be a tight subgradient (tight at b, affine lower bound on f (x)) for all b

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 9 / 31

SLIDE 15

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Concave Functions and Tight Supergradients

b fb(b) = f(b) fb(a) ≥ f(a) x f(x) fb(x)

A concave f has a supergradient at any in-domain point b, namely there exists f b such that f (x) − f (b) ≤ f b, x − b, ∀x. (3) we have f b(x) = f (b) + f b, x − b We have that f (x) is concave, f b(x) is affine, and can be a tight supergradient (tight at b, affine upper bound on f (x)) for all b

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 10 / 31

SLIDE 16

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Two Sides of Submodularity

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 11 / 31

SLIDE 17

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Two sides of Submodularity

Convex aspects (Fujishige (1984, 2005), Frank (1982)) Minimization: Poly-time. Convex continuous extension - Lov´ asz extension. Subgradients and Subdifferential. Convex duality, discrete seperation etc. Optimality Conditions for Submodular Min Concave aspects (Vondrak (2007), This Work) Max: constant-factor approx! Multilinear extension - concave in a direction. Supergradients and Superdifferential. Under restricted settings, duality, separation etc. Optimality Conditions for Submodular Max

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 12 / 31

SLIDE 18

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodularity, Convexity and Concavity

Many interesting polyhedral and algorithmic connections between convexity and submodularity well known over the years.

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 13 / 31

SLIDE 19

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodularity, Convexity and Concavity

Many interesting polyhedral and algorithmic connections between convexity and submodularity well known over the years. There are well-known concave aspects of submodular functions as well: The definition ∇j∇kf (X) ≤ 0 where ∇jf (X) = f (j|X), concave over modular is submodular, efficient approximate maximization, 1 − 1/e or 1/2 for many problems (Vondr´ ak and many others).

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 13 / 31

SLIDE 20

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodularity, Convexity and Concavity

Many interesting polyhedral and algorithmic connections between convexity and submodularity well known over the years. There are well-known concave aspects of submodular functions as well: The definition ∇j∇kf (X) ≤ 0 where ∇jf (X) = f (j|X), concave over modular is submodular, efficient approximate maximization, 1 − 1/e or 1/2 for many problems (Vondr´ ak and many others). Question: Can one provide a principled theoretical characterization (similar to the the convex aspects of submodular functions), from a concave perspective?

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 13 / 31

SLIDE 21

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Outline

1

Introduction

2

Two sides of Submodularity

3

Submodular Polyhedra

4

Submodular Sub-differentials and Super-differentials

5

Other Convex/Concave Aspects

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 14 / 31

SLIDE 22

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Lower Polyhedron

A submodular f : 2V → R, has a polyhedron called the submodular (lower) polyhedron and a base (lower) polytope: Pf = {x ∈ RV : x(S) ≤ f (S), ∀S ⊆ V } (4) Bf = Pf ∩ {x : x(V ) = f (V )}. (5) where x(S) =

i∈S xi is seen as a modular function

x1 x2

(0, 0)

Pf Bf

0.5 1 0.5 1 0.2 0.4 0.6 0.8 1 1.2 1.4 e1 e2 e3 0.5 1 0.5 1 0.2 0.4 0.6 0.8 1 1.2 1.4 e1 e2 e3

Bf Bf

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 15 / 31

SLIDE 23

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Chains & Extreme Points of the Submodular Polyhedron

Notation: Given a permutation σ = (σ(1), σ(2), . . . , σ(n)) of V , define chain ∅ = Sσ

0 ⊂ Sσ 1 ⊂ · · · ⊂ Sσ n = V where

Sσ

i = {σ(1), σ(2), . . . , σ(i)}

(6)

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 16 / 31

SLIDE 24

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Chains & Extreme Points of the Submodular Polyhedron

Notation: Given a permutation σ = (σ(1), σ(2), . . . , σ(n)) of V , define chain ∅ = Sσ

0 ⊂ Sσ 1 ⊂ · · · ⊂ Sσ n = V where

Sσ

i = {σ(1), σ(2), . . . , σ(i)}

(6) These chains define all extreme points.

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 16 / 31

SLIDE 25

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Chains & Extreme Points of the Submodular Polyhedron

Notation: Given a permutation σ = (σ(1), σ(2), . . . , σ(n)) of V , define chain ∅ = Sσ

0 ⊂ Sσ 1 ⊂ · · · ⊂ Sσ n = V where

Sσ

i = {σ(1), σ(2), . . . , σ(i)}

(6) These chains define all extreme points. (Edmonds, 1970) Define hσ ∈ RV as, hσ(σ(i)) = f (Sσ

i ) − f (Sσ i−1)

(7) Then, hσ is an extreme point of Pf .

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 16 / 31

SLIDE 26

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Upper Polyhedron

Define Submodular Upper Polyhedron as follows: Pf = {x ∈ Rn : x(S) ≥ f (S), ∀S ⊆ V } (8) Lemma 1 Given a submodular function f , Pf = {x ∈ Rn : x(j) ≥ f (j)} (9)

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 17 / 31

SLIDE 27

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Upper Polyhedron

Pf = {x ∈ Rn : x(S) ≥ f (S), ∀S ⊆ V }

x1 x2

(0, 0)

Pf

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 18 / 31

SLIDE 28

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Upper Polyhedron

Pf = {x ∈ Rn : x(S) ≥ f (S), ∀S ⊆ V }

x1 x2

(0, 0)

Pf

Immediate facts:

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 18 / 31

SLIDE 29

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Upper Polyhedron

Pf = {x ∈ Rn : x(S) ≥ f (S), ∀S ⊆ V }

x1 x2

(0, 0)

Pf

Immediate facts: Membership problem: x ∈ Pf same as maxX⊆V f (X) − x(X) ≤ 0, submodular maximization which is hard,

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 18 / 31

SLIDE 30

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Upper Polyhedron

Pf = {x ∈ Rn : x(S) ≥ f (S), ∀S ⊆ V } = {x ∈ Rn : x(j) ≥ f (j)}

x1 x2

(0, 0)

Pf

Immediate facts: Membership problem: x ∈ Pf same as maxX⊆V f (X) − x(X) ≤ 0, submodular maximization which is hard, but in fact same as maxX⊆V

i∈X f (i) − x(X) ≤ 0, identical to checking singletons

f (i) − x(i) < 0.

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 18 / 31

SLIDE 31

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Outline

1

Introduction

2

Two sides of Submodularity

3

Submodular Polyhedra

4

Submodular Sub-differentials and Super-differentials

5

Other Convex/Concave Aspects

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 19 / 31

SLIDE 32

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Subdifferential

Analogous to convex functions, submodular functions have subdifferential structure (Fujishige’84,’05) at each X ⊆ V . ∂f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≥ f (X) − x(X) ∀Y ⊆ V } (10)

x1 x2

∂ f(∅) ∂ f({v1}) ∂ f({v2}) ∂ f({v1, v2})

(0, 0)

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 20 / 31

SLIDE 33

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Subdifferential

Analogous to convex functions, submodular functions have subdifferential structure (Fujishige’84,’05) at each X ⊆ V . ∂f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≥ f (X) − x(X) ∀Y ⊆ V } (10)

x1 x2

∂ f(∅) ∂ f({v1}) ∂ f({v2}) ∂ f({v1, v2})

(0, 0)

Each hX ∈ ∂f (X) defines modular lower bound of f tight at X:

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 20 / 31

SLIDE 34

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Subdifferential

Analogous to convex functions, submodular functions have subdifferential structure (Fujishige’84,’05) at each X ⊆ V . ∂f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≥ f (X) − x(X) ∀Y ⊆ V } (10)

x1 x2

∂ f(∅) ∂ f({v1}) ∂ f({v2}) ∂ f({v1, v2})

(0, 0)

Each hX ∈ ∂f (X) defines modular lower bound of f tight at X: Easy to characterize extreme points, facets and the membership problem!

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 20 / 31

SLIDE 35

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Subdifferential Redundancy

Define three polyhedra based on a partition of the constraints: ∂1

f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≥ f (X) − x(X), ∀Y ⊆ X}

(11) ∂2

f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≥ f (X) − x(X), ∀Y ⊇ X}

(12) ∂3

f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≥ f (X) − x(X), ∀Y : Y ⊆ X, Y ⊇ X}

(13)

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 21 / 31

SLIDE 36

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Subdifferential Redundancy

Define three polyhedra based on a partition of the constraints: ∂1

f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≥ f (X) − x(X), ∀Y ⊆ X}

(11) ∂2

f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≥ f (X) − x(X), ∀Y ⊇ X}

(12) ∂3

f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≥ f (X) − x(X), ∀Y : Y ⊆ X, Y ⊇ X}

(13) Immediately ∂f (X) = ∂1

f (X) ∩ ∂2 f (X) ∩ ∂3 f (X) but more interestingly:

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 21 / 31

SLIDE 37

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Subdifferential Redundancy

Define three polyhedra based on a partition of the constraints: ∂1

f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≥ f (X) − x(X), ∀Y ⊆ X}

(11) ∂2

f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≥ f (X) − x(X), ∀Y ⊇ X}

(12) ∂3

f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≥ f (X) − x(X), ∀Y : Y ⊆ X, Y ⊇ X}

(13) Immediately ∂f (X) = ∂1

f (X) ∩ ∂2 f (X) ∩ ∂3 f (X) but more interestingly:

Lemma 2 (Fujishige’84) Given a submodular function, ∂f (X) = ∂1

f (X) ∩ ∂2 f (X) for all X ⊆ V .

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 21 / 31

SLIDE 38

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Subdifferential Redundancy

Define three polyhedra based on a partition of the constraints: ∂1

f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≥ f (X) − x(X), ∀Y ⊆ X}

(11) ∂2

f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≥ f (X) − x(X), ∀Y ⊇ X}

(12) ∂3

f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≥ f (X) − x(X), ∀Y : Y ⊆ X, Y ⊇ X}

(13) Immediately ∂f (X) = ∂1

f (X) ∩ ∂2 f (X) ∩ ∂3 f (X) but more interestingly:

Lemma 2 (Fujishige’84) Given a submodular function, ∂f (X) = ∂1

f (X) ∩ ∂2 f (X) for all X ⊆ V .

So for X / ∈ {∅, V } many of the subdifferen- tial inequalities are redundant.

∅ V X

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 21 / 31

SLIDE 39

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Superdifferentials

Analogous to concave functions, we define Submodular Superdifferentials at each X ⊆ V : ∂f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≤ f (X) − x(X), ∀Y ⊆ V } (14)

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 22 / 31

SLIDE 40

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Superdifferentials

Analogous to concave functions, we define Submodular Superdifferentials at each X ⊆ V : ∂f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≤ f (X) − x(X), ∀Y ⊆ V } (14)

x1 x2

∂f(∅) ∂f({v2}) ∂f({v1}) ∂f({v1, v2})

(0, 0)

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 22 / 31

SLIDE 41

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Superdifferentials

Analogous to concave functions, we define Submodular Superdifferentials at each X ⊆ V : ∂f (X) = {x ∈ Rn : f (Y ) − x(Y ) ≤ f (X) − x(X), ∀Y ⊆ V } (14)

x1 x2

∂f(∅) ∂f({v2}) ∂f({v1}) ∂f({v1, v2})

(0, 0)

Each gX ∈ ∂f (X) defines modular upper bound of f tight at X: ∂f (X) = {x ∈ Rn : f (Y ) ≤ f (X) − x(X) + x(Y ); ∀Y ⊆ V } so mX(Y ) f (X) − gX(X) + gX(Y ) ≥ f (Y ) and mX(X) = f (X).

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 22 / 31

SLIDE 42

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Superdifferential Redundancy

Define three polyhedra: ∂f

1(X) = {x ∈ Rn : f (Y ) − x(Y ) ≤ f (X) − x(X), ∀Y ⊆ X}

(15) ∂f

2(X) = {x ∈ Rn : f (Y ) − x(Y ) ≤ f (X) − x(X), ∀Y ⊇ X}

(16) ∂f

3(X) = {x ∈ Rn : f (Y ) − x(Y ) ≤ f (X) − x(X), ∀Y : Y ⊆ X, Y ⊇ X}

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 23 / 31

SLIDE 43

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Superdifferential Redundancy

Define three polyhedra: ∂f

1(X) = {x ∈ Rn : f (Y ) − x(Y ) ≤ f (X) − x(X), ∀Y ⊆ X}

(15) ∂f

2(X) = {x ∈ Rn : f (Y ) − x(Y ) ≤ f (X) − x(X), ∀Y ⊇ X}

(16) ∂f

3(X) = {x ∈ Rn : f (Y ) − x(Y ) ≤ f (X) − x(X), ∀Y : Y ⊆ X, Y ⊇ X}

Immediately ∂f (X) = ∂f

1(X) ∩ ∂f 2(X) ∩ ∂f 3(X). Also, we have

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 23 / 31

SLIDE 44

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Submodular Superdifferential Redundancy

Define three polyhedra: ∂f

1(X) = {x ∈ Rn : f (Y ) − x(Y ) ≤ f (X) − x(X), ∀Y ⊆ X}

(15) ∂f

2(X) = {x ∈ Rn : f (Y ) − x(Y ) ≤ f (X) − x(X), ∀Y ⊇ X}

(16) ∂f

3(X) = {x ∈ Rn : f (Y ) − x(Y ) ≤ f (X) − x(X), ∀Y : Y ⊆ X, Y ⊇ X}

Immediately ∂f (X) = ∂f

1(X) ∩ ∂f 2(X) ∩ ∂f 3(X). Also, we have

Lemma 3 For submodular f , ∂f

1(X) and ∂f 2(X)’s irredundant representation is:

∂f

1(X) = {x ∈ Rn : f (j|X\j) ≥ x(j), ∀j ∈ X}

(17) ∂f

2(X) = {x ∈ Rn : f (j|X) ≤ x(j), ∀j /

∈ X}. (18)

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 23 / 31

SLIDE 45

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Hardness of Characterizing Super-differentials

Superdifferential membership problem is hard: Given a submodular function f and a set Y : ∅ ⊂ Y ⊂ V , the membership problem y ∈ ∂f (Y ) is NP hard. However, it is possible to provide outer and inner bounds of the super-differentials. Sequential Outerbounds by considering upto k inequalities of ∂f

3(X).

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 24 / 31

SLIDE 46

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Inner and Outer Bounds of the Super-differential

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 25 / 31

SLIDE 47

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

The Three Supergradients (Iyer et al, 2013)

Define three vectors ∈ RV as follows: ˆ gX(j) =

f (j|X − j)

if j ∈ X f (j) if j / ∈ X (19) ˇ gX(j) =

f (j|V − j)

if j ∈ X f (j|X) if j / ∈ X (20) ¯ gX(j) =

f (j|V − j)

if j ∈ X f (j) if j / ∈ X (21) Theorem 4 For a submodular function f , ˆ gX, ˇ gX, ¯ gX ∈ ∂f (X). Hence for every submodular function f and set X, ∂f (X) is non-empty.

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 26 / 31

SLIDE 48

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Outline

1

Introduction

2

Two sides of Submodularity

3

Submodular Polyhedra

4

Submodular Sub-differentials and Super-differentials

5

Other Convex/Concave Aspects

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 27 / 31

SLIDE 49

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Optimality Conditions for Submodular Minimization

Lemma 5 (Fujishige’91,’05) A set A ⊆ V is a minimizer of f : 2V → R if and only if: 0 ∈ ∂f (A) (22) Lemma 6 (Fujishige’91,’05) A set A minimizes a submodular function f if and only if f (A) ≤ f (B) for all sets B such that B ⊆ A or A ⊆ B.

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 28 / 31

SLIDE 50

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Frank’s discrete separation theorem (DST)

Lemma 7 (Frank’82) Given a submodular function f and a supermodular function g such that f (X) ≥ g(X), ∀X (and which satisfy f (∅) = g(∅) = 0), there exists a modular function h such that f (X) ≥ h(X) ≥ g(X). Furthermore, if f and g are integral so may be h.

g(x) x f(x) m(x)

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 29 / 31

SLIDE 51

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Optimality Conditions for Submodular Maximization

Theorem 8 Given a submodular function f , a set X is the global maxima iff 0 ∈ ∂f (X) Unfortunately, hard to characterize the superdifferential – even checking if X is a global maximizer is NP hard! Not surprising since submodular maximization is NP hard. However, we can relax the optimality conditions to instead test if 0 belongs to approximations of the super-differentials!

0 belongs to Inner Bound of the Super-differential gives a sufficient condition for optimality. 0 belonging to the Outer-Bounds gives approximate maxima. For example, if 0 belongs to the outerbound ∂f

1(X) ∩ ∂f 2(X), we get a 1/3

approximate solution.

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 30 / 31

SLIDE 52

Introduction Two sides of Submodularity Submodular Polyhedra Sub/Super Differentials Other Convex/Concave Aspects

Concave Discrete Separation Theorem

Lemma 9 Given submodular f and supermodular g, with f (X) ≤ g(X), ∀X ⊆ V , and f (∅) = g(∅) or f (V ) = g(V ). There exists modular h such that f (X) ≤ h(X) ≤ g(X), ∀X ⊆ V . When f and g are also integral, there exists an integral h satisfying the above. Possible to also obtain similar restricted results for Fenchel Duality Theorem.

R. Iyer & J. Bilmes

ISIT 2020: Concave Aspects of Submodular Functions page 31 / 31