( ) Outline Submodular - - PowerPoint PPT Presentation

岩田 覚 (京都大学数理解析研究所)

劣モジュラ最適化

Outline

• Submodular Functions

Examples Discrete Convexity

• Minimizing Submodular Functions
• Symmetric Submodular Functions
• Maximizing Submodular Functions
• Approximating Submodular Functions

Submodular Functions

• Cut Capacity Functions
• Matroid Rank Functions
• Entropy Functions

) ( ) ( ) ( ) ( Y X f Y X f Y f X f ∪ + ∩ ≥ +

V Y X ⊆ ∀ ,

V

X

Y

R 2 : →

V

f

: V

Finite Set

Cut Capacity Function

Cut Capacity

∑

= } leaving : | ) ( { ) ( X a a c X κ X

s t

Max Flow Value＝Min Cut Capacity

) ( ≥ a c

Matroid Rank Functions

: ) ( ) ( | | ) ( , ρ ρ ρ ρ Y X Y X X X V X ≤ ⇒ ⊆ ≤ ⊆ ∀

] , [ rank ) ( X U A X = ρ

Matrix Rank Function Submodular

= A

X

V

U

Whitney (1935)

Entropy Functions

: ) (X h ) ( = φ h

X

Information Sources

≥

) ( ) ( ) ( ) ( Y X h Y X h Y h X h ∪ + ∩ ≥ +

Entropy of the Joint Distribution Conditional Mutual Information

Positive Definite Symmetric Matrices

) ( = φ f

= A

] [X A ] [ det log ) ( X A X f = X

) ( ) ( ) ( ) ( Y X f Y X f Y f X f ∪ + ∩ ≥ + Ky Fan’s Inequality Extension of the Hadamard Inequality

∏

∈

≤

V i ii

A A det

Discrete Concavity

}) ({ }) { ( ) ( }) { ( u f u T f S f u S f T S − ∪ ≥ − ∪ ⇒ ⊆

V

T

Diminishing Returns

S

u

Discrete Convexity

) ( ) ( ) ( ) ( Y X f Y X f Y f X f ∪ + ∩ ≥ + V X

Y

Convex Function

x

y

Discrete Convexity

} , { b a

} , , { c b a V = } {b

} {a

} {c

} , { c b

φ

Lovász (1983)

f ˆ

f

f ˆ

： Linear Interpolation

： Convex ： Submodular

Murota (2003)

Discrete Convex Analysis

Submodular Function Minimization

X

) (X f

? } | ) ( min{ V Y Y f ⊆

Minimization Algorithm Evaluation Oracle Minimizer ) ( = φ f

Assumption: Ellipsoid Method Grötschel, Lovász, Schrijver (1981)

Base Polyhedra

Submodular Polyhedron

)} ( ) ( , , | { ) ( Y f Y x V Y x x f P

V

≤ ⊆ ∀ ∈ = R

∑

∈

=

Y v

v x Y x ) ( ) (

} | { R R → = V x

V

)} ( ) ( ), ( | { ) ( V f V x f P x x f B = ∈ =

Base Polyhedron

Greedy Algorithm

v

⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ = ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ )) ( ( )) ( ( )) ( ( ) ( ) ( ) ( 1 1 1 1 1 1

2 1 2 1 n n

v L f v L f v L f v y v y v y M M L O M M M O L

}) { ) ( ( )) ( ( ) ( v v L f v L f v y − − =

Edmonds (1970) Shapley (1971) Extreme Base

) (v L

) ( V v∈

: y

2

v

1

v

n

v

Theorem

Min-Max Theorem

)} ( | ) ( max{ ) ( min f B x V x Y f

V Y

∈ =

− ⊆

)} ( , min{ : ) ( v x v x =

−

) ( ) ( ) ( Y f Y x V x ≤ ≤

−

Edmonds (1970)

Combinatorial Approach

Extreme Base Convex Combination Cunningham (1985)

L L L y

x ∑

Λ ∈

= λ

) ( f B yL ∈

x

| ) ( | max X f M

V X⊆

=

) log (

6

nM M n O γ

Submodular Function Minimization

) (

8 7

n n O + γ

) log (

5

M n O γ ) log (

7

n n O γ

Grötschel, Lovász, Schrijver (1981, 1988) Iwata, Fleischer, Fujishige (2000) Schrijver (2000) Iwata (2003) Fleischer, Iwata (2000)

) log ) ((

5 4

M n n O + γ

Orlin (2007)

) (

6 5

n n O + γ

Iwata (2002)

Fully Combinatorial Ellipsoid Method

Cunningham (1985) Iwata, Orlin (2009)

) ( subject to Minimize

2

f B x x ∈

The Minimum-Norm Base

sol.

• pt.

:

∗

x } ) ( | { : < =

∗ −

v x v X } ) ( | { : ≤ =

∗ v

x v X

Minimal Minimizer Maximal Minimizer

∗

x

Theorem

Fujishige (1984)

Wolfe’s Algorithm Practically Efficient

Evacuation Problem （Dynamic Flow）

: ) (X

• :

) (a τ

Hoppe, Tardos (2000)

: ) (v b

: ) (a c S X ∩ X T \ T S X X

• X

b ∪ ⊆ ∀ ≤ ), ( ) ( T S

Capacity Transit Time Supply/Demand Maximum Amount of Flow from to . Feasible

Multiterminal Source Coding

X

X

R

Y

Encoder Encoder

Y

R

Decoder

) (Y H

) (X H

) | ( X Y H

) | ( Y X H ) , ( Y X H

) , ( Y X H

X

R

Y

R

Slepian, Wolf (1973)

Multiclass Queueing Systems

i

μ

i

λ

Server

Control Policy Arrival Rate Service Rate Multiclass M/M/1 Preemptive Arrival Interval Service Time

Performance Region

V X S

X i i X i i i X i i i

⊆ ∀ − ≥

∑ ∑ ∑

∈ ∈ ∈

, 1 ρ μ ρ ρ

: s

, :

i i i

μ λ ρ =

Y

R

j

s

i

s

:

j

s

Expected Staying Time of a Job in j Achievable

Coffman, Mitrani (1980)

: s

1 <

∑

∈V i i

ρ

A Class of Submodular Functions

V

z y x

+

∈R , ,

)) ( ( ) ( ) ( ) ( X x h X y X z X f − =

: h

V X S

X i i X i i i X i i i

⊆ ∀ − ≥

∑ ∑ ∑

∈ ∈ ∈

, 1 ρ μ ρ ρ

Nonnegative, Nondecreasing, Convex

) ( V X ⊆

i i i

y μ ρ = : x x h − = 1 1 : ) (

i i i

S z ρ = :

i i

x ρ = :

Submodular Itoko & Iwata (2005)

Zonotope in 3D

)) ( ), ( ), ( ( ) ( X z X y X x X w =

} | ) ( { conv V X X w Z ⊆ =

) ( ) , , ( ~ x yh z z y x f − =

Zonotope

}

• f

Point Extreme Lower : ) , , ( | ) , , ( ~ min{ } | ) ( min{ Z z y x z y x f V X X f = ⊆

Remark:

) , , ( ~ z y x f

is NOT concave!

z

Line Arrangement

β

i i i

z y x = + β α

α Topological Sweeping Method Edelsbrunner, Guibas (1989)

) (

2

n O

Enumerating All the Cells

Symmetric Submodular Functions

. ), \ ( ) ( V X X V f X f ⊆ ∀ =

? } , | ) ( min{ V X V X X f ≠ ⊂ ≠ φ

) ( ) ( ) ( ) ( , Y X f Y X f Y f X f V Y X Y X ∩ + ∪ ≥ + ⇒ ≠ ∪ ≠ ∩ φ

R →

V

f 2 :

Symmetric Crossing Submodular Symmetric Submodular Function Minimization

Maximum Adjacency Ordering

• Minimum Cut Algorithm by MA-ordering

Nagamochi & Ibaraki (1992)

• Simpler Proofs

Frank (1994), Stoer & Wagner (1997)

• Symmetric Submodular Functions

Queyranne (1998)

• Alternative Proofs

Fujishige (1998), Rizzi (2000)

Minimum Degree Ordering

Nagamochi (2007)

ISAAC’07, Sendai, Japan

Finding the family of all extreme sets for symmetric crossing submodular functions in time.

) (

3γ

n O

Symmetric Submodular Function Minimization

Extreme Sets

. : ), ( ) ( X Z X Z X f Z f ≠ ≠ ⊂ ∀ > φ

) \ ( ) \ ( ) ( ) ( X Y f Y X f Y f X f + ≥ +

: f

Symmetric Crossing Submodular Function Extreme Set

: X

The family of all extreme sets forms a laminar.

X

Y

X

Y

Flat Pair for Symmetric Submodular Functions

, } | ) ( min{ ) ( X x x f X f ∈ ≥ . 1 | } , { | s.t. = ∩ ⊆ ∀

v u X V X

) ( } , { v u V v u ≠ ⊆

Flat Pair

u v X

} , { v u

No Extreme Sets Separate and

u . v

Shrink into a single vertex.

} , { v u

MD-Ordering for Symmetric Submodular Functions

X

) \ ( ) ( ) ( : ) (

i i i

V V X X V f X f X f ⊆ ∪ + =

i

V Symmetric, Crossing Submodular

MD-ordering

V v v v v

n n

∈

− ,

,..., ,

1 2 1

j

v

Each has minimum value of among

) (

1 v

f j− . \

1 −

∈

j

V V v } ,..., { :

1 i i

v v V =

MD-Ordering for Symmetric Submodular Functions

n n

v v ,

1 −

1 − n

v

2 − = n i

The last two vertices of an MD-ordering form a flat pair.

Proof by Induction:

: } , {

1 n n

v v −

i

f

. , 1 ,..., 2 − = n i

Flat Pair for on

n

v

i

V V \

Time Complexity

• Finding an MD-ordering in time.
• Finding all the extreme sets in time.
• Minimizing symmetric submodular functions

in time.

) (

3γ

n O ) (

2γ

n O

) (

3γ

n O

Application to Clustering

A

V

A V \

n

X X ,...,

1

) , ( ) ( ) ( ) | ( ) ( ) ; (

B A B A A B B B A

X X H X H X H X X H X H X X I − + = − =

Random variables

V

Partition into and as independent as possible

) ; (

\ A V A X

X I

Minimize subject to

V A ≠ ≠ φ

A V \

A

Application to Clustering

) ( j i V V

j i

≠ = ∩ φ

∑

= k i i

V f

1

) (

Minimize subject to

k

V V V ∪ ∪ = L

1

Greedy Split

) 1 2 ( k −

• Approximation

Submodurlar Function Maximization

Approximation Algorithms

Nemhauser, Wolsey, Fisher (1978) Monotone SF / Cardinality Constraint (1-1/e)-Approximation Feige, Mirrokni, Vondrák (FOCS 2007) Nonnegative SF 2/5-Approximation Vondrák (STOC 2008) Monotone SF / Matroid Constraint (1-1/e)-Approximation

Approximate Maximization

) ( ) ( : } { : }) { ( max arg : :

1 1 1 − − −

− = ∪ = ∪ = =

j j j j j j j j

T f T f v T T v T f v T ρ φ

1 − j

v

1

v

Greedy Algorithm

) (S f

Maximize subject to

k S ≤ | |

k j ,..., 1 =

1 − j

T

( )

k S S S f T f

k

≤ ∀ − ≥ | :| ), ( e 1 1 ) (

Approximate Maximization

) ,..., 1 ( k j =

) ( :

*

S f = η

:

*

S

∑

− =

+ ≤

1 1 j i i j

k ρ ρ η

[ ]

j j T S u j j j j

k T f T f u T f T f T S f S f

j

ρ + ≤ − ∪ + ≤ ∪ ≤

− ∈ − − − −

∑

−

) ( ) ( }) { ( ) ( ) ( ) ( )

1 \ 1 1 1 1 * *

1 *

Q

Optimal Solution

∑

− = 1 1 j i i

ρ

∑

=

=

k i i k

T f

1

) ( ρ

Approximate Maximization

) ,..., 1 ( k j

j

= ≥ ρ

1

1 1 : ˆ

−

⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − =

j j

k k η ρ

) ,..., 1 ( k j =

⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ ≥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ η η η ρ ρ ρ M M L O O M M O L

k

k k k

2 1

1 1 1

Minimize ∑

= k i i 1

ρ

subject to

⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − ≥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎣ ⎡ ⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − − =

∑

=

e 1 1 1 1 1 ˆ

1

η η ρ

k k i i

k

) ( :

*

S f = η

∑

=

=

k i k i

T f

1

) ( ρ

Feature Selection

n

X X Y ,..., ,

1

A

X

) | ( ) ( : ) ; (

A A

X Y H Y H Y X I − =

Random Variables

1

X

Y

Predict from subset

2

X

3

X

Y

“Sick” “Fever” “Rash” “Cough”

) ; ( Y X I

A

Maximize subject to

k A ≤ | |

: X

Conditionally Independent

) | ( ) ( Y X H X H

A A −

=

Submodular Krause & Guestrin (2005)

Submodular Welfare Problem

(Monotone, Submodular)

k

f f ,...,

1

Utility Functions

) ( j i V V

j i

≠ = ∩ φ

∑

= k i i

V f

1

) (

Maximize subject to

k

V V V ∪ ∪ = L

1

e) 1 1 ( −

• Approximation

Vondrák (2008)

Approximating Submodular Functions

X

) (X f

Algorithm Evaluation Oracle Function

f ˆ

Y

) ( ˆ Y f

Construct a set function such that For what function is this possible?

Approximating Submodular Functions

. ), ( ˆ ) ( ) ( ) ( ˆ V X X f n X f X f ⊆ ∀ ≤ ≤ α

α , ) ( = φ f

Assumption

. 0, ) ( V X X f ⊆ ∀ ≥

Problem

f ˆ

1 ) ( = n α n n = ) ( α

Remarks for cut capacity functions for general monotone submodular functions

Approximating Submodular Functions

• Algorithm with

for matroid rank functions.

• Algorithm with

for monotone submodular functions.

• No polynomial algorithm can achieve

a factor better than even for matroid rank functions.

) log ( ) ( n n O n = α ) log ( ) ( n n n Ω = α 1 ) ( + = n n α

Goemans, Harvey, Iwata, Mirrokni (2009)

Submodular Load Balancing

? ) ( max min

} ,..., { 1 j j j V V

V f

m

∑

∈

=

X v jv j

p X f : ) (

: ,...,

1 m

f f

Svitkina & Fleischer (2008) Monotone Submodular Functions Scheduling 2-Approximation Algorithm Lenstra, Shmoys, Tardos (1990)

) log ( n n O

• Approximation Algorithm

Pointers

• S. Fujishige: Submodular Functions and

Optimization, Elsevier, 2005.

• ICML 2008 Tutorial Session

http://www.submodularity.org

• 永野清仁，河原吉伸，岡本吉央：離散凸最適化

手法による機械学習の諸問題へのアプローチ， 回路とシステム軽井沢ワークショップ，2009.