Frank-Wolfe Splitting via Augmented Lagrangian Method Fabian - - PowerPoint PPT Presentation

Frank-Wolfe Splitting via Augmented Lagrangian Method

Gauthier Gidel1 Fabian Pedregosa2 Simon Lacoste-Julien1

1MILA, DIRO Université de Montréal 2UC Berkeley & ETH Zurich

April 2018

Gauthier Gidel FW Splitting via ALM April 2018

Why Frank-Wolfe is wonderful.

◮ Constrained optimization algorithm:

min

x∈C f(x)

f convex, C convex compact.

◮ Interesting for highly structured constraint sets:

Alignment constraint: [Alayrac et al., 2016] Permutahedron: [Lancia and Serafini, 2018] [Evangelopoulos et al., 2017]

Gauthier Gidel FW Splitting via ALM April 2018

Why Frank-Wolfe is wonderful.

◮ Constrained optimization algorithm:

min

x∈C f(x)

f convex, C convex compact.

◮ Interesting for highly structured constraint sets:

Alignment constraint: [Alayrac et al., 2016] Permutahedron: [Lancia and Serafini, 2018] [Evangelopoulos et al., 2017]

Gauthier Gidel FW Splitting via ALM April 2018

Why Frank-Wolfe is wonderful.

◮ Constrained optimization algorithm:

min

x∈C f(x)

f convex, C convex compact.

◮ Interesting for highly structured constraint sets:

Alignment constraint: [Alayrac et al., 2016] Permutahedron: [Lancia and Serafini, 2018] [Evangelopoulos et al., 2017]

Gauthier Gidel FW Splitting via ALM April 2018

Why Frank-Wolfe is wonderful.

◮ Constrained optimization algorithm:

min

x∈C f(x)

f convex, C convex compact.

◮ Interesting when projection is not practical:

Projection Linear Minimization Oracle

◮ When projection is practical better use projected gradient

method.

Gauthier Gidel FW Splitting via ALM April 2018

Why Frank-Wolfe sometimes is not enough.

◮ FW requires linear minimization (LMO) over these set.

LMO(d) := arg min

x∈C

d, x

◮ Intersection of constraint sets: C1 ∩ C2. ◮ LMOC1∩C2(d) may be too expensive. ◮ FW-AL just requires LMOC1(d) and LMOC2(d).

Gauthier Gidel FW Splitting via ALM April 2018

Why Frank-Wolfe sometimes is not enough.

◮ FW requires linear minimization (LMO) over these set.

LMO(d) := arg min

x∈C

d, x

◮ Intersection of constraint sets: C1 ∩ C2. ◮ LMOC1∩C2(d) may be too expensive. ◮ FW-AL just requires LMOC1(d) and LMOC2(d).

Gauthier Gidel FW Splitting via ALM April 2018

Why Frank-Wolfe sometimes is not enough.

◮ FW requires linear minimization (LMO) over these set.

LMO(d) := arg min

x∈C

d, x

◮ Intersection of constraint sets: C1 ∩ C2. ◮ LMOC1∩C2(d) may be too expensive. ◮ FW-AL just requires LMOC1(d) and LMOC2(d).

Gauthier Gidel FW Splitting via ALM April 2018

Simultaneously sparse and low rank matrix recovery

Proposed by Richard et al. [2012]: min

S0,S1≤β1,S∗≤β2

S − ˆ Σ2

2 . ◮ Sparcity constraint: C1 := {S 0, S1 ≤ β1},

LMOC1(D) = Largest coefficient of the matrix: O(d2)

◮ Low rank constraint: C2 := {S 0, S∗ ≤ β2}.

LMOC2(D) = Largest eigenvector: O(d2/√ǫ)

Gauthier Gidel FW Splitting via ALM April 2018

Simultaneously sparse and low rank matrix recovery

Proposed by Richard et al. [2012]: min

S0,S1≤β1,S∗≤β2

S − ˆ Σ2

2 . ◮ Sparcity constraint: C1 := {S 0, S1 ≤ β1},

LMOC1(D) = Largest coefficient of the matrix: O(d2)

◮ Low rank constraint: C2 := {S 0, S∗ ≤ β2}.

LMOC2(D) = Largest eigenvector: O(d2/√ǫ)

Gauthier Gidel FW Splitting via ALM April 2018

Simultaneously sparse and low rank matrix recovery

Proposed by Richard et al. [2012]: min

S0,S1≤β1,S∗≤β2

S − ˆ Σ2

2 . ◮ Sparcity constraint: C1 := {S 0, S1 ≤ β1},

LMOC1(D) = Largest coefficient of the matrix: O(d2)

◮ Low rank constraint: C2 := {S 0, S∗ ≤ β2}.

LMOC2(D) = Largest eigenvector: O(d2/√ǫ)

Gauthier Gidel FW Splitting via ALM April 2018

Multiple sequence alignment

Proposed by Yen et al. [2016a]: min

W∈A∩P W, D ◮ W: alignment the sequences. D: cost matrix. ◮ A : alignment constraint. Each alignment with the

consensus sequence is valid.

◮ P : consensus constraint. Alignments consistent between

each other.

Gauthier Gidel FW Splitting via ALM April 2018

Multiple sequence alignment

Proposed by Yen et al. [2016a]: min

W∈A∩P W, D ◮ W: alignment the sequences. D: cost matrix. ◮ A : alignment constraint. Each alignment with the

consensus sequence is valid.

◮ P : consensus constraint. Alignments consistent between

each other.

Gauthier Gidel FW Splitting via ALM April 2018

Multiple sequence alignment

Proposed by Yen et al. [2016a]: min

W∈A∩P W, D ◮ W: alignment the sequences. D: cost matrix. ◮ A : alignment constraint. Each alignment with the

consensus sequence is valid.

◮ P : consensus constraint. Alignments consistent between

each other.

Gauthier Gidel FW Splitting via ALM April 2018

Multiple sequence alignment

Proposed by Yen et al. [2016a]: min

W∈A∩P W, D ◮ W: alignment the sequences. D: cost matrix. ◮ A : alignment constraint. Each alignment with the

consensus sequence is valid.

◮ P : consensus constraint. Alignments consistent between

each other.

Gauthier Gidel FW Splitting via ALM April 2018

Structured SVM

Proposed by Yen et al. [2016b]: dual problem: min

αf∈∆|Yf |

1 2

• F∈T

AF α2

2 −

• j∈V

δ⊤

j αj

s.t. Mfi αf = αi , f ∈ F, F ∈ T , i ∈ N(f) .

◮ V : Variables. T : Factor templates. N(f): neighbors of f. ◮ Consistency constraint: M11x(1) = α1, M12x(1) = α2, . . .

α1 α2 α3 x(1) x(2)

Gauthier Gidel FW Splitting via ALM April 2018

Structured SVM

Proposed by Yen et al. [2016b]: dual problem: min

αf∈∆|Yf |

1 2

• F∈T

AF α2

2 −

• j∈V

δ⊤

j αj

s.t. Mfi αf = αi , f ∈ F, F ∈ T , i ∈ N(f) .

◮ V : Variables. T : Factor templates. N(f): neighbors of f. ◮ Consistency constraint: M11x(1) = α1, M12x(1) = α2, . . .

α1 α2 α3 x(1) x(2)

Gauthier Gidel FW Splitting via ALM April 2018

Structured SVM

Proposed by Yen et al. [2016b]: dual problem: min

αf∈∆|Yf |

1 2

• F∈T

AF α2

2 −

• j∈V

δ⊤

j αj

s.t. Mfi αf = αi , f ∈ F, F ∈ T , i ∈ N(f) .

◮ V : Variables. T : Factor templates. N(f): neighbors of f. ◮ Consistency constraint: M11x(1) = α1, M12x(1) = α2, . . .

α1 α2 α3 x(1) x(2)

Gauthier Gidel FW Splitting via ALM April 2018

General Formulation

minimize

x(1),...,x(k) f(x(1), . . . , x(k)) ,

x(k) ∈ Ck, k ∈ [K],

K

• k=1

Akx(k) = 0 .

◮ f is convex and smooth (gradient Lipschitz). ◮ Ck, k ∈ {1, . . . , K} are convex compact.

Gauthier Gidel FW Splitting via ALM April 2018

Augmented Lagrangian Method

◮ Augmented Lagrangian trick to get rid of K k=1 Akx(k) = 0. ◮ M s.t. Mx = 0 ⇔ K k=1 Akx(k) = 0 and the functions,

L(x, y) := f(x) + y, Mx + λ

2Mx2.

p(x) := max

y∈Rd L(x, y) =

• f(x)

if Mx = 0 , +∞

• therwise.

◮ Augmented Lagrangian formulation of our problem,

minimize

x

max

y∈Rd L(x, y)

s.t. x ∈ X := ×K

k=1Ck .

Gauthier Gidel FW Splitting via ALM April 2018

Augmented Lagrangian Method

◮ Augmented Lagrangian trick to get rid of K k=1 Akx(k) = 0. ◮ M s.t. Mx = 0 ⇔ K k=1 Akx(k) = 0 and the functions,

L(x, y) := f(x) + y, Mx + λ

2Mx2.

p(x) := max

y∈Rd L(x, y) =

• f(x)

if Mx = 0 , +∞

• therwise.

◮ Augmented Lagrangian formulation of our problem,

minimize

x

max

y∈Rd L(x, y)

s.t. x ∈ X := ×K

k=1Ck .

Gauthier Gidel FW Splitting via ALM April 2018

Augmented Lagrangian Method

◮ Augmented Lagrangian trick to get rid of K k=1 Akx(k) = 0. ◮ M s.t. Mx = 0 ⇔ K k=1 Akx(k) = 0 and the functions,

L(x, y) := f(x) + y, Mx + λ

2Mx2.

p(x) := max

y∈Rd L(x, y) =

• f(x)

if Mx = 0 , +∞

• therwise.

◮ Augmented Lagrangian formulation of our problem,

minimize

x

max

y∈Rd L(x, y)

s.t. x ∈ X := ×K

k=1Ck .

Gauthier Gidel FW Splitting via ALM April 2018

FW-AL algorithm

minimize

x

max

y∈Rd L(x, y)

s.t. x ∈ X := ×K

k=1Ck . ◮ Standard AL method:

  

xt+1 = arg min

x∈X

L(x, yt) (argmin step) , yt+1 = yt + ηtMxt+1 (Gradient ascent step) .

◮ Replace arg min steps by FW steps. FW-AL:

• xt+1 = FW(xt; L(·, yt))

(Frank-Wolfe step) , yt+1 = yt + ηtMxt+1 (Gradient ascent step) .

Gauthier Gidel FW Splitting via ALM April 2018

FW-AL algorithm

minimize

x

max

y∈Rd L(x, y)

s.t. x ∈ X := ×K

k=1Ck . ◮ Standard AL method:

  

xt+1 = arg min

x∈X

L(x, yt) (argmin step) , yt+1 = yt + ηtMxt+1 (Gradient ascent step) .

◮ Replace arg min steps by FW steps. FW-AL:

• xt+1 = FW(xt; L(·, yt))

(Frank-Wolfe step) , yt+1 = yt + ηtMxt+1 (Gradient ascent step) .

Gauthier Gidel FW Splitting via ALM April 2018

The FW algorithm

Algorithm 1 One Frank-Wolfe step

1: Let x(t) ∈ M 2: Compute r(t) = ∇f(x(t)) 3: Compute s(t) ∈ argmin

s∈C

• s, r(t)

4: Compute gt :=

• x(t) − s(t), r(t)

5: if gt ≤ ǫ then return x(t) 6: Let γ =

2 2+t (or do line-search)

7: Update x(t+1) := (1 − γ)x(t) + γs(t)

α f(α) M

f

Gauthier Gidel FW Splitting via ALM April 2018

The FW algorithm

Algorithm 2 One Frank-Wolfe step

1: Let x(t) ∈ M 2: Compute r(t) = ∇f(x(t)) 3: Compute s(t) ∈ argmin

s∈C

• s, r(t)

4: Compute gt :=

• x(t) − s(t), r(t)

5: if gt ≤ ǫ then return x(t) 6: Let γ =

2 2+t (or do line-search)

7: Update x(t+1) := (1 − γ)x(t) + γs(t)

α f(α) M

f

f(α) +

• s0 − α, rf(α)
• Gauthier Gidel

FW Splitting via ALM April 2018

The FW algorithm

Algorithm 3 One Frank-Wolfe step

1: Let x(t) ∈ M 2: Compute r(t) = ∇f(x(t)) 3: Compute s(t) ∈ argmin

s∈C

• s, r(t)

4: Compute gt :=

• x(t) − s(t), r(t)

5: if gt ≤ ǫ then return x(t) 6: Let γ =

2 2+t (or do line-search)

7: Update x(t+1) := (1 − γ)x(t) + γs(t)

α f(α) M

f

f(α) +

• s0 − α, rf(α)
• Gauthier Gidel

FW Splitting via ALM April 2018

The FW algorithm

Algorithm 4 One Frank-Wolfe step

1: Let x(t) ∈ M 2: Compute r(t) = ∇f(x(t)) 3: Compute s(t) ∈ argmin

s∈C

• s, r(t)

4: Compute gt :=

• x(t) − s(t), r(t)

5: if gt ≤ ǫ then return x(t) 6: Let γ =

2 2+t (or do line-search)

7: Update x(t+1) := (1 − γ)x(t) + γs(t)

α f(α) M

f

s

f(α) +

• s0 − α, rf(α)
• Gauthier Gidel

FW Splitting via ALM April 2018

Related work: GDMM

◮ Replace arg min step by a FW step initially proposed by

Yen et al. [2016a] to solve MSA problem.

◮ Afterwards used for Structured SVM [Yen et al., 2016b]

and MAP inference [Huang et al., 2017].

◮ Restricted to polytopes and simple (linear and quadratic)

functions. Contributions:

◮ Extension of GDMM for general convex sets. (e.g. Trace

norm ball)

◮ Fix a crucial missing part in the previous proofs.

Gauthier Gidel FW Splitting via ALM April 2018

Related work: GDMM

◮ Replace arg min step by a FW step initially proposed by

Yen et al. [2016a] to solve MSA problem.

◮ Afterwards used for Structured SVM [Yen et al., 2016b]

and MAP inference [Huang et al., 2017].

◮ Restricted to polytopes and simple (linear and quadratic)

functions. Contributions:

◮ Extension of GDMM for general convex sets. (e.g. Trace

norm ball)

◮ Fix a crucial missing part in the previous proofs.

Gauthier Gidel FW Splitting via ALM April 2018

Related work: GDMM

◮ Replace arg min step by a FW step initially proposed by

Yen et al. [2016a] to solve MSA problem.

◮ Afterwards used for Structured SVM [Yen et al., 2016b]

and MAP inference [Huang et al., 2017].

◮ Restricted to polytopes and simple (linear and quadratic)

functions. Contributions:

◮ Extension of GDMM for general convex sets. (e.g. Trace

norm ball)

◮ Fix a crucial missing part in the previous proofs.

Gauthier Gidel FW Splitting via ALM April 2018

Related work: GDMM

◮ Replace arg min step by a FW step initially proposed by

Yen et al. [2016a] to solve MSA problem.

◮ Afterwards used for Structured SVM [Yen et al., 2016b]

and MAP inference [Huang et al., 2017].

◮ Restricted to polytopes and simple (linear and quadratic)

functions. Contributions:

◮ Extension of GDMM for general convex sets. (e.g. Trace

norm ball)

◮ Fix a crucial missing part in the previous proofs.

Gauthier Gidel FW Splitting via ALM April 2018

Theoretical contribution

Additional assumption: Slater’s condition: ∃ x(k) ∈ relint(Ck), k ∈ [K] s.t.

K

• k=1

Akx(k) = 0 . New lemma: Let d be the augmented dual function, d(y) := min

x∈X L(x, y) .

There exist a constant α > 0 such that close enough to Y∗, d∗ − d(y) ≥ αdist(y, Y∗)2.

Gauthier Gidel FW Splitting via ALM April 2018

Theoretical contribution

Additional assumption: Slater’s condition: ∃ x(k) ∈ relint(Ck), k ∈ [K] s.t.

K

• k=1

Akx(k) = 0 . New lemma: Let d be the augmented dual function, d(y) := min

x∈X L(x, y) .

There exist a constant α > 0 such that close enough to Y∗, d∗ − d(y) ≥ αdist(y, Y∗)2.

Gauthier Gidel FW Splitting via ALM April 2018

Convergence results

◮ For general convex sets:

With decreasing step size ηt := O

• 1

t+1

• ,

subopt: ∆t ≤ O(1) t , feasibility: min

t0≤s≤tMxs2 ≤ O(1)

t .

◮ For X a polytope:

With small enough constant step size ηt: ∆t ≤ ∆t0 (1 + ρ)t−t0 , Mxt+12 ≤ O(1) (1 + ρ)t−t0 . Only holds for generalized strongly convex function and uses a variant of FW with away-step.

◮ Standard splitting algorithms have faster rate per

iteration in practice.

◮ Advantage only comes from the cheaper oracle !

Gauthier Gidel FW Splitting via ALM April 2018

Convergence results

◮ For general convex sets:

With decreasing step size ηt := O

• 1

t+1

• ,

subopt: ∆t ≤ O(1) t , feasibility: min

t0≤s≤tMxs2 ≤ O(1)

t .

◮ For X a polytope:

With small enough constant step size ηt: ∆t ≤ ∆t0 (1 + ρ)t−t0 , Mxt+12 ≤ O(1) (1 + ρ)t−t0 . Only holds for generalized strongly convex function and uses a variant of FW with away-step.

◮ Standard splitting algorithms have faster rate per

iteration in practice.

◮ Advantage only comes from the cheaper oracle !

Gauthier Gidel FW Splitting via ALM April 2018

Experiments

Simultaneously sparse and low rank matrix recovery: min

S0,S1≤β1,S∗≤β2

S − ˆ Σ2

2 . ◮ Sparcity constraint: C1 := {S 0, S1 ≤ β1},

LMOC1(D) = Largest coefficient of the matrix: O(d2)

◮ Low rank constraint: C2 := {S 0, S∗ ≤ β2}.

LMOC2(D) = Largest eigenvector: O(d2/√ǫ)

Gauthier Gidel FW Splitting via ALM April 2018

Experiments

LMO vs. projection for trace norm ball:

102 103 104 105 106 107 108 Dimension 10−3 10−2 10−1 100 101 102 103 Time (in s) FW-AL Vs. Baseline Linear oracle on B∗ Projection on B∗

Gauthier Gidel FW Splitting via ALM April 2018

Experiments

Support recovered by FW-AL and the generalized forward backward algorithm as a function of time:

100 101 102 Seconds 0.0 0.2 0.4 0.6 0.8 1.0 Support recovery

Forward-backward FW-AL λ = 1, η = 10−4 FW-AL λ = 20, η = 10−4 FW-AL λ = 1, η = 1

Gauthier Gidel FW Splitting via ALM April 2018

Conclusion

Task: Minimize a function over an intersection of convex sets. Problem:

◮ Projections or linear minimization oracle (LMO) over the

intersection is expensive.

◮ Projection onto each individual set is expensive.

Our solution:

◮ Requires linear minimization oracles over individual

constraints.

◮ Based on the Augmented Lagrangian Method.

Contributions:

◮ Extension of GDMM for general convex sets. ◮ Fix a missing part of the previous proofs.

Gauthier Gidel FW Splitting via ALM April 2018

Conclusion

Task: Minimize a function over an intersection of convex sets. Problem:

◮ Projections or linear minimization oracle (LMO) over the

intersection is expensive.

◮ Projection onto each individual set is expensive.

Our solution:

◮ Requires linear minimization oracles over individual

constraints.

◮ Based on the Augmented Lagrangian Method.

Contributions:

◮ Extension of GDMM for general convex sets. ◮ Fix a missing part of the previous proofs.

Gauthier Gidel FW Splitting via ALM April 2018

Conclusion

Task: Minimize a function over an intersection of convex sets. Problem:

◮ Projections or linear minimization oracle (LMO) over the

intersection is expensive.

◮ Projection onto each individual set is expensive.

Our solution:

◮ Requires linear minimization oracles over individual

constraints.

◮ Based on the Augmented Lagrangian Method.

Contributions:

◮ Extension of GDMM for general convex sets. ◮ Fix a missing part of the previous proofs.

Gauthier Gidel FW Splitting via ALM April 2018

Conclusion

Task: Minimize a function over an intersection of convex sets. Problem:

◮ Projections or linear minimization oracle (LMO) over the

intersection is expensive.

◮ Projection onto each individual set is expensive.

Our solution:

◮ Requires linear minimization oracles over individual

constraints.

◮ Based on the Augmented Lagrangian Method.

Contributions:

◮ Extension of GDMM for general convex sets. ◮ Fix a missing part of the previous proofs.

Gauthier Gidel FW Splitting via ALM April 2018

Conclusion

Task: Minimize a function over an intersection of convex sets. Problem:

◮ Projections or linear minimization oracle (LMO) over the

intersection is expensive.

◮ Projection onto each individual set is expensive.

Our solution:

◮ Requires linear minimization oracles over individual

constraints.

◮ Based on the Augmented Lagrangian Method.

Contributions:

◮ Extension of GDMM for general convex sets. ◮ Fix a missing part of the previous proofs.

Gauthier Gidel FW Splitting via ALM April 2018

Conclusion

Task: Minimize a function over an intersection of convex sets. Problem:

◮ Projections or linear minimization oracle (LMO) over the

intersection is expensive.

◮ Projection onto each individual set is expensive.

Our solution:

◮ Requires linear minimization oracles over individual

constraints.

◮ Based on the Augmented Lagrangian Method.

Contributions:

◮ Extension of GDMM for general convex sets. ◮ Fix a missing part of the previous proofs.

Gauthier Gidel FW Splitting via ALM April 2018

Conclusion

Task: Minimize a function over an intersection of convex sets. Problem:

◮ Projections or linear minimization oracle (LMO) over the

intersection is expensive.

◮ Projection onto each individual set is expensive.

Our solution:

◮ Requires linear minimization oracles over individual

constraints.

◮ Based on the Augmented Lagrangian Method.

Contributions:

◮ Extension of GDMM for general convex sets. ◮ Fix a missing part of the previous proofs.

Gauthier Gidel FW Splitting via ALM April 2018

Thank You !

Slides available on www.di.ens.fr/~gidel and www-ens.iro.umontreal.ca/~gidelgau.

Bibliography

J.-B. Alayrac, P. Bojanowski, N. Agrawal, I. Laptev, J. Sivic, and S. Lacoste-Julien. Unsupervised learning from narrated instruction videos. In CVPR, 2016.

• X. Evangelopoulos, A. J. Brockmeier, T. Mu, and J. Y. Goulermas. A graduated non-convexity

relaxation for large scale seriation. In Proceedings of the 2017 SIAM International Conference

• n Data Mining, pages 462–470. SIAM, 2017.
• X. Huang, I. E.-H. Yen, R. Zhang, Q. Huang, P. Ravikumar, and I. Dhillon. Greedy direction

method of multiplier for MAP inference of large output domain. In AISTATS, 2017.

• G. Lancia and P. Serafini. Compact Extended Linear Programming Models. Springer, 2018.
• E. Richard, P.-A. Savalle, and N. Vayatis. Estimation of simultaneously sparse and low rank
• matrices. In ICML, 2012.
• I. Yen, X. Huang, K. Zhong, R. Zhang, P. Ravikumar, and I. Dhillon. Dual decomposed

learning with factorwise oracle for structural SVM with large output domain. In NIPS, 2016b.

• I. E.-H. Yen, X. Lin, J. Zhang, P. Ravikumar, and I. Dhillon. A convex atomic-norm approach

to multiple sequence alignment and motif discovery. In ICML, 2016a. Gauthier Gidel FW Splitting via ALM April 2018