Conditional Gradient Methods via Stochastic Path-Integrated - - PowerPoint PPT Presentation

Conditional Gradient Methods via Stochastic Path-Integrated Differential Estimator

Alp Yurtsever

alp.yurtsever@epfl.ch Massachusetts Institute of Technology (MIT) Ecole Polytechnique Fédérale de Lausanne (EPFL) joint work with Suvrit Sra & Volkan Cevher

MIT EPFL

ICML2019 - Long Beach

Algorithm 1 CGM for smooth minimization Input: x1 2 X for k = 1, 2, . . . , do ηk = 2/(k + 1) sk = arg minx∈X ⌦ rf(xk), x ↵ xk+1 = xk + ηk(sk xk) end for

X

{x : f(x) Æ f(xk)} ≠Òf(xk) xk sk xk+1

min

x∈X f(x)

Conditional Gradient Method (CGM)

(Frank & Wolfe, 1956) (Hazan, 2008) (Jaggi, 2013)

. X ⊂ Rd is a convex compact set . f : X → R is a smooth function

Stochastic Templates

. X ⊂ Rd is a convex compact set . f and fi are differentiable and possibly non-convex . ⇠ ∼ P is a random variable

minimize

x∈X

F(x)

F(x) := 8 < : Eξf(x, ξ) (expectation)

1 n

Pn

i=1 fi(x)

(finite-sum)

Stochastic Templates

. X ⊂ Rd is a convex compact set . f and fi are differentiable and possibly non-convex . ⇠ ∼ P is a random variable

minimize

x∈X

F(x)

F(x) := 8 < : Eξf(x, ξ) (expectation)

1 n

Pn

i=1 fi(x)

(finite-sum)

Erf(x, ξ) = rF(x)

unbiased estimates

Ekrf(x, ξ) rF(x)k2  σ2 < +1, 8x 2 X

bounded variance

Ekrf(x, ξ) rf(y, ξ)k2  Lkx yk2, 8(x, y) 2 X 2

averaged smoothness

Assumptions

we study the theoretical complexity of stochastic and finite-sum Frank-Wolfe variants Stochastic Templates

. X ⊂ Rd is a convex compact set . f and fi are differentiable and possibly non-convex . ⇠ ∼ P is a random variable

minimize

x∈X

F(x)

F(x) := 8 < : Eξf(x, ξ) (expectation)

1 n

Pn

i=1 fi(x)

(finite-sum)

Erf(x, ξ) = rF(x)

unbiased estimates

Ekrf(x, ξ) rF(x)k2  σ2 < +1, 8x 2 X

bounded variance

Ekrf(x, ξ) rf(y, ξ)k2  Lkx yk2, 8(x, y) 2 X 2

averaged smoothness

Assumptions

Stochastic first-order oracle (sfo) Incremental first-order oracle (ifo) Linear minimization oracle (lmo)

for stochastic function Eξf(x, ξ)

with ξ ∼ P

(sfo) returns (f(x, ξ0), rf(x, ξ0))

where ξ0

is an iid sample from P

for finite-sum, (ifo) draws an index i

{1, 2, . . . , n}

from uniformly random and returns (fi(x), rfi(x))

Oracle Models

given a gradient estimate v ∈ Rd

(lmo) returns such that

s ∈ Rd

s 2 argmin

x∈X

hv, xi

✔ Frank-Wolfe Algorithm (FW)

(Frank & Wolfe, 1956) (Jaggi, 2013) (Lacoste-Julien, 2016)

O(✏−1)

(lmo) and gradient complexity

O(✏−2)

State of the Art Deterministic variants

in the convex setting ……… in the non-convex setting

✔ Frank-Wolfe Algorithm (FW)

(Frank & Wolfe, 1956) (Jaggi, 2013) (Lacoste-Julien, 2016)

O(✏−1)

(lmo) and gradient complexity

O(✏−2)

State of the Art Deterministic variants

in the convex setting ……… in the non-convex setting

✔ Conditional Gradient Sliding (CGS)

(Lan & Zhou, 2016)

use accelerated gradient method approximately solve projection step using FW

O(✏−1)

(lmo)

O(✏−1/2)

(gradient) in the convex setting in the non-convex setting we provide new results

✔ Stochastic FW with constant batch size

(Mokhtari et al., 2018)

✔ Stochastic FW

(Hazan & Luo, 2016) (Reddi et al., 2016)

✔ Online FW

(Hazan & Kale, 2012)

✔ Stochastic CGS

(Lan & Zhou, 2016)

✔ SVRF / SVFW ✔ STORC

(Hazan & Luo, 2016) (Hazan & Luo, 2016) (Reddi et al., 2016)

Variance reduced based on SVRG

{

(Johnson & Zhang, 2013)

State of the Art Stochastic variants

CGM with SPIDER

SPIDER: Stochastic Path-Integrated Differential Estimator

(Fang et al., 2018)

Lemma (Variance bound):

EkrF(xk) vkk2  L2 Sk kxk xk−1k2 + krF(xk−1) vk−1k2  (LDηk)2 Sk + krF(xk−1) vk−1k2

vk = rSk(xk) rSk(xk−1) + vk−1

we introduce SPIDER-FW

best known rates in the non-convex setting

O(√n✏−2)

O(✏−2)

O(✏−3)

(finite-sum) (expectation) (sfo) (lmo)

O(✏−2)

(ifo) (lmo)

Poster today: Pacific Ballroom #85

convex non-convex finite-sum expectation finite-sum expectation (ifo) (lmo) (sfo) (lmo) (ifo) (lmo) (sfo) (lmo) FW O(n✏−1) O(✏−1)

• O(n✏−2)

O(✏−2)

• CGS

O(n✏−1/2) O(✏−1)

• O(n✏−2)

O(✏−2)

• SFW

O(✏−3) O(✏−1) O(✏−3) O(✏−1) O(✏−4) O(✏−2) O(✏−4) O(✏−2) SFW-1 O(✏−3) O(✏−3) O(✏−3) O(✏−3)

• Online-FW

O(✏−4) O(✏−2) O(✏−4) O(✏−2)

• SCGS

O(✏−2) O(✏−1) O(✏−2) O(✏−1) O(✏−4) O(✏−2) O(✏−4) O(✏−2) SVRF / SVFW O(n ln(✏−1) + ✏−2) O(✏−1)

• O(n + n2/3✏−2)

O(✏−2) O(✏−10/3) O(✏−2) STORC† O(n ln(✏−1) + ✏−3/2) O(✏−1)

• SPIDER-FW

O(n ln(✏−1) + ✏−2) O(✏−1) O(✏−2) O(✏−1) O(n1/2✏−2) O(✏−2) O(✏−3) O(✏−2) SPIDER-CGS O(n ln(✏−1) + ✏−2) O(✏−1) O(✏−2) O(✏−1) O(n1/2✏−2) O(✏−2) O(✏−3) O(✏−2)

Table 1: Comparison of conditional gradient methods for stochastic optimization. Contribution of this work is highlighted with blue font.

See Section 6 for more details.

FW (Frank & Wolfe, 1956; Jaggi, 2013) , CGS (Lan & Zhou, 2016) , SFW (Hazan & Luo, 2016; Reddi et al., 2016) , SFW-1 (Mokhtari et al., 2018) , Online-FW (Hazan & Kale, 2012) , SCGS (Lan & Zhou, 2016) , SVRF / SVFW (Hazan & Luo, 2016; Reddi et al., 2016) , STORC (Hazan & Luo, 2016)

Comparison