Projected primal-dual splitting for solving constrained convex - - PowerPoint PPT Presentation

Motivation Convergence Numerics Extensions

Projected primal-dual splitting for solving constrained convex optimization1

• L. M. Brice˜

no-Arias Universidad T´ ecnica Federico Santa Mar´ ıa LAWOC 2018, Quito 5 September 2018

1Joint work with Sergio L´

• pez Rivera, USM

LAWOC 2018 1/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

1 Motivation Stationary Mean Field Games 2 Algorithm and convergence 3 Numerical experiences Stationary Mean Field Games Numerical simulation 4 Extensions and acceleration

LAWOC 2018 2/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Motivation: Mean Field Games (MFG)

Introduced by Lasry-Lions (2006) : Given T > 0 and ν ≥ 0 − ∂tuT − ν∆uT + 1 2|∇uT |2 = V (x, mT ) in T2 × [0, T] ∂tmT − ν∆mT − div

• mT ∇uT
• = 0

in T2 × [0, T] mT (x, 0) = m0(x), uT (x, T) = Φ(x, mT (x, T)), in T2. The first is a Hamilton-Jacobi equation backward in time. It characterizes the value of an optimal control problem solved by “a typical small” player and whose cost function V (and final cost Φ) depends on the density of other players at each t ∈ [0, T] (at time T). The second is a Fokker-Plank equation forward in time. It models the evolution of initial density m0 at the Nash Equilibrium.

LAWOC 2018 3/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Motivation: Stationary MFG

It is interpreted as the limit behavior of the rescaled MFG when T → ∞ (see Cardaliaguet et. al 2012). SMFG −ν∆u(x) + 1 2|∇u(x)|2 + λ = V (x, m(x)) in T2 −ν∆m(x) − div

• m(x)∇u(x)
• = 0

in T2 0 ≤ m,

• T2 m(x)dx = 1,
• T2 u(x)dx = 0.

Well-posedness is studied in the case of smooth and weak solutions: Lasry, Lions (2006-07), Cirant (2015-16), Gomes, Patrizi,Voskanyan (2014), Gomes, Mitaki (2015)...

LAWOC 2018 4/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Motivation: Discretized SMFG

If ν > 0, V (x, ·) is increasing and we suppose that the sta- tionary system admits a unique classical solution, in Achdou, Camilli & Capuzzo Dolcetta (2013) the convergence of a dis- cretized SMFG to the unique solution to the stationary sys- tem as h → 0 is proved (Lp some p < 2). To solve the discretized system, Newton’s method can be used (see Achdou, Camilli & Capuzzo Dolcetta, 2013 and Cacace & Camilli, 2016) if the initial guess is close enough to the solution. The performance of Newton’s method depends heavily on the values of ν: for small values of ν the convergence is much slower and cannot be guaranteed in general since mh can become negative.

LAWOC 2018 5/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Motivation: Discretized SMFG is FOC of (Ph)2

Discrete optimization problem (Ph) inf

(m,w)∈Mh×Wh

h2

Nh−1

• i,j=1
• ˆ

b(mi,j, wi,j) + F(xi,j, mi,j)

• s.t.
• −ν(∆hm)i,j + (divhw)i,j = 0,

∀ 0 ≤ i, j, ≤ Nh − 1 h2

i,j mi,j = 1.

h > 0, Nh = 1/h, Mh = RNh×Nh, Wh = R4(Nh×Nh). ∆h : Mh → Mh, divh : Wh → Mh are linear (A,B) Define

ˆ b: (m,w)→

    

|w|2 2 m ,

if m>0, w∈K, 0, if (m,w)=(0,0), +∞,

• therwise.

F : (x,m)→ m V (x,m′)dm′.

K := R+ × R− × R+ × R− (w = −mDhu)

2Joint work with F.J. Silva (U. Limoges) y D. Kalise (I. College) LAWOC 2018 6/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Motivation: (Ph)’s structure

Assume V (x, ·) increasing (F(x, ·) convex). f : m →

i,j F(xi,j, mi,j) it is convex and smooth.

g: (m, w) →

i,j ˆ

b(mi,j, wi,j) is convex, l.s.c., and nonsmooth. We recall −∆h = A and divh = B. Reformulation of (Ph) (P) inf

(m,w)∈Mh×Wh

f(m) + g(m, w) s.t. L(m, w) = (0, 1), where L := A B h21⊤

• LAWOC 2018

7/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Convex non-differentiable optimization problem

Problem (P) minimize

x∈RN

f(x) + g(x) + h(Lx). f : RN → R is differentiable and ∇f is β−1−Lipschitz. g ∈ Γ0(RN) and h ∈ Γ0(RM) (l.s.c. convex proper). L is a M × N real matrix. The set of solutions is nonempty. ri(domh) ∩ L(ri(domg)) = ∅. Important case: h = ι{b} : x →

• 0,

if x = b +∞,

• therwise,, b ∈ RM

minimize

Lx=b

f(x) + g(x).

LAWOC 2018 8/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Classic approach: ADMM

An equivalent formulation is min

Lx=y f(x) + g(x) + h(y)

from which we define the Augmented Lagrangian (γ > 0): Lγ(x, y, u) = f(x) + g(x) + h(y) + u · (Lx − y) + γ

2Lx − y2.

Under qualification conditions x solves (P) iff (x, Lx, u) is a saddle point of Lγ. From an alternating minimization-maximization method we obtain the classical Alternating Direction method of Multipliers (ADMM) (Gabay-Mercier 80’s):

LAWOC 2018 9/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Classic approach: ADMM

xk+1 = argminxLγ(x, yk, uk) yk+1 = argminyLγ(xk+1, y, uk) uk+1 = uk + γ(Lxk+1 − yk+1).

LAWOC 2018 10/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Classic approach: ADMM

xk+1 = argminx

• f(x) + g(x) + uk · Lx + γ

2Lx − yk2 yk+1 = argminy

• h(y) − uk · y + γ

2Lxk+1 − y2 uk+1 = uk + γ(Lxk+1 − yk+1).

LAWOC 2018 10/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Classic approach: ADMM

xk+1 = argminx

• f(x) + g(x) + uk · Lx + γ

2Lx − yk2 yk+1 = argminy

• h(y) − uk · y + γ

2Lxk+1 − y2 uk+1 = uk + γ(Lxk+1 − yk+1). In the case when h = ι{b} for some b ∈ RM, we have xk+1 = argminx

• f(x) + g(x) + uk · Lx + γ

2Lx − b2 uk+1 = uk + γ(Lxk+1 − b).

LAWOC 2018 10/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Drawbacks ADMM

The primal iterates (xk)k∈N do not satisfy the constraints (Lxk = b). Moreover, the first step it is not easy in general (involves L and f + g). It can be solved efficiently only in specific instances: f + g quadratic, L⊤L = αId. Idea: Try to split the influence of L, f and g in the first step.

LAWOC 2018 11/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Drawbacks ADMM

The primal iterates (xk)k∈N do not satisfy the constraints (Lxk = b). Moreover, the first step it is not easy in general (involves L and f + g). It can be solved efficiently only in specific instances: f + g quadratic, L⊤L = αId. Idea: Try to split the influence of L, f and g in the first step. Given x ∈ RN, proxfx is the unique solution to minimize

y∈RN

f(y) + 1 2x − y2. Several functions f have an explicit or efficently computable proxf. Examples: · 1, ιC (proxιC = PC), dC, etc...

LAWOC 2018 11/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Other approaches

Problem (P) min

x∈RN f(x) + g(x) + h(Lx).

Combettes-Pesquet (2012) Let 0 < γ < (L + β)−1, x0 ∈ RN and u0 ∈ RM and iterate       pk

1 = proxγg(xk − γ(∇f(xk) + L⊤uk))

pk

2 = proxγh∗(uk + γLxk)

xk+1 = pk

1 − γ(L⊤pk 2 + ∇f(pk 1) − L⊤uk − ∇f(xk))

uk+1 = pk

2 + γ(Lpk 1 − Lxk).

In the case f = 0, is the method proposed in BA-Combettes (2011).

LAWOC 2018 12/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Other approaches

Problem (P) case h = ι{b} min

Lx=b f(x) + g(x).

Combettes-Pesquet (2012) Let 0 < γ < (L + β)−1, x0 ∈ RN and u0 ∈ RM and iterate       pk

1 = proxγg(xk − γ(∇f(xk) + L⊤uk))

pk

2 = uk + γ

• Lxk − b
• xk+1 = pk

1 − γ(L⊤pk 2 + ∇f(pk 1) − L⊤uk − ∇f(xk))

uk+1 = pk

2 + γ(Lpk 1 − Lxk).

proxγh∗ = Id − γproxh/γ ◦ (Id/γ) = Id − γb. Also the influences of L, f and g have been split, but primal iterates do not satisfy the constraints. The method does not exploit cocoercivity of ∇f.

LAWOC 2018 12/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Other approaches

Problem (P) min

x∈RN f(x) + g(x) + h(Lx).

Condat-V˜ u (2013) x0 = ¯ x0 ∈ RN and u0 ∈ RM, τ, γ > 0 such that τγL2 < 1 − τ

2β

    uk+1 = proxγh∗(uk + γL¯ xk) xk+1 = proxτg(xk − τ(∇f(xk) + L⊤uk+1)) ¯ xk+1 = 2xk+1 − xk. The case f = 0, the method was first proposed by Chambolle-Pock (2011).

LAWOC 2018 13/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Other approaches

Problem (P) case h = ι{b} min

Lx=b f(x) + g(x).

Condat-V˜ u (2013) x0 = ¯ x0 ∈ RN and u0 ∈ RM, τ, γ > 0 such that τγL2 < 1 − τ

2β

    uk+1 = uk + γ

• L¯

xk+1 − b

• xk+1 = proxτg(xk − τ(∇f(xk) + L⊤uk+1))

¯ xk+1 = 2xk+1 − xk. Now the influences of L, f and g have been split, but the primal iterates do not satisfy the constraints. Same with

• ther Aug. Lagrangian approach proposed by Chen &

Teboulle (1994).

LAWOC 2018 13/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Case h = ι{b} and PL−1b is computable...

Suppose that it is possible to compute PL−1b = proxh◦L. If LL⊤ is invertible, PL−1bx = x − L⊤(LL⊤)−1(Lx − b). In this case we can avoid splitting L from h and use several methods for solving optimization problems without involving linear operators. In all these methods, primal iterates satisfy the constraints. But, in several cases LL⊤ is not invertible or it is bad

• conditioned. In those cases...

LAWOC 2018 14/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Goal of this talk

Provide an algorithm which can ensure that primal iterates satisfy some of the constraints by adding a projection. Provide some numerical experiences showing the advantage

• f projecting (SMFG).

Study acceleration schemes in the presence of strong convexity. Explore the extension to more general optimization problems...

LAWOC 2018 15/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Goal of this talk

Provide an algorithm which can ensure that primal iterates satisfy some of the constraints by adding a projection. Provide some numerical experiences showing the advantage

• f projecting (SMFG).

Study acceleration schemes in the presence of strong convexity. Explore the extension to more general optimization problems... ... and to monotone inclusions.

LAWOC 2018 15/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

1 Motivation Stationary Mean Field Games 2 Algorithm and convergence 3 Numerical experiences Stationary Mean Field Games Numerical simulation 4 Extensions and acceleration

LAWOC 2018 16/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Projected primal-dual splitting

Let C be a nonempty closed convex set such that PC is easy to compute. Problem (P) min

x∈RN f(x) + g(x) + h(Lx).

Condat-V˜ u (2013) x0 = ¯ x0 ∈ RN and u0 ∈ RM, τ, γ > 0 such that τγL2 < 1 − τ

2β

      uk+1 = proxγh∗(uk + γL¯ xk) pk+1 = proxτg(xk − τ(∇f(xk) + L⊤uk+1)) xk+1 = pk+1 ¯ xk+1 = xk+1 + pk+1 − xk. Then, xk → x solution to (P)

LAWOC 2018 17/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Projected primal-dual splitting

Let C be a nonempty closed convex set such that PC is easy to compute. Problem (P) with apriori information C (Q) find ˆ x ∈ C ∩ arg min

x∈RN f(x) + g(x) + h(Lx) = ∅.

Projected primal-dual splitting x0 = ¯ x0 ∈ RN and u0 ∈ RM, τ, γ > 0 such that τγL2 < 1 − τ

2β

      uk+1 = proxγh∗(uk + γL¯ xk) pk+1 = proxτg(xk − τ(∇f(xk) + L⊤uk+1)) xk+1 = PCpk+1 ¯ xk+1 = xk+1 + pk+1 − xk. Then, (xk)k∈N ⊂ C and xk → ˆ x solution to (Q).

LAWOC 2018 17/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Projected primal-dual splitting: case h = ι{b}

In the case when h = ι{b}, suppose that M = r + s, L: x → (Rx, Sx), where R and S are r × N and s × N real matrices, and b = (c, d), where c ∈ Rr and d = Rs. Suppose we can project onto C := R−1c ⊂ L−1b. Problem (P) min

Lx=b f(x) + g(x).

Condat-V˜ u (2013) x0 = ¯ x0 ∈ RN and u0 ∈ RM, τ, γ > 0 such that τγL2 < 1 − τ

2β

    uk+1 = uk + γ(L¯ xk − b) xk+1 = proxτg(xk − τ(∇f(xk) + L⊤uk+1)) ¯ xk+1 = 2xk+1 − xk. Then, xk → x solution to (P)

LAWOC 2018 18/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Projected primal-dual splitting: case h = ι{b}

In the case when h = ι{b}, suppose that M = r + s, L: x → (Rx, Sx), where R and S are r × N and s × N real matrices, and b = (c, d), where c ∈ Rr and d = Rs. Suppose we can project onto C := R−1c ⊂ L−1b. Problem (P) with C = R−1c (Q) find ˆ x ∈ R−1c ∩ arg min

Rx=c Sx=d

f(x) + g(x). Projected primal-dual splitting x0 = ¯ x0 ∈ RN and u0 ∈ RM, τ, γ > 0 such that τγL2 < 1 − τ

2β

      uk+1 = uk + γ(L¯ xk − b) pk+1 = proxτg(xk − τ(∇f(xk) + L⊤uk+1)) xk+1 = pk+1 − R⊤(RR⊤)−1(Rpk+1 − c) ¯ xk+1 = xk+1 + pk+1 − xk. Then, (xk)k∈N ⊂ C and xk → ˆ x solution to (Q).

LAWOC 2018 18/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

1 Motivation Stationary Mean Field Games 2 Algorithm and convergence 3 Numerical experiences Stationary Mean Field Games Numerical simulation 4 Extensions and acceleration

LAWOC 2018 19/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Numerics 2: SMFG

By setting x = (m, w), b = (0, 1), (Ph) inf

Lx=b f(x) + g(x)

L = A B h21⊤

• (Ph) is equivalent to

(Ph) inf

x∈RN f(x) + g(x) + h(Mx)

proxγ(f+g) is easy to compute: solve cubic equation ∀i, j. (Split)

• h = ι{(0,1)}

M = L

• r

(Unsplit)

• h = ιL−1(0,1)

M = Id.

LAWOC 2018 20/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Numerical 1: SMFG

“Unsplit” decomposition: ψ = ι{L−1(0,1)} and for computing proxγψ∗ or proxγψ we need to compute (LL∗)−1 (more precisely (ν2AA∗ + BB∗)−1). Depending on the parameter ν, this matrix can be very bad conditioned and difficult to invert. “Split” decomposition: ψ = ι{(0,1)} and proxγψ∗ = Id − γ(0, 1). Then, all previous methods include a Lagrange multiplier step of the form uk+1 = uk + γ(Lxk − b) The primal iterates (xk)k∈N are not feasible ! Numerically the methods were very slow... ...we include projection onto density constraint and method improved.

LAWOC 2018 21/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Numerical 1: SMFG

We consider the first-order stationary MFG system (Almulla-Ferreira-Gomes, 2015) 1 2|∇u|2 − λ = log m − sin(2πx) − sin(2πy), div(m∇u) = 0,

• T2 mdx = 1,
• T2 udx = 0,

with explicit solution u(x, y) = 0, m(x, y) = esin(2πx)+sin(2πy)−λ , λ = log

• T2 esin(2πx)+sin(2πy)dxdy
• .

LAWOC 2018 22/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

1 1 2 3 Exact mass m(x, y) Test 1 x 4 y 0.5 0.5 5 1 103 104 DoF 10-5 10-4 10-3 10-2 10-1 L2 error MS-SP PCPM-SP CP-SP MS-U PCPM-U CP-U O(DoF)

103 104 DoF 100 101 102 103 104 CPU time MS-SP PCPM-SP CP-SP MS-U PCPM-U CP-U

10-5 10-4 10-3 10-2 10-1 L2 error 100 101 102 103 104 CPU time MS-SP PCPM-SP CP-SP MS-U PCPM-U CP-U

LAWOC 2018 23/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Numerical 2

min

Rx=c Sx=d

x1 Is the case f = 0, h = ι{(c,d)} and g = · 1. Dimensions: N = 1000, s = 100 and r ∈ {1, 10, 30}. We consider 20 random realizations of matrices R, S and vectors c and d and we measure the relative error

Rk =

• uk+1−uk2+xk+1−xk2

uk2+xk2

. r = 1, s = 100 e = 10−4 e = 5 · 10−5 e = 10−5 iter time (s) iter time (s) iter time (s) PCP 9265 22.28 14570 37.02 46191 116.26 CP 9732 23.04 15718 39.21 50544 125.49 %improv. 4.8 3.3 7.3 5.6 8.6 7.4 Table: Average time and number of iterations when m = 1 for

• btaining Rk < e.

LAWOC 2018 24/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Numerical 2

min

Rx=c Sx=d

x1 Is the case f = 0, h = ι{(c,d)} and g = · 1. Dimensions: N = 1000, s = 100 and r ∈ {1, 10, 30}. We consider 20 random realizations of matrices R, S and vectors c and d and we measure the relative error

Rk =

• uk+1−uk2+xk+1−xk2

uk2+xk2

. r = 10, s = 100 e = 10−4 e = 5 · 10−5 e = 10−5 iter time (s) iter time (s) iter time (s) PCP 6865 18.65 10229 27.86 22855 65.05 CP 9280 23.72 16033 39.13 49526 129.78 %improv. 26.0 21.4 36.2 28.8 53.9 49.9 Table: Average time and number of iterations when m = 10 for

• btaining Rk < e.

LAWOC 2018 24/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Numerical 2

min

Rx=c Sx=d

x1 Is the case f = 0, h = ι{(c,d)} and g = · 1. Dimensions: N = 1000, s = 100 and r ∈ {1, 10, 30}. We consider 20 random realizations of matrices R, S and vectors c and d and we measure the relative error

Rk =

• uk+1−uk2+xk+1−xk2

uk2+xk2

. r = 30, s = 100 e = 10−4 e = 5 · 10−5 e = 10−5 iter time (s) iter time (s) iter time (s) PCP 5146 7.68 7143 10.67 13421 19.70 CP 9941 12.93 16438 21.37 50841 64.23 %improv. 48.2 40.6 56.5 50.1 73.6 69.3 Table: Average time and number of iterations when m = 30 for

• btaining Rk < e.

LAWOC 2018 24/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

1 Motivation Stationary Mean Field Games 2 Algorithm and convergence 3 Numerical experiences Stationary Mean Field Games Numerical simulation 4 Extensions and acceleration

LAWOC 2018 25/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Chambolle-Pock’s acceleration

The algorithm in Chambolle-Pock (2011) has a worst-case convergence rate of O(1/k) (G¨ uler, 1991). In the strongly convex case they prove: Problem (P) when g and/or h∗ is/are strongly convex Suppose that g and h∗ are ρ− and χ− strongly convex, with ρ ≥ 0 and χ ≥ 0. min

x∈RN g(x) + h(Lx)

Chambolle-Pock (2011) x0 ∈ RN and u0 ∈ RM, (τk)k∈N and (γk)k∈N are strictly positive sequences such that τ0γ0 = 1/L2 and θk ∈]0, 1]. (∀k ∈ N)     uk+1 = proxγkh∗(uk + γkL¯ xk) xk+1 = proxτkg(xk − τkL∗uk+1) ¯ xk+1 = xk+1 + θk(xk+1 − xk).

LAWOC 2018 26/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Chambolle-Pock’s acceleration

Suppose that ρ > 0 and χ = 0. If we set (∀k ∈ N) θk = 1 √1 + 2ρτk , τk+1 = θkτk, γk+1 = γk/θk, (1) we have a convergence rate of O(1/k2) on iterates. Suppose that ρ > 0 and χ > 0 and define µ = 2√ρχ L . (2) If we set θk ≡ θ ∈ ((1 + µ)−1, 1], τk ≡ τ and γk ≡ γ with τ = µ 2ρ and γ = µ 2χ, (3) we obtain linear convergence O(ωk/2) with ω = (1 + θ)/(2 + µ) < 1.

LAWOC 2018 27/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Accelerated projected primal-dual splitting

g and h∗ are ρ− and χ− strongly convex, with ρ ≥ 0 and χ ≥ 0. Problem (P) when g and/or h∗ is/are strongly convex min

x∈RN g(x) + h(Lx)

Chambolle-Pock (2011) x0 = ¯ x0 ∈ RN and u0 ∈ RM, (τk)k∈N and (γk)k∈N are strictly positive sequences such that τ0γ0L2 = 1 and θk ∈]0, 1]. (∀k ∈ N)     uk+1 = proxγkh∗(uk + γkL¯ xk) xk+1 = proxτkg(xk − τkL∗uk+1) ¯ xk+1 = xk+1 + θk(xk+1 − xk).

LAWOC 2018 28/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Accelerated projected primal-dual splitting

g and h∗ are ρ− and χ− strongly convex, ρ ≥ 0,χ ≥ 0. f and ℓ∗ are differentiable with β−1 and δ−1 Lipschitz gradients. Problem (Q) find ˆ x ∈ C ∩ arg minx∈RN f(x) + g(x) + h ℓ(Lx) Accelerated projected primal-dual x0 = ¯ x0 ∈ RN and u0 ∈ RM, (τk)k∈N ⊂]0, 2β[ and (γk)k∈N ⊂]0, 2δ[ are such that τ0γ0L2 = (1 − τ0

2β)(1 − γ0 2δ) and

θk ∈]0, 1]. (∀k ∈ N)       uk+1 = proxγkh∗(uk + γk(L¯ xk − ∇ℓ∗(uk))) pk+1 = proxτkg(xk − τk(L∗uk+1 + ∇f(xk))) xk+1 = PCpk+1 ¯ xk+1 = xk+1 + θk(pk+1 − xk).

LAWOC 2018 28/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Projected primal-dual acceleration

Suppose that ρ > 0 and χ = 0. If we set (∀k ∈ N) θk = 1 √1 + 2ρτk , τk+1 = θkτk, γk+1 = γk/θk, (4) we have a convergence rate of O(1/k2) on iterates. Suppose that ρ > 0 and χ > 0 and define µ = 2√ρχ L . (5) If we set θk ≡ θ ∈ ((1 + µ)−1, 1], τk ≡ τ and γk ≡ γ with τ = µ 2ρ and γ = µ 2χ, (6) we obtain linear convergence O(ωk/2) with ω = (1 + θ)/(2 + µ) < 1.

LAWOC 2018 29/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Projected primal-dual acceleration

Suppose that ρ > 0, δ = +∞ (ℓ∗ = 0) and χ = 0. If we set (∀k ∈ N) θk = 1 √1 + 2ρτk , τk+1 = θkτk, γk+1 = γk/θk, (4) we have a convergence rate of O(1/k2) on iterates. Suppose that ρ > 0 and χ > 0 and define µ = 2√ρχ L and α = min

• µρ

ρ + µ

4β

, µχ χ + µ

4δ

• .

(5) If we set θk ≡ θ ∈ ((1 + α)−1, 1], τk ≡ τ and γk ≡ γ with τ = 2βµ µ + 4βρ and γ = 2µδ µ + 4δχ, (6) we obtain linear convergence O(ωk/2) with ω = (1 + θ)/(2 + α) < 1.

LAWOC 2018 29/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

Further contributions...

We work in general real Hilbert spaces. We include an extension to monotone inclusions with similar acceleration in the strongly monotone case. Monotone inclusion find x ∈ Fix T such that 0 ∈ Ax + B D(x) + Cx, A and B are maximally monotone. C and D−1 are cocoercive.

LAWOC 2018 30/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa

Motivation Convergence Numerics Extensions

References

• L. M. Brice˜

no-Arias and S. L´

• pez Rivera, Projected primal-dual splitting for

solving constrained composite convex optimization: Acceleration and exten- sions, https://arxiv.org/abs/1805.11687.

• L. M. Brice˜

no-Arias, D. Kalise, and F. J. Silva, Proximal methods for sta- tionary Mean Field Games with local couplings, SIAM J. Control Optim. 56, 801–836, 2018.

• L. M. Brice˜

no-Arias and P. L. Combettes, A monotone + skew splitting model for composite monotone inclusions in duality, SIAM J. Optim. 21, 1230–1250, 2011.

• A. Chambolle and T. Pock, A first order primal dual algorithm for convex

problems with applications to imaging, J. Math. Imaging Vis. 40, 120–145, 2011.

• L. Condat, A primal-dual splitting method for convex optimization involving

lipschitzian, proximable and linear composite terms, J. Optim. Theory Appl. 158, 460–479, 2013.

• G. Chen and M. Teboulle, A proximal-based decomposition method for convex

minimization problems, Math. Programming Ser. A 64, 81–101, 1994. J.-M. Lasry and P.-L. Lions. Jeux ` a champ moyen I. Le cas stationnaire. C.

• R. Math. Acad. Sci. Paris, 343:619–625, 2006.

B.C. V˜ u, A splitting algorithm for dual monotone inclusions involving coco- ercive operators. Adv. Comput. Math. 38, 667-681, 2013.

LAWOC 2018 31/ 31

• L. M. Brice˜

no-AriasUniversidad T´ ecnica Federico Santa Mar´ ıa