A Parallel Forward-backward Splitting Method for Multiterm Composite - - PowerPoint PPT Presentation

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

Maicon Marques Alves Joint work with Samara C. Lima Federal University of Santa Catarina, Florian´

• polis.

DIMACS Workshop on ADMM and Proximal Splitting Methods in Optimization DIMACS, June 11–13, 2018

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

Forward-backward splitting

In a real Hilbert space H, min

x∈H {f(x) + ϕ(x)}

where

◮ f : H → R is convex and differentiable with a L-Lipschitz

continuous gradient: ∇f(x) − ∇f(y) ≤ Lx − y ∀x, y ∈ H.

◮ ϕ : H → R ∪ {∞} is proper, convex and closed with an easily

computable proximity operator/resolvent: proxλϕ(x) := arg min

y∈H

• ϕ(y) + 1

2λy − x2

• .

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

Forward-backward splitting

◮ ϕ = δC

⇒ proxλϕ = PC.

◮ The forward-backward/proximal gradient method:

x+ = proxλϕ backward (x − λ∇f(x)

• forward

), λ > 0.

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

The problem

In a real Hilbert space H, min

x∈H m

• i=1

{fi(x) + ϕi(x)} where, for all i = 1, . . . , m,

◮ fi : H → R is convex and differentiable with a L-Lipschitz

continuous gradient.

◮ ϕi : H → R ∪ {∞} is proper, convex and closed with an easily

computable proximity operator/resolvent.

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

The problem

◮ Interesting applications (see, e.g., N. He, A. Juditsky and A.

Nemirovski, 2015; E. Ryu and W. Yin, 2017).

◮ Under standard regularity conditions, it is equivalent to

0 ∈

m

• i=1

(∇fi + ∂ϕi

• =:Ti

)(x).

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

The proximal point method

◮ Monotone inclusion problem:

find z ∈ H such that 0 ∈ T(z) where T : H ⇒ H is maximal monotone.

◮ Rockafellar (1976):

zk − (λkT + I)−1zk−1 ≤ ek,

∞

• k=1

ek < ∞.

◮ Rockafellar (1976): (weak) convergence and applications.

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

The hybrid proximal extragradient (HPE) method

◮ z+ = (λT + I)−1z ⇐

⇒ v ∈ T(z+), λv + z+ − z = 0.

◮ ε–enlargements (Burachik-Sagastiz´

abal-Svaiter): T ε(z) := {v ∈ H | z − z′, v − v′ ≥ −ε ∀v′ ∈ T(z′)}.

◮ T(z) ⊂ T ε(z). ◮ ∂εf(z) ⊂ (∂f)ε(z).

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

The hybrid proximal extragradient (HPE) method

Problem: 0 ∈ T(z). Hybrid proximal extragradient (HPE) method (Solodov and Svaiter, 1999): (0) Let z0 ∈ H, σ ∈ [0, 1) and set k = 1. (1) Find (˜ zk, vk, εk) ∈ H × H × R+ and λk > 0 such that vk ∈ T εk(˜ zk), λkvk + ˜ zk − zk−12 + 2λkεk ≤ σ2˜ zk − zk−12. (2) Define zk = zk−1 − λkvk, set k ← k + 1 and go to Step 1. End

◮ σ = 0

⇒ zk = (λkT + I)−1zk−1.

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

The hybrid proximal extragradient (HPE) method

◮ Some special instances: Forward-backward, Tseng’s modified

forward-backward, Korpolevich, ADMM.

◮ Termination criterion (Monteiro-Svaiter, 2010): Given a

precision ρ > 0, find (z, v) and ε ≥ 0 such that v ∈ T ε(z), max{v, ε} ≤ ρ.

◮ Monteiro and Svaiter (2010): Iteration-complexity; global

O(1/ √ k) pointwise and O(1/k) ergodic convergence rates.

◮ A., Monteiro and Svaiter (2016): Regularized HPE method

with O(ρ−1 log(ρ−1)) pointwise iteration-complexity.

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

The hybrid proximal extragradient (HPE) method

Theorem (Monteiro and Svaiter, 2010) (a) For any k ≥ 1, there exists i ∈ {1, . . . , k} such that vi ∈ T εi(˜ zi), vi ≤ d0 λ √ k

• 1 + σ

1 − σ, εi ≤ σ2d2 2(1 − σ2)λk . (b) For any k ≥ 1, va

k ∈ T εa

k(˜

za

k),

va

k ≤ 2d0

λk , εa

k ≤ 2(1 + σ/

√ 1 − σ2)d2 λk .

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

The partial inverse method of Spingarn

◮ Spingarn (1983): find x, y ∈ H such that

x ∈ V, y ∈ V ⊥ and y ∈ T(x) where V is a closed subspace of H and T : H ⇒ H is maximal monotone.

◮ V = H

⇒ 0 ∈ T(x).

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

The partial inverse method of Spingarn

◮ Spingarn (1983):

0 ∈

m

• i=1

Ti(x).

◮ It is equivalent to

yi ∈ Ti(xi),

m

• i=1

yi = 0, x1 = x2 = · · · = xm.

◮ In this case,

V = {(x1, x2, . . . , xm) ∈ Hm | x1 = x2 = · · · = xm}, V ⊥ = {(y1, y2, . . . , ym) ∈ Hm | y1 + y2 + · · · + ym = 0}, T : Hm ⇒ Hm, (x1, x2, . . . , xm) → T1(x1) × T2(x2) × Tm(xm).

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

The partial inverse method of Spingarn

Problem: x ∈ V, y ∈ V ⊥, y ∈ T(x). Partial inverse method: (0) Let x0 ∈ V , y0 ∈ V ⊥ and set k = 1. (1) Find ˜ xk, ˜ yk ∈ H and λk > 0 such that 1 λk PV (˜ yk) + PV ⊥(˜ yk) ∈ T

• PV (˜

xk) + 1 λk PV ⊥(˜ xk)

• ,

˜ yk + ˜ xk = xk−1 + yk−1. (2) Define xk = PV (˜ xk), yk = PV ⊥(˜ yk), set k ← k + 1 and go to Step 1. End

◮ λk = 1

⇒ ˜ xk = (T + I)−1(xk−1 + yk−1).

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

The partial inverse method of Spingarn

◮ Spingarn (1983): The partial inverse of T w.r.t. V is defined

as TV : H ⇒ H, v ∈ TV (z) ⇐ ⇒ PV (v) + PV ⊥(z) ∈ T

• PV (z) + PV ⊥(v)
• .

◮ In particular,

0 ∈ TV (z) ⇐ ⇒ PV ⊥(z)

y

∈ T(PV (z)

x

) and xk + yk

zk

= (λkTV + I)−1(xk−1 + yk−1

• zk−1

).

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

Spingarn’s operator splitting

Problem: 0 ∈ m

i=1 Ti(x).

Spingarn’s operator splitting: (0) Let (x0, y1,0, . . . , ym,0) ∈ Hm+1 be such that y1,0 + · · · + ym,0 = 0 and set k = 1. (1) For each i = 1, . . . , m, find ˜ xi, k, ˜ yi, k ∈ H such that ˜ yi, k ∈ Ti(˜ xi, k), ˜ yi, k + ˜ xi, k = xk−1 + yi,k−1. (2) Define xk = 1 m

m

• i=1

˜ xi, k, yi, k = yi, k−1 + xk − ˜ xi, k for i = 1, . . . , m, set k ← k + 1 and go to step 1.

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

SPDG (Mahey, Oualibouch and Tao, 1995)

Problem: x ∈ V, y ∈ V ⊥, y ∈ T(x). (0) Let x0 ∈ V , y0 ∈ V ⊥, γ > 0 be given and set k = 1. (1) Find ˜ xk, ˜ yk ∈ H such that ˜ yk ∈ T(˜ xk), γ˜ yk + ˜ xk = xk−1 + γyk−1. (2) Define xk = PV (˜ xk), yk = PV ⊥(˜ yk), set k ← k + 1 and go to step 1. End

◮ Fixed point theory: (xk, γyk) = J ((xk−1, γyk−1)),

J firmly nonexp.

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

SPDG (Mahey, Oualibouch and Tao, 1995)

◮ A. and Lima (2017):

xk + γyk = ((γT)V + I)−1(xk−1 + γyk−1) and T strongly monotone and Lipschitz ⇒ TV strongly monotone.

◮ Rockafellar (2017): Progressive decoupling algorithm for

monotone variational inequalities.

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

Relative-error inexact Spingarn’s operator splitting (A. and Lima, 2017)

Problem: 0 ∈ T1(x) + T2(x) + · · · + Tm(x). (0) Let (x0, y1,0, . . . , ym,0) ∈ Hm+1 be such that y1,0 + · · · + ym,0 = 0 and σ ∈ [0, 1[ be given and set k = 1. (1) For each i = 1, . . . , m, find ˜ xi, k, ˜ yi, k ∈ H and εi, k ≥ 0 such that ˜ yi, k ∈ T εi, k

i

(˜ xi, k), γ˜ yi, k + ˜ xi, k = xk−1 + γyi,k−1, 2γεi, k ≤ σ2˜ xi, k − xk−12. (2) Define xk = 1 m

m

• i=1

˜ xi, k, yi, k = yi, k−1 + 1 γ (xk − ˜ xi, k) for i = 1, . . . , m, set k ← k + 1 and go to step 1.

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

Relative-error inexact Spingarn’s operator splitting

◮ It is a special instance of the HPE method applied to solve

0 ∈ (γT)V (z) where V = {(x1, x2, . . . , xm) ∈ Hm | x1 = x2 = · · · = xm}, V ⊥ = {(y1, y2, . . . , ym) ∈ Hm | y1 + y2 + · · · + ym = 0}, T : Hm ⇒ Hm, (x1, x2, . . . , xm) → T1(x1) × T2(x2) × Tm(xm).

◮ Similar approach by R. Burachik, C. Sagastiz´

abal and S. Scheimberg (2006): convergence if σ < 1/ √ 5 ≈ 0.447.

◮ σ = 0.99 seems to work well for some HPE-based algorithms

(Monteiro-Ortiz-Svaiter, 2014; Eckstein-Yao, 2017).

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

Parallel forward-backward (A. and Lima, 2017)

◮ Problem:

min

x∈H m

• i=1

{fi(x) + ϕi(x)}

◮ Monotone inclusion problem:

0 ∈

m

• i=1

Ti(x), Ti := ∇fi + ∂ϕi ∀i = 1, . . . , m.

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

Parallel forward-backward

(0) Let (x0, y1,0, . . . , ym,0) ∈ Hm+1 such that y1,0 + · · · + ym,0 = 0 and σ ∈]0, 1[ be given and set γ = σ2/L and k = 1. (1) For each i = 1, . . . , m, compute ˜ xi, k = proxγϕi

• xk−1 − γ∇fi(xk−1) + γyi, k−1
• .

(2) Define xk = 1 m

m

• i=1

˜ xi, k, yi, k = yi, k−1 + 1 γ (xk − ˜ xi, k) for i = 1, . . . , m, set k ← k + 1 and go to step 1.

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

Parallel forward-backward

◮ It is closely related to the Proximal-proximal-gradient (PPG)

method, by E. Ryu and W. Yin (2017).

◮ PPG: distributed computing, stochastic PPG, applications,

etc.

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

Parallel forward-backward

Theorem (pointwise global convergence, A. and Lima, 2017) For all k ≥ 1, there exists j ∈ {1, . . . , k} such that ˜ yi,j ∈

• ∂εi,jfi + ∂ϕi
• (˜

xi,j) ∀i = 1, . . . , m,

• m
• i=1

˜ yi,j

• ≤

√m L d0 σ2√ k

• 1 + σ

1 − σ, ˜ xi,j − ˜ xℓ,j ≤ 2 d0 √ k

• 1 + σ

1 − σ ∀i, ℓ = 1, . . . , m,

m

• i=1

εi,j ≤ L d 2 2(1 − σ2)k .

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

Parallel forward-backward

Theorem (ergodic global convergence, A. and Lima, 2017) For all k ≥ 1, ˜ y a

i, k ∈

• ∂ε′ a

i, kfi + ∂(ε a i, k−ε′ a i, k)ϕi

• (˜

xa

i, k)

∀i = 1, . . . , m,

• m
• i=1

˜ ya

i,k

• ≤ 2√m L d0

σ2k , ˜ xa

i, k − ˜

xa

ℓ,k ≤ 4d0

k ∀i, ℓ = 1, . . . , m,

m

• i=1

ε a

i, k ≤ 2(1 + σ/

√ 1 − σ2) L d 2 σ2k .

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

Some references

• M. Marques Alves, R. D. C. Monteiro, and B. F. Svaiter.

Regularized HPE-type methods for solving monotone inclusions with improved pointwise iteration-complexity bounds. SIAM J. Optim., 26(4):2730–2743, 2016.

• M. Marques Alves and S. C. Lima.

An inexact Spingarn’s partial inverse method with applications to operator splitting and composite optimization. JOTA, 175(3):818-847, 2017.

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization

Thank you for your attention! maicon.alves@ufsc.br

A Parallel Forward-backward Splitting Method for Multiterm Composite Convex Optimization