Introductory Course on Non-smooth Optimisation Lecture 04 - - - PowerPoint PPT Presentation

Introductory Course on Non-smooth Optimisation

Lecture 04 - Backward–Backward splitting Jingwei Liang

Department of Applied Mathematics and Theoretical Physics

Table of contents

1

Problem

2

Forward–Backward splitting revisit

3

MAP continue

4

Backward–Backward splitting

5

Numerical experiments

Monotone inclusion pronblem Problem

Let B : Rn → Rn be β-cocoercive for some β > 0, s > 1 be a positive integer, such that for each i ∈ {1, ..., s}: Ai : Rn ⇒ Rn is maximal monotone. Consider the problem Find x ∈ Rn such that 0 ∈ B(x) + s

i=1Ai(x).

Ai can be composed with linear mapping, e.g. L∗ ◦ A ◦ L. Even if the resolvents of B and each Ai are simple, the resolvent of B +

i Ai in most cases is

not solvable. Use the properties of operators and structure of problem to derive operator splitting schemes.

Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Outline

1 Problem 2 Forward–Backward splitting revisit 3 MAP continue 4 Backward–Backward splitting 5 Numerical experiments

Monotone inclusion problem Monotone inclusion

Find x ∈ Rn such that 0 ∈ A(x) + B(x). Assump Assumptions tions A : Rn ⇒ Rn is maximal monotone. B : Rn → Rn is β-cocoersive. zer(A + B) = ∅. Characterisation of minimiser: γ > 0 x⋆ − γB(x⋆) ∈ x⋆ + γA(x⋆) ⇔ x⋆ = JγA ◦ (Id − γB)(x⋆). Ex Example ample Let R ∈ Γ0 and F ∈ C1

L,

min

x∈Rn R(x) + F(x). Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Forward–Backward splitting

Fixed-point operator: γ ∈]0, 2β[ TFB = JγA ◦ (Id − γB). JγA is firmly non-expansive. Id − γB is

γ 2β -averaged non-expansive.

TFB is

2β 4β−γ -averaged non-expansive.

fix(TFB) = zer(A + B).

Forward–Backward splitting

Let γ ∈ ]0, 2β[, λk ∈ [0, 4β−γ

2β ]:

xk+1 = (1 − λk)xk + λkTFB(xk). Special case of Krasnosel’ski˘ ı-Mann iteration. Recovers proximal point algorithm when B = 0.

Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Outline

1 Problem 2 Forward–Backward splitting revisit 3 MAP continue 4 Backward–Backward splitting 5 Numerical experiments

Method of alternating projection

Let X, Y ⊆ Rn be closed convex and non-empty, such that X ∩ Y = ∅ min

x∈Rn ιX (x) + ιY(x).

Method of alternating projection (MAP)

Let x0 ∈ X: yk+1 = PY(xk), xk+1 = PX (yk+1). Fixed-point operator: xk+1 = TMAP(xk), TMAP

def

= PX ◦ PY. PX , PY are firmly non-expansive. TMAP is 2

3-averaged non-expansive.

fix(TMAP) = X ∩ Y.

Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Derive MAP

Feasibility problem is equivalent to min

x,y∈Rn ιX (x) + 1

2||x − y||2 + ιY(y).

Optimality condition 0 ∈ NY(y⋆) + y⋆ − x⋆, 0 ∈ NX (x⋆) + x⋆ − y⋆. Fixed-point characterisation y⋆ = PY(x⋆), x⋆ = PX (y⋆). Fixed-point iteration yk+1 = PY(xk), xk+1 = PX (yk+1).

Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Example: SDP feasibility SDP feasibility

Find X ∈ Sn such that X 0 and Tr(AiX) = bi, i = 1, ..., m. Two sets and projection: X = Sn

+ is the positive semidefinite cone. Let Yk = n i=1 σiuiuT i be the eigenvalue

decomposition of Yk, then PX (Yk) = n

i=1 max{0, σi}uiuT i .

Y is the affine set in Sn define by the linear inequalities, PY(Xk) = Xk − m

i=1uiAi,

where ui are found from the normal equations Gu =

• Tr(AiXk) − bi, · · · , Tr(AiXk) − bm
• , Gi,j = Tr(AiAj).

Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Convergence rate

Let X, Y be two subspaces, and assume 1 ≤ p

def

= dim(X) ≤ q

def

= dim(Y) ≤ n − 1. Principal Principal angles angles The principal angles θk ∈ [0, π

2 ], k = 1, . . . , p between X and Y are defined by,

with u0 = v0

def

= 0, and cos(θk)

def

= uk, vk = maxu, v s.t. u ∈ X, v ∈ Y, ||u|| = 1, ||v|| = 1, u, ui = v, vi = 0, i = 0, · · · , k − 1. Friedrichs Friedrichs angle angle The Friedrichs angle θF ∈]0, π

2 ] between X and Y is

cos

• θF(X, Y)

def = maxu, v s.t. u ∈ X ∩ (X ∩ Y)⊥, ||u|| = 1, v ∈ Y ∩ (X ∩ Y)⊥, ||v|| = 1.

Lemma

The Friedrichs angle is θd+1 where d

def

= dim(X ∩ Y). Moreover, θF(X, Y) > 0.

Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Convergence rate

Example X, Y are defined by X = {x : Ax = 0}, Y = {x : Bx = 0}. Projection onto subspace PX (x) = x − AT(AAT)−1Ax.

Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Convergence rate

Define diagonal matrices c = diag

• cos(θ1), · · · , cos(θp)
• ,

s = diag

• sin(θ1), · · · , sin(θp)
• .

Suppose p + q < n, then there exists orthogonal matrix U such that PX = U      Idp 0p 0q−p 0n−p−q      U∗, PY = U      c2 cs cs c2 Idq−p 0n−p−q      U∗.

Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Convergence rate

Fixed-point operator TMAP = PX ◦ PY = U      c2 cs 0p 0q−p 0n−p−q      U∗. Consider relaxation T λ

MAP = (1 − λ)Id + λTMAP

= U    (1 − λ)Idp + λc2 λcs (1 − λ)Idp (1 − λ)Idn−2p    U∗.

Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Convergence rate

Eigenvalues σ(T λ

MAP) =

• 1 − λsin2(θi)|i = 1, ..., p
• ∪ {1 − λ}.

Spectral radius ρ(T λ

MAP) = max

• 1 − λsin2(θF), |1 − λ|
• .

No relaxation ρ(TMAP) = cos2(θF). Convergence rate, C > 0 is some constant ||xk − x⋆|| = ||TMAPxk−1 − TMAPx⋆|| = ... = ||T k

MAP(x0 − x⋆)||

≤ C||TMAP||k||x0 − x⋆||.

Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Outline

1 Problem 2 Forward–Backward splitting revisit 3 MAP continue 4 Backward–Backward splitting 5 Numerical experiments

Best pair problem

When X ∩ Y = ∅, MAP returns xk, yk → x⋆ ∈ X ∩ Y.

Best pair problem

Let X, Y ⊆ Rn be closed and convex, such that X ∩ Y = ∅. Consider finding two points in X and Y such that they are the closest, that is min

x∈X,y∈Y ||x − y||.

MAP can be applied and (xk, yk) → (x⋆, y⋆) where (x⋆, y⋆) is a best pair.

Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Backward–Backward splitting

Consider Find x, y ∈ Rn such that 0 ∈ A(x) + B(y), A, B : Rn ⇒ Rn are maximal monotone. The set of solition is non-empty. There exists x⋆, y⋆ ∈ Rn and γ > 0 such that y⋆ − x⋆ ∈ γA(x⋆), x⋆ − y⋆ ∈ γB(y⋆).

Backward–Backward splitting

Let x0 ∈ Rn, γ > 0: yk+1 = JγB(xk), xk+1 = JγA(yk+1).

Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Regularised monotone inclusion

Yosida

• sida appr

approxima ximation tion

γA = 1

γ (Id − JγA).

which is γ-cocoercive. Regularised egularised monot monotone

• ne inclusion

inclusion Find x ∈ Rn such that 0 ∈ A(x) + γB(x). For

• rwar

ard–Backw d–Backwar ard split splitting ting τ ∈]0, 2γ] xk+1 = JτA ◦ (Id − τ γB)(xk). BB BB as as special special case ase of

• f FB

FB let τ = γ xk+1 = JγA ◦ (Id − γγB)(xk) = JγA ◦

• Id − γ 1

γ (Id − JγB)

• (xk)

= JγA ◦ JγB(xk).

Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Inertial BB splitting An inertial Backward–Backward splitting

Initial Initial : x0 ∈ Rn, x−1 = x0 and γ > 0, τ ∈]0, 2γ]; yk = xk + a0,k(xk − xk−1) + a1,k(xk−1 − xk−2) + · · · , xk+1 = JγA ◦ JγB(yk), λk ∈ [0, 1].

An inertial BB splitting based on Yosida approximation

Initial Initial : x0 ∈ Rn, x−1 = x0 and γ > 0; yk = xk + a0,k(xk − xk−1) + a1,k(xk−1 − xk−2) + · · · , zk = xk + b0,k(xk − xk−1) + b1,k(xk−1 − xk−2) + · · · , xk+1 = JτA ◦

• yk − τ γB(zk)
• , λk ∈ [0, 1].

Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Outline

1 Problem 2 Forward–Backward splitting revisit 3 MAP continue 4 Backward–Backward splitting 5 Numerical experiments

Numerical experiment

Feasibility problem for two subspaces: a = [−4/5, 1] and b = [−1/5, 1]

20 40 60 80 100 120 140 10-12 10-8 10-4 100 MAP Inertial MAP, a = 0.1 Inertial MAP, a = 0.2 Inertial MAP, a = 0.3 Inertial MAP, a = 0.4 Inertial MAP, a = 0.5 Inertial MAP, a = 0.6 Inertial MAP, a = 0.9

Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Reference

• S. Boyd. “Alternating projection”, lecture notes.
• H. H. Bauschke, J. Y. Bello Cruz, T. T. A. Nghia, H. M. Pha, and X. Wang. “Optimal rates of linear

convergence of relaxed alternating projections and generalized Douglas–Rachford methods for two subspaces”. Numerical Algorithms, 73(1):33–76, 2016.

• P. L. Lions and B. Mercier. “Splitting algorithms for the sum of two nonlinear operators”. SIAM

Journal on Numerical Analysis, 16(6):964–979, 1979.