Asymmetric Proximal Point Algorithms with Moving Proximal Centers - - PowerPoint PPT Presentation

▶
asymmetric proximal point algorithms with moving proximal
SMART_READER_LITE
LIVE PREVIEW

Asymmetric Proximal Point Algorithms with Moving Proximal Centers - - PowerPoint PPT Presentation

Sep 19, 2023 263 likes •713 views

Asymmetric Proximal Point Algorithms with Moving Proximal Centers Deren Han (handeren@njnu.edu.cn) School of Mathematical Sciences, Nanjing Normal University Nanjing, 210023, China. September 2, 2014 Deren Han (NJNU) Asymmetric PPA September

slide-1
SLIDE 1

Asymmetric Proximal Point Algorithms with Moving Proximal Centers

Deren Han (handeren@njnu.edu.cn)

School of Mathematical Sciences, Nanjing Normal University Nanjing, 210023, China.

September 2, 2014

Deren Han (NJNU) Asymmetric PPA September 2, 2014 1 / 26

slide-2
SLIDE 2

Outline

1

Introduction

2

APPA with Moving Proximal Centers

3

Worst-case Rate of Convergence

4

Linear Convergence

5

Two Concrete Algorithms The Saddle Point Problem Multiblock Decomposible Convex Optimization

6

Conclusion

Deren Han (NJNU) Asymmetric PPA September 2, 2014 2 / 26

slide-3
SLIDE 3

VI

(x − x∗)TF(x∗) ≥ 0, ∀x ∈ Ω, (1) Ω: a nonempty, closed and convex set in RN; F: a continuous and monotone mapping defined on RN.

Deren Han (NJNU) Asymmetric PPA September 2, 2014 3 / 26

slide-4
SLIDE 4

VI

(x − x∗)TF(x∗) ≥ 0, ∀x ∈ Ω, (1) Ω: a nonempty, closed and convex set in RN; F: a continuous and monotone mapping defined on RN. Examples: Equations, Complementarity Problems, Constrained Optimization Problems, Saddle Point Problems, etc.

Deren Han (NJNU) Asymmetric PPA September 2, 2014 3 / 26

slide-5
SLIDE 5

Proximal Point Algorithms

Classical PPA: (x − xk+1)T

  • F(xk+1) + 1

ck (xk+1 − xk)

  • ≥ 0, ∀x ∈ Ω,

(2)

1 ck is the proximal parameter and xk is the proximal center.

Deren Han (NJNU) Asymmetric PPA September 2, 2014 4 / 26

slide-6
SLIDE 6

Proximal Point Algorithms

Classical PPA: (x − xk+1)T

  • F(xk+1) + 1

ck (xk+1 − xk)

  • ≥ 0, ∀x ∈ Ω,

(2)

1 ck is the proximal parameter and xk is the proximal center.

A General Version: (x − xk+1)T F(xk+1) + Mk(xk+1 − xk)

  • ≥ 0, ∀x ∈ Ω,

(3) where the metric proximal parameter Mk ∈ Rn×n is positive definite and symmetric.

Deren Han (NJNU) Asymmetric PPA September 2, 2014 4 / 26

slide-7
SLIDE 7

Proximal Point Algorithms

Classical PPA: (x − xk+1)T

  • F(xk+1) + 1

ck (xk+1 − xk)

  • ≥ 0, ∀x ∈ Ω,

(2)

1 ck is the proximal parameter and xk is the proximal center.

A General Version: (x − xk+1)T F(xk+1) + Mk(xk+1 − xk)

  • ≥ 0, ∀x ∈ Ω,

(3) where the metric proximal parameter Mk ∈ Rn×n is positive definite and symmetric. Mk := 1/ckM, (x − xk+1)T ckF(xk+1) + M(xk+1 − xk)

  • ≥ 0, ∀x ∈ Ω.

(4)

Deren Han (NJNU) Asymmetric PPA September 2, 2014 4 / 26

slide-8
SLIDE 8

Role of M in Algorithms:

Make the subproblems easier: Preconditioner: T. Pock and A. Chambolle [1]. Decomposable of the subproblems. [1] T. Pock and A. Chambolle. Diagonal preconditioning for first order primal-dual algorithms in convex optimization, IEEE Inter. Con.

  • Comput. Vis., 2011, pp. 1762-1769.

Deren Han (NJNU) Asymmetric PPA September 2, 2014 5 / 26

slide-9
SLIDE 9

A simple example

The saddle point problem: min

u∈U max v∈V Φ(u, v) := f(u) + vTAu − g(v).

(5)

Deren Han (NJNU) Asymmetric PPA September 2, 2014 6 / 26

slide-10
SLIDE 10

A simple example

The saddle point problem: min

u∈U max v∈V Φ(u, v) := f(u) + vTAu − g(v).

(5) PDHG scheme:                      ˆ uk+1 = uk − τATvk, uk+1 = arg minu∈U f(u) + 1

2τ u − ˆ

uk+12, ¯ uk+1 = uk+1 + θ(uk+1 − uk), ˆ vk+1 = vk + σA¯ uk+1, vk+1 = arg minv∈V g(v) + 1

2σv − ˆ

vk+12 (6)

Deren Han (NJNU) Asymmetric PPA September 2, 2014 6 / 26

slide-11
SLIDE 11

A simple example

The saddle point problem: min

u∈U max v∈V Φ(u, v) := f(u) + vTAu − g(v).

(5) PDHG scheme:                      ˆ uk+1 = uk − τATvk, uk+1 = arg minu∈U f(u) + 1

2τ u − ˆ

uk+12, ¯ uk+1 = uk+1 + θ(uk+1 − uk), ˆ vk+1 = vk + σA¯ uk+1, vk+1 = arg minv∈V g(v) + 1

2σv − ˆ

vk+12 (6) PPA point of view M :=  

1 τ Im

−AT −θA

1 σIn

  . (7)

Deren Han (NJNU) Asymmetric PPA September 2, 2014 6 / 26

slide-12
SLIDE 12

Difficulty in Analysis:

Can not bound the progress of consecutive iterations using M-norm;

Deren Han (NJNU) Asymmetric PPA September 2, 2014 7 / 26

slide-13
SLIDE 13

Difficulty in Analysis:

Can not bound the progress of consecutive iterations using M-norm; Remedies: Correction steps;

Deren Han (NJNU) Asymmetric PPA September 2, 2014 7 / 26

slide-14
SLIDE 14

Difficulty in Analysis:

Can not bound the progress of consecutive iterations using M-norm; Remedies: Correction steps; Our Motivation: Schemes without corrections.

Deren Han (NJNU) Asymmetric PPA September 2, 2014 7 / 26

slide-15
SLIDE 15

Definitions

Let F(·) be a mapping from RN into RN. Then, F(·) is said to be Monotone: if (F(x) − F(y))T(x − y) ≥ 0 ∀x, y ∈ RN; Strongly monotone with modulus µ if (F(x) − F(y))T(x − y) ≥ µx − y2 ∀x, y ∈ RN. Lipschitz continuous: if F(x) − F(y) ≤ Lx − y ∀x, y ∈ RN; Co-coercive with modulus σ > 0 if (F(x) − F(y))T(x − y) ≥ σF(x) − F(y)2 ∀x, y ∈ RN. (8)

Deren Han (NJNU) Asymmetric PPA September 2, 2014 8 / 26

slide-16
SLIDE 16

Reformulation

Ms ≡ 1

2(M + MT): The symmetric part of M. Define

C := M−1/2

s

(M − Ms)M−1/2

s

; K ≡ M1/2

s

Ω = {M1/2

s

x : x ∈ Ω}; K∗ ≡ M1/2

s

Ω∗ = {M1/2

s

x∗ : x∗ ∈ Ω∗}; ˜ FM(y) ≡ M−1/2

s

F(M−1/2

s

y), ∀y ∈ K, ; For any x ∈ Ω, we define y := M1/2

s

  • x. Thus,

y′ ≡ M1/2

s

x′, yk+1 ≡ M1/2

s

xk+1, and yk ≡ M1/2

s

xk. (9) Then, a vector x∗ is a solution of the variational inequality (1) if and only if y∗ := M1/2

s

x∗ is a solution of the variational inequality of finding y ∈ K such that (y′ − y∗)T ˜ FM(y∗) ≥ 0, ∀y′ ∈ K. (10) The solution set of (10) is thus given by K∗.

Deren Han (NJNU) Asymmetric PPA September 2, 2014 9 / 26

slide-17
SLIDE 17

A Useful Lemma

Lemma 1

Let the mapping G : Rn → Rn be co-coercive on a nonempty, closed, convex subset W in Rn with modulus σ > 1/2, Then, for any x, y, z ∈ W, we have (x − y)T(G(z) − G(y)) ≥ −ν 2x − z2, (11) where ν is an arbitrary number satisfying 0 < 1 2σ < ν < 1. (12)

Deren Han (NJNU) Asymmetric PPA September 2, 2014 10 / 26

slide-18
SLIDE 18

Co-Coerciveness

Lemma 2

[Proposition 12.5.3](Facchinei and Pang 2003) The mapping α˜ FM − C is co-coercive over K with modulus greater than 1/2 if either one of the following two conditions holds:

1

F is Lipschitz continuous on Ω with modulus L and a τ ∈ (0, 1) exists such that, for all y1, y2 ∈ K, α˜ FM(y2) − α˜ FM(y1) − M−1/2

s

MM−1/2

s

(y2 − y1) ≤ τy2 − y1;

2

F(x) = Qx + q, for some positive semidefinite matrix Q ≡ D + E with D positive definite and E symmetric, and 0 < In + H < 2, where H ≡ D−1/2

s

ED−1/2

s

.

Deren Han (NJNU) Asymmetric PPA September 2, 2014 11 / 26

slide-19
SLIDE 19

The Exact Version

Deren Han (NJNU) Asymmetric PPA September 2, 2014 12 / 26

slide-20
SLIDE 20

The Exact Version

Remark: The exact version of APPA-MPC (“APPA-MPC-E" in short): The proximal center in xk in classical PPA is shifted to xk − αM−1F(xk) in (3.1).

Deren Han (NJNU) Asymmetric PPA September 2, 2014 12 / 26

slide-21
SLIDE 21

Convergence

Lemma 3

The subproblem (3.1) of the APPA-MPC-E can be rewritten as (y′ − yk+1)T[˜ FM(yk+1) + (I + C)(yk+1 − yk) + α˜ FM(yk)] ≥ 0, ∀y′ ∈ K. (13)

Deren Han (NJNU) Asymmetric PPA September 2, 2014 13 / 26

slide-22
SLIDE 22

Convergence

Lemma 3

The subproblem (3.1) of the APPA-MPC-E can be rewritten as (y′ − yk+1)T[˜ FM(yk+1) + (I + C)(yk+1 − yk) + α˜ FM(yk)] ≥ 0, ∀y′ ∈ K. (13)

Lemma 4

Let y∗ be an arbitrary solution point in K∗.. Then, we have yk+1 − y∗2 ≤ yk − y∗2 − (1 − ν)yk − yk+12, (14) where ν is an arbitrary number satisfying (12).

Deren Han (NJNU) Asymmetric PPA September 2, 2014 13 / 26

slide-23
SLIDE 23

The Inexact Version

Deren Han (NJNU) Asymmetric PPA September 2, 2014 14 / 26

slide-24
SLIDE 24

Convergence

Lemma 5

Let {xk} be the sequence generated by APPA-MPC-I. Then, there exist a positive scalar Γ such that the following inequality holds for any y′ ∈ K: (y′−¯ yk+1)T(˜ FM(¯ yk+1)+(I+C)(¯ yk+1−yk)+α˜ FM(yk)) ≥ −ǫky′−¯ yk+12−Γǫk. (15)

Deren Han (NJNU) Asymmetric PPA September 2, 2014 15 / 26

slide-25
SLIDE 25

Convergence

Lemma 5

Let {xk} be the sequence generated by APPA-MPC-I. Then, there exist a positive scalar Γ such that the following inequality holds for any y′ ∈ K: (y′−¯ yk+1)T(˜ FM(¯ yk+1)+(I+C)(¯ yk+1−yk)+α˜ FM(yk)) ≥ −ǫky′−¯ yk+12−Γǫk. (15)

Theorem 6

The sequence {xk} generated by the APPA-MPC-I globally converges to a solution point of the variational inequality (1).

Deren Han (NJNU) Asymmetric PPA September 2, 2014 15 / 26

slide-26
SLIDE 26

APPA-MPC-E

Theorem 7

Let {xk} be the sequence generated by the APPA-MPC-E. For an integer t > 0, let yt := 1 t + 1

t

  • k=0

yk+1, (16) then we have yt ∈ K and (yt − y′)T ˜ FM(y′) ≤ 1 2(1 + α)(t + 1)y′ − y02, ∀y′ ∈ K. (17)

Deren Han (NJNU) Asymmetric PPA September 2, 2014 16 / 26

slide-27
SLIDE 27

APPA-MPC-E

Theorem 7

Let {xk} be the sequence generated by the APPA-MPC-E. For an integer t > 0, let yt := 1 t + 1

t

  • k=0

yk+1, (16) then we have yt ∈ K and (yt − y′)T ˜ FM(y′) ≤ 1 2(1 + α)(t + 1)y′ − y02, ∀y′ ∈ K. (17)

Lemma 8 (Theorem 2.3.5, Facchinei and Pang (2003))

The solution set K∗ of the variational inequality is convex and it can be characterized as K∗ =

  • y′∈K

{y∗ ∈ K | (y′ − y∗)T ˜ FM(y′) ≥ 0.} (18)

Deren Han (NJNU) Asymmetric PPA September 2, 2014 16 / 26

slide-28
SLIDE 28

APPA-MPC-I

Theorem 9

Let {xk} be the sequence generated by the APPA-MPC-I. For an integer t > 0, let yt := 1 t + 1

t

  • k=0

¯ yk+1, (19) then we have yt ∈ K and (yt − y′)T ˜ FM(y′) ≤ 1 2(1 + α)(t + 1)

  • y′ − y02 + N(y′)

t

  • k=0

ǫk

  • , ∀y′ ∈ K.

Deren Han (NJNU) Asymmetric PPA September 2, 2014 17 / 26

slide-29
SLIDE 29

Theorem 10

Suppose that F is strongly monotone with the modulus µf > 0 and Lipschitz continuous with the constant Lf . Let {xk} be the sequence generated by the proposed APPA-MPC-E. Then it converges to a solution point in Ω∗ linearly.

Deren Han (NJNU) Asymmetric PPA September 2, 2014 18 / 26

slide-30
SLIDE 30

Theorem 10

Suppose that F is strongly monotone with the modulus µf > 0 and Lipschitz continuous with the constant Lf . Let {xk} be the sequence generated by the proposed APPA-MPC-E. Then it converges to a solution point in Ω∗ linearly.

Theorem 11

Suppose that F is strongly monotone with the modulus µf > 0 and Lipschitz continuous with the constant Lf . Moreover, assume that ǫk ≤ 3(1 − ν) 8Γ + 2 yk − ¯ yk+12, ∀k ≥ 0. Let {xk} be the sequence generated by the proposed APPA-MPC-I. Then it converges to a solution point in Ω∗ linearly.

Deren Han (NJNU) Asymmetric PPA September 2, 2014 18 / 26

slide-31
SLIDE 31

8¹

1

Introduction

2

APPA with Moving Proximal Centers

3

Worst-case Rate of Convergence

4

Linear Convergence

5

Two Concrete Algorithms The Saddle Point Problem Multiblock Decomposible Convex Optimization

6

Conclusion

Deren Han (NJNU) Asymmetric PPA September 2, 2014 19 / 26

slide-32
SLIDE 32

The saddle-point problem

Recall that the problem is min

u∈U max v∈V Φ(u, v) := f(u) + vTAu − g(v),

(20) where f : U → R, g : V → R are convex but not necessarily smooth functions; U ⊆ Rn and V ⊆ Rm are two nonempty, closed and convex sets; and A ∈ Rm×n.

Deren Han (NJNU) Asymmetric PPA September 2, 2014 20 / 26

slide-33
SLIDE 33

The algorithm

Deren Han (NJNU) Asymmetric PPA September 2, 2014 21 / 26

slide-34
SLIDE 34

8¹

1

Introduction

2

APPA with Moving Proximal Centers

3

Worst-case Rate of Convergence

4

Linear Convergence

5

Two Concrete Algorithms The Saddle Point Problem Multiblock Decomposible Convex Optimization

6

Conclusion

Deren Han (NJNU) Asymmetric PPA September 2, 2014 22 / 26

slide-35
SLIDE 35

The Problems

min m

  • i=1

θi(xi)

  • m
  • i=1

Aixi = b, xi ∈ Xi; i = 1, · · · , m

  • ,

(21) where θi : Rni → R are closed proper convex functions (not necessarily smooth); Ai ∈ Rl×ni; Xi ⊂ Rni are closed and convex nonempty sets; b ∈ Rl; and m

i=1 ni = n.

Deren Han (NJNU) Asymmetric PPA September 2, 2014 23 / 26

slide-36
SLIDE 36

The Decomposition Algorithm

M =            κI −βAT

1A2

· · · −βAT

1Am

κI · · · −βAT

2Am

. . . . . . ... . . . . . . κI

1 βI

           . (22)

Deren Han (NJNU) Asymmetric PPA September 2, 2014 24 / 26

slide-37
SLIDE 37

The algorithm

Deren Han (NJNU) Asymmetric PPA September 2, 2014 25 / 26

slide-38
SLIDE 38

We have proposed an APPA-MPC(Exact& Inexact);

Deren Han (NJNU) Asymmetric PPA September 2, 2014 26 / 26

slide-39
SLIDE 39

We have proposed an APPA-MPC(Exact& Inexact); Proved the global convergence ;

Deren Han (NJNU) Asymmetric PPA September 2, 2014 26 / 26

slide-40
SLIDE 40

We have proposed an APPA-MPC(Exact& Inexact); Proved the global convergence ; Worst-case convergence rate & Asymptotical linear rate of convergence.

Deren Han (NJNU) Asymmetric PPA September 2, 2014 26 / 26

slide-41
SLIDE 41

We have proposed an APPA-MPC(Exact& Inexact); Proved the global convergence ; Worst-case convergence rate & Asymptotical linear rate of convergence. Two concrete algorithms;

Deren Han (NJNU) Asymmetric PPA September 2, 2014 26 / 26

slide-42
SLIDE 42

We have proposed an APPA-MPC(Exact& Inexact); Proved the global convergence ; Worst-case convergence rate & Asymptotical linear rate of convergence. Two concrete algorithms; Numerical performances?

Deren Han (NJNU) Asymmetric PPA September 2, 2014 26 / 26

slide-43
SLIDE 43

We have proposed an APPA-MPC(Exact& Inexact); Proved the global convergence ; Worst-case convergence rate & Asymptotical linear rate of convergence. Two concrete algorithms; Numerical performances?

Deren Han (NJNU) Asymmetric PPA September 2, 2014 26 / 26