Introductory Course on Non-smooth Optimisation Lecture 04 - - PowerPoint PPT Presentation

Introductory Course on Non-smooth Optimisation Lecture 04 - Backward–Backward splitting Jingwei Liang Department of Applied Mathematics and Theoretical Physics

Table of contents Problem 1 2 Forward–Backward splitting revisit MAP continue 3 4 Backward–Backward splitting 5 Numerical experiments

Monotone inclusion pronblem Problem Let B : R n → R n be β -cocoercive for some β > 0, s > 1 be a positive integer, such that for each i ∈ { 1 , ..., s } : A i : R n ⇒ R n is maximal monotone. Consider the problem Find x ∈ R n such that 0 ∈ B ( x ) + � s i = 1 A i ( x ) . A i can be composed with linear mapping, e.g. L ∗ ◦ A ◦ L . Even if the resolvents of B and each A i are simple, the resolvent of B + � i A i in most cases is not solvable. Use the properties of operators and structure of problem to derive operator splitting schemes. Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Outline 1 Problem 2 Forward–Backward splitting revisit 3 MAP continue 4 Backward–Backward splitting 5 Numerical experiments

Monotone inclusion problem Monotone inclusion Find x ∈ R n such that 0 ∈ A ( x ) + B ( x ) . Assump Assumptions tions A : R n ⇒ R n is maximal monotone. B : R n → R n is β -cocoersive. zer ( A + B ) � = ∅ . Characterisation of minimiser: γ > 0 x ⋆ − γ B ( x ⋆ ) ∈ x ⋆ + γ A ( x ⋆ ) x ⋆ = J γ A ◦ ( Id − γ B )( x ⋆ ) . ⇔ ample Let R ∈ Γ 0 and F ∈ C 1 Ex Example L , min x ∈ R n R ( x ) + F ( x ) . Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Forward–Backward splitting Fixed-point operator: γ ∈ ] 0 , 2 β [ T FB = J γ A ◦ ( Id − γ B ) . J γ A is firmly non-expansive. γ Id − γ B is 2 β -averaged non-expansive. 2 β T FB is 4 β − γ -averaged non-expansive. fix ( T FB ) = zer ( A + B ) . Forward–Backward splitting Let γ ∈ ] 0 , 2 β [ , λ k ∈ [ 0 , 4 β − γ 2 β ] : x k + 1 = ( 1 − λ k ) x k + λ k T FB ( x k ) . Special case of Krasnosel’ski˘ ı-Mann iteration. Recovers proximal point algorithm when B = 0. Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Outline 1 Problem 2 Forward–Backward splitting revisit 3 MAP continue 4 Backward–Backward splitting 5 Numerical experiments

Method of alternating projection Let X , Y ⊆ R n be closed convex and non-empty, such that X ∩ Y � = ∅ x ∈ R n ι X ( x ) + ι Y ( x ) . min Method of alternating projection (MAP) Let x 0 ∈ X : y k + 1 = P Y ( x k ) , x k + 1 = P X ( y k + 1 ) . Fixed-point operator: x k + 1 = T MAP ( x k ) , T MAP def = P X ◦ P Y . P X , P Y are firmly non-expansive. T MAP is 2 3 -averaged non-expansive. fix ( T MAP ) = X ∩ Y . Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Derive MAP Feasibility problem is equivalent to 2 || x − y || 2 + ι Y ( y ) . x , y ∈ R n ι X ( x ) + 1 min Optimality condition 0 ∈ N Y ( y ⋆ ) + y ⋆ − x ⋆ , 0 ∈ N X ( x ⋆ ) + x ⋆ − y ⋆ . Fixed-point characterisation y ⋆ = P Y ( x ⋆ ) , x ⋆ = P X ( y ⋆ ) . Fixed-point iteration y k + 1 = P Y ( x k ) , x k + 1 = P X ( y k + 1 ) . Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Example: SDP feasibility SDP feasibility Find X ∈ S n such that X � 0 Tr ( A i X ) = b i , i = 1 , ..., m . and Two sets and projection: X = S n + is the positive semidefinite cone. Let Y k = � n i = 1 σ i u i u T i be the eigenvalue decomposition of Y k , then P X ( Y k ) = � n i = 1 max { 0 , σ i } u i u T i . Y is the affine set in S n define by the linear inequalities, P Y ( X k ) = X k − � m i = 1 u i A i , where u i are found from the normal equations � � Gu = Tr ( A i X k ) − b i , · · · , Tr ( A i X k ) − b m , G i , j = Tr ( A i A j ) . Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Convergence rate Let X , Y be two subspaces, and assume 1 ≤ p def = dim( X ) ≤ q def = dim( Y ) ≤ n − 1 . angles The principal angles θ k ∈ [ 0 , π 2 ] , k = 1 , . . . , p between X and Y are defined by, Principal Principal angles with u 0 = v 0 def = 0, and cos ( θ k ) def = � u k , v k � = max � u , v � s . t . u ∈ X , v ∈ Y , || u || = 1 , || v || = 1 , � u , u i � = � v , v i � = 0 , i = 0 , · · · , k − 1 . angle The Friedrichs angle θ F ∈ ] 0 , π 2 ] between X and Y is Friedrichs Friedrichs angle � def u ∈ X ∩ ( X ∩ Y ) ⊥ , || u || = 1 , � θ F ( X , Y ) = max � u , v � s . t . cos v ∈ Y ∩ ( X ∩ Y ) ⊥ , || v || = 1 . Lemma def = dim( X ∩ Y ) . Moreover, The Friedrichs angle is θ d + 1 where d θ F ( X , Y ) > 0 . Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Convergence rate Example X , Y are defined by X = { x : Ax = 0 } , Y = { x : Bx = 0 } . Projection onto subspace P X ( x ) = x − A T ( AA T ) − 1 Ax . Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Convergence rate Define diagonal matrices � � c = diag cos ( θ 1 ) , · · · , cos ( θ p ) , � sin ( θ 1 ) , · · · , sin ( θ p ) � s = diag . Suppose p + q < n , then there exists orthogonal matrix U such that   Id p 0 0 0   0 0 p 0 0  U ∗ , P X = U     0 0 0 q − p 0  0 0 0 0 n − p − q   c 2 cs 0 0 c 2   cs 0 0  U ∗ . P Y = U     0 0 Id q − p 0  0 0 0 0 n − p − q Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Convergence rate Fixed-point operator T MAP = P X ◦ P Y   c 2 cs 0 0   0 0 p 0 0  U ∗ . = U     0 0 0 q − p 0  0 0 0 0 n − p − q Consider relaxation T λ MAP = ( 1 − λ ) Id + λ T MAP   ( 1 − λ ) Id p + λ c 2 λ cs 0  U ∗ . = U ( 1 − λ ) Id p   0 0  ( 1 − λ ) Id n − 2 p 0 0 Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Convergence rate Eigenvalues σ ( T λ � � MAP ) = 1 − λ sin 2 ( θ i ) | i = 1 , ..., p ∪ { 1 − λ } . Spectral radius ρ ( T λ � � MAP ) = max 1 − λ sin 2 ( θ F ) , | 1 − λ | . No relaxation ρ ( T MAP ) = cos 2 ( θ F ) . Convergence rate, C > 0 is some constant || x k − x ⋆ || = ||T MAP x k − 1 − T MAP x ⋆ || = ... MAP ( x 0 − x ⋆ ) || = ||T k ≤ C ||T MAP || k || x 0 − x ⋆ || . Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Outline 1 Problem 2 Forward–Backward splitting revisit 3 MAP continue 4 Backward–Backward splitting 5 Numerical experiments

Best pair problem When X ∩ Y � = ∅ , MAP returns x k , y k → x ⋆ ∈ X ∩ Y . Best pair problem Let X , Y ⊆ R n be closed and convex, such that X ∩ Y = ∅ . Consider finding two points in X and Y such that they are the closest, that is x ∈X , y ∈Y || x − y || . min MAP can be applied and ( x k , y k ) → ( x ⋆ , y ⋆ ) where ( x ⋆ , y ⋆ ) is a best pair. Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Backward–Backward splitting Consider Find x , y ∈ R n such that 0 ∈ A ( x ) + B ( y ) , A , B : R n ⇒ R n are maximal monotone. The set of solition is non-empty. There exists x ⋆ , y ⋆ ∈ R n and γ > 0 such that y ⋆ − x ⋆ ∈ γ A ( x ⋆ ) , x ⋆ − y ⋆ ∈ γ B ( y ⋆ ) . Backward–Backward splitting Let x 0 ∈ R n , γ > 0: y k + 1 = J γ B ( x k ) , x k + 1 = J γ A ( y k + 1 ) . Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Regularised monotone inclusion Yosida osida appr approxima ximation tion γ A = 1 γ ( Id − J γ A ) . which is γ -cocoercive. Regularised egularised monot monotone one inclusion inclusion Find x ∈ R n such that 0 ∈ A ( x ) + γ B ( x ) . ting τ ∈ ] 0 , 2 γ ] For orwar ard–Backw d–Backwar ard split splitting x k + 1 = J τ A ◦ ( Id − τ γ B )( x k ) . FB let τ = γ BB BB as as special special case ase of of FB x k + 1 = J γ A ◦ ( Id − γ γ B )( x k ) = J γ A ◦ � Id − γ 1 γ ( Id − J γ B ) � ( x k ) = J γ A ◦ J γ B ( x k ) . Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Inertial BB splitting An inertial Backward–Backward splitting Initial : x 0 ∈ R n , x − 1 = x 0 and γ > 0 , τ ∈ ] 0 , 2 γ ] ; Initial y k = x k + a 0 , k ( x k − x k − 1 ) + a 1 , k ( x k − 1 − x k − 2 ) + · · · , x k + 1 = J γ A ◦ J γ B ( y k ) , λ k ∈ [ 0 , 1 ] . An inertial BB splitting based on Yosida approximation Initial : x 0 ∈ R n , x − 1 = x 0 and γ > 0; Initial y k = x k + a 0 , k ( x k − x k − 1 ) + a 1 , k ( x k − 1 − x k − 2 ) + · · · , z k = x k + b 0 , k ( x k − x k − 1 ) + b 1 , k ( x k − 1 − x k − 2 ) + · · · , � y k − τ γ B ( z k ) � x k + 1 = J τ A ◦ , λ k ∈ [ 0 , 1 ] . Jingwei Liang, DAMTP Introduction to Non-smooth Optimisation November 21, 2019

Outline 1 Problem 2 Forward–Backward splitting revisit 3 MAP continue 4 Backward–Backward splitting 5 Numerical experiments