Optimal algorithms for large scale quadratic programming problems - PowerPoint PPT Presentation

Optimal algorithms for large scale quadratic programming problems Zden ě k DOSTÁL Department of Applied Mathematics FEI VŠB-Technical University of Ostrava TU Graz, May 2006 http://www.am.vsb.cz

with David Horák, Vít Vondrák, Dalibor Lukáš, Marta Domorádová Department of Applied Mathematics FEI VŠB-TU Ostrava P. Avery, C. Farhat Stanford M. Lesoinne UC Boulder

Outline 1. Motivation, optimal algorithms 2. SMALE (semimonotonic augmented Lagrangians) for equality constrained quadratic programming 3. MPRGP-optimal algorithm for bound constrained quadratic programming 4. SMALBE (semimonotonic augmented Lagrangians) for bound and equality constrained quadratic programming 5. Numerical experiments

Motivation: scalable algorithms for PDE Elliptic problems ( ) = − ∈ Ω f u a u u b u u H 1 1 ( , ) ( ), ( ) 2 0 > 2 ≠ = a u u C u u o a u v a v u ( , ) for , ( , ) ( , ) ( ) ∈ Ω f u u H 1 (QP) Find: for min ( ) 0 Discretization and multigrid or FETI (Fedorenko 60’s, … , Farhat 90’s, …) ( ) = − T f x x A x b x (QP ) Find: 1 min h h h h 2 2 ≥ ≥ 2 T C x x A x C x h 2 1 ⇒ Solvable in iterations (1) O

Our goal: develop tools for extending the results to constrained problems Challenges: � Identify the active constraints for free � Get rate of convergence independent of conditioning of constraints � Use only preconditioners that preserve bound constraints (e.g. lecture M. Domorádová, Thursday), not considered here

Equality constrained problems ∈ i T For let ( ) = − T T f x x A x b x 1 i i i 2 { } Ω = = ≤ x B x o B C : , i i i 0 = T A A B , possibly not full rank i i i 2 ≤ ≤ 2 T C x x A x C x i 1 2 ( ) f x (QPE ) Find: min i i Ω i Goal: find approximate solution at O(1) iterations !!! Note: we do not assume full row rank of B

Prolog: penalty method 1 2 ( ) ( ) = + ρ − f x f x Bx c ρ 2 ( ) ( ) = Ω f x f x on ρ = f x c ( ) ρ ∇ f � x � ρ x � 0 x = f x c Ω ( )

Penalty approximation of the Lagrange multipliers 1 1 2 = + ρ T T f x x Ax -b x Bx -c ( ) 2 2 ( ) ∇ = ρ T f ρ x Ax -b + B Bx -c ( ) ( ) �� �� � λ

b ε Optimal estimate ≤ ( ) x b ρ f ρ ∇ ε min + λ 0, 1 > ≤ ρ c − 0, x B > ε ⇒ .: Th

ρ Non optimal but linear in estimate ε > ρ > ∇ ≤ ε Th f x b .: 0, 0, ( ) ρ β − T BA B 1 the smallest nonzero eigenvalue of + ε 1 ⇒ − ≤ − + ρ − B x c b BA b c 1 1 + βρ 1

Optimality of dual penalty for FETI1 ρ Bx b H h / for varying and fixed / ρ \ n 1152 139392 2130048 1 1.32e-1 1.20e-1 1.12e-1 1000 1.40e-3 1.28e-3 1.19e-3 100 000 1.40e-5 1.28e-5 1.19e-5

A ugmented Lagrangian and gradient 1 2 ( ) ( ) μ ρ = + μ + ρ T L x f x Bx -c Bx -c , , ( ) 2 ( ) ( ) ( ) μ ρ = ∇ μ ρ = μ + ρ T g x L x Ax -b + B Bx -c , , , , ( ) x �� � ��� � μ �

Ω Augmented Lagrangians f ∇ � c x = ) ρ 1 , + 1 k + x k λ ( , k x x L c = ) ρ , k λ ( , x L

c x λ = ( , ) L KKT conditions f ∇ x � f ∇ T λ B c = ( ) x f

SMALE-Semimonotonic Augmented Lagrangians {Initialization} < β ρ > η > > μ Step M 0 0 1 , 0, 0, 0, 0 {Approximate solution of bound constrained pr ob l em} { } ( ) ρ ≤ − η k k k k Step x g x μ M Bx c 1 Find such that , , min , k {Tes } t ( ) k k ρ k − k Step g x μ B x c x 2 If , , and are sma ll then is solution k {Update Lagrange multipliers} ( ) + = + ρ − k k k Step μ μ Bx c 1 3 k {Update penalty parame ter} ρ ( ) ( ) 2 + + ρ ≤ ρ + + − k k k k + k Step L x μ L x μ k Bx c 1 1 1 1 4 If , , , , + k k 1 2 ρ = βρ then + k 1 k ρ = ρ else + k k 1 {Re peat loop} = + Step k k 5 1 and return to Step 1

Basic relations for SMALE Theorem : { } { } { } − μ ρ α ∈ k k 1 x A k Let , and be generated with (0, ] Γ and >0. ρ ≥ λ (i) M A 2 If / ( ) then k min ρ ( ) ( ) 2 + μ + ρ ≥ μ ρ + + k k k k + k L x L x k Dx 1 1 1 1 , , , , + k k 1 2 = C C C C M (ii) There is ( , , ) such that 1 2 ρ ∞ ∑ 2 ≤ k k B x C 2 = k 1 Z.D. SINUM (200 ), Z.D. Computing (200 ) , 6 6 ρ

Optimality of SMALE Corol lary : { } { } { } − μ ρ α ∈ 1 k k x A Let , and be generated with (0, ], i β Γ >0, M>0 and >0. (i) ρ ≤ β λ M A 2 / ( ) k min x k (ii) SMALE generates that satisfies ≤ ε ≤ ε k g x Bx k ( ) b and b O at (1) outer iterations x k (iii) SMALE with CG in inner loop generates that sati sfies ≤ ε ≤ ε k g x Bx k ( ) b and b O at (1) matrix-vector multiplications Z.D. OMS (2005), COA ( 0 2 07) ρ

Convergence of Lagrange multipliers (i) Lagrange multipliers converge even for dependent constraints (ii) The convergence is linear for sufficiently large ρ

CG iterace – string system on Winkler support, multipoint constraints, cond=5 G

Bound constrained problems ∈ i T For let ( ) { } = − Ω = ≥ T T f x x A x b x x x c 1 , : , i i i i i 2 = > ≠ T T A A x A x x o , 0 for i i i + 2 ≤ ≤ 2 ≤ T C x x A x C x c C and i i 1 2 3 ( ) f x (QPB ) Find: min i i Ω i Goal: find approximate solution at O(1) iterations !!!

Projected gradient Ω k g x − k g x ( ) ( ) − β k x ( ) β = o k x ( ) − ϕ k x − ϕ k ( ) x k P k k ( ) x g x x ( ) − k g x = g k P k g x x ( ) ( ) ( )

Deleting indices from active set- proportioning T x 2 Γ ϕ ϕ ≥ β � x proportional: x x 2 ( ) ( ) ( ) Reduction of the active set + k x 1 Ω − β − k k x g x ( ) ( ) − ϕ k x ( ) k x W 1 Γ k g x ( )

Proportional iterations = ϕ β = k k k g x x x ( ) ( ), ( ) 0 Ω Projection step: + k k x x 1 W expansion of the active set − α k g x ( ) Ω β − k k x g x ( ) ( ) Feasible conjugate + W k x k 1 x 1 gradient step: − ϕ k x ( ) k g x ( ) Γ

MPRGP- Modified Proportioning with Reduced Gradient Projection Initialization { } − ∈ Ω α ∈ 1 Γ > x A 0 Given , (0, ], 0 Proportioning { } x x k k +1 Step 1: if is not proportional, then define by proportionalization − β x k i. e. minimalization in direction ( ) conjugate gradie nt { } + x x k k 1 Ste p 2 : if is proportional, then generate by trial cg step project ion { } + ∈ Ω x k 1 Step 3 : if then use it, + = − αϕ + x x x k 1 k k else ( ( ))

Rate of convergence of MPRGP Theorem : − > = α = λ ˆ Γ Γ Γ Γ 1 x A Let max solution of 0, { , }, (QPB), ( ), 1 min { } − ⎤ α ∈ 1 x k A generated with Then 0, . : ⎦ 2 = T x x Ax The R -linear rate of convergence in the energy norm is given by (i) A αα ( ) 2 − ≤ − = − < k k x x η f x f x η 0 1 2 ( ) ( ) with 1 1 + Γ ˆ 2 A 2 2 (ii) The R -linear rate of convergence of the projected gradient is given by α α − − ( ) 1 1 36 2 ≤ − = P k k 1 g x a η f x f x a 0 ( ) ( ) ( ) , with η − η (1 ) Z.D., J. Schoeberl, Comput. Opt. Appl. (2005), Z.D. NA (2004)

Optimality of MPRGP Theorem : − > = ˆ Γ Γ Γ Γ x 1 Let max solution of (QPB 0, { , }, ), i i { } − α ∈ = k x C x c o 1 0 generated with and max (0, ] { , }. i i i 2 k x Then that satisfies i − ≤ ε ≤ ε k P k x x b g x b and ( ) i i i i i is found at O (1) matrix-vector multiplications Z.D., J. Schoeberl, Comput. Opt. Appl. (2 005),

Finite termination Theorem : { } x x k Let denote the solution of (QPB) generated with , − ⎤ α ∈ > 1 A Γ and Then (0, 0. ⎦ = = ≥ = k x g x k x x (i) If implies then there is such that 0 ( ) 0 0 i i ( ) ≥ κ + ≥ = k Γ A k x x (ii) If then there is s uch that 2 ( ) 1 0 (i) More Z.D. SIOPT (1996), (ii) Z.D., Schoeberl, COA (2005)

CG iterace – string system on Winkler support, bound constraints, cond=5

Bound and equality constrained problems ∈ i T For let ( ) = − T T f x x A x b x 1 i i i 2 { } Ω = ≥ = ≤ x x c B x o B C : and , i i i i 0 = T A A , i i ≤ ≤ + ≤ 2 T 2 C x x A x C x c C and i i 1 2 3 ( ) f x (QPBE ) Find: min i i Ω i Goal: find approximate solution at O(1) iterations !!! Note: we do not assume full row rank of D!!!