Optimal algorithms for large scale quadratic programming problems - - PowerPoint PPT Presentation

Optimal algorithms for large scale quadratic programming problems Zdeněk DOSTÁL

Department of Applied Mathematics FEI VŠB-Technical University of Ostrava

TU Graz, May 2006

http://www.am.vsb.cz

with

David Horák, Vít Vondrák, Dalibor Lukáš, Marta Domorádová Department of Applied Mathematics FEI VŠB-TU Ostrava

• P. Avery, C. Farhat

Stanford

• M. Lesoinne

UC Boulder

Outline

• 1. Motivation, optimal algorithms
• 2. SMALE (semimonotonic augmented Lagrangians)

for equality constrained quadratic programming

• 3. MPRGP-optimal algorithm for bound constrained

quadratic programming

• 4. SMALBE (semimonotonic augmented

Lagrangians) for bound and equality constrained quadratic programming

• 5. Numerical experiments

Motivation: scalable algorithms for PDE

( )

1 1 2 2

( , ) ( ), ( ) ( , ) for , ( , ) ( , ) = − ∈ Ω > ≠ = f a b H a C a a u u u u u u u u u

• u v

v u

2 2 2 1

≥ ≥

T h

C C x x A x x

( )

1

min ( ) (QP) Find: for ∈ Ω f H u u

Discretization and multigrid or FETI (Fedorenko 60’s, … , Farhat 90’s, …) Elliptic problems

( )

1 2

min

h

(QP ) Find: = −

T h h h

f x x A x b x (1) O ⇒ Solvable in iterations

Our goal: develop tools for extending the results to constrained problems

Identify the active constraints for free Get rate of convergence independent of conditioning of constraints Use only preconditioners that preserve bound

constraints (e.g. lecture M. Domorádová, Thursday), not considered here

Challenges:

Equality constrained problems

( ) { } ( )

1 2 2 2

: , , possibly not full rank 1 2 min

T T i i i i i i T i i i T i i i

i f C C C f x x A x b x x B x

• B

A A B x x A x x x ∈ = − Ω = = ≤ = ≤ ≤ Ω

i

For let (QPE ) Find: T

Goal: find approximate solution at O(1) iterations !!! Note: we do not assume full row rank of B

Prolog: penalty method

( ) ( ) ( ) ( )

2

1 2

• n

f f f f

ρ ρ

ρ = + − = Ω x x Bx c x x

• x

f ∇ ( ) f c = x

Ω

( ) f c

ρ

= x

ρ

x x

Penalty approximation of the Lagrange multipliers

( )

2

1 1 ( ) 2 2 ( ) ( )

T T T

f fρ ρ ρ λ = + ∇ = x x Ax -b x Bx -c x Ax -b + B Bx -c

Optimal estimate

min

.: 0, 0, ( ) 1 Th f x

ρ

ε ρ ε ε λ ρ > > ∇ ≤ + ⇒ − ≤ x b B c b

Non optimal but linear in estimate

1 1 1

.: 0, 0, ( ) the smallest nonzero eigenvalue of 1 1

T

Th f x

ρ

ε ρ ε β ε ρ βρ

− − −

> > ∇ ≤ + ⇒ − ≤ + + x b BA B B c b BA b c

ρ

Optimality of dual penalty for FETI1

1152 139392 2130048 1 1.32e-1 1.20e-1 1.12e-1 1000 1.40e-3 1.28e-3 1.19e-3 100 000 1.40e-5 1.28e-5 1.19e-5

/ for varying and fixed / H h Bx b ρ

\ n ρ

Augmented Lagrangian and gradient

( ) ( ) ( ) ( ) ( )

2

1 , , ( ) 2 , , , , ( )

T T

L f L

x

x x Bx -c Bx -c g x x Ax -b + B Bx -c μ ρ μ ρ μ ρ μ ρ μ ρ μ = + + = ∇ = +

Augmented Lagrangians

• x

f ∇ ( , , )

k

L c λ ρ = x

Ω

k

x

1 + k

x

1

( , , )

k

L c λ ρ

+

= x

KKT conditions

Tλ

B

• x

f ∇ ( ) f c = x ( , ) L c x λ = f ∇

SMALE-Semimonotonic Augmented Lagrangians

( )

{ }

( )

0 1 , 0, 0, 0, 1 Find such that , , min , 2 If , , and are sma

k k k k k k k k k

Step M Step M Step β ρ η μ ρ η ρ < > > > ≤ − − {Initialization} {Approximate solution of bound constrained pr x g x μ Bx

• b

c l g x μ B em} x t c {Tes }

( ) ( ) ( )

1 2 1 1 1 1 1 1 k 1

ll then is solution 3 4 If , , , , 2 then else

k k k k k k k k k k k k k k k k

Step Step L L ρ ρ ρ ρ ρ βρ ρ ρ

+ + + + + + + +

= + − ≤ + − = = {Update Lagrange multipliers} {Update penalty parame x μ μ Bx c x μ x μ Bx c ter} {Re 5 1 and return to Step 1 Step k k = + peat loop}

Basic relations for SMALE

{ } { } { }

1 k 1 2

Let , and be generated with (0, ] and >0. (i) (ii) There is ( , , ) such that , 6 6

k k

C C C C M Theorem : Z.D. SINUM (200 ), Z.D. Computing (200 ) x A μ ρ α

−

∈ Γ =

2 1

2

k k k

C ρ

∞ =

≤

∑

B x

ρ

( ) ( )

2 min 2 1 1 1 1 1

If / ( ) then , , , , 2

k k k k k k k k k

M L L ρ λ ρ μ ρ μ ρ

+ + + + +

≥ ≥ + A x x Dx

Optimality of SMALE

{ } { } { }

1 k k k

Let , and be generated with (0, ], >0, M>0 and >0. (i) (ii) SMALE generates that satisfies ( ) b and b (iii) SMALE with CG in inner loop generates that sati

k k i k

Corol x A x g x Bx x lary : μ ρ α β ε ε

−

∈ Γ ≤ ≤

k

sfies ( ) b and b

k

Z.D. OMS (2005), COA g x ( 0 Bx 2 07) ε ε ≤ ≤

2 min

/ ( )

k

M ρ β λ ≤ A

at (1)

• uter iterations

O

ρ

at (1) matrix-vector multiplications O

Convergence of Lagrange multipliers

(i) Lagrange multipliers converge even for dependent constraints (ii) The convergence is linear for sufficiently large ρ

CG iterace – string system on Winkler support, multipoint constraints, cond=5 G

Bound constrained problems

( ) { } ( )

1 2 2 2 1 2 3 i

For let , : , , 0 for and (QPB ) Find: min

T T i i i i i T T i i i T i i i i

i f C C C f

+

∈ = − Ω = ≥ = > ≠ ≤ ≤ ≤ Ω x x A x b x x x c A A x A x x

• x

x A x x c x T

Goal: find approximate solution at O(1) iterations !!!

Projected gradient

Ω

) (

k

x β − ) (

k

x g −

k

x

) (

k

x ϕ −

( ) ( )

k P k

g x x = g ( )

k

x β = o ( )

k

g x

k

x

) (

k

x ϕ −

( )

k

g x − ( )

P k

x g

Deleting indices from active set- proportioning

2 2

( ) ( ) ( )

T x

x x ϕ ϕ β Γ ≥

• x proportional:

Reduction of the active set ) (

k

x g −

k

x

) (

k

x ϕ −

1 ) (

k

x β −

1 + k

x

Ω W

) (

k

x g Γ

Proportional iterations

Projection step: expansion of the active set Feasible conjugate gradient step:

k

x ) ( ), ( ) ( = =

k k k

x x x g β ϕ

1 + k

x

Ω W

) (

k

x g α −

) (

k

x g −

k

x 1

1 + k

x

Ω W

) (

k

x β ) (

k

x g Γ ) (

k

x ϕ −

MPRGP- Modified Proportioning with Reduced Gradient Projection

1 +1

Given , (0, ], 1: if is not proportional, then define by proportionalization

• i. e. minimalization in direction

{ } { } ) } ( {

k k k

Step Ste α β

−

∈ Ω ∈ Γ > − Initialization Proportioning conjugate gradie x nt x A x x

1 1 1

2 : if is proportional, then generate by trial cg step 3 : if then use it, else ( ( )) { }

k k k k k k

p Step αϕ

+ + + +

∈ Ω = − project x x x x x x ion

Rate of convergence of MPRGP

{ }

1 1 min 1 k 2

ˆ 0, { , }, (QPB), ( ), 0, . : (i)

T

Γ Γ Γ Γ

A

x A x Theorem : A x x Ax α λ α

− −

> = = ⎤ ∈ ⎦ = Let max solution of generated with Then The R -linear rate of convergence in the energy norm is given by (ii) The R -linear rate of Z.D., J. Schoeberl, Comput. Opt. Appl. (2005), Z.D. NA (2004) convergence of the projected gradient is given by

( )

1 1 2

36 ( ) ( ) ( ) , with (1 )

P k k

a η f f a α α η η

− − 1

≤ − = − g x x x

( )

2 1 2

2 ( ) ( ) with 1 1 ˆ 2 2

k k

η f f η αα − ≤ − = − < + Γ

A

x x x x

Optimality of MPRGP { }

1 1 2

ˆ 0, { , }, ), (0, ] { , }.

i i k i i i k i

Γ Γ Γ Γ C Theorem : Z.D., J. Schoeberl, Comput. Opt. x x x c o

• Appl. (2

x 005), α

− −

> = ∈ = Let max solution of (QPB generated with and max Then that satisfies is found at

(1) matrix-vector multiplications O

and ( )

k P k i i i i i

g ε ε − ≤ ≤ x x b x b

Finite termination

{ }

( )

k 1

, (0, 0. ( ) 2 ( ) 1

k i i

Γ x g k Γ k Theorem : x x A x x x A α κ

− ⎤

∈ > ⎦ = = ≥ = ≥ + ≥ Let denote the solution of (QPB) generated with and Then (i) If implies then there is such that (ii) If then there is s

k

(i) More Z.D. SIOPT (1996), (ii) Z.D., Schoeberl, x x COA (2005) = uch that

CG iterace – string system on Winkler support, bound constraints, cond=5

Bound and equality constrained problems

( ) { } ( )

1 2 2 2

: and , , and 1 2 3 min

T T i i i i i i i T i i T i i i i

i f C C C C f

+

∈ = − Ω = ≥ = ≤ = ≤ ≤ ≤ Ω x x A x b x x x c B x

• B

A A x x A x x c x

i

For let (QPBE ) Find: T

Goal: find approximate solution at O(1) iterations !!! Note: we do not assume full row rank of D!!!

Augmented Lagrangian and projected

gradient

( ) ( ) ( ) ( ) ( ) ( ) ( )

2

1 , , 2 , , , , , , , , , ,

T P x P P

L f L μ ρ μ ρ μ ρ μ ρ μ ρ ϕ μ ρ β μ ρ = + + = ∇ = = + x x Bx Bx g x x g g x x x

SMALBE-Semimonotonic augmented Lagrangians

( )

{ }

( )

0 1 , 0, 0, 0, 1 Find such that , , min , 2 If , , and are small

k P k k k k P k k k k

Step M Step x M Step x β ρ η μ μ ρ η μ ρ < > > > ≤ {Initializa g x Bx g tion} {Approximate solution of bound Bx constrained problem} {Test}

( ) ( ) ( )

1 2 1 1 1 1 1 1 k 1

then is solution 3 4 If , , , , 2 then else

k k k k k k k k k k k k k k k k

x Step Step L L μ μ ρ ρ μ ρ μ ρ ρ βρ ρ ρ

+ + + + + + + +

= + ≤ + = = {Update Lagrange multipliers} {Update penalty parameter} {Repe Bx a x x Bx t lo 5 1 and return to Step 1 Step k k = +

• p}

Basic relations for SMALBE

{ } { } { }

1 k 1 2

Let , and be generated with (0, ] and >0. (i) (ii) There is ( , , , , ) such that

k k

C C C C M μ ρ α α

−

∈ Γ = Γ Theorem : Z.D. SINUM (2005),Z.D.( x A 2006)

2 1 2 k k k

C ρ

∞ =

≤

∑

Bx

ρ

( ) ( )

2 min 2 1 1 1 1 1

If / ( ) then , , , , 2

k k k k k k k k k

M L L ρ λ ρ μ ρ μ ρ

+ + + + +

≥ ≥ + A x x Bx

Optimality of SMALBE

{ } { } { }

1 k k k

Let , and be generated with (0, ], >0, M>0 and >0. (i) (ii) SMALBE generates that satisfies ( ) b and b (ii) SMALBE with MPRGP in inner loop generates that

k k i P k

g Corollary : x A x x Bx x μ ρ α β ε ε

−

∈ Γ ≤ ≤

k

satisfies ( ) b and b

P k

g Z.D. SINUM (2006), Z.D. Computing x ( 0 B 07) x 2 ε ε ≤ ≤

2 min

/ ( )

k

M ρ β λ ≤ A

at (1)

• uter iterations

O

ρ

at (1) matrix-vector multiplications O

CG iterations – string system on Winkler support, bound and multipoint constraints, cond=5

Solution and numerical scalability of TFETI for n ranging from 50 to 2 130 048 (C/PETSc)

10

5

10

6

10 20 30 40 50 60 70 80 90 100 110

Solution and numerical scalability of FETI 2D semicoercive benchmark, 6 bodies

Subdomains dof Contact conditions It FETI-1 It FETI-DP 96 118098 565 103 82 384 466578 1125 129 90

Related work

• 1. Projectors introduced by Calamai, More, Toraldo
• 2. Efficiency of inexact working set strategy with

preconditioning in face considered by O’Leary

• 3. Adaptive precision control introduced by

Friedlander and Martinez

• 4. Basic algorithm for bound and equality constraints

was introduced by Conn, Gould and Toint and used in LANCELOT

• 5. Precision control that we use introduced Hager,

used by Z.D., Friedlander, Santos and Gomes

Conlusions

• 1. New algorithms for bound and equalityconstrained

problems were introduced

• 2. Qualitatively new results were proved
• 3. Theoretical results demonstrated by numerical

experiments

• 4. The results were applied to develop scalable

algorithms for elliptic boundary variational inequalities

• 5. Current reserach: preconditioning with improved

rate of convergence (Thursday – Domorádová)