Computational Optimization Constrained Optimization m R b , m - - PowerPoint PPT Presentation

Computational Optimization

Constrained Optimization

Easiest Problem

Linear equality constraints

min ( ) . . ,

n m n m

f x f R s t Ax b A R b R

×

∈ = ∈ ∈

Null Space Representation

Let x* be a feasible point, Ax*=b. Any other feasible point can be written as x=x*+P where Ap=0 The feasible region {x : x*+p p∈N(A)} where N(A) is null space of A

Example

Solve by substitution becomes

( )

1 2 2 3 1 2 2 2 1 2 3

min . . 3 4 4 x x x s t x x x + + + + =

( )

1 2 2 3 1 2 2 2 1 2 3

min . . 4 3 4 x x x s t x x x + + = − −

( )

( )

3

2 1 2 3 2 3 2 2 2

min 4 3 4 x x x x − − + +

Null Space Method

x*= [4 0 0]’ x=x*+v becomes

( )

1 2 2 3 1 2 2 2 1 2 3

min . . 3 4 4 x x x s t x x x + + + + =

( )

( )

3

2 1 2 3 1 2 1 3 2

min 4 3 4 v v v v − − + +

3 4 1 1 Z − − ⎡ ⎤ ⎢ ⎥ = ⎢ ⎥ ⎢ ⎥ ⎣ ⎦

1 1 1 1 2 2

4 3 4 4 3 4 1 1 v v v v v v − − − − ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ + = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦

Variable Reduction Method

Let A=[B N] for x* (a basic feasible solution with at most m nonzero variables corresponding to columns of B) assumes m < n is a basis matrix for null space of A

[ ]

1 1 r r

B B A AA B N I

− −

⎡ ⎤ ⎡ ⎤ − = ⇒ = = + ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦

1

B N Z I

−

⎡ ⎤ − = ⎢ ⎥ ⎣ ⎦

( ) ( ) ( ) m n m m m n m n m n m

A R B R N R I R

× × × − − × −

∈ ∈ ∈ ∈

Where did Z come from?

A=[1 3 4] x* = [4 0 0] A=[B N] B=[1] N = [3 4]

1

3 4 1*[3 4] 1 1 1 1 B N Z I

−

− − − ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ − ⎢ ⎥ ⎢ ⎥ = = = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦

General Method

There exists a Null Space Matrix The feasible region is: Equivalent “Reduced” Problem

n r

Z R r n m

×

∈ ≥ −

{ }

| * x x Zv +

min ( * )

v f x

Zv +

Practice Problem

( )

1 2 2 2 1 2 3 2 1 2 3 2 3

min . . 3 4 4 x x x s t x x x x x + + + + = − =

Optimality Conditions

Assume feasible point and convert to null space formulation

2 2 2

( ) ( * ) ( ) ' ( * ) ' ( ) * ( ) ' ( * ) ' ( ) g v f x Zv g v Z f x Zv Z f y where z x Zv g v Z f x Zv Z Z f y Z = + ∇ = ∇ + = ∇ = = + ∇ = ∇ + = ∇

Lemma 14.1 Necessary Conditions (Nash + Sofer)

If x* is a local min of f over {x|Ax=b}, and Z is a null matrix Or equivalently use KKT Conditions

2

' ( *) ' ( *) . . . Z f x and Z f x Z is p s d ⇒ ∇ = ∇

2

( *) ' * ' ( *) . . . f x A has a solution Ax b Z f x Z is p s d λ ⇒ ∇ − = = ⇒ ∇

Lemma 14.2 Sufficient Conditions (Nash + Sofer)

If x* satisfies (where Z is a basis matrix for Null(A)) then x* is a strict local minimizer

2

* ' ( *) ' ( *) . . Ax b Z f x Z f x Z is p d = ∇ = ∇

Lemma 14.2 Sufficient Conditions (KKT form)

If (x*,λ*) satisfies (where Z is a basis matrix for Null(A)) then x* is a strict local minimizer

2

* ( *) ' ' ( *) . . Ax b f x A Z f x Z is p d λ = ∇ − = ∇

Lagrangian Multiplier

λ* is called the Lagrangian Multiplier It represents the sensitivity of solution to small perturbations of constraints

* 1

ˆ ˆ ( ) ( *) ( *)' ( *) ˆ ( *) ( *)' ' * ˆ ( *) ' * ( *)

m i i i

f x f x x x f x f x x x A by KKT OC Nowlet Ax b f x f x λ δ δ λ δ λ

=

≈ + − ∇ = + − = + = + = +∑

Optimality conditions

Consider min (x2+4y2)/2 s.t. x-y=10

( ) ' 1 4 1 10 * * 8, * 2, f x A Ax b x y x y x y λ λ λ ∇ − = = ⎡ ⎤ ⎡ ⎤ = ⎢ ⎥ ⎢ ⎥ − ⎣ ⎦ ⎣ ⎦ − = = = =

Optimality conditions

Find KKT point Check SOSC

( ) ' 1 4 1 10 * * 8, * 2, f x A Ax b x y x y x y λ λ λ ∇ − = = ⎡ ⎤ ⎡ ⎤ = ⎢ ⎥ ⎢ ⎥ − ⎣ ⎦ ⎣ ⎦ − = = = = −

• 2

2

' [1 1] 1 ( ) 4 ' ( ) . . So SOSC satisfied Or we could just observe that it is a convex program so FONC are sufficient Z f x Z f x Z is p d = ⎡ ⎤ ∇ = ⎢ ⎥ ⎣ ⎦ ∇

( )

[ ]

10 1 1 4 1 1 A 4x x f(x) b Ax ) ( 10 s.t. 4 2 1 min

2 1 2 1 2 1 2 1 2 2 2 1

= − ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ − = ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ ∴ − = ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ = ∇ = = ∇ = − + x x x x A x f x x x x

T

λ λ

Linear Equality Constraints - I

Linear Equality Constraints - II

point KKT 8 * , 2 8 * 8 2 8 10 5 10 4 4 4 x : Solve

2 1 2 2 2 2 1 1 2 1

← = ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ − = = − = = = − ⇒ = − − − = ⇒ − = ⇒ = λ λ λ x x x x x x x x x x

[ ] [ ]

1 1 4 1 1 1 ) ( 4 1 ) ( 1 1 Z 1

• 1

A SOSC

2 2

> ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ = ∇ ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ = ∇ ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ = = Z x f Z x f

T

so SOSC satisfied, and x* is a strict local minimum Objective is convex, so KKT conditions are sufficient.

Linear Equality Constraints - III

Handy ways to compute Null Space

Variable Reduction Method Orthogonal Projection Matrix QR factorization (best numerically) Z=Null(A) in matlab

Orthogonal Projection Method

Use optimization. Minimize distance between given point c and null space of A.

2

1 min 2 . .

p

p c s t Ap − =

( *) ' * ( * ) ' f p A Ap

• r equivalently

p c A * Ap λ λ ∇ = = − = =

Orthogonal Projection Method

Optimality conditions give us the solution

( ) ( ) ( )

1 1 1

( * ) ' * * ' ' * ' ' ' ( ' ' ) FONC is p c A Ap Ap Ac AA AA Ac p A c A AA Ac c I A AA A c λ λ λ λ

− − −

− = = ⇒ − = ⇒ = − ⇒ = + = − + = −

Orthogonal Projection Method

Final result is: Note null space matrix is not unique

( )

1

( ' ' ) I A AA A Null Matrices of A

−

− ∈

Try it in Matlab for A= [1 3 5; 2 4 -1] Compare with Null(A) Null(‘A’,r)

Get Lagrangian Multipliers for free!

The matrix is the right inverse matrix for A. For general problems

( ) ( )

1 1

' ' ' '

r r

A A AA where AA AA AA I

− −

= = =

'

min ( ) . . * ( *)

r

f x s t Ax b A f x λ = = ∇

Let’s try it

For Projection matrix

1 2 3 4

1 2 2 2 2 2 1 2 3 4

min ( ) . . 1 f x x x x x s t x x x x ⎡ ⎤ = + + + ⎣ ⎦ + + + =

( ) [ ] [ ]

1 1 3 1 1 1 4 4 4 4 1 3 1 1 4 4 4 4 1 1 3 1 4 4 4 4 1 1 1 3 4 4 4 4

1 1 1 1 1 1 ' ' 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Z I A AA A

− − − − − − − − − − − − − −

⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ = − = − ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦

Solve FONC for Optimal Point

FONC

1 2 4

1 2 3 4

1 1 ( ) ' 1 1 1 x x f x A x x x x x x λ λ ⎡ ⎤ ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ∇ − = = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎢ ⎥ ⎣ ⎦ + + + =

Check Optimality Conditions

For Using Lagrangian

3 1 1 1 4 4 4 4 1 3 1 1 4 4 4 4 1 1 3 1 4 4 4 4 1 1 1 3 4 4 4 4

1 1 1 ( *) * 1 4 1 Z f x

− − − − − − − − − − − −

⎡ ⎤ ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ∇ = = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦

* [1111]/ 4 ( *) [1111]/ 4 * x f x Ax b = ∇ = =

( )

1

1 ' ' [1111]' 4 ( *) 1/ 4 ( *) '

r r

A A AA A f x Clearly f x A λ λ

−

= = = ∇ = ∇ =

You try it

For Find projection matrix Confirm optimality conds are Z’Cx*=0, Ax* = b Find x* Compute Lagrangian multipliers Check Lagrangian form of the multipliers.

1 2

min ( ) ' . . 13 6 3 13 23 9 3 2 1 2 1 2 6 9 12 1 1 1 3 1 3 3 3 1 3 f x x Cx s t Ax b C A b = = − − − ⎡ ⎤ ⎢ ⎥ − − ⎡ ⎤ ⎡ ⎤ ⎢ ⎥ = = = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ − − − − ⎣ ⎦ ⎣ ⎦ ⎢ ⎥ − ⎢ ⎥ ⎣ ⎦

Variable Reduction Method

Let A=[B N] A is m by n B is m by m assume m < n is a basis matrix for null space of A

[ ]

1 1 r r

B B A AA B N I

− −

⎡ ⎤ ⎡ ⎤ = ⇒ = = + ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦

1

B N Z I

−

⎡ ⎤ − = ⎢ ⎥ ⎣ ⎦

Try on our example

Take for example first two columns for B Then Condition number of Z’ CZ = 158 better but not great

[ ]

2 1 2 1 2 1 2 1 1 1 3 1 1 1 3 1 A B N ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ = = = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ − − ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦

1 2 1 1 4 3 1 2 1 1

r

Z A − − ⎡ ⎤ ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ − − ⎢ ⎥ ⎢ ⎥ = = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦

QR Factorization

Use Gram-Schmidt algorithm to make

• rthogonal factorize A’=QR with Q
• rthogonal and R upper triangular

[ ]

1 1 2 1 2 1 2 1 1

' , , ( ),

T r

R A QR Q Q where A m n Q n m Q n n m R m m Z Q A Q R− ⎡ ⎤ = = ⎢ ⎥ ⎣ ⎦ ∈ × ∈ × ∈ × − ∈ × = =

QR on problem

Use matlab command QR [Q R ] = qr(A’) Q2 = Q(:,3:4) Cond(Q2’*C*Q2) = 9.79

In Class Practice

Find optimal solution and verify FONC and SOSC of the following: Let the perimeter of a rectangle be fixed to 4. Find the shape of the rectangle with largest area. Solve the problem max x1x2+x2x3+x1x3 s.t. x1+x2+x3=3