Computational Optimization Convexity and Unconstrained Optimization - - PowerPoint PPT Presentation

Computational Optimization

Convexity and Unconstrained Optimization 1/29/08 and 2/1(revised)

Convex Sets

A set S is convex if the line segment joining any two points in the set is also in the set, i.e., for any x,y∈S, λx+(1- λ)y ∈S for all 0≤ λ ≤ 1 }.

convex convex not convex not convex not convex

Proving Convexity

Prove {x|Ax<=b} is convex.

{ }

Let x and y be elements of C= x|Ax b . For any (0,1), ( (1 ) ) (1 ) (1 ) (1 ) A x y Ax Ay b b b x y C λ λ λ λ λ λ λ λ λ ≤ ∈ + − = + − ≤ + − = ⇒ + − ∈

You Try

Prove D= {x | ||x||≤1} is convex.

Convex Functions

A function f is (strictly) convex on a convex set S, if and only if for any x,y∈S, f(λx+(1- λ)y)(<) ≤ λ f(x)+ (1- λ)f(y) for all 0≤ λ ≤ 1.

x y f(y) f(x)

λx+(1- λ)y f(λx+(1- λ)y)

Proving Function Convex

Linear functions

1

( ) '

n n i i i

f x w x w x where x R

=

= = ∈

∑

, (0,1) ( (1 ) ) '( (1 ) ) ' (1 ) ' ( ) (1 ) ( )

n

For any x y R f x y w x y w x w y f x f y λ λ λ λ λ λ λ λ λ ∈ ∈ + − = + − = + − ≤ + −

You Try

2 2 1 2 1 2

( , ) 2 f x x x x = +

Hint: x2 is convex

( ) ( ) ( )

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

Consider any two points x,y and (0,1) (1 ) (1 ) (1 ) (1 ) (1 ) 2 (1 ) (1 ) (1 ) (1 ) (1 )( ) (1 ) First line uses (1 ) an x y x y x y x y xy x y x y x y x y x x x λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ λ ∈ + − = + − + − + − = + − − − + − + − = + − + − − ≥ + − = + −

2 2 2 2 2 2

d similarly for (1 ) . Second line completes the square of (1 ) . Third line observes the remaining terms are a square. Fouth line follows since (1 )( ) 0. y x y x y λ λ λ λ λ − + − − − ≥

Handy Facts

Let be convex functions And a>0. Then is convex. And is convex.

) ( , ), (

1

x g x g

m

…

∑

=

=

m i i x

g x f

1

) ( ) (

1

( ) ( ) h x ag x =

Convexity and Curvature

Convex functions have positive curvature everywhere. Curvature can be measured by the second derivative or Hessian. Properties of the Hessian indicate if a function is convex or not.

Convex Functions

A function f is (strictly) convex on a convex set S, if and only if for any x,y∈S, f(λx+(1- λ)y)(<) ≤ λ f(x)+ (1- λ)f(y) for all 0≤ λ ≤ 1.

x y f(y) f(x)

λx+(1- λ)y f(x+(1- λ)y)

Theorem

Let f be twice continuously differentiable. f(x) is convex on S if and only if for all x∈S, the Hessian at x is positive semi-definite.

2 ( )

f x ∇

Definition

The matrix H is positive semi-definite (p.s.d.) if and only if for any vector y The matrix H is positive definite (p.d.) if and

• nly if for any nonzero vector y

Similarly for negative (semi-) definite.

y Hy ′ ≥ y Hy ′ >

Theorem

Let f be twice continuously differentiable. f(x) is strictly convex on S if and only if for all x∈X, the Hessian at x is positive definite.

2 ( )

f x ∇

Checking Matrix H is p.s.d/p.d.

Manually

[ ]

1 2 2 1 2 1 2 1 1 2 2 2 2 2 1 1 2 2 2 2 1 2 1 2 1, 2

4 1 4 3 1 3 4 2 3 ( ) ^ 2 3 2 [ ] so matrix is positive definite x x x x x x x x x x x x x x x x x x x x − ⎡ ⎤ ⎡ ⎤ = − − + ⎢ ⎥ ⎢ ⎥ − ⎣ ⎦ ⎣ ⎦ = − + = − + + > ∀ ≠

Useful facts

The sum of convex functions is convex The composition of convex functions is convex.

via eigenvalues

The eigenvalues of

4 1 are 4.618 and 2.382 1 3 so matrix is positive definite − ⎡ ⎤ ⎢ ⎥ − ⎣ ⎦

Summary: using eigenvalues

If all eigenvalues are positive, then matrix is positive definite, p.d. If all eigenvalues are nonnegative, then matrix is positive semi-definite, p.s.d If all eigenvalues are negative, then matrix is negative definite, n.d. If all eigenvalues are nonpositive, then matrix is negative semi-definite, n.s.d Otherwise the matrix is indefinite.

Try with Hessians

2 2 1 2 1 2 1 2 2 2 2

( , ) 2 2 ( ) 4 2 ( ) 4 2 [ ] 2 4 [ ] 4 f x x x x x f x x f x a ab a b for any ab b StrictlyConvex = + ⎡ ⎤ ∇ = ⎢ ⎥ ⎣ ⎦ ⎡ ⎤ ∇ = ⎢ ⎥ ⎣ ⎦ ⎡ ⎤ ⎡ ⎤ = + > ≠ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦

Check Hessian

H = [ 2 0; 0 4] Eigs(H) are 2 and 4 So Hessian matrix is always p .d. So function is strictly convex

Differentiability and Convexity

For convex function, linear approximation underestimates function

f(x)

( ) ( *) ( *) ( *) g x f x x x f x ′ = + − ∇

(x*,f(x*))

Theorem

Assume f is continuously differentiable on a Set S. f is convex on S if and only if

( ) ( ) ( )' ( ) , f y f x y x f x x y S ≥ + − ∇ ∀ ∈

Theorem

Consider problem min f(x) unconstrained. If and f is convex, then is a global minimum. Proof: ( ) f x ∇ =

x

( ) ( ) ( )' ( ) byconvexityof ( ) since ( ) 0. y f y f x y x f x f f x f x ∀ ≥ + − ∇ = ∇ =

Unconstrained Optimality Conditions

Basic Problem: (1) Where S is an open set e.g. Rn

min ( )

x S

f x

∈

First Order Necessary Conditions

Theorem: Let f be continuously differentiable. If x* is a local minimizer of (1), then

( * ) f x ∇ =

Stationary Points

Note that this condition is not sufficient

( * ) f x ∇ =

Also true for local max and saddle points

Proof

Assume false, e.g.,

Let ( *), then d f x = −∇

( * ) ( *) ( *) ( *, ) ( * ) ( *) ( *) ( *, ) ( * ) ( *) 0 for sufficiently small since ( *) 0and ( *, ) 0. !! * is a local min. f x d f x d f x d x d f x d f x d f x d x d f x d f x d f x x d CONTRADICTION x λ λ λ α λ λ α λ λ λ λ α λ ′ + = + ∇ + ⇓ + − ′ = ∇ + ⇓ + − < ′∇ < →

( *) f x ∇ ≠

Second Order Sufficient Conditions

Theorem: Let f be twice continuously differentiable. If and then x* is a strict local minimizer of (1).

( * ) f x ∇ =

2

( *) is positive definite f x ∇

Proof

Any point x in neighborhood of x* can be written as x*+λd for some vector d with norm 1 and λ<= λ*. Since f is twice continuously differentiable, we can choose λ* such that

2 2 2 2

1 , *, ( * ) ( *) ( *) ' ( ) 2 1 ( * ) ( *) ' ( *) 2 therefore * is a strict local min. d f x d f x d f x d f d f x d f x d f x d x λ λ λ λ λ ε λ λ ′ ∀ ≤ + = + ∇ + ∇ ⇓ + − = ∇ > ⇓ s i n c e ( * ) f x ∇ =

2 ( )is p.d. for all

such that * * f x ε ε ε λ ∇ − ≤

Second Order Necessary Conditions

Theorem: Let f be twice continuously differentiable. If x* is a local minimizer of (1) then

( * ) f x ∇ =

2

( *) is positive semi definite f x ∇

Proof by contradiction

Assume false, namely there exists some d such that

2 2 2 2 2 2 2

( *) 0 then 1 ( * ) ( *) ( *) ' ( *) ( *, ) 2 ( * ) ( *) 1 ' ( *) ( *, ) 2 ( * ) ( *) 0 for sufficiently small since ( *) 0for some and ( *, ) 0. d f x d f x d f x d f x d f x d d x d f x d f x d f x d d x d f x d f x d f x d d x d Contrad λ λ λ λ α λ λ α λ λ λ λ α λ ′∇ < ′ + = + ∇ + ∇ + ⇓ + − = ∇ + ⇓ + − < ′∇ < → !!! x* is a local min. iction

Example

Say we are minimizing

2 2 1 2 1 1 2 2 1 2

1 ( , ) 2 15 4 2 f x x x x x x x x = − + − −

[8,2]???

Solve FONC

Solve FONC to find stationary point *

1 1 2 1 2 1 2 2 1 1 1 2 1 2 2

2 15 ( , ) 4 4 2 * 15 8 * 4 2 4 x f x x x x x

− − − − −

⎡ ⎤⎡ ⎤ ⎡ ⎤ ∇ = − = ⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎢ ⎥⎣ ⎦ ⎣ ⎦ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⇒ = = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦

Check SOSC

The Hessian at x*

1 2 2 1 2 1 2

2 ( *, *) 4 is p.d. since the eigenvalues, 4.118 and 1.882, are positive. Therefore SOSC are satisfied. x* is a strictly local min; f x x

− −

⎡ ⎤ ∇ =⎢ ⎥ ⎢ ⎥ ⎣ ⎦

Alternative Argument

The Hessian at every value x is

1 2 2 1 2 1 2

2 ( , ) 4 which is p.d. since the eigenvalues, 4.118 and 1.882, are positive. Therefore the function is strictly convex. Since f(x*)=0 and f is a strictly convex, x* is the unique f x x

− −

⎡ ⎤ ∇ =⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ∇ strict global minimum.

You Try

Use FONC and matlab to find solution

• f

Are SOSC satisfied? Are SONC? Is f convex?

2 2 2 1 2 3 1 2 3 1 2 2 3 2 1 3

min ( , , ) 10 5 3 2 4 f x x x x x x x x x x x x x = + + − + − − +

2 2 2 1 2 3 1 2 3 1 2 2 3 2 1 3

min ( , , ) 10 5 3 2 4 f x x x x x x x x x x x x x = + − − + − − +

Optimality Conditions for 1-

• dimen. functions

First Order Necessary Condition If x* is a local min then f’(x*)=0. If f’(x*)=0 then ??????????

2nd Derivatives - 1D Case

Sufficient conditions

If f’(x*)=0 and f’’(x*) >0, then x* is a strict local min. If f’(x*)=0 and f’’(x*) <0, then x* is a strict local

max.

Necessary conditions

If x* is a local min, then f’(x*)=0 and f’’(x*) >=0. If x* is a local max, then f’(x*)=0 and f’’(x*) <=0.

Optimality Conditions for function of Rn

First Order Necessary Condition If x* is a local min then If then ??????????

*) ( = ∇ x f *) ( = ∇ x f

Second Order Conditions

Sufficient conditions

If and is p.d.

then x* is a strict local min.

If and is n.d.

then x* is a strict local max.

*) ( = ∇ x f

2 ( *)

f x ∇ *) ( = ∇ x f

2 ( *)

f x ∇

Second Order Conditions

Necessary conditions

If x* is a local min,

then and is p.s.d.

If x* is a local max,

then and is n.s.d.

*) ( = ∇ x f

2 ( *)

f x ∇ *) ( = ∇ x f

2 ( *)

f x ∇

Optimality Conditions for Convex Convexity

Let f be continuously differentiable convex function, x* is a global minimum of f if and only if Let f be continuously differentiable strictly convex function, If then x* is the unique global minimum of f.

*) ( = ∇ x f *) ( = ∇ x f

Works similarly for max and concave.

Line Search

Assume f function maps the vector f to a scalar: Current point is Have interval Want to find:

:

n

f R R → [ , ] a b λ∈

n

x R ∈

[ , ]

min ( ) ( )

a b f x

d g

λ

λ λ

∈

+ =

Example

Say we are minimizing Solution is [8 2]’ Say we are at [ 0, -1]’ and we want to do a linesearch in direction d=[1 0]’

[ ]

2 2 1 2 1 1 2 2 1 2 1 1 1 2 1 2 1 2 2 2

1 ( , ) 2 15 4 2 2 1[ ] 15 4 4 2 f x x x x x x x x x x x x x x

− −

= − + − − ⎡ ⎤⎡ ⎤ ⎡ ⎤ = − ⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦ ⎣ ⎦

Line Search

We are at [ 0, -1]’ and we want to do a linesearch in direction d=[1 0]’

[8,2] [0,-1] d [29/4,-1]

Descent Directions

If the directional derivative is negative then linesearch will lead to decrease in the function

[8,2] [0,-1] d

( ) f x d ′ ∇ <

( ) f x −∇

Example continued

The exact stepsize can be found

2 ' 1 2 ' 1 2

1 1 1 ( ) ( ) 2 15 4 2 2 15 1 '( ) '( ) ( ) 1 4 4 1 2 15 2 29 4 x d g f x d g f x d f x d λ λ λ λ λ λ λ λ λ λ λ λ λ

− −

⎡ ⎤ ⎡ ⎤ ⎡ ⎤ + = + = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ − − ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ = + = + + − + ⎛ ⎞ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎜ ⎟ = + = ∇ = − ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎜ ⎟ − ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ ⎢ ⎥ ⎣ ⎦ ⎝ ⎠ = + − = ⇒ =

Example continued

So new point is

2 9 2 9 4 4

1 + 1 1 f([0 ,-1 ]= 6 f([2 9 /4 , -1 ])= -4 6 .5 f([8 ,2 ])= -6 4 x d λ ⎡ ⎤ ⎡ ⎤ ⎡ ⎤ + = = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ − − ⎣ ⎦ ⎣ ⎦ ⎣ ⎦

In this case, is a convex function (verify) so this is a Global min.

( ) f x d λ +

Example 2

Consider Find all points satisfy FONC What can you say about that point based on SONC?

3 2 2 1 1 2 2

min 2 x x x x − +

FONC

The first order necessary conditions are So both (0,0) and (6,9) satisfy FONC

4 2 3

2 2 1 2 1 2 1

= + − = − x x x x x

Second Order Conditions

The Hessian is The Hessian at (6,9) Is indefinite so (6,9) is not a local min (or max).

⎥ ⎦ ⎤ ⎢ ⎣ ⎡ − − − = ∇ 4 2 2 2 6 ) (

1 1 2 1 2

x x x x x f

2

18 12 (6,9) 12 4 f − ⎡ ⎤ ∇ = ⎢ ⎥ − ⎣ ⎦

Second Order Conditions

The Hessian at (0,0) is psd. So this point satisfies the SONC But not the SOSC. It might be a local min, (but in fact it is not try (-ε,0) for any ε>0

2

(0,0) 4 f ⎡ ⎤ ∇ = ⎢ ⎥ ⎣ ⎦

Do Lab 2