Optimization (ND Methods) What is the optimal solution? (ND) f- ( x - - PowerPoint PPT Presentation

Optimization (ND Methods)

What is the optimal solution? (ND)

(First-order) Necessary condition (Second-order) Sufficient condition

1D: !## " > 0

1D: !′ " = 0

! *∗ = min

$ ! * f-(x)

HI)

= A

NI

Q - gives

stationary solution

NI

t

)

is

positive

definite

→ x

minimizer

Taking derivatives…

f : IR

f-(Xn) = f-(Xi , Xz ,

• , Xn)

¥

÷

• :# ⇒

I:÷÷÷l

gradient

• f

f

CnxD

¥

µ ,±,

Ein Eas

• Tian Hi .
• ¥

,

:*

. :*

:#

• x.
• o¥oxn a

¥÷⇒

K

YE'

• ¥a

.. ¥¥

.

• -
• . g¥)

nth

From linear algebra:

I.i

ta

• O

y

e::÷

.I :

.

! *∗ = min

$ ! *

First order necessary condition: +! * = , Second order sufficient condition: - * is positive definite How can we find out if the Hessian is positive definite? la.eight)

THW→ ( X, y)

jyxyty

• x " yn:

→ "5fa¥:*

.

* ki so

ti

/)

y

THy

minimizer

* Xi Lo fi ⇒ y '

Hy

< o

Hy → His neg def ⇒ x* is

maximizer

* Tgi,Yo }→ H is indefinite

psaddetde

• int

Types of optimization problems

Gradient-free methods Gradient (first-derivative) methods

Evaluate ! * , +! * , +%! *

Second-derivative methods

! *∗ = min

$ ! *

Evaluate ! ' Evaluate ! ' , !" #

+HH

Consider the function " -E, -: = 2-E

Find the stationary point and check the sufficient condition

Example (ND)me

E-if

"-244¥:(

"

• "

: )

8×2+2

1)of -

• e

→ (GI!:{

6×2--24

8×2=-2

• O
• 25

stationary points :

x# III.as]

x'

• fo?⇐]

''et:

Hina

.fi#:.u.fttfo.-t-l::Ih::::i.n

Optimization in ND:

Steepest Descent Method

Given a function ! * : ℛ7 → ℛ at a point *, the function will decrease its value in the direction of steepest descent: −+! *

" -E, -: = (-E − 1)O+(-: − 1)O

What is the steepest descent direction?

min FG)

x

EI

• FIFA

£

Xz

• -
• • ×

⇒if

"

(Xo)

X1

Steepest Descent Method

" -E, -: = (-E − 1)O+(-: − 1)O

Start with initial guess:

!M = 3 3

Check the update:

¥2

• Of

XoKD

es

I

It

Etan

• IT

missed

±

ft;]

• i

Steepest Descent Method

" -E, -: = (-E − 1)O+(-: − 1)O Update the variable with: !ILE = !I − PIQ" !I How far along the gradient should we go? What is the “best size” for PI? =

I ,

• 0=5 OfGo)

• µ

12=0.57

How

can

we get

a ?

I

i

:

"

÷÷÷÷÷:÷÷÷

.

.

05¥

Steepest Descent Method

Algorithm: Initial guess: *3 Evaluate: ;4= −+! *4 Perform a line search to obtain <4 (for example, Golden Section Search) <4 = argmin

8

! *4 + < ;4 Update: *456 = *4 + <4 ;4

#

• f

=①

÷÷

.

D

O

ret ask

Line Search

Xktt

• Ak Tf(

Xn)

de St .

f-(Hett )

min f( Xn

• ④DfC)

x

he

1st order condition ¥70

→ gives a)

DI

= O

da

OXKH

pg(XnH)•Tf¥

f(Xkti) is

• rthogonal

to

zigpoEEfrwnergefa-a.GR#

Example

Consider minimizing the function ! "!, "" = 10("!)# − "" " + "! − 1 Given the initial guess "! = 2, ""= 2 what is the direction of the first step of gradient descent?

Miu far, , Xz)

=@ ±

• I:]

a- =p

if

Iska

• III ]

4un±÷÷t→fi;

Newton’s Method

Using Taylor Expansion, we build the approximation:

f-(Ets)

• f

t ELITE tf

It④

• -

t

nonlinear

quadratic

+ St order condition

approx off

I⇒t⇒

→ His :p

.my#etric--/tfH.)s---TfGTf

→ solve

liusys

to find

• Newton slip

s

Newton’s Method

Algorithm: Initial guess: *3 Solve: -9 *4 ;4 = −+! *4 Update: *456 = *4 + ;4

Note that the Hessian is related to the curvature and therefore contains the information about how large the step should be.

,#ii=f¥

any

• → solve

0¥ ⑤

• T

G

Try this out!

" -, R = 0.5-: + 2.5R:

When using the Newton’s Method to find the minimizer of this function, estimate the number of iterations it would take for convergence?

A) 1 B) 2-5 C) 5-10 D) More than 10 E) Depends on the initial guess

Newton’s Method Summary

Algorithm: Initial guess: *3 Solve: -9 *4 ;4 = −+! *4 Update: *456 = *4 + ;4 About the method…

• Typical quadratic convergence J
• Need second derivatives L
• Local convergence (start guess close to solution)
• Works poorly when Hessian is nearly indefinite
• Cost per iteration: >(@:)

=