Optimization (Introduction) : IR IR f ( x ) ID Optimization " - - PowerPoint PPT Presentation

▶
optimization introduction
SMART_READER_LITE
LIVE PREVIEW

Optimization (Introduction) : IR IR f ( x ) ID Optimization " - - PowerPoint PPT Presentation

May 15, 2023 177 likes •412 views

Optimization (Introduction) : IR IR f ( x ) ID Optimization " 112 FCI ) : IR NE Goal: Find the minimizer ! that minimizes the objective (cost) function " ! : " - fatal * Unconstrained Optimization x :

slide-1
SLIDE 1

Optimization (Introduction)

slide-2
SLIDE 2 Goal: Find the minimizer !∗that minimizes the objective (cost) function " ! : ℛ" → ℛ

Optimization

Unconstrained Optimization ID f(x) : IR → IR NE FCI) : IR " → 112
  • fatal
*

/H¥nfGY

  • r
x: arg yi±yf
slide-3
SLIDE 3 Goal: Find the minimizer !∗that minimizes the objective (cost) function " ! : ℛ" → ℛ

Optimization

Constrained Optimization f- (x*) = min f Cx) X s .t . hi G) ⇐ o →equality gig

:# so

→ inequality

(

in

slide-4
SLIDE 4

Unconstrained Optimization

  • What if we are looking for a maximizer !∗?
" !∗ = max # " ! t÷

f(x*)

= main ( -fan )
slide-5
SLIDE 5

Calculus problem: maximize the rectangle area subject to perimeter constraint

max ! ∈ ℛ!

✓

area
  • Max Area
①
  • perimeter
→perimeter constraint ②ditto ③dzflo }

di.dz#Lwithoutpericoust)-s/di--dz=l0-sA=IooT

slide-6
SLIDE 6 '$ '$ '% '% ()*+ = '$'%
  • *)./*0*) = 2('$ + '%)
Area

[

mata -100

perimeter

t.a.pe

(violates"

  • f#¥Lr
.

D

D

slide-7
SLIDE 7 sizes A

¥

"

pig

¥ .dz) :÷÷÷ILi€¥⇐

:*

:i÷÷÷

.

* i;

(d, .dz) d' dies
slide-8
SLIDE 8

What is the optimal solution? (1D)

(First-order) Necessary condition (Second-order) Sufficient condition ! "∗ = min " ! "

IT

,

max
  • gifted, points

f

" txt) > o → x* is minimum f- "(x*) co - X* is maximum
slide-9
SLIDE 9

Does the solution exists? Local or global solution?

"°*^#

a

IEEE

slide-10
SLIDE 10

Example (1D)

  • 6
  • 4
  • 2
2 4 6
  • 200
  • 100
100 Consider the function " ! = 7! 8 − 7" 9 − 11 -: + 40-. Find the stationary point and check the sufficient condition miufcx) X * lstarder necessary condition max : : ' I f- '( x) .
  • 4¥
  • 3¥
  • 22×+40
; min f)G) = o ⇒ x3 - E - 22×+40=0 ! solutions ⇒ x =/
  • 25
! * min 4 * 2nd order condition :

'

''as.ae#-aftIfa7::ii:i:.::.oIEhin

,

f- "f-5) =3 ( 25) +10-2220 (Min)
slide-11
SLIDE 11

Types of optimization problems

Gradient-free methods Gradient (first-derivative) methods Evaluate ! " , !′ " , !′′ " Second-derivative methods ! "∗ = min " ! " Evaluate ! " Evaluate ! " , !′ " 5: nonlinear, continuous and smooth

If

  • ff
slide-12
SLIDE 12

Optimization in 1D:

Golden Section Search

  • Similar idea of bisection method for root finding
  • Needs to bracket the minimum inside an interval
  • Required the function to be unimodal
A function ": ℛ → ℛ is unimodal on an interval [4, 5] ü There is a unique !∗ ∈ [4, 5] such that "(!∗) is the minimum in [4, 5] ü For any -E, -: ∈ [4, 5] with -E < -: §
  • : < !∗ ⟹ "(-E) > "(-:)
§
  • E > !∗ ⟹ "(-E) < "(-:)
HuhfK¥,if 'm

I

::¥:*"i÷

/ / 4 I

fix.

✓ *

OI

  • r
  • r
slide-13
SLIDE 13 + F G H$ H% H' H( 5 $ 5 % 5 ' 5 ( + F G H$ H% H' H( 5 $ 5 % 5 ' 5 (

¥7

(FIT ta

fa

Fb Fb
  • B.
+* *22 B A

*a

*2×* B

Ifif@HsxIIfIf1xisxIXttE-La.XzTfI.t

[ 4 , b ]

slide-14
SLIDE 14

K£7

Propose the point asks
  • t.

IT

x , e- at Cl -E) hk / XI Xz

HI

, Xz
  • - at Chic

Ill

  • C) hk
: The 1 at the start h$⇐ (b -a) i I

1←Ehk'#£E)hI

f , > f,
  • r
f, sfz

!¥%,

9k

Ex, , b ] [a. xD

T.ee/-fEoery

iteration

aFI¥EIb

* hkti-E.hn

interval gets '

Em!i hatch

..ch#..EfET;db" I '

(I - E) hrs

  • Ehr

¥t%h* ,

Cte

)

  • E - 12=0.6-187
I

t

slide-15
SLIDE 15

KI

interval Caio) 12-0.618-1 h . .
  • Cb - a)
  • !
I I → x, = at ( t - E) ho

I

,

hofz-t-GD-qh.pe#qs.--cyh-/

if f , sfz :
  • →x*E[a,Xz]
" I

9k

b=X2

µh-#

xz=x, → fz -fi a:④TTbtf+ ,

III.Ethan..

t¥h¥¥/(

f, - FCK) if f , > fz : → x*E Ex, , b]

¥+,¥¥¥hµ

,

That

Ej÷¥÷nn-f=fz

  • Xz= At Chu
fz=f(Xz)
slide-16
SLIDE 16

Golden Section Search

slide-17
SLIDE 17

Golden Section Search

What happens with the length of the interval after one iteration? ℎ! = ( ℎ" Or in general: ℎ#$! = ( ℎ# Hence the interval gets reduced by ) (for bisection method to solve nonlinear equations, (=0.5) For recursion: ( ℎ! = (1 − () ℎ" ( ( ℎ" = (1 − () ℎ" (% = (1 − () ) = .. 012
slide-18
SLIDE 18
  • Derivative free method!
  • Slow convergence:
lim I→K |@ILE| @I = 0.618 D = 1 (EFG@4D HIGJ@DK@GH@)
  • Only one function evaluation per iteration

Golden Section Search II

→ the < tot

① I

evaluate FG) HE ha
  • ¥
  • hee

ef÷=Y÷

  • Chien
  • re I
→ T
  • Xi
cheap ,
slide-19
SLIDE 19

Example

A = - to → ho = 20

TT

b
  • to

h ,

= ? he , = I ko → 0.618×20 = 12.36
slide-20
SLIDE 20

Newton’s Method

Using Taylor Expansion, we can approximate the function " with a quadratic function about -M " - ≈ " -M + "N -M (- − -M) + E : "N′ -M (- − -M): And we want to find the minimum of the quadratic function using the first-order necessary condition

Yeti = Xk t h

n¥near"

= I

* = O

→

point

5- ' Go) tyzf '' Go) Cx
  • to)¢=of

stationary

sin:÷±÷÷÷µ÷¥¥÷7¥

slide-21
SLIDE 21

Newton’s Method

  • Algorithm:
"3 = starting guess "456 = "4 − !′ "4 /!′′ "4
  • Convergence:
  • Typical quadratic convergence
  • Local convergence (start guess close to solution)
  • May fail to converge, or converge to a maximum or
point of inflection

i

  • - -
slide-22
SLIDE 22

Newton’s Method (Graphical Representation)

get A tho) HA

IED

'• I

µ.¥i

l l l

#i→

X3 Xz Xl Xo X sequence of opt. using

quad

. approx I
slide-23
SLIDE 23

Example

Consider the function " - = 4 -9 + 2 -: + 5 - + 40 If we use the initial guess -M = 2, what would be the value of - after one iteration of the Newton’s method?
  • x , = ?
f- ' G) = 12×2+4×+5 f- "G) = 24 X t 4

h

=
  • fifty
  • CMg¥£¥t#
=
  • ft
X , = Xo th → X, = 2
  • ¥2 -1×1--0.82697