[PDF] - Example - Newton-Raphson Method We now consider the following PDF Document

SLIDE 1

Example - Newton-Raphson Method

We now consider the following example: minimize Since and we form the following iteration: . We can now try the iteration for different initial guesses. initial guess: initial guess: initial guess: initial guess: We can now evaluate f(x) at the two stationary points we’ve found: , and since therefore . In fact you can convince yourself that it is the global minimum. (Note that the global maximum is at .)

f x ( ) x3 – 3 4

x4

+ = f′ x ( ) 3x2 – 3x3 + = f″ x ( ) 6x – 9x2 + = xn

1 +

xn 3 xn ( )

3 3 xn ( )

2 – 9 xn ( )

2 6xn –

–

xn xn ( )

2 xn – 3xn 2 –

–

2 xn ( )

2 xn – 3xn 2 –

=

= = x 0

( )

= x 1

( )

= ( ) stationary point ⇒ x 0

( )

0.5 = x 1

( )

= ( ) stationary point ⇒ x 0

( )

.9 = x 1

( )

1.029 = x 2

( )

1.0015 = x 3

( )

1.0000045 = ( ) 1.0 ⇒ stationary point = x 0

( )

1.0 = x 1

( )

1.0 = f 0 ( ) = f 1 ( ) 1 4 ⁄ – = f″ 1 ( ) 3 > = local minimum ⇒ x* 1 local minimum ⇒ = x* ∞ ± =

SLIDE 2

Interval Reduction Using Function Samples Only

1) consider , 2) consider . Algorithm: Interval Halving Search 1. input a, b, ∆min // input the initial interval 2. set: , , 3. while repeat 4. set: , 5. , 6. if then 7. set , , , 8. else if 9. set , , , 10. else 11. set , , 12. end if 13. end if

14. end while
15. output [a, b]

// final interval size is less than ∆min After 2n+1 function evaluation the interval is reduced by , where is the size of the initial interval.

initial bracketed interval I0 = [a, b]

f2 f1

a c b x d

f1: f1 d ( ) f1 c ( ) < I1 ⇒ c b [ , ] = f2 : f2 d ( ) f2 c ( ) > I1 ⇒ a d [ , ] = a b xL xm xu x ∆ b a – = xm a ∆ 2 ⁄ + = fm f xm ( ) = ∆ ∆min > xL a ∆ 4 ⁄ + = fL f xL ( ) = xu b ∆ 4 ⁄ – = fu f xu ( ) = fL fm < b xm = xm xL = fm fL = ∆ b a – = fu fm < a xm = xm xu = fm fu = ∆ b a – = a xL = b xu = ∆ b a – = I2n

1 +

I0 1 2 ⁄ ( )n = n 1 2 3… , , = Io

SLIDE 3

Fibonacci Search Technique

Question How should we pick N successive points in an interval to perform function evaluations such that the final reduced interval is the smallest possible irrespective of the properties of ? Best location for two function evaluations Examples of optimal sampling locations Fibonacci sequence of numbers: , , . Given an interval of proportion and choosing two sample points at .

f x ( )

x 2 1 1+δ initial interval: [0, 2] new interval: [0, 1+δ] x 3 1 1+δ initial interval: [0, 3] final interval: [1, 2] 2 N = 3 x 5 1 4+δ initial interval: [0, 5] final interval: [4, 5] 3 N = 4 x 8 2 1+δ initial interval: [0, 8] final interval: [4, 5] 3 N = 5 4 2 4 5 1

F0 F1 1 = = Fi Fi

1 –

Fi

2 –

+ = i 2 1 ( )∞ = Fi Fi

2 –

Fi

1 –

,

SLIDE 4

Fibonacci type sampling The new interval will be either

r

the first has size , the second has size and therefore they are the same size. The next point is chosen as either

r

. The interval reduction is equal to after N function evaluations. It has been shown that this is the best possible reduction for a given number of function evaluations. At each step the reduction is equal to . The drawback of this procedure is that you must start with and work backwards down the sequence.

Golden Section Search

Given a normalized interval where should two points be chosen such that: 1) size of reduced interval is independent of function. 2) only one function evaluation per interval reduction. Golden section interval reduction, initial interval is [0, 1]. x Fi Fi-2 initial interval: [0, Fi] new interval: [0, Fi-1] or [Fi-2, Fi] Fi-1

0 Fi

1 –

[ , ] Fi

2 –

Fi [ , ] Fi

1 –

Fi Fi

2 –

– Fi

1 –

= Fi

3 –

Fi Fi

3 –

– IN I0 ⁄ 1 FN ⁄ = Fi Fi

1 +

⁄ FN 0 1 [ , ] x1 and x2

x2 x1 1 if f1 < f2 then new interval is [x2, 1] if f2 < f1 then new interval is [0, x1] x1 1 x2

SLIDE 5

Therefore by condition 1 above

r

and by condition 2 from where we call the golden section and the golden mean. After each function evaluation and comparison

f the interval is eliminated or of the original

interval remains. Thus after N function evaluations an interval of original size L will be reduced to a size Algorithm: Golden Section search 1. input a1, b1, tol 2. set 3. 4. for 5. if then 6. set 7. 8. 9. else 10. set 11. 12. 13. end

14. end until

Notes: iterate until

x1 – 1 x2 – = x2 1 x1 – = x1 x2 – x1

1

x1 – 1

=

x1 x2 – x1 x1

2 – x1

2 ⇒ x2 = = x1

2 x1 1 – + x1 ⇒ τ .61803 = = = x2 1 x1 – .38197 θ 1 τ – τ2 = = = = = τ θ θ τ LτN

1 –

c1 a1 1 τ – ( ) b1 a1 – ( ), Fc + F c1 ( ) = = d1 b1 1 τ – ( ) b1 a1 – ( ), Fd – F d1 ( ) = = K 1 2 …repeat , , = Fc Fd < ak

1 +

ak bk

1 +

, dk dk

1 +

, ck = = = ck

1 +

ak

1 +

1 τ – ( ) + bk

1 +

ak

1 +

– ( ) = Fd Fc = Fc , F ck

1 +

( ) = ak

1 +

ck bk

1 +

, bk ck

1 +

, dk = = = dk

1 +

bk

1 +

1 τ – ( ) – bk

1 +

ak

1 +

– ( ) = Fc Fd Fd , F dk

1 +

( ) = = bk

1 +

ak

1 +

– tol < interval tol <

SLIDE 6

Polynomial Interpolation Methods

In an interval of uncertainty the function is approximated by a polynomial and the minimum of the polynomial is used to predict the minimum of the function.

Quadratic Interpolation

If is the interpolating quadratic we need to find the coefficients p, q, and r given the end points of the interval and c such that with , . The coefficients satisfy the matrix equation which can be easily solved analytically. Once we have the coefficients we need to find the minimum of F(x) and thus and therefore we only need the q and p coefficients. These can be solved using Cramer’s rule: and therefore

F x ( ) px2 qx r + + = a b [ , ] a c b < < Fa F a ( ) = Fb F b ( ) = Fc F c ( ) = a2 a 1 b2 b 1 c2 c 1 p q r Fa Fb Fc = x d d F x ( ) g x* ( ) 2px* q + = = = x* q 2p

–

= p 1 ∆

Fa a 1

Fb b 1 Fc c 1 Fa b c – ( ) Fb c a – ( ) Fc a b – ( ) + + ∆

=

= q 1 ∆

a2 Fa 1

b2 Fb 1 c2 Fc 1 Fa b2 c2 – ( ) Fb a2 c2 – ( ) Fc a2 b2 – ( ) – + ∆

–

= = x* q 2p

–

1 2

Fa b2

c2 – ( ) Fb c2 a2 – ( ) Fc a2 b2 – ( ) + + Fa b c – ( ) Fb c a – ( ) Fc a b – ( ) + +

=

=

SLIDE 7

Notice that must lie in if Fc < Fa and Fc < Fb bracketing interval. Also note that and therefore if local minimum of quadratic whereas if local maximum. We now evaluate and compare it to in order to reduce the interval of uncertainty.

f(x*) > fc k

new interval is [ak+1, bk+1] and new c value is ck+1

f(x*) < fc k

new interval is [ak+1, bk+1] and new c value is ck+1

x* a b [ , ] ⇐ f″ x* ( ) 2p x* ( ) = p x* ( ) 0 ⇒ > p x* ( ) 0 ⇒ < f x* ( ) f c ( ) f(x) x ak x* ck bk f(x*) > fc k ak+1 ck+1 bk+1 ⇒

f(x) < fc k f(x) x ak x ck bk ak+1 ck+1 bk+1

⇒

SLIDE 8

Algorithm: Quadratic interpolation search

1. input a1, b1, c1, xtol, ftol 2. set 3. for k = 1,2, . . . repeat 4. set 5. 6. if and then 7. set , , 8. , 9. else if and then 10. set , , 11. 12. else if and then 13. set , , 14. 15. else 16. set , , 17. , 18. end

19. end until
r

Notes: stop when interval reduced to xtol or when relative change in function value < ftol

Fa F a1 ( ) Fb , F b1 ( ) Fc , F c1 ( ) = = = x* 1 2

Fa bk

2 ck

2 – ( ) Fb ck

2 ak

2 – ( ) Fc ak

2 bk

2 – ( ) + + Fa bk ck – ( ) Fb ck ak – ( ) Fc ak bk – ( ) + +

=

Fx F x* ( ) = x* ck > Fx Fc < ak

1 +

ck = bk

1 +

bk = ck

1 +

x* = Fa Fc = Fc Fx = x* ck > Fx Fc > ak

1 +

ak = bk

1 +

x* = ck

1 +

ck = Fb Fx = x* ck < Fx Fc > ak

1 +

x* = bk

1 +

bk = ck

1 +

ck = Fa Fx = ak

1 +

ak = bk

1 +

ck = ck

1 +

x* = Fb Fc = Fc Fx = bk

1 +

ak

1 +

– ( ) xtol < F ck ( ) F ck

1 +

( ) – ( ) F ck ( ) ftol < ⁄

SLIDE 9

Bounding Phase

Question: how do we determine this initial interval?

In general we perform a coarse search to bound or bracket the minimum x*. This is also called interval location. We will discuss two types of interval location: 1) function comparison (Swann’s Bracketing Method); and 2) polynomial extrapolation (Powell’s Method). Both of these methods are heuristic in nature but are about the best we can provide in terms of an

algorithm. We start with a description of Swann’s bracketing method.

Function comparison: Swann’s Bracketing Method

Given an initial step length

and starting point we check the point a distance away on the right side of .

If

then we will move to the right.

If

then we will move to the left (lets say we move to the right).

We now magnify the step size by a magnification factor, say 2, and evaluate the function at the

point away from the latest point.

As soon as any new function evaluation shows an increase we can say that the last two inter-

vals provide a bound or interval of uncertainty for our minimum.

Example of Swann’s bracketing method, magnification factor is 2.0.

∆ a1 ∆ a1 f a1 ∆ + ( ) f a1 ( ) < f a1 ∆ + ( ) f a1 ( ) > 2∆ 8∆ 4∆ 2∆ x F(x) move to right until function evaluation increase, then last two intervals are interval of uncertainty. a1 b1 a2 b2 a3 b3 a4 b4 ∆

SLIDE 10

Algorithm: Swann’s Bracketing Method - based on a heuristic expanding pattern.

1. input: , // initial point and step size 2. set: // lower test point. 3. // upper test point. 4. // lower function value 5. // upper function value 6. // central function value 7. // expanding exponent 8. if // case 1: move to the right 9. loop-up: 10. set: , , , 11. // shift up in x by a magnified amount 12. 13. if go to loop-up 14. else output

15. if

// case 2: move to the left 16. loop-down: 17. set: , , , 18. // shift down in x by a magnified amount 19. 20. if go to loop-down 21. else output

22. if

// case 3: initial interval is bracket 23.

utput
24. if

error: non-unimodal function

x0 ∆ xl x0 ∆ – = xu x0 ∆ + = fl f xl ( ) = fu f xu ( ) = f0 f x0 ( ) = i 1 = fl f0 fu ≥ ≥ i i 1 + = xl x0 = x0 xu = fl f0 = f0 fu = xu xu 2i ∆ + = fu f xu ( ) = fu f0 < xl xu , ( ) fl f0 fu ≤ ≤ i i 1 + = xu x0 = x0 xl = fu f0 = f0 fl = xl xl 2i ∆ – = fl f xl ( ) = fl f0 < xl xu , ( ) fl f0 fu ≤ ≥ xl xu , ( ) fl f0 fu ≥ ≤

SLIDE 11

Polynomial Extrapolation: Powell’s Method

we use polynomial extrapolate from the initial starting point.
Given a starting point a, and two more points

apart we extrapolate a polynomial through them (we will use a quadratic function).

Now in this case the fitted quadratic may have either a maximum or minimum and this

may be determined by evaluating the second derivative.

As before if the polynomial is represented as

then the second derivative is given as and in terms of the three function values Fa, Fb, Fc, evaluated at x = a, b, and c we can evaluate p as

.

Now how do we use this to find an interval?

Starting from three points, a1, c1, and b1, a ∆ apart we fit a quadratic to these points, say F(x).
If we find that this quadratic has a maximum, that is p < 0, then the next point we take is ∆max

away from the smallest function value. (See figure below) where for sake of an example we are assuming that we move to the right.

Extrapolated polynomial has a maximum therefore use the ∆max.

We discard the point with smallest function value and take new point

away from b1. Discarding the lowest function value gives us more of a chance of fitting a polynomial with a minimum in the next iteration of the algorithm. ∆ F x ( ) px2 qx r + + = F″ x ( ) G x ( ) 2p = = p c b – ( )Fa a c – ( )Fb b a – ( )Fc + + b c – ( ) c a – ( ) a b – ( )

=

x* a1 c1 b1 a2 c2 b2 ∆max f(x) case 1: F(x) has a maximum, i.e. p < 0 F(x) ∆max

SLIDE 12

If the polynomial does have a minimum then we either use the minimum of the polynomial as

shown in case 3 or we again use ∆max if the minimum is farther than ∆max from b1 as shown in case 2.

Extrapolated polynomial has a minimum but it is too far away so use ∆max. Extrapolated polynomial has a minimum and it is closer than ∆max from b1.

Algorithm: Interval location by Powell’s Method

1. input , , 2. set , , 3. if then 4. set , 5. forward = true 6. else 7. set , , 8. , , 9. forward = false

∆max f(x) a1 c1 b1 a2 c2 b2 F(x) x* case 2: F(x) has a minimum, i.e. p > 0 ∆max f(x) a1 c1 b1 a2 c2 b2 F(x) x* case 3: F(x) has a minimum, i.e. p > 0, and x* - b1 < ∆max a1 ∆ ∆max c1 a1 ∆ + = Fa F a1 ( ) = Fc F c1 ( ) = Fa Fc > b1 a1 2∆ + = Fb F b1 ( ) = b1 c1 = c1 a1 = a1 a1 ∆ – = Fb Fc = Fc Fa = Fa F a1 ( ) =

SLIDE 13

10. end
11. for

12. set 13. if then 14. set 15. end if 16. if forward then // moving forward 17. if then // quadratic has a maximum 18. set , 19. , 20. else 21. if then // quadratic minimum is too far 22. set 23. else // quadratic minimum is O.K. 24. set 25. end 26. set , 27. , , 28. end 29. else // moving backward (forward = false) 30. if then // quadratic has a maximum 31. set , 32. , 33. else 34. if then // quadratic minimum is too far 35. set 36. else // quadratic minimum is O.K. 37. set 38. end 39. set , 40. , , 41. end // if 42. end

43. end until

and

K 1 2 … repeat , , = p cK bK – ( )Fa aK cK – ( )Fb bK aK – ( )Fc + + bk cK – ( ) cK aK – ( ) aK bK – ( )