SLIDE 1
Computational Optimization Convergence Rates Lecture 3 1/24/08 - - PowerPoint PPT Presentation
Computational Optimization Convergence Rates Lecture 3 1/24/08 - - PowerPoint PPT Presentation
Computational Optimization Convergence Rates Lecture 3 1/24/08 Golden Section Search Basic idea: Economize the number of function evaluations by trapping a min solutions in a set of nested intervals. Begin by evaluating: f(a o ), f(left o
SLIDE 2
SLIDE 3
Which interval contains x*?
a left b right a left b right a left b right
f(left) < f(right) → [a,right] f(left) > f(right) f(left) = f(right) → [left,right] → [left,b]
SLIDE 4
Choice of τ
a0 l0 b0 r0 Iteration 0 a1 a0 r1 l0 b0 b1 r0 l1
- Want l0 = r1
= a1+τ(b1-a1) = a0+τ(r0-a1) = a0+τ(a0+τ(b0-a0) –a0) = a0+ τ2(b0-a0)
Iteration 1
SLIDE 5
Choice of τ Continued
Have: l0 = b0-τ(b0-a0)= a0 + τ2(b0-a0) Therefore 0=(τ2 + τ -1)(b0-a0) Which implies
τ is Golden Section
1 5 .6 1 8 2 τ − + = ≈
SLIDE 6
Golden Section Search Alg.
1.
ao, bo ,f,tol, k=0
2.
r0 = a0+τ(b0-a0), l0 = b0-τ(b0-a0)
3.
fl=f(lk),fr=f(rk), fa=f(a0 ),fb=f(b0 ),
4.
while [ak, bk] > tol
1.
If f(lk) < f(rk) then ak+1= ak, bk+1=r k rk+1=lk, lk+1= bk+1-τ(bk+1-ak+1), fa=fa, fb =fr, fr=fl, fl=f( lk+1)
2.
else if f(lk) >= f(rk) then ak+1= lk, bk+1=bk lk+1= rk, rk+1=ak+1+τ(bk+1-ak+1), fa=fl, fb =fb, fl=fr, fr=f( rk+1)
3.
k=k+1
5. x*= (ak+bk)/2
SLIDE 7
Challenges
Define an algorithm to optimize problem Does the algorithm converge? What does it converge to? Under what assumptions is the algorithm valid? How fast does the algorithm work in theory? How well does the algorithm work in practice?
SLIDE 8
Analysis of GS Search
The length of the interval of uncertainty,bk – ak,is reduced by τ at each interval. Length of interval of uncertainty. (b1 – a1)= τ (b0 – a0). (b2 – a2)= τ(b1 – a1)= τ2 (b0 – a0). …. (bk– ak)= τ (bk-1 – ak-1)= τk(b0 – a0).
SLIDE 9
What Is the Quality of the Solution?
Program halts when (ak - bk)<acc=tol(b0 – a0) We know |xk-x*|<acc/2. What other measures are there of the. Quality of a solution?
|f(xk)-f(x*)|. |f’(xk)| if differentiable since
f’(x*)=0 at min for diff. funct.
SLIDE 10
Error Analysis
The midpoint xk = (ak + bk)/2 of the interval of uncertainty is the best current estimate of x*. Why? Error(0) = |x0 – x*| = (b0 – a0)/2. Error(1) = |x1 – x*| = (b1 – a1)/2=τ (b0 – a0)/2. …… Error(k) = |xk – x*| = (bk – ak)/2=τk(b0 – a0)/2. Error reduces by τ each iteration. Called linear or geometric convergence.
SLIDE 11
Convergence Analysis
Interval shrinks by τ each iteration. So algorithm must terminate. Algorithm halts when (bk – ak)<acc. For acc = tol(b0 – a0), tol(b0 – a0)≥(bk – ak)= τk(b0 – a0), tol>= τK implies log(tol) ≥ k log(τ), So halts when k >= ceil(log(tol)/log(τ)).
SLIDE 12
How Efficient?
For acc = tol(b0 – a0), 1/log(τ)=-4.784. Halts when k = ceil(-log(tol)4.784). Dominate computation cost is function evaluations. Requires 4 + k function evaluations. Does golden section search perform like this in practice?
SLIDE 13
Rates of Convergence
Local analysis: How does the algorithm behave near solution? Assume xk→x*
- r
f(xk )→f(x*) ek=|| xk-x*|| or ek=| f(xk)-f(x*)| In GS search ek+1 ≤ τek = τkeo
SLIDE 14
Linear Convergence See page 619-621 in MW
A sequence converges linearly or geometrically if ∃ q>0 and β∈[0,1] such that ek≤qβk for k sufficiently large
- r equivalently
1
lim
k k k
e e β
+ → ∞
≤
SLIDE 15
General Convergence
We say {xk} converges to x* with rate r and rate constant c if If r=1 0<c<1 then linear rate. If (r=1 and c=0) or (1<r<2) then superlinear rate. If r=2 then quadratic rate.
1
li m ( )
k r k k
e c e
+ → ∞
=
SLIDE 16
Sample convergence rates
Linear 1, .1, .01, .001, .0001, .00001,… Quadratic 1, .1, .001, 1e-9,1e-27,….
SLIDE 17
How fast does it converge?
Let xk+1=(xk+1)/2 (implies x*=1) Define ek+1=|xk+1-1|=|(xk+1)/2-1| =|(xk-1)/2| Thus So linear convergence Doesn’t work for r>1, so not superlinear.
1
( 1) / 2 1 lim 1 2 1
k k r r k k k
x e for r e x
+ →
− = = = −
SLIDE 18
You try
Consider What does {xk} converge to? What is the convergence rate?
2
(0,1)
k
k
x a for a = ∈
SLIDE 19
Quadratic Convergence
[ ]
1 1 1
2 2 1 2 2 2 2 2 1 2
lim 2 2 .
k k k k k k k
k k r r k r r k k
e a e a e a a a e a provided r
- r
r
+ + +
+ ⎡ ⎤ − − + ⎣ ⎦ →∞
= = = = = <∞ ⎡ ⎤ ⎣ ⎦ − ≥ ≥
SLIDE 20