SLIDE 1
Line Search type Algorithm
Start with x0 For k =1,…,K
If xk is optimal then stop Pick descent direction pk Perform linesearch to determine
Computational Optimization Practical Linesearch 2/12 Line Search - - PowerPoint PPT Presentation
Computational Optimization Practical Linesearch 2/12 Line Search - - PowerPoint PPT Presentation
Computational Optimization Practical Linesearch 2/12 Line Search type Algorithm Start with x 0 For k =1,,K If x k is optimal then stop Pick descent direction p k Perform linesearch to determine x k+1 =x k + k p k Outline
SLIDE 2
SLIDE 3
Outline
Linesearch Conditions
Armijo + backtracking search Wolfe Conditions Strong Wolfe Conditions Goldstein (not covered in class)
Linesearch Algorithms
SLIDE 4
Why linesearch?
Decide how far to step along a descent direction. Just decreasing is not enough Satisfy certain properties to guarentee global convergence of an algorithm Exact properties depends on the algorithm being used
SLIDE 5
Armijo Condition
( ) ( ) '(0 ) ( ) '
k k
k k
g f x p g f x p α α = + = ∇
g(α) α g(0)+ α g’(α) g(0)+ c1μg’(α) 1 1/2
1
( ) ( ) ( ) '
k k k
k k
f x p f x c f x p α α + ≤ + ∇
SLIDE 6
Backtracking Search
Key point: Stepsize cannot be allowed to go to 0 unless gradient is going to 0. Must have sufficient decrease. Fix δ>0 β∈(0,1) μ ∈(0,1)
1 2
max( , , ,...) which satisfies ( ) ( ) ( ) '
k k
k k k k k k
f x p f x f x p α δ β β β α μα = + ≤ + ∇
SLIDE 7
Backtracking Search
Step is fixed value or small enough to satisfy Armijo condition and not too small Not too small guarentees sufficient decrease for convergence. Easy to implement Initial step size may be too small Good for Newton’s method. Not so hot for steepest descent
SLIDE 8
Step-Length Selection
Approximately solve efficiently to satisfy some condition Two phases
Bracketing – find interval Zoom – find step in interval
( ) ( ) ( ) '
k k
k k
g f x p fo r f x p α α = + ∇ <
SLIDE 9
Quadratic Interpolation
Approximate g(α) with h(0) h(0)=g(0), h’(0) =g’(0), g(α0)
g(α0) α α1
) ( ) ( ' ) ( ' ) ( ) ( ) (
2 2
g g g g g h + + ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ − − = α α α α α α α
SLIDE 10
Quadratic Interpolation
Potential step
g(α0) α α1
2 1
'( ) 2( ( ) (0) '( )) g g g g α α α α α α = − −
SLIDE 11
Cubic Interpolation
3 2 2 2 1 1 1 2 2 3 3 1 1 1 2 2
( ) '(0) (0) ( ) (0) '(0) 1 ( ) ( ) (0) '(0) 3 '(0) 3 h a b g g a g g g b g g g b b ag a α α α α α α α α α α α α α α α α α = + + + ⎛ ⎞⎛ ⎞ − − − ⎡ ⎤ = ⎜ ⎟⎜ ⎟ ⎢ ⎥ − − − − ⎣ ⎦ ⎝ ⎠⎝ ⎠ − + − =
SLIDE 12
Initial Step Length
Assume this step will be on the order of last step Interpolate Sometimes min of
k k k k k
- k
k k k k
- p
f p f
- r
p f p f ' ' ' '
1 1 1 1 1 1
∇ ∇ = ∇ = ∇
− − − − − −
α α α α
1 1
, , ' and take as min 2( ) '(0)
k k k k
- k
k
- f
f f p f f g α α
− −
∇ − =
α
min(1.01 ,1)
- α
SLIDE 13