Overview of Line Search Topics Problem Definition Problem - - PowerPoint PPT Presentation

▶
overview of line search topics problem definition problem
SMART_READER_LITE
LIVE PREVIEW

Overview of Line Search Topics Problem Definition Problem - - PowerPoint PPT Presentation

Jan 22, 2023 119 likes •233 views

Overview of Line Search Topics Problem Definition Problem definition f ( ) Line search algorithms Uniform search Dichotomous search The line search problem: find a scalar R 1 such that Golden section search

slide-1
SLIDE 1

Motivation

α f(α)

  • We saw earlier that there was an optimal smoothness parameter

for each of our smoothers

  • We could pick the smoothness parameter to optimize the

estimated prediction error

  • Calculating the prediction error can be time consuming
  • How do we do this efficiently?
  • This is an example application for line search algorithms
  • Can also be used to optimize design parameters to maximize some

metric of performance

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

3

Overview of Line Search Topics

  • Problem definition
  • Line search algorithms

– Uniform search – Dichotomous search – Golden section search – Quadratic fit search

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

1

Convexity

  • A function f(α) is convex[2, pp. 79] if

f(λα1 + (1 − λ)α2) ≤ λf(α1) + (1 − λ)f(α2) for all 0 ≤ λ ≤ 1

  • A function f(α) is quasiconvex[2, p. 108] if

f(λα1 + (1 − λ)α2) ≤ max [f(α1), f(α2)] for all 0 ≤ λ ≤ 1

  • A differentiable function f(α) is pseudoconvex[2, pp. 113–114] if

for every ∇xf(x1)(x2 − x1) ≥ 0, we have f(x2) ≥ f(x1)

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

4

Problem Definition

α f(α)

  • The line search problem: find a scalar α ∈ R1 such that

α∗ = argmin

α

f(α) using as few evaluations of f(α) as possible (see [1, 7.1–7.4] and [2, 8.1–8.4])

  • The optimization problem: find a vector a ∈ Rp such that

a∗ = argmin

α

f(a) using as few evaluations of f(a) as possible – A generalization of the line search problem to multiple dimensions

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

2

slide-2
SLIDE 2

Interval Bounding Algorithm This is one popular algorithm for finding the upper limit for a line search algorithm. The user specifies the initial step α1 and picks the expansion rate c.

  • Evaluate the function at α0 = 0
  • Evaluate the function at α1
  • Evaluate the function at α2 = α1 × c
  • k = 1
  • Until f(αk−1) > f(αk) and f(αk) < f(αk+1)

– k := k + 1 – αk+1 := αk × c

  • There is a minimum between αk−1 and αk+1

Note that this algorithm may also increase the lower bound from its initial value of 0.

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

7

Goal

α f(α)

  • There is no line search algorithm that is guaranteed to find the

global minimum for any function f(α)

  • All of the line search algorithms try to find a possibly local

minimum as quickly as possible

  • They are designed with functions with some type of convexity in

mind

  • Work with non-convex functions, but very few compare or try to

find the deepest local minimum

  • Many heuristic techniques for searching for the global minimum,

but none provide any guarantees or any claims of optimality

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

5

Example 1: Interval Bounding

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.2 0.4 0.6 0.8 1

1 2 3 4

α f(α)

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

8

Interval Bounding Problem

α f(α)

  • Many line search algorithms try to find a local minimum in the

range [0, ∞] as quickly as possible

  • If the criteria contains multiple local minima, they find one of

them quickly

  • Otherwise they find the global minima
  • Most of these algorithms require that the minima be constrained

to a finite range: α∗ ∈ [αmin, αmax]

  • By the nature of the problem, the lower limit is usually known to

be zero: αmin = 0

  • Often the upper limit has to be found
  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

6

slide-3
SLIDE 3

Bounding the Line Search Interval More generally, the line search interval can be found by “interval doubling”. Assume that α is constrained to being a positive number.

  • 1. Pick initial values of αmin
  • 2. While f(αmin) < f(αmax), αmin = αmin/c
  • 3. While f(αmin) > f(αmax), αmax = αmax × c

Typically c = 2, but any c > 1 could be used.

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

11

Example 1: MATLAB Code

function [] = IntervalBound ; st = 2; % Step size x = zeros (100 ,1); x(1) = 0; x(2) = 0.1; % First step x(3) = x(2)* st; cnt = 3; while ¬(LSFn(x(cnt -2)) > LSFn(x(cnt -1)) & LSFn(x(cnt -1)) < LSFn(x(cnt ))) cnt = cnt + 1; x(cnt) = x(cnt -1)* st; end x = x(1: cnt ); x0 = x(cnt -2); x1 = x(cnt ); figure; FigureSet (1,’Slides ’); u = 0:0 .01 :1; h = patch ([x0 x1 x1 x0],[0 0 1 1],’k’); set(h,’LineStyle ’,’None ’); set(h,’FaceColor ’,0.8 *[1 1 1]); % Light gray region hold on; h = plot(u,LSFn(u),’r’,x,LSFn(x),’.’); set(h(2),’MarkerSize ’ ,8); set(h,’LineWidth ’,1.2); hold

  • ff;
  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

9

Uniform Search

α f(α)

  • Pick the search points α1, α2, . . . , αn so that they are uniformly

spaced over some preset range

  • Then pick the best α

α∗ = argminα∈{a1,a2,...,an} f(α) + No assumptions about convexity or shape of f(α) + Finds (nearly) a global minimum − Relatively inefficient

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

12

for c1 = 1: length(x), h = text(x(c1 )+0.01 ,LSFn(x(c1)), num2str(c1 -1)); set(h,’HorizontalAlignment ’,’Left ’); set(h,’VerticalAlignment ’,’Middle ’); end box off; FigureLatex; xlabel(’α’); ylabel(’f(α)’); AxisSet (8); print -depsc IntervalBound ;

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

10

slide-4
SLIDE 4

Dichotomous Search

  • Suppose you know that α is in the range of αmin to αmax
  • 1. Calculate the following evaluation points

b = αmin + αmax 2 − ǫ c = αmin + αmax 2 + ǫ

  • 2. If f(b) < f(c), set αmax = c

Otherwise, set αmin = b

  • 3. Repeat until convergence
  • If the derivative df(α)

dα

can be calculated, the computation can be reduced to one evaluation of the derivative per an iteration

  • If the derivative is used, this is called the bisection method

− Converges to a local minimum (global if f(α) is quasiconvex) + Faster than a uniform search

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

15

Example 2: Uniform Search

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.2 0.4 0.6 0.8 1 Resolution:0.2222 α f(α)

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

13

Example 3: Dichotomous Search

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.2 0.4 0.6 0.8 1

1 3 5 7 9

α f(α) Resolution:0.0332

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

16

Example 2: MATLAB Code

function [] = UniformSearch ; ll = 0; % Lower limit ul = 1; % Upper limit np = 10; % No. of points x = ll:(ul -ll)/(np -1): ul; f = LSFn(x); xrng = (ul -ll )/(np -1); [fmin ,id] = min(f); xmin = x(id); figure; FigureSet (1,’Slides ’); u = 0:0 .01 :1; h = plot(u,LSFn(u),’r’,x,f,’.’,xmin ,fmin ,’g.’); set(h(2),’MarkerSize ’ ,8); set(h,’LineWidth ’,1.2); set(h(3),’MarkerSize ’ ,15); box off; st = sprintf(’Resolution :%6 .4f ’ ,2*min(diff(x))); title(st); FigureLatex; xlabel(’α’); ylabel(’f(α)’); AxisSet (8); print -depsc UniformSearch ;

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

14

slide-5
SLIDE 5

[fmin ,id] = min(f); xmin = x(id); xrng = ul -ll; figure; FigureSet (1,’Slides ’); u = 0:0 .01 :1; h = plot(u,LSFn(u),’r’,x,f,’.’,xmin ,fmin ,’g.’); set(h(2),’MarkerSize ’ ,8); set(h,’LineWidth ’,1.2); set(h(3),’MarkerSize ’ ,15); for c1 = 1:2: length(x), h = text(x(c1),f(c1 )+0.01 ,num2str(c1 )); set(h,’HorizontalAlignment ’,’Center ’); set(h,’VerticalAlignment ’,’Bottom ’); end; FigureLatex; xlabel(’α’); ylabel(’f(α)’); st = sprintf(’Resolution :%6 .4f\n’,xrng ); title(st); box off; AxisSet (8); print -depsc DichotomousSearch ; figure; FigureSet (2,’Slides ’); k = 1: length(f); u = 0:0 .01 :1; h = stem(k,f,’r’); set(h,’MarkerFaceColor ’,’r’);

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

19

Example 3: Dichotomous Error

1 2 3 4 5 6 7 8 9 10 0.05 0.1 0.15 0.2 0.25 Iteration f(αi)

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

17

xlim ([0 .5 length(f)+0 .5 ]); ylim ([0 1.05*max(f)]); FigureLatex; ylabel(’f(αk)’); xlabel(’Iteration (k)’); box off; AxisSet (8); print -depsc DichotomousError;

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

20

Example 3: MATLAB Code

function [] = DichotomousSearch ; clear all; close all; ll = 0; % Lower limit ul = 1; % Upper limit np = 10; % No. of points eta = 0.001; % Dither cnt = 0; x = zeros(np ,1); f = zeros(np ,1); for c1 = 1:np/2, mp = (ul+ll )/2; b = mp -eta; fb = LSFn(b); c = mp+eta; fc = LSFn(c); if fb <fc , ul = c; else ll = b; end; cnt = cnt + 1; x(cnt) = b; f(cnt) = fb; cnt = cnt + 1; x(cnt) = c; f(cnt) = fc; end;

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

18

slide-6
SLIDE 6

Example 4: Golden Section Search

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.2 0.4 0.6 0.8 1

1 2 3 4 5 6 7 8 9 10

α f(α) Resolution:0.0213

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

23

Golden Section Line Search Suppose you know that α is in the range of αmin to αmax

  • 1. Evaluate f(α) at αmin,b,c, and αmax where

b0 = αmin + (1 − γ)(αmax − αmin) c0 = αmin + γ(αmax − αmin)

  • 2. If f(bk) > f(ck),

αmax,k+1 = αmax,k αmin,k+1 = bk bk+1 = ck ck+1 = αmin,k+1 + γ(αmax,k+1 − αmin,k+1)

Else,

αmin,k+1 = αmin,k αmax,k+1 = ck ck+1 = bk bk+1 = αmin,k+1 + (1 − γ)(αmax,k+1 − αmin,k+1)

  • 3. Loop to 2 until convergence
  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

21

Example 4: Golden Section Error

1 2 3 4 5 6 7 8 9 10 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Iteration (k) f(αk)

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

24

Golden Section Line Search Comments

α f(α)

  • γ ≈ 0.618
  • This is the inverse of the “Golden ratio”: φ = 1

2(1 +

√ 5)

  • Not a rational number
  • Consider a line segment from A to C
  • Place a point B such that AB

BC = BC AC

  • This is the golden ratio
  • For more info, search the web

− Converges to a local minimum (global if quasi-convex) + More efficient than dichotomous search

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

22

slide-7
SLIDE 7

xlabel(’α’); ylabel(’f(α)’); st = sprintf(’Resolution :%6 .4f\n’,xrng ); title(st); box off; AxisSet (8); print -depsc GoldenSectionSearch; figure; FigureSet (2,’Slides ’); k = 1: length(f); u = 0:0 .01 :1; h = stem(k,f,’r’); set(h,’MarkerFaceColor ’,’r’); xlim ([0 .5 length(f)+0 .5 ]); ylim ([0 1.05*max(f)]); FigureLatex; ylabel(’f(αk)’); xlabel(’Iteration (k)’); box off; AxisSet (8); print -depsc GoldenSectionError;

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

27

Example 4: MATLAB Code

function [] = GoldenSection ; clear all; close all; ll = 0; % Lower limit ul = 1; % Upper limit np = 10; % No. of points r = 0.618; % Dither b = ll + (1-r)*(ul -ll); c = ll + r*(ul -ll); fb = LSFn(b); fc = LSFn(c); x = zeros(np ,1); f = zeros(np ,1); x(1) = b; x(2) = c; f(1) = fb; f(2) = fc; cnt = 2; for cnt = 3:np , if fb >fc , ll = b; b = c; fb = fc; c = ll + r*(ul -ll);

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

25

Quadratic Fit Line Search

  • 1. Find α1, α2, and α3 such that f(α1) ≥ f(α2) and f(α2) ≤ f(α3)
  • 2. Find α∗ as the minimum to the quadratic fit of f(α1), f(α2), and

f(α3)

  • 3. If α∗ > α2,

If f(α∗) > f(α2), then αnew = {α1, α2, α∗} If f(α∗) ≤ f(α2), then αnew = {α2, α∗, α3} Else, If f(α∗) > f(α2), then αnew = {α∗, α2, α3} If f(α∗) ≤ f(α2), then αnew = {α1, α∗, α2}

  • 4. Loop to 2 until convergence
  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

28

fc = LSFn(c); x(cnt) = c; f(cnt) = fc; else ul = c; c = b; fc = fb; b = ll + (1-r)*(ul -ll); fb = LSFn(b); x(cnt) = b; f(cnt) = fb; end; end; [fmin ,id] = min(f); xmin = x(id); xrng = ul -ll; figure; FigureSet (1,’Slides ’); u = 0:0 .01 :1; h = plot(u,LSFn(u),’r’,x,f,’.’,xmin ,fmin ,’g.’); set(h(2),’MarkerSize ’ ,8); set(h,’LineWidth ’,1.2); set(h(3),’MarkerSize ’ ,15); for c1 = 1: length(x), h = text(x(c1),f(c1 )+0.01 ,num2str(c1 )); set(h,’HorizontalAlignment ’,’Center ’); set(h,’VerticalAlignment ’,’Bottom ’); end; FigureLatex;

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

26

slide-8
SLIDE 8

Example 5: Quadratic Error

1 2 3 4 5 6 7 8 9 10 0.2 0.4 0.6 0.8 1 Iteration (k) f(αk)

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

31

Quadratic Fit Line Search Comments

α f(α)

  • Very popular
  • Often includes a safeguard technique to handle ill-conditioning

effects + Very fast convergence − Can be unstable if f(α) is not quasi-convex and safeguard technique is not used − Finds a local minimum

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

29

Example 5: MATLAB Code

function [] = QuadraticSearch ; ll = 0; % Lower limit ul = 1; % Upper limit np = 10; % No. of points x = zeros(np ,1); f = zeros(np ,1); xl = ll; fl = LSFn(xl); % Left edge xc = (ll+ul )/2; fc = LSFn(xc); % Center xr = ul; fr = LSFn(xr); % Right edge x(1) = xl; f(1) = fl; x(2) = xc; f(2) = fc; x(3) = xr; f(3) = fr; for cnt = 4:np , if fc <fl & fc <fr , % Criteria

  • kay - do

quadratic search iteration acr = xc - xr; bcr = xc^2 - xr^2; arl = xr - xl; brl = xr^2 - xl^2; alc = xl - xc; blc = xl^2 - xc^2; xn = 0.5*( bcr*fl + brl*fc + blc*fr )/( acr*fl + arl*fc + alc*fr ); fn = LSFn(xn); x(cnt) = xn; f(cnt) = fn; if xn >xc , if fn ≥ fc ,

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

32

Example 5: Quadratic Search

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.2 0.4 0.6 0.8 1

1 2 3 4 5 6 7 8 9 10

α f(α) Resolution: 0.0071

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

30

slide-9
SLIDE 9

xlim ([0 .5 length(f)+0 .5 ]); ylim ([0 1.05*max(f)]); FigureLatex; ylabel(’f(αk)’); xlabel(’Iteration (k)’); box off; AxisSet (8); print -depsc QuadraticError;

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

35

xr = xn; fr = fn; else xl = xc; fl = fc; xc = xn; fc = fn; end; else if fn ≥ fc , xl = xn; fl = fn; else xr = xc; fr = fc; xc = xn; fc = fn; end; end; else % Safety switchover to bisection

  • safeguard

technique x2 = xn + 0.001 *(xr -xl); f2 = LSFn(x2); x(cnt) = x2; f(cnt) = f2; if fn <f2 , xr = x2; fr = f2; else % fc >fr xl = xc; fl = fc; end; xc = (xl+xr )/2; fc = LSFn(xc); % Center cnt = cnt + 1; x(cnt) = xc; f(cnt) = fc; end; end;

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

33

Line Search Algorithm Comments

  • If the No. iterations is known in advance, there is yet another

algorithm (Fibonacci search) that is more efficient than the golden section

  • If the No. iterations is not known, golden section is more efficient

than Fibonacci search, dichotomous search, or uniform search

  • If the second derivative can be calculated, there is another line

search algorithm called Newton’s method

  • How do you know when the algorithm has converged?
  • There are a several different popular criteria
  • In practice, the algorithms are often stopped after

– Some user-specified number of iterations – The interval of uncertainty is reduced to some threshold

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

36

[fmin ,id] = min(f); xmin = x(id); xrng = xr -xl; figure; FigureSet (1,’Slides ’); u = 0:0 .01 :1; h = plot(u,LSFn(u),’r’,x,f,’.’,xmin ,fmin ,’g.’); set(h(2),’MarkerSize ’ ,8); set(h,’LineWidth ’,1.2); set(h(3),’MarkerSize ’ ,15); for c1 = 1: length(x), h = text(x(c1),f(c1 )+0.01 ,num2str(c1 )); set(h,’HorizontalAlignment ’,’Center ’); set(h,’VerticalAlignment ’,’Bottom ’); end; FigureLatex; xlabel(’α’); ylabel(’f(α)’); st = sprintf(’Resolution: %6 .4f\n’,xrng ); title(st); box off; AxisSet (8); print -depsc QuadraticSearch ; figure; FigureSet (2,’Slides ’); k = 1: length(f); u = 0:0 .01 :1; h = stem(k,f,’r’); set(h,’MarkerFaceColor ’,’r’);

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

34

slide-10
SLIDE 10

Line Search Algorithm Comments Continued

α f(α)

  • All of these methods may get stuck in a local minimum
  • Uniform search is the least likely to get stuck, but is also the least

efficient

  • Hybrid searches are often used for better convergence
  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

37

References

[1] David G. Luenberger. Linear and Nonlinear Programming. Adison-Wesley, second edition, 1989. [2] Mokhtar S. Bazaraa, Hanif D. Sherali, and C. M. Shetty. Nonlinear Programming: Theory and Algorithms. John Wiley & Sons, Inc., second edition, 1993.

  • J. McNames

Portland State University ECE 4/557 Line Search Algorithms

  • Ver. 1.14

38