NLO: April 18, 2013 Dr. Thomas M. Surowiec Humboldt University of - PowerPoint PPT Presentation

Step Size Strategies and Algorithms The Wolfe-Powell Rule NLO: April 18, 2013 Dr. Thomas M. Surowiec Humboldt University of Berlin Department of Mathematics Summer 2013 Dr. Thomas M. Surowiec BMS Course NLO, Summer 2013

Step Size Strategies and Algorithms The Wolfe-Powell Rule Armijo Rule Definition 1.1 Let σ ∈ ( 0 , 1 ) be fixed. The Armijo rule is a condition which ensures sufficient descent in the sense that f ( x + α d ) ≤ f ( x ) + σα ∇ f ( x ) T d . discussion On the board. Dr. Thomas M. Surowiec BMS Course NLO, Summer 2013

Step Size Strategies and Algorithms The Wolfe-Powell Rule Armijo Step-Size Strategy Algorithm 1 Armijo Step Size Strategy Descent Direction d , l := 0, α ( 0 ) = 1, 0 < ν 1 ≤ ν 2 < 1 Input: 1: while Armijo Rule not fulfilled do Determine α ( l + 1 ) ∈ [ ν 1 α ( l ) , ν 2 α ( l ) ] 2: Set l := l + 1. 3: 4: end while Set α k := α ( l ) . Dr. Thomas M. Surowiec BMS Course NLO, Summer 2013

Step Size Strategies and Algorithms The Wolfe-Powell Rule Armijo Step-Size Strategy Algorithm 2 Armijo Step Size Strategy Descent Direction d , l := 0, α ( 0 ) = 1, 0 < ν 1 ≤ ν 2 < 1 Input: 1: while Armijo Rule not fulfilled do Determine α ( l + 1 ) ∈ [ ν 1 α ( l ) , ν 2 α ( l ) ] 2: Set l := l + 1. 3: 4: end while Set α k := α ( l ) . Dr. Thomas M. Surowiec BMS Course NLO, Summer 2013

Step Size Strategies and Algorithms The Wolfe-Powell Rule Analysis of the Armijo Step-Size Strategy Lemma 1.1 Let f ∈ C 1 , 1 ( R n , R ) , L Lipschitz constant for ∇ f. Let x k � � be generated by Algorithm 1 (cf. Lecture Notes Algorithm 1) using Algorithm 2 for choice of M k � � step size. Let be a sequence of symmetric pos. def. matrices such that: d k = − M k ∇ f ( x k ) 1 2 There exist λ 1 , λ 2 such that 0 < λ 1 ≤ λ 2 < + ∞ with λ 1 ≤ λ ( k ) ≤ λ ( k ) ≤ λ 2 , ∀ k ∈ N . s g Then α k fulfills α k ≥ α ∗ = 2 ν 1 λ 1 ( 1 − σ ) , ∀ k ∈ N L κ ∗ with κ ∗ = λ 2 /λ 1 . Furthermore, in every iteration of Algorithm (cf. Lecture Notes Algorithm 1) there will be at most m ≤ log ( 2 λ 1 ( 1 − σ ) ) / log ( ν 2 ) , m ∈ N Lk ∗ step size reductions necessary. Dr. Thomas M. Surowiec BMS Course NLO, Summer 2013

Step Size Strategies and Algorithms The Wolfe-Powell Rule How Important is the Lipschitz Assumption? Without the assumption that ∇ f : R n → R n is Lipschitz continuous, there � � might be subsequences of step sizes α k ( l ) such that α k ( l ) → 0. Thus, the main assumption of Lemma 1.1 is violated. Discussion on the board. Dr. Thomas M. Surowiec BMS Course NLO, Summer 2013

Step Size Strategies and Algorithms The Wolfe-Powell Rule Convergence without Sufficient Descent Theorem 1.1 Let f R n → R be continuously differentiable and let M k � � fulfill the � f ( x k ) � assumptions of Lemma 1.1. It holds that either is unbounded from below or k →∞ ∇ f ( x k ) = 0 , lim � x k � is a stationary point of f in R n . In and thus every accumulation point of � f ( x k ) � particular, it holds ture that if is bounded from below and lim l → + ∞ x k ( l ) = x ∗ , then ∇ f ( x ∗ ) = 0 . Proof. This will be a homework question. Refer back to the proofs of the previous three results going back to the lecture notes from April 16. Note: there is no guarantee that a unique accumulation point exists! Dr. Thomas M. Surowiec BMS Course NLO, Summer 2013

Step Size Strategies and Algorithms The Wolfe-Powell Rule Alternatives and Extensions Another possibility for choosing α that can be analyzed as above is: Let r > 0 be a scaling factor, set � r β l : l = 0 , 1 , 2 , . . . � α = max unitl the Armijo rule is satisfied. Both variants are known as backtracking strategies. The drawback is that one imediately chooses a reduction after the first step. Dr. Thomas M. Surowiec BMS Course NLO, Summer 2013

Step Size Strategies and Algorithms The Wolfe-Powell Rule Armijo-Goldstein Rule A more flexible strategy: Given σ, µ with 0 σ < 1 / 2 < µ < 1 one tests f ( x + α d ) ≤ f ( x ) + σα ∇ f ( x ) T d . (1) f ( x + α d ) ≥ f ( x ) + µα ∇ f ( x ) T d . (2) Equation (1) ensures that α is not too large, whereas Equation (2) ensures that α is not too small. Discussion of an implementation strategy on the board. Dr. Thomas M. Surowiec BMS Course NLO, Summer 2013

Step Size Strategies and Algorithms The Wolfe-Powell Rule Using Polynomial Models Apart from simple backtracking methods, there are other strategies for minimizing ϕ ( α ) = f ( x + α d ) Here, we use the given data to define a model (function), which locally approximates ϕ . Derviation and Discussion on the board. Dr. Thomas M. Surowiec BMS Course NLO, Summer 2013

Step Size Strategies and Algorithms The Wolfe-Powell Rule Intro to the Wolfe-Powell Rule Definition 2.1 Let σ ∈ ( 0 , 1 2 ) and ρ ∈ [ σ, 1 ) be fixed. The following relations are known as the Wolfe-Powell conditions: For x , d ∈ R n with ∇ f ( x ) T d < 0 determine a step size α > 0 such that f ( x + α d ) ≤ f ( x ) + σα ∇ f ( x ) T , (3) ∇ f ( x + α d ) T d ≥ ρ ∇ f ( x ) T d . (4) Discussion on the board. Dr. Thomas M. Surowiec BMS Course NLO, Summer 2013