The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods Adaptive Low Complexity Algorithms for Unconstrained Minimization Carmine Di Fiore, Stefano Fanelli, Paolo Zellini mailto:difiore@mat.uniroma2.it Cortona, September 2004 Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods The minimization problem and classical solvers 1 Previous contribution: L QN descent methods 2 New contribution: Adaptive L QN descent methods 3 Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods The minimization problem and classical solvers f ( x ∗ ) = min x ∈ R n f ( x ), find x ∗ Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods The minimization problem and classical solvers f ( x ∗ ) = min x ∈ R n f ( x ), find x ∗ Descent methods generate a minimizing sequence { x k } + ∞ k =0 by the iterative scheme: x 0 ∈ R n , g 0 = ∇ f ( x 0 ) , d 0 = − g 0 For k = 0 , 1 , . . . x k +1 = x k + λ k d k λ k > 0 g k +1 = ∇ f ( x k +1 ) B k +1 = n × n matrix, positive definite (pd) − B − 1 d k +1 = k +1 g k +1 � �� � descent direction Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods The Newton descent method B k +1 = ∇ 2 f ( x k +1 ) A quadratic rate of convergence O ( n 3 ) arithmetic operations to compute x k +1 from x k Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods The Newton descent method B k +1 = ∇ 2 f ( x k +1 ) A quadratic rate of convergence O ( n 3 ) arithmetic operations to compute x k +1 from x k Quasi-Newton (QN) descent methods B k +1 defined in terms of ∇ f A superlinear rate of convergence Convergence under weak analytical assumptions O ( n 2 ) arithmetic operations to compute x k +1 from x k O ( n 2 ) memory allocations for implementation Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods The Newton descent method B k +1 = ∇ 2 f ( x k +1 ) A quadratic rate of convergence O ( n 3 ) arithmetic operations to compute x k +1 from x k Quasi-Newton (QN) descent methods B k +1 defined in terms of ∇ f A superlinear rate of convergence Convergence under weak analytical assumptions O ( n 2 ) arithmetic operations to compute x k +1 from x k O ( n 2 ) memory allocations for implementation Main example: the BFGS method (Broyden et al.’70) Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods BFGS x 0 ∈ R n , d 0 = − g 0 For k = 0 , 1 , . . . λ k | s T x k +1 = x k + λ k d k k y k > 0 B k +1 = ϕ ( B k , x k +1 − x k , g k +1 − g k ) � �� � � �� � s k y k d k +1 = − B − 1 k +1 g k +1 Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods BFGS x 0 ∈ R n , d 0 = − g 0 For k = 0 , 1 , . . . λ k | s T x k +1 = x k + λ k d k k y k > 0 B k +1 = ϕ ( B k , x k +1 − x k , g k +1 − g k ) � �� � � �� � s k y k d k +1 = − B − 1 k +1 g k +1 ϕ properties ⇒ • B k +1 inherites positive definiteness from B k s T y > 0 Proof: B pd & ⇒ ϕ ( B , s , y ) pd • B k +1 ( x k +1 − x k ) = g k +1 − g k Proof: ϕ ( B , s , y ) s = y Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods The updating function ϕ in B k +1 = ϕ ( B k , s k , y k ) is 1 1 y T syy T − s T B s B ss T B ϕ ( B , s , y ) = B + ⇒ BFGS is a secant method: B k +1 ( x k +1 − x k ) = g k +1 − g k secant equation � �� � � �� � s k y k Proof (independent on B) : � � y T s yy T − 1 s T B s B ss T B 1 ϕ ( B , s , y ) s = B + s 1 1 y T s y ( y T s ) − s T B s B s ( s T B s ) = B s + = y Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods Quasi-Newton (QN) descent methods for large scale problems B k +1 defined in terms of ∇ f A fast rate of convergence Convergence under weak analytical assumptions less than O ( n 2 ) arithmetic operations to compute x k +1 from x k less than O ( n 2 ) memory allocations for implementation Classical example: the Limited memory BFGS method (Nocedal et al. ’80) Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods Quasi-Newton (QN) descent methods for large scale problems B k +1 defined in terms of ∇ f A fast rate of convergence Convergence under weak analytical assumptions less than O ( n 2 ) arithmetic operations to compute x k +1 from x k less than O ( n 2 ) memory allocations for implementation Classical example: the Limited memory BFGS method (Nocedal et al. ’80) A recent proposal: the L QN method (Di Fiore, Fanelli, Zellini et al. ’00) Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods Previous contribution: L QN descent methods Replace the matrix B k in B k +1 = ϕ ( B k , s k , y k ) with a matrix A k of a low complexity space L Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods Previous contribution: L QN descent methods Replace the matrix B k in B k +1 = ϕ ( B k , s k , y k ) with a matrix A k of a low complexity space L Choice of L B k ∈ sd U for some unitary matrix U , where z 1 0 · · · 0 0 z 2 0 sd U = { Ud ( z ) U ∗ : z ∈ C n } , d ( z ) = . . ... . . . . 0 · · · 0 z n ⇒ choose L = sd U , U = fast unitary transform ( U = F ourier, H artley, . . . ) Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods Choice of A k in L A k = the best least squares fit to B k in L = sd U , i.e. A k = L B k where �L B k − B k � F = min X ∈L � X − B k � F The L QN algorithm x 0 ∈ R n , d 0 = − g 0 For k = 0 , 1 , . . . λ k | s T x k +1 = x k + λ k d k k y k > 0 B k +1 = ϕ ( L B k , x k +1 − x k , g k +1 − g k ) � �� � � �� � s k y k d k +1 = − B − 1 k +1 g k +1 Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods B k +1 = ϕ ( L B k , s k , y k ) • B k +1 inherites positive definiteness from B k Proof: B pd ⇒ L B pd Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
The minimization problem and classical solvers Previous contribution: L QN descent methods New contribution: Adaptive L QN descent methods B k +1 = ϕ ( L B k , s k , y k ) • B k +1 inherites positive definiteness from B k Proof: B pd ⇒ L B pd • B k +1 s k = y k , i.e. L QN is a secant method Cortona, September 2004 C. Di Fiore, S. Fanelli, P. Zellini Adaptive Low Complexity Algorithms for Unconstrained Min
Recommend
More recommend
