Transporting iterative algorithms from Euclidean space to manifolds - PowerPoint PPT Presentation

Transporting iterative algorithms from Euclidean space to manifolds Jochen Trumpf Jochen.Trumpf@anu.edu.au Department of Information Engineering Research School of Information Sciences and Engineering The Australian National University and National ICT Australia Ltd. Transporting iterative algorithms from Euclidean space to manifolds – p. 1/18

overview the Newton iteration Transporting iterative algorithms from Euclidean space to manifolds – p. 2/18

overview the Newton iteration parametrisations Transporting iterative algorithms from Euclidean space to manifolds – p. 2/18

overview the Newton iteration parametrisations the new algorithm Transporting iterative algorithms from Euclidean space to manifolds – p. 2/18

overview the Newton iteration parametrisations the new algorithm convergence properties Transporting iterative algorithms from Euclidean space to manifolds – p. 2/18

overview the Newton iteration parametrisations the new algorithm convergence properties an example Transporting iterative algorithms from Euclidean space to manifolds – p. 2/18

overview the Newton iteration parametrisations the new algorithm convergence properties an example general iterates Transporting iterative algorithms from Euclidean space to manifolds – p. 2/18

overview the Newton iteration parametrisations the new algorithm convergence properties an example general iterates joint work with J. Manton Transporting iterative algorithms from Euclidean space to manifolds – p. 2/18

the Newton iteration Newton’s method x k +1 = x k − { Hess f ( x k ) } − 1 grad f ( x k ) , x 0 ∈ R n is an iteration x 0 ∈ R n x k +1 = N( f )( x k ) , which is defined for any twice differentiable function f : R n − → R . Transporting iterative algorithms from Euclidean space to manifolds – p. 3/18

the Newton iteration The sequence x k = { N( f ) } k ( x 0 ) it generates converges locally quadratic to non-degenerate critical points of f . Transporting iterative algorithms from Euclidean space to manifolds – p. 4/18

the Newton iteration The sequence x k = { N( f ) } k ( x 0 ) it generates converges locally quadratic to non-degenerate critical points of f . In particular, it converges locally to any (isolated) strict local maximum of f . Transporting iterative algorithms from Euclidean space to manifolds – p. 4/18

parametrisations Sometimes a to be maximised function is not naturally defined on an R n but rather on some smooth manifold (curved space), e.g. the sphere. Transporting iterative algorithms from Euclidean space to manifolds – p. 5/18

parametrisations Sometimes a to be maximised function is not naturally defined on an R n but rather on some smooth manifold (curved space), e.g. the sphere. One description of manifolds is that they look locally like an R n . This means that the manifold can be covered by a collection of subsets for each of which there is a homeomorphism (coordinate chart) onto an open set in R n . Transporting iterative algorithms from Euclidean space to manifolds – p. 5/18

parametrisations Sometimes a to be maximised function is not naturally defined on an R n but rather on some smooth manifold (curved space), e.g. the sphere. One description of manifolds is that they look locally like an R n . This means that the manifold can be covered by a collection of subsets for each of which there is a homeomorphism (coordinate chart) onto an open set in R n . The whole atlas has to fit nicely together, i.e. via diffeomorphisms in overlapping regions. Transporting iterative algorithms from Euclidean space to manifolds – p. 5/18

parametrisations This implies that for each point p of the manifold M there exists a local parametrisation , i.e. a smooth injective map µ p : R n − → M, µ p (0) = p Transporting iterative algorithms from Euclidean space to manifolds – p. 6/18

parametrisations This implies that for each point p of the manifold M there exists a local parametrisation , i.e. a smooth injective map µ p : R n − → M, µ p (0) = p We consider the special case where µ p varies locally smoothly with the base point, which might only be possible in a small neighborhood of a given point p ∗ (hedgehog theorem). Transporting iterative algorithms from Euclidean space to manifolds – p. 6/18

parametrisations Take e.g. the sphere and the operation of the special orthogonal group on it φ : SO ( n + 1) × S n − → S n , ( Q, p ) �→ Qp and consider the exponential map exp : so ( n + 1) − → SO ( n + 1) , Ω �→ exp Ω . Transporting iterative algorithms from Euclidean space to manifolds – p. 7/18

parametrisations It can be shown that φ (exp( . ) , p ∗ ) : so ( n + 1) − → S n is locally injective around 0 when restricted to the subspace     0 Z  | Z ∈ R k × ( n − k )    . − Z ⊤  0  This defines a local parametrisation µ p ∗ which can be “moved around” S n by applying φ . Transporting iterative algorithms from Euclidean space to manifolds – p. 8/18

the new algorithm Let µ p and ν p be two families of local parametrisations and consider the iteration p k +1 = ν p k (N( f ◦ µ p k )(0)) , p 0 ∈ M which is defined for every twice differentiable function f : M − → R . Transporting iterative algorithms from Euclidean space to manifolds – p. 9/18

the new algorithm Let µ p and ν p be two families of local parametrisations and consider the iteration p k +1 = ν p k (N( f ◦ µ p k )(0)) , p 0 ∈ M which is defined for every twice differentiable function f : M − → R . Note that for M = R n and ν p = µ p the obvious parametrisation x �→ p + x this is the standard Newton method. Transporting iterative algorithms from Euclidean space to manifolds – p. 9/18

convergence properties Theorem: If µ p and ν p are smooth around a non-degenerate critical point p ∗ of f and if moreover µ ′ p ∗ (0) = ν ′ p ∗ (0) then our algorithm converges locally quadratic to p ∗ . Transporting iterative algorithms from Euclidean space to manifolds – p. 10/18

convergence properties Theorem: If µ p and ν p are smooth around a non-degenerate critical point p ∗ of f and if moreover µ ′ p ∗ (0) = ν ′ p ∗ (0) then our algorithm converges locally quadratic to p ∗ . In general, nothing is said (and known) about global convergence. Transporting iterative algorithms from Euclidean space to manifolds – p. 10/18

an example Consider a real symmetric n × n matrix N with eigenvalues λ 1 ≥ · · · ≥ λ k > λ k +1 ≥ · · · ≥ λ n . Its k -dimensional principal eigenspace is the subspace spanned by the eigenvectors to λ 1 , . . . , λ k . Transporting iterative algorithms from Euclidean space to manifolds – p. 11/18

an example Consider a real symmetric n × n matrix N with eigenvalues λ 1 ≥ · · · ≥ λ k > λ k +1 ≥ · · · ≥ λ n . Its k -dimensional principal eigenspace is the subspace spanned by the eigenvectors to λ 1 , . . . , λ k . Consider the function (generalised Rayleigh quotient) → R , [ X ] �→ tr X ⊤ NX f : Grass( k, n ) − Transporting iterative algorithms from Euclidean space to manifolds – p. 11/18

an example µ p is given by      I p =  Q   0       0 Z  I µ p ( Z ) =  Q exp − Z ⊤     0 0 where Q ∈ O ( n ) and Z is k × ( n − k ) . Transporting iterative algorithms from Euclidean space to manifolds – p. 12/18

an example Then        I 0  0 − N 12  =  Q ⊤ NQ, grad( f ◦ µ p )(0) = N ⊤   0 0 0 12 Transporting iterative algorithms from Euclidean space to manifolds – p. 13/18

an example Then        I 0  0 − N 12  =  Q ⊤ NQ, grad( f ◦ µ p )(0) = N ⊤   0 0 0 12 and   0 ZN 22 − N 11 Z Hess( f ◦ µ p )(0) Z = Z ⊤ N 11 − N 12 Z ⊤   0 Transporting iterative algorithms from Euclidean space to manifolds – p. 13/18

an example So computing N( f ◦ µ p )(0) amounts to solving the Sylvester equation N 11 Z − ZN 22 = − N 12 Transporting iterative algorithms from Euclidean space to manifolds – p. 14/18

an example So computing N( f ◦ µ p )(0) amounts to solving the Sylvester equation N 11 Z − ZN 22 = − N 12 This Z could than be plugged into       0 Z  I ν p ( Z ) =  Q exp − Z ⊤     0 0 to get a new Q . Transporting iterative algorithms from Euclidean space to manifolds – p. 14/18

an example It’s much better though to use an orthogonal projection onto O ( n ) instead by computing a QR -decomposition of    I  = Q Z R − Z ⊤ and to use QQ Z as the new Q . Transporting iterative algorithms from Euclidean space to manifolds – p. 15/18

general iterates Replacing the Newton iteration N( f ) : R n − → R n by any other iteration G( f ) : R n − → R n that is locally order q converging to non-degenerate critical points of f , we can derive sufficient conditions on a family µ p of local parametrisations that guarantee local order q convergence of the “transported algorithm” p k +1 = µ p k (G( f ◦ µ p k )(0)) , p 0 ∈ M Transporting iterative algorithms from Euclidean space to manifolds – p. 16/18