Solving large scale eigenvalue problems Lecture 8, April 18, 2018: - PowerPoint PPT Presentation

Solving large scale eigenvalue problems Solving large scale eigenvalue problems Lecture 8, April 18, 2018: Krylov spaces http://people.inf.ethz.ch/arbenz/ewp/ Peter Arbenz Computer Science Department, ETH Z¨ urich E-mail: arbenz@inf.ethz.ch Large scale eigenvalue problems, Lecture 8, April 18, 2018 1/37

Solving large scale eigenvalue problems Survey Survey of today’s lecture We are back at single vector iterations. But now we want to extract more information from the data we generate. ◮ Krylov (sub)spaces ◮ Orthogonal bases for Krylov spaces Large scale eigenvalue problems, Lecture 8, April 18, 2018 2/37

Solving large scale eigenvalue problems Introduction Introduction ◮ In power method: we contruct sequence of the form (up to normalization x , A x , A 2 x , . . . ◮ Information at k -th iteration step: x ( k ) = A k x / � A k x � . ◮ All other information discarded! ◮ What about keeping all the information (vectors)? More memory space required! ◮ Can we extract more information from all the vectors? Less computational work! Large scale eigenvalue problems, Lecture 8, April 18, 2018 3/37

Solving large scale eigenvalue problems Introduction Introductory example   2 − 1 − 1 2 − 1   � 51 � 2   ... ... ...   ∈ R 50 × 50 . T =   π   − 1 2 − 1   − 1 2 ◮ Initial vector x = [1 , . . . , 1] ∗ . ◮ Compute first three iterates of IVI: x (1) = x , x (2) = T − 1 x , and x (3) = T − 2 x . ◮ Compute Rayleigh quotients ρ ( i ) = x ( i ) T T x ( i ) / � x ( i ) � 2 . ◮ Compute Ritz values ϑ ( k ) obtained by Rayleigh-Ritz procedure j with span( x (0) , . . . , x ( k ) ), k = 1 , 2 , 3, Large scale eigenvalue problems, Lecture 8, April 18, 2018 4/37

Solving large scale eigenvalue problems Introduction Introductory example (cont.) ϑ ( k ) ϑ ( k ) ϑ ( k ) ρ ( k ) k 1 2 3 1 10.541456 10.541456 2 1.012822 1.009851 62.238885 3 0.999822 0.999693 9.910156 147.211990 The three smallest eigenvalues of T are 0 . 999684, 3 . 994943, and 8 . 974416. The approximation errors are thus ρ (3) − λ 1 ≈ 0 . 000 ′ 14 and ϑ (3) − λ 1 ≈ 0 . 000 ′ 009, which is 15 times smaller. 1 Results show that cost of three matrix vector multiplications can be much better exploited than with plain (inverse) vector iteration – at the expense of more memory space. Large scale eigenvalue problems, Lecture 8, April 18, 2018 5/37

Solving large scale eigenvalue problems Krylov spaces: definition and basic properties Krylov spaces: definition and basic properties Definition 1 Krylov matrix generated by vector x ∈ R n and A : K m ( x ) = K m ( x , A ) := [ x , A x , . . . , A m − 1 x ] ∈ R n × m (1) Krylov (sub)space: � � K m ( x ) = K m ( x , A ) := span x , A x , A 2 x , . . . , A m − 1 x ⊂ R n . (2) We can also write K m ( A , x ) = { p ( A ) x | p ∈ P m − 1 } where P d denotes set of polynomials of degree at most d . Large scale eigenvalue problems, Lecture 8, April 18, 2018 6/37

Solving large scale eigenvalue problems Krylov spaces: definition and basic properties Krylov spaces: definition and basic properties (cont.) The Arnoldi and Lanczos algorithms are methods to compute an orthonormal basis of the Krylov space. Let � � x , A x , . . . , A k − 1 x = Q ( k ) R ( k ) be QR factorization of Krylov matrix K m ( x , A ). The Ritz values and Ritz vectors of A in K m ( x , A ) are obtained by means of the k × k eigenvalue problem Q ( k ) ∗ AQ ( k ) y = ϑ ( k ) y . (3) If ( ϑ ( k ) , y j ) is an eigenpair of (3) then ( ϑ ( k ) , Q ( k ) y j ) is a Ritz pair j j of A in K m ( x ). Large scale eigenvalue problems, Lecture 8, April 18, 2018 7/37

Solving large scale eigenvalue problems Krylov spaces: definition and basic properties Krylov spaces: definition and basic properties (cont.) Simple properties of Krylov spaces [2, p.238] 1. Scaling . K m ( x , A ) = K m ( α x , β A ) , α, β � = 0 . 2. Translation . K m ( x , A − σ I ) = K m ( x , A ). 3. Change of basis . If U is unitary then U K m ( U ∗ x , U ∗ AU ) = K m ( x , A ). In fact, K m ( x , A ) = [ x , A x , . . . , A m − 1 x ] = U [ U ∗ x , ( U ∗ AU ) U ∗ x , . . . , ( U ∗ AU ) m − 1 U ∗ x ] , = UK m ( U ∗ x , U ∗ AU ) . Notice that the scaling and translation invariance hold only for the Krylov subspace, not for the Krylov matrices. Large scale eigenvalue problems, Lecture 8, April 18, 2018 8/37

Solving large scale eigenvalue problems Krylov spaces: definition and basic properties Dimension of K k ( x , A ) What is the dimension of K k ( x )? Clearly, dim( K k ( x )) ≤ k ≤ n . There must be a m for which K 1 � K 2 � · · · � K m = K m +1 = · · · We have A m x = α 0 x + α 1 A x + α 2 A 2 x + · · · + α p − 1 A m − 1 x Thus, K m +1 ( x ) has linearly depending columns. If we reach m there cannot be a further increase of dimension later. Large scale eigenvalue problems, Lecture 8, April 18, 2018 9/37

Solving large scale eigenvalue problems Krylov spaces: definition and basic properties Dimension of K k ( x , A ) (cont.) m � Let A be diagonalizable and x = q i , where A q i = λ i q i , q i � = 0 , i =1 with distinct λ i . Then,   λ k 1 λ 1 · · · 1 λ k 1 λ 2 · · ·   2 [ x , A x , · · · , A k x ]   = [ q 1 , q 2 , · · · , q m ] . . .  . . .  . . . � �� � � �� �   n × ( k +1) n × m λ k 1 λ m · · · m � �� � m × ( k +1) For k < m , the m × ( k +1) matrix on the right has linearly independent columns. (Relation to Vandermonde matrices!) dim K k ( x , A ) = min { k , m } Large scale eigenvalue problems, Lecture 8, April 18, 2018 10/37

Solving large scale eigenvalue problems Polynomial basis for K m Polynomial basis for K m Now we assume A to be Hermitian. Let s ∈ K j ( x ). Then j − 1 j − 1 � � c i A i x = π ( A ) x , c i ξ i . s = π ( ξ ) = i =0 i =0 Let P j be the space of polynomials of degree ≤ j . Then K j ( x ) = { π ( A ) x | π ∈ P j − 1 } . Let m be the smallest index for which K m ( x ) = K m +1 ( x ). Then, � � P j − 1 ∋ c i ξ i → c i A i x ∈ K j ( x ) is bijective for j ≤ m , while it is only surjective for j > m . Large scale eigenvalue problems, Lecture 8, April 18, 2018 11/37

Solving large scale eigenvalue problems Polynomial basis for K m Polynomial basis for K m (cont.) Let Q ∈ R n × j be matrix with orthonormal basis of K j ( x ) Let ˜ A = Q ∗ AQ . The spectral decomposition X ∗ ˜ A ˜ ˜ X = ˜ ˜ X Θ , X = I , Θ = diag( ϑ i , . . . , ϑ j ) , of ˜ A provides the Ritz values of A in K j ( x ). The columns y i of Y = Q ˜ X are the Ritz vectors. By construction the Ritz vectors are mutually orthogonal. Furthermore, A y i − ϑ i y i ⊥ K j ( x ) (4) Large scale eigenvalue problems, Lecture 8, April 18, 2018 12/37

Solving large scale eigenvalue problems Polynomial basis for K m Polynomial basis for K m (cont.) It is easy to represent a vector in K j ( x ) that is orthogonal to y i . Lemma 2 Let ( ϑ i , y i ) , 1 ≤ i ≤ j be Ritz values and Ritz vectors of A in K j ( x ) , j ≤ m. Let ω ∈ P j − 1 . Then ω ( A ) x ⊥ y k ⇐ ⇒ ω ( ϑ k ) = 0 . ( ∗ ) Large scale eigenvalue problems, Lecture 8, April 18, 2018 13/37

Solving large scale eigenvalue problems Polynomial basis for K m Polynomial basis for K m (cont.) Proof. “ ⇐ =” Let ω ∈ P j with ω ( x ) = ( x − ϑ k ) π ( x ) , π ∈ P j − 1 . Then y ∗ k ω ( A ) x = y ∗ here we use that A = A ∗ k ( A − ϑ k I ) π ( A ) x , (5) (4) = ( A y k − ϑ k y k ) ∗ π ( A ) x = 0 . “= ⇒ ” Let S k ⊂ K j ( x ) be defined by S k := ( A − ϑ k I ) K j − 1 ( x ) = { τ ( A ) x | τ ∈ P j − 1 , τ ( ϑ k ) = 0 } , (5) = ⇒ y k is orthogonal to S k . S k has dimension j − 1. As the dimension of a subspace of K j ( x ) that is orthogonal to y k is j − 1, it must coincide with S k . Large scale eigenvalue problems, Lecture 8, April 18, 2018 14/37

Solving large scale eigenvalue problems Polynomial basis for K m Polynomial basis for K m (cont.) We define the polynomials j j µ ( ξ ) � � µ ( ξ ) := ( ξ − ϑ i ) ∈ P j , π k ( ξ ) := ( ξ − ϑ k ) = ( ξ − ϑ i ) ∈ P j − 1 . i =1 i =1 i � = k (Normalized) Ritz vector y k can be represented in the form π k ( A ) x y k = � π k ( A ) x � , (6) as π k ( ϑ i ) = 0 for all i � = k . According to the Lemma π k ( A ) x is perpendicular to all y i with i � = k . Large scale eigenvalue problems, Lecture 8, April 18, 2018 15/37

Solving large scale eigenvalue problems Polynomial basis for K m Polynomial basis for K m (cont.) By the first part of Lemma 2 µ ( A ) x ∈ K j +1 ( x ) is orthogonal to K j ( x ). As each monic ω ∈ P j can be written in the form ψ ∈ P j − 1 , ω ( ξ ) = µ ( ξ ) + ψ ( ξ ) , we have � ω ( A ) x � 2 = � µ ( A ) x � 2 + � ψ ( A ) x � 2 , as ψ ( A ) x ∈ K j ( x ). Let u 1 , · · · , u m be the eigenvectors of A corresponding to λ 1 < · · · < λ m that span K m ( x ). Let � x � = 1. Let ϕ := ∠ ( x , u 1 ). Then � u 1 u ∗ � ( I − u 1 u ∗ 1 x � = cos ϕ, 1 ) x � = sin ϕ. Large scale eigenvalue problems, Lecture 8, April 18, 2018 16/37