Solving large scale eigenvalue problems Lecture 6, March 28, 2018: - PowerPoint PPT Presentation

Solving large scale eigenvalue problems Solving large scale eigenvalue problems Lecture 6, March 28, 2018: Simple vector iterations http://people.inf.ethz.ch/arbenz/ewp/ Peter Arbenz Computer Science Department, ETH Z¨ urich E-mail: arbenz@inf.ethz.ch Large scale eigenvalue problems, Lecture 6, March 28, 2018 1/42

Solving large scale eigenvalue problems Survey Survey of today’s lecture The power method (aka. vector iteration) is the simplest method to compute a single eigenvector of a matrix. ◮ Simple vector iteration (power method) ◮ Inverse vector iteration ◮ Rayleigh quotient iteration (RQI) Large scale eigenvalue problems, Lecture 6, March 28, 2018 2/42

Solving large scale eigenvalue problems Simple vector iteration Simple vector iteration Let A ∈ R n × n . Starting with arbitrary initial vector x (0) ∈ R n we form the vector � x ( k ) � ∞ sequence k =0 defined by x ( k ) := A x ( k − 1) , k = 1 , 2 , . . . ( ∗ ) Clearly, x ( k ) := A k x (0) . We will show that the x ( k ) ‘converge’ to ‘the’ eigenvector associated with the eigenvalue of largest magnitude. Large scale eigenvalue problems, Lecture 6, March 28, 2018 3/42

Solving large scale eigenvalue problems Simple vector iteration Algorithm: Simple vector iteration 1: Choose a starting vector x (0) ∈ R n with � x (0) � = 1. 2: k = 0. 3: repeat k := k + 1; 4: y ( k ) := A x ( k − 1) ; 5: µ k := � y ( k ) � ; 6: x ( k ) := y ( k ) /µ k ; 7: 8: until a convergence criterion is satisfied x ( k ) � ∞ Vectors x ( k ) have all norm (length) one. � k =0 is a sequence on the unit sphere of R n . Here, the maximum norm is polular as well: � y � ∞ = max i | y i | . Large scale eigenvalue problems, Lecture 6, March 28, 2018 4/42

Solving large scale eigenvalue problems Simple vector iteration Important note ◮ Let A = USU ∗ be the Schur decomposition of A . Then, U ∗ x ( k ) := SU ∗ x ( k − 1) U ∗ x ( k ) := S k U ∗ x (0) . and ◮ U unitary: � x ( k ) � � U ∗ x ( k ) � = 1 for all k . = ⇒ � x ( k ) � ∞ ◮ If sequence k =0 converges to x ∗ then sequence � y ( k ) = U ∗ x ( k ) � ∞ k =0 converges to y ∗ = U ∗ x ∗ . ◮ So, for convergence analysis: can assume w.l.o.g. that A is upper triangular. ◮ If we assumed that A is symmetric then for a convergence analysis we could restrict ourselves to diagonal matrices. ◮ Note that some performance issues are excluded here. Large scale eigenvalue problems, Lecture 6, March 28, 2018 5/42

Solving large scale eigenvalue problems Simple vector iteration Angles between vectors Intermezzo: Angles between vectors Let q 1 and q 2 be unit vectors. Angle between vectors q 1 and q 2 : q 2 q 1 Large scale eigenvalue problems, Lecture 6, March 28, 2018 6/42

Solving large scale eigenvalue problems Simple vector iteration Angles between vectors Intermezzo: Angles between vectors (cont.) The length of the orthogonal projection of q 2 on span { q 1 } is: c := � q 1 q 1 ∗ q 2 � = | q 1 ∗ q 2 | ≤ 1 . The length of the orthogonal projection of q 2 on span { q 1 } ⊥ is s := � ( I − q 1 q 1 ∗ ) q 2 � . (+) As q 1 q ∗ 1 is an orthogonal projection, by Pythagoras’ formula: 1 = � q 2 � 2 = � q 1 q 1 ∗ q 2 � 2 + � ( I − q 1 q 1 ∗ ) q 2 � 2 = s 2 + c 2 . s 2 � ( I − q 1 q 1 ∗ ) q 2 � 2 From (+): = q 2 ∗ ( I − q 1 q 1 ∗ ) q 2 = q 2 ∗ q 2 − ( q 2 ∗ q 1 )( q 1 ∗ q 2 ) = 1 − c 2 = Large scale eigenvalue problems, Lecture 6, March 28, 2018 7/42

Solving large scale eigenvalue problems Simple vector iteration Angles between vectors Intermezzo: Angles between vectors (cont.) So, there is a number, say, ϑ , 0 ≤ ϑ ≤ π 2 , such that c = cos ϑ and s = sin ϑ . This uniquely determined number ϑ is the angle between the vectors q 1 and q 2 : ϑ = ∠ ( q 1 , q 2 ) . The generalization to arbitrary vectors is straightforward. Definition The angle θ between two nonzero vectors x and y is given by � y � | x ∗ y | �� � � � � I − xx ∗ � � ϑ = ∠ ( x , y ) = arcsin = arccos . � � � x � 2 � y � � x �� y � � � Large scale eigenvalue problems, Lecture 6, March 28, 2018 8/42

Solving large scale eigenvalue problems Simple vector iteration Angles between vectors Convergence analysis Assume that � λ 1 � s ∗ 1 S = , ( S 2 upper triangular) (1) 0 S 2 has eigenvalues | λ 1 | > | λ 2 | ≥ | λ 3 | ≥ · · · ≥ | λ n | . Eigenvector of S corresponding to largest eigenvalue λ 1 is e 1 . We will show that the iterates x ( k ) converge to e 1 . More precisely, we will show that ∠ ( x ( k ) , e 1 ) − → 0 as k → ∞ . Large scale eigenvalue problems, Lecture 6, March 28, 2018 9/42

Solving large scale eigenvalue problems Simple vector iteration Angles between vectors Convergence analysis (cont.) Let   x ( k ) 1 � � x ( k )   x ( k ) x ( k ) =   2 1 =:   . x ( k ) .   . 2   x ( k ) n with � x ( k ) � = 1. Then, � � � � � n i =2 | x ( k ) n | 2 � � sin ϑ ( k ) := sin( ∠ ( x ( k ) , e 1 )) = � | x ( k ) | 2 = i | 2 . � � i � n i =1 | x ( k ) i =2 i The last expression is for non-normalized vectors x ( k ) , cf. ( ∗ ). Large scale eigenvalue problems, Lecture 6, March 28, 2018 10/42

Solving large scale eigenvalue problems Simple vector iteration Angles between vectors Convergence analysis (cont.) First, we simplify the form of S in (1), by eliminating s ∗ 1 , � 1 � � λ 1 � � 1 � − 1 − t ∗ s ∗ − t ∗ 1 0 I 0 S 2 0 I � 1 � � λ 1 � � 1 � λ 1 � � − t ∗ s ∗ t ∗ 0 ∗ 1 = = . 0 I 0 S 2 0 I 0 S 2 The vector t that realizes this transformation has to satisfy λ 1 t ∗ + s ∗ 1 − t ∗ S 2 = 0 ∗ s ∗ 1 = t ∗ ( S 2 − λ 1 I ) . ⇐ ⇒ This equation has a solution if and only if λ 1 �∈ σ ( S 2 ) which is the case by assumption. Remark: [1 , − t ∗ ] is left eigenvector of S associated with λ 1 . Large scale eigenvalue problems, Lecture 6, March 28, 2018 11/42

Solving large scale eigenvalue problems Simple vector iteration Angles between vectors Convergence analysis (cont.) So, we have � λ 1 � λ 1 � � � � k x ( k ) s ∗ s ∗ x ( k ) = x ( k − 1) = · · · = 1 1 x (0) 1 = x ( k ) 0 S 2 0 S 2 2 � 1 � � λ 1 � k � 1 � � � x (0) t ∗ 0 ∗ − t ∗ 1 = . x (0) 0 0 0 I S 2 I 2 We define � 1 � − t ∗ y ( k ) := 1 x ( k ) (2) λ k 0 I 1 Large scale eigenvalue problems, Lecture 6, March 28, 2018 12/42

Solving large scale eigenvalue problems Simple vector iteration Angles between vectors Convergence analysis (cont.) � 1 � − t ∗ y ( k ) = 1 S x ( k − 1) λ k 0 I 1 � 1 � λ 1 � 1 �� � 0 ∗ − t ∗ 1 x ( k − 1) = 0 S 2 λ k − 1 0 I λ 1 1 � 1 � 1 � � � y ( k − 1) � 0 ∗ 0 ∗ y ( k − 1) . 1 = = 1 1 y ( k − 1) 0 λ 1 S 2 0 λ 1 S 2 2 Let us assume that y (0) = 1. Then, y ( k ) = 1 for all k . 1 1 Need to show that y ( k ) goes to zero as k → ∞ , and how fast. 2 Large scale eigenvalue problems, Lecture 6, March 28, 2018 13/42

Solving large scale eigenvalue problems Simple vector iteration Angles between vectors Convergence analysis (cont.) = 1 y ( k ) S 2 y ( k − 1) 2 2 λ 1   µ 2 ∗ . . . ∗ µ 3 . . . ∗   1 | µ k | = | λ k |   S 2 =  , | λ 1 | < 1 . . ...   . λ 1 .  µ n Large scale eigenvalue problems, Lecture 6, March 28, 2018 14/42

Solving large scale eigenvalue problems Simple vector iteration Angles between vectors Convergence analysis (cont.) Theorem Let ||| · ||| be any matrix norm. Then k →∞ ||| M k ||| 1 / k = ρ ( M ) = max lim | λ i ( M ) | . (3) i Proof. See Horn-Johnson, Matrix Analysis , 1985, pp.297-299. So, for any ε > 0 there is an integer K ( ε ) such that ||| M k ||| 1 / k ≤ ρ ( M ) + ε, for all k > K ( ε ) . (4) Large scale eigenvalue problems, Lecture 6, March 28, 2018 15/42

Solving large scale eigenvalue problems Simple vector iteration Angles between vectors Convergence analysis (cont.) So, for any ε > 0 there is an integer K ( ε ) such that ||| M k ||| 1 / k ≤ ρ ( M ) + ε, for all k > K ( ε ) . (4) In our case: � 1 � ρ S 2 = | µ 2 | < 1 . λ 1 Can choose ε such that | µ 2 | + ε < 1. For any such ε we have sin( ∠ ( y ( k ) , e 1 )) = � y ( k ) � y ( k ) 2 � 2 � � y ( k ) � = � 1 + � y ( k ) 2 � 2 � ≤ � 1 ≤ � y ( k ) S � k � y (0) 2 � ≤ ( | µ 2 | + ε ) k � y (0) 2 � . λ 1 Large scale eigenvalue problems, Lecture 6, March 28, 2018 16/42

Solving large scale eigenvalue problems Simple vector iteration Angles between vectors Convergence analysis (cont.) Analogous result for x ( k ) : � 1 � � � � � x ( k ) y ( k ) t ∗ 1 = λ k 1 , 1 x ( k ) y ( k ) 0 I 2 2 So, | x ( k ) 1 | y ( k ) + t ∗ y ( k ) 1 | 1 + t ∗ y ( k ) 1 | = λ k 2 | = λ k 2 | . 1 Since � y ( k ) 2 � ≤ ( | µ 2 | + ε ) k � y (0) 2 � , there is a ˜ K ≥ K ( ε ) such that 2 | < 1 | t ∗ y ( k ) ∀ k > ˜ 2 , K and 2 | > 1 | 1 + t ∗ y ( k ) ∀ k > ˜ 2 , K . Large scale eigenvalue problems, Lecture 6, March 28, 2018 17/42