IDR a brief introduction Martin H. Gutknecht ETH Zurich, Seminar - PowerPoint PPT Presentation

IDR — a brief introduction Martin H. Gutknecht ETH Zurich, Seminar for Applied Mathematics Minisymposium “Induced Dimension Reduction (IDR) methods” SIAM Conf. on Applied Linear Algebra, Monterey, CA, USA, Oct. 27, 2009

Prerequisites History IDR basics Case s=1 Case s>1 Conclusions Outline Prerequisites History IDR basics Case s=1 Case s>1 Conclusions M.H. Gutknecht SIAM-ALA09 p. 2

Prerequisites History IDR basics Case s=1 Case s>1 Conclusions Prerequisites: Krylov (sub)space solvers Given: linear system Ax = b , initial approx. x 0 ∈ C N . Construct: approximate solutions (“iterates”) x n and corresponding residuals r n : ≡ b − Ax n with x n ∈ x 0 + K n ( A , r 0 ) , r n ∈ r 0 + A K n ( A , r 0 ) where r 0 : ≡ b − Ax 0 is the initial residual, and K n : ≡ K n ( A , r 0 ) : ≡ span { r 0 , Ar 0 , . . . , A n − 1 r 0 } is the n th Krylov subspace generated by A from r 0 . We can, e.g., construct x n such that � r n � is minimal. Conjugate Residual ( CR ) method (Stiefel, 1955), � GCR and GMR ES . � M.H. Gutknecht SIAM-ALA09 p. 3

Prerequisites History IDR basics Case s=1 Case s>1 Conclusions Some further Krylov space solvers are based on other orthogonal or oblique projections: Conjugate Gradient ( CG ) method (Hestenes/Stiefel, 1952): r n ∈ r 0 + A K n , r n ⊥ K n . Biconjugate Gradient ( B I CG ) method (Lanczos, 1952; Fletcher, 1976): r n ∈ r 0 + A K n , r n ⊥ � K n : ≡ K n ( A r 0 ) . ⋆ , � ML ( s ) B I CG method (M.-C. Yeung and T. F. Chan, 1999): s � r sj ∈ r 0 + A K sj , r sj ⊥ K j ( A R 0 ) : ≡ K j ( A r ( i ) ⋆ , � ⋆ , � 0 ) . i = 1 M.H. Gutknecht SIAM-ALA09 p. 4

Prerequisites History IDR basics Case s=1 Case s>1 Conclusions Prerequisites: residual polynomials r n ∈ r 0 + A K n ( A , r 0 ) implies that r n = ρ n ( A ) r 0 . ∃ ρ n ∈ P n , ρ n ( 0 ) = 1 : Means roughly: � r n � is small if | ρ n ( t ) | is small at the eigenvalues of A . Means also: “everything” can be formulated in terms of residual polynomials. M.H. Gutknecht SIAM-ALA09 p. 5

Prerequisites History IDR basics Case s=1 Case s>1 Conclusions Special cases: In CG the residual polynomials are orthogonal polynomials ( OPs ) w.r.t. a weight function determined by EVals of A (symmetric) and by r 0 . In B I CG the residual polynomials are formal orthogonal polynomials ( FOPs ). � Lanczos polynomials . In (Bi)Conjugate Gradient Squared ( CGS ) , � � 2 BICG ρ CGS = ρ . n n BiCGSTAB BICG in B I CGS TAB , ρ = ρ Ω n , n n where Ω n ( t ) : ≡ ( 1 − ω 1 t ) · · · ( 1 − ω n t ) . Here, at step n , ω n is chosen to minimize the residual on a straight line. M.H. Gutknecht SIAM-ALA09 p. 6

Prerequisites History IDR basics Case s=1 Case s>1 Conclusions History of IDR: references P. W ESSELING AND P. S ONNEVELD , Numerical experiments with a multiple grid and a preconditioned Lanczos type method , in Approximation Methods for Navier-Stokes Problems, R. Rautmann, ed., Lecture Notes in Mathematics, vol. 771, Springer, 1980, pp. 543–562. Introduce Induced Dimension Reduction (IDR) method , attributed to Sonneveld (“Public. in preparation”), on 3 1 2 pages: a “Lanczos-type method” for nonsymmetric linear systems which does not require A T . P. S ONNEVELD , CGS, a fast Lanczos-type solver for nonsymmetric linear systems , SIAM J. Sci. Statist. Comput., 10 (1989), pp. 36–52. Received Apr. 24, 1984. Introduces (Bi)CG Squared . M.H. Gutknecht SIAM-ALA09 p. 7

Prerequisites History IDR basics Case s=1 Case s>1 Conclusions H. A. VAN DER V ORST , Bi-CGSTAB: a fast and smoothly converging variant of Bi-CG for the solution of nonsymmetric linear systems , SIAM J. Sci. Statist. Comput., 13 (1992), pp. 631–644. Received May 21, 1990. Presented at Householder Tylosand. 1st preprint: “ CGSTAB : A more smoothly converging variant of CG-S”, coauthored by P . Sonneveld. M.-C. Y EUNG AND T. F. C HAN , ML ( k ) BiCGSTAB: a BiCGSTAB variant based on multiple Lanczos starting vectors , SIAM J. Sci. Comput., 21 (1999), pp. 1263–1290. Received May 16, 1997. Introduce first ML ( k ) BiCG and then ML ( k ) BiCGSTAB — versions of B I CG and B I CGS TAB , resp., with “multiple left (shadow) residuals”. Fundamental idea. Astonishing numerical results. Some details complicated and not so well explained. M.H. Gutknecht SIAM-ALA09 p. 8

Prerequisites History IDR basics Case s=1 Case s>1 Conclusions P. S ONNEVELD AND M. B. VAN G IJZEN , IDR(s): a family of simple and fast algorithms for solving large nonsymmetric systems of linear equations , Report 07-07, Department of Applied Mathematical Analysis, Delft University of Technology; SIAM J. Sci. Comput., 31 (2008), pp. 1035–1062. Generalizing IDR ≈ IDR(1) to IDR( s ) . A fundamental generalization. Very good numerical results. Detailed description of method, connection to B I CGS TAB . G. S LEIJPEN , P. S ONNEVELD , AND M. B. VAN G IJZEN , Bi-CGSTAB as an induced dimension reduction method , Report 08-07, Department of Applied Mathematical Analysis, Delft University of Technology. Partly new view; explores connection to BiCGSTAB and ML ( k ) BiCGSTAB from a partly different point of view. M.H. Gutknecht SIAM-ALA09 p. 9

Prerequisites History IDR basics Case s=1 Case s>1 Conclusions M. B. VAN G IJZEN AND P. S ONNEVELD , An elegant IDR(s) variant that efficiently exploits bi-orthogonality properties , Report 08-21, Department of Applied Mathematical Analysis, Delft University of Technology. Uses the freedom in the choice of the “intermediate” residuals to come up with a new version of IDR( s ) that is slightly more efficient and particularly ingenious. M.H. Gutknecht SIAM-ALA09 p. 10

Prerequisites History IDR basics Case s=1 Case s>1 Conclusions This talk is based on: M. H. G UTKNECHT , IDR explained . To appear in ETNA. Further papers by: K UNIYOSHI A BE AND G ERARD S LEIJPEN , T IJMEN C OLLIGNON , Y USUKE O NOUNE AND S EIJI F UJINO , V ALERIA S IMONCINI AND D ANIEL S ZYLD , M ASAAKI T ANIO AND M ASAAKI S UGIHARA � MS38, Wes 2:45pm , M AN -C HUNG Y EUNG , MHG AND J ENS -P ETER Z EMKE , ... plus many more on applications. M.H. Gutknecht SIAM-ALA09 p. 11

Prerequisites History IDR basics Case s=1 Case s>1 Conclusions IDR( s ) basics: the setting Given: linear system Ax = b ∈ C N , initial approx. x 0 . Let: r 0 : ≡ b − Ax 0 , K m : ≡ K m ( A , r 0 ) : ≡ span { r 0 , Ar 0 , . . . , A m − 1 r 0 } , ν such that G 0 : ≡ K ν invariant , S ⊂ C N linear subspace of dimension N − s , for j = 1 , 2 , . . . : choose ω j � = 0 and let G j : ≡ ( I − ω j A )( G j − 1 ∩ S ) , for n = n j , . . . , n j + 1 − 1 : choose x n such that r n ∈ G j ∩ ( r 0 + A K n ) . � �� � ⊂ K n + 1 Note: Typically n j + 1 := n j + s + 1 . M.H. Gutknecht SIAM-ALA09 p. 12

Prerequisites History IDR basics Case s=1 Case s>1 Conclusions IDR( s ) basics: the spaces G j (case s = 1) G 0 = R 3 I − ω 1 A G 1 I − ω 2 A G 1 ∩ S G 0 ∩ S = S G 2 G 2 ∩ S = { 0 } G j : ≡ ( I − ω j A )( G j − 1 ∩ S ) M.H. Gutknecht SIAM-ALA09 p. 13

Prerequisites History IDR basics Case s=1 Case s>1 Conclusions IDR( s ) basics: IDR theorem G j : ≡ ( I − ω j A )( G j − 1 ∩ S ) , r n ∈ G j ∩ ( r 0 + A K n ) . Recall: contains no eigenvector of A . Genericness assumption: S ∩ G 0 T HEOREM (IDR T HEOREM (W ES /S ON 80,S ON / V G I 07)) G j � G j − 1 unless G j − 1 = { o } . Consequently: G j = { o } for some j ≤ N . IDR Thm. suggests: r n = o once r n ∈ G N , i.e., j = N . But typically r n ∈ G j when n = j ( s + 1 ) . However, normally r n = o once n = N . Hence: IDR Thm. strongly underestimates convergence rate. M.H. Gutknecht SIAM-ALA09 p. 14

Prerequisites History IDR basics Case s=1 Case s>1 Conclusions IDR( s ) basics: what’s different? Most currently used KSS (= Krylov subspace solvers) are based on a different kind of “induced dimension reduction”: r n ∈ L ⊥ n ∩ ( r 0 + A K n ( A , r 0 )) , where, e.g., L n = K n ( A , r 0 ) ( CG ) , L n = A K n ( A , r 0 ) ( CR , GCR , GMRES ) , K n : ≡ K n ( A r 0 ) L n = � ⋆ , � ( BiCG ) . IDR: G j is not an orthogonal complement of a Krylov subspace. However, due to form of the recursion for {G j } , G j turns out to be the image of an orthogonal complement of a Krylov subspace. M.H. Gutknecht SIAM-ALA09 p. 15

Prerequisites History IDR basics Case s=1 Case s>1 Conclusions IDR( s ) basics: recursions for { r n } G j : ≡ ( I − ω j A )( G j − 1 ∩ S ) . Recall: r n + 1 ∈ G j ∩ ( r 0 + A K n + 1 ) . Wanted: r n + 1 := ( I − ω j A ) v n , v n ∈ G j − 1 ∩ S ∩ ( r 0 + A K n ) , = ⇒ ι ( n ) � v n := r n − γ ( n ) ∆ r n − i = r n − ∆ R n c n , = ⇒ (1) i i = 1 ( � ∆ r n − i ∈ G j − 1 ) where s ≤ ι ( n ) ≤ n − n j − 1 , ∆ r n : ≡ r n + 1 − r n , � ∆ r n − 1 � ∆ R n : ≡ ∆ r n − ι ( n ) . . . , � � T c n : ≡ γ ( n ) γ ( n ) . . . . 1 ι ( n ) M.H. Gutknecht SIAM-ALA09 p. 16