Linear System of Equations - Conditioning Numerical experiments - PowerPoint PPT Presentation

Linear System of Equations - Conditioning

Numerical experiments Input has uncertainties: • Errors due to representation with finite precision • Error in the sampling Once you select your numerical method , how much error should you expect to see in your outp output? ut? Is Is y your m method s sens nsitive t to e errors ( (perturbation) n) i in t n the i inp nput? t ⇒ → Demo “HilbertMatrix-ConditionNumber”

= b ✓ A) random A x - B) Hilbert A ① Defining ( NXN ) know exact solution : with ② Start a ( np.ones.CN 2) . , I ] + true = [ 1 , I , - - . A @ xtrue ③ Compute be ④ Ab Is Solve → X solve - X true H H X solve ⑤ Compute error

Sensitivity of Solutions of Linear Systems Suppose we start with a non-singular system of linear equations ! " = $ . 1 F We change the right-hand side vector $ (input) by a small amount Δ$ . How much the solution " (output) changes, i.e., how large is Δ" ? FT - ra s Output Relative error Δ1 / 1 Δ1 3 Input Relative error = = Δ3 / 3 Δ3 - - - → exact At =b = 5 A- I → pert b = b t Sb I = x tsx = btsb → /ADX=bb A- ( xtsx )

Sensitivity of Solutions of Linear Systems TAXIS Output Relative error Δ1 / 1 Δ1 3 A- Ax - Ab Input Relative error = = Δ3 / 3 Δ3 1 ⑧ - A- Ab HA"H Output Relative error if "iII= Input Relative error = " a .is . " , in ::i÷÷÷ :

Sensitivity of Solutions of Linear Systems i÷s - Δ" Δ% $ &' ≤ $ " % - - - T o F

Sensitivity of Solutions of Linear Systems We can also add a perturbation to the matrix ! (input) by a small amount ( , such that • O (! + () , " = $ and in a similar way obtain: Δ" ( ! !" ≤ ! " ! - T Cornell A)

Condition number The condition number is a measure of sensitivity of solving a linear system of equations to variations in the input. The condition number of a matrix ! : ! !" ./01 ! = ! - - Recall that the induced matrix norm is given by ! = max # $" !" p p And since the condition number is relative to a given norm, we should be precise and for example write: ① ./01 % ! or ./01 & ! - Demo “HilbertMatrix-ConditionNumber”

A- → sing Condition number loud CA ) =D Δ" Δ$ - ≤ ./01 ! " $ Small condition numbers mean not a lot of error amplification. Small condition numbers are good! But how small? ' 'll = HAH HA coned CA ) 1) H X H cord CA ) so > O - - ' II f HAH HA ' 'll 2) l l X Y H f Il XII 11411 → HAA HAITTIA ' ' Il > HI 11 Hell = hfffx.gl/IHI--1/HAHHA-'H3aTf →

Condition number Δ" Δ$ ≤ ./01 ! " $ Small condition numbers mean not a lot of error amplification. Small condition numbers are good! Recall that 5 = max # $" 5 " = 1 Which provides with a lower bound for the condition number: ! !" ! !" ! ./01 ! = ! ≥ = 5 = 1 If ! !" does not exist, then ./01 ! = ∞ (by convention)

Recall Induced Matrix Norms 8 < 4 = max > ? 65 Maximum absolute column sum of the matrix ! 5 674 = 8 < 9 = max > ? 65 Maximum absolute row sum of the matrix ! 6 574 < : = max @ ; ; 9 ' are the singular value of the matrix !

yo ! ' " B A- e ' o÷s } { too its - { 100 , 13 , 0.5 } 11 A Hz = 100 ' ' 112 = 2 HA = 2001 cand CA ) = 100 × 2

Condition Number of Orthogonal Matrices What is the 2-norm condition number of an orthogonal matrix A? ! !" ( ! ) ./01 ! = ! ( = ( ! ( = 1 That means orthogonal matrices have optimal conditioning. They are very well-behaved in computation. small → well - conditioned & condra , { " - conditioned N large → ill

About condition numbers For any matrix ! , ./01 ! ≥ 1 1. For the identity matrix 5, ./01 5 = 1 2. For any matrix ! and a nonzero scalar ; , ./01 ;! = ./01 ! 3. *+# , ! For any diagonal matrix < , ./01 < = 4. *-. , ! 5. The condition number is a measure of how close a matrix is to being singular: a matrix with large condition number is nearly singular, whereas a matrix with a condition number close to 1 is far from being singular 6. The determinant of a matrix is NOT a good indicator is a matrix is near → sing detf A) = o singularity

Demo “HilbertMatrix-ConditionNumber” Residual versus error O Our goal is to find the solution " to the linear system of equations ! " = $ - Let us recall the solution of the perturbed problem , " = " + Δ" e te which could be the solution of ! , " = $ + Δ$ , ! + ( , " = $, (! + () , " = $ + Δ$ - - - - And the error vector as - = -0 = = Δ" = , " − " ③ Its Eel We can write the residual vector as ¥ ? = $ − ! , "

→ ! Relative residual: " # - (How well the solution satisfies the problem) - = Relative error: $# # (How close the approximated solution is from the exact one)

Residual versus error It is possible to show that the residual satisfy the following inequality: ± ? ≤ . @ / ! , " Where . is “large” constant when LU/Gaussian elimination is performed without pivoting and “small” with partial pivoting. Therefore, Gaussian elimination with partial pivoting yields small relative residual regardless of conditioning of the system . When solving a system of linear equations via LU with partial pivoting, the relative residual is guaranteed to be small!

⇒ ' b = A- Residual versus error At = b ⇒ x Let us first obtain the norm of the error: = H Ajax ' ' - ' fAb ) H b l l = HA - A = 11 I - x H Il DX It r ' ' II l l r l l ' r l l Hittin H A H A- H DX It * Tix ' = - IHA " YEI ' " an cord CA ) florid CA ) Hrd lls x H H x H HAH

Rule of thumb for conditioning Suppose we want to find the solution " to the linear system of equations ! " = $ using LU factorization with partial pivoting and backward/forward substitutions. 's per -1%4%10 Suppose we compute the solution , ". -0-0 If the entries in ! and $ are accurate to S decimal digits, gerehffyy-fcondcmttfbbuI-le.no?YoIs-w and ./01 ! = BC 0 , then the elements of the solution vector , " will be accurate to about D − E ' decimal digits Donee