Linear System of Equations - Conditioning Numerical experiments - - PowerPoint PPT Presentation

Linear System of Equations - Conditioning

Numerical experiments

Input has uncertainties:

• Errors due to representation with finite precision
• Error in the sampling

Once you select your numerical method , how much error should you expect to see in your outp

• utput?

ut? Is Is y your m method s sens nsitive t to e errors ( (perturbation) n) i in t n the i inp nput?

Demo “HilbertMatrix-ConditionNumber”

t⇒→

A x

= b ✓

A) random

①

Defining

A

• B) Hilbert

(NXN)

②

Start

with

a

know exact solution :

+true = [ 1 , I ,

• .
• . , I ]

(np.ones.CN2)

③ Compute

be A@ xtrue

④

Solve

AbIs

→ Xsolve

⑤ Compute

error

H Xsolve

• Xtrue H

Sensitivity of Solutions of Linear Systems

Suppose we start with a non-singular system of linear equations ! " = $. We change the right-hand side vector $ (input) by a small amount Δ$. How much the solution " (output) changes, i.e., how large is Δ"?

Output Relative error Input Relative error = Δ1 / 1 Δ3 / 3 = Δ1 3 Δ3 1 F

s

• ra

FT

• At =b

→ exact

A- I

= 5

→ pert

I

= x tsx

b = b t Sb

A- ( xtsx)

Sensitivity of Solutions of Linear Systems

Output Relative error Input Relative error = Δ1 / 1 Δ3 / 3 = Δ1 3 Δ3 1 Output Relative error Input Relative error =

TAXIS

A- Ax - Ab

HA"H

⑧ - A- Ab

" a.is. " ,

if"iII=

in::i÷÷÷:

Δ" " ≤ $&' $ Δ% %

Sensitivity of Solutions of Linear Systems

i÷s -

• - -

F

T

Sensitivity of Solutions of Linear Systems

We can also add a perturbation to the matrix ! (input) by a small amount (, such that (! + () , " = $ and in a similar way obtain: Δ" " ≤ !!" ! ( !

• O
• T

CornellA)

Condition number

Demo “HilbertMatrix-ConditionNumber”

The condition number is a measure of sensitivity of solving a linear system

• f equations to variations in the input.

The condition number of a matrix !: ./01 ! = !!" ! Recall that the induced matrix norm is given by ! = max

# $" !"

And since the condition number is relative to a given norm, we should be precise and for example write: ./01% ! or ./01& !

• p

p

①

Condition number

Δ" " ≤ ./01 ! Δ$ $ Small condition numbers mean not a lot of error amplification. Small condition numbers are good! But how small?

A- → sing

loudCA) =D

• conedCA)

1)

H X H

> O -

cord CA) so

2) ll X Y H f Il XII 11411 → HAA

• ' II f HAH HA

HAITTIA

Hell = hfffx.gl/IHI--1/HAHHA-'H3aTf

→

Condition number

Δ" " ≤ ./01 ! Δ$ $ Small condition numbers mean not a lot of error amplification. Small condition numbers are good! Recall that 5 = max

# $" 5 "

= 1 Which provides with a lower bound for the condition number: ./01 ! = !!" ! ≥ !!"! = 5 = 1 If !!" does not exist, then ./01 ! = ∞ (by convention)

Recall Induced Matrix Norms

< 4 = max

5

>

674 8

?65 < 9 = max

6

>

574 8

?65 < : = max

;

@;

9' are the singular value of the matrix ! Maximum absolute column sum of the matrix ! Maximum absolute row sum of the matrix !

=

A-

"B

yo!

• { 100 , 13 , 0.5 }

{ too its

11 A Hz = 100

HA

112 = 2

cand CA)

= 100

× 2

Condition Number of Orthogonal Matrices

What is the 2-norm condition number of an orthogonal matrix A? ./01 ! = !!" ( ! ( = !)

( ! ( = 1

That means orthogonal matrices have optimal conditioning. They are very well-behaved in computation.

condra, {

small → well - conditioned &

large

→ ill

• conditionedN

About condition numbers

1. For any matrix !, ./01 ! ≥1 2. For the identity matrix 5, ./01 5 = 1 3. For any matrix ! and a nonzero scalar ;, ./01 ;! = ./01 ! 4. For any diagonal matrix <, ./01 < =

*+# ,! *-. ,!

5. The condition number is a measure of how close a matrix is to being singular: a matrix with large condition number is nearly singular, whereas a matrix with a condition number close to 1 is far from being singular 6. The determinant of a matrix is NOT a good indicator is a matrix is near singularity

detfA) = o

→ sing

Residual versus error

Our goal is to find the solution " to the linear system of equations ! " = $ Let us recall the solution of the perturbed problem , " = " + Δ" which could be the solution of ! , " = $ + Δ$ , ! + ( , " = $, (! + () , " = $ + Δ$ And the error vector as = = Δ" = , " − " We can write the residual vector as ? = $ − ! , "

Demo “HilbertMatrix-ConditionNumber”

O

• te

e

• = -0

③Its Eel

¥

Relative residual:

! " #

(How well the solution satisfies the problem) Relative error: $#

#

(How close the approximated solution is from the exact one)

→

• =

Residual versus error

It is possible to show that the residual satisfy the following inequality: ? ! , " ≤ . @/ Where . is “large” constant when LU/Gaussian elimination is performed without pivoting and “small” with partial pivoting. Therefore, Gaussian elimination with partial pivoting yields small relative residual regardless of conditioning of the system.

When solving a system of linear equations via LU with partial pivoting, the relative residual is guaranteed to be small!

±

Residual versus error

Let us first obtain the norm of the error:

At = b ⇒ x

Il DX It

• x H

• A

b ll

• 'fAb)H

r

H DX It

H A-

' r ll

H A

* Tix '

Hittin

• YEI ' "

IHA

"

an

cordCA)

llsx H

⇒

floridCA) Hrd

H xHHAH

Rule of thumb for conditioning

Suppose we want to find the solution " to the linear system of equations ! " = $ using LU factorization with partial pivoting and backward/forward substitutions. Suppose we compute the solution , ". If the entries in ! and $ are accurate to S decimal digits, and ./01 ! = BC0, then the elements of the solution vector , " will be accurate to about D − E decimal digits

per-1%4%10

's

• 0-0

gerehffyy-fcondcmttfbbuI-le.no?YoIs-w

'

Donee