Linear Systems CS3220 Summer 2008 Jonathan Kaldor Systems of - - PowerPoint PPT Presentation

Linear Systems

CS3220 Summer 2008 Jonathan Kaldor

Systems of Linear Equations

• Want to find x, y, z such that

2x + 2z = 6 y - 3z = 4 2x + 4y = 2

• We have 3 linear equations, and 3

unknowns

Systems of Linear Equations

• More formally, a system of n linear

equations in n variables is: f1(x1, x2, ..., xn) = b1 f2(x1, x2, ..., xn) = b2 ... fn(x1, x2, ..., xn) = bn

• where each of the fi is linear in each of the

unknowns xi

What makes it linear?

• Formally, a function is linear iff

f(x + x’) = f(x) + f(x’) f(c x) = c f(x)

• Functions that satisfy above are of the form

f(x1, x2, ..., xn) = a1 x1 + a2 x2 +...+ an xn

• Note: can also express as

f(x) = aTx where x and a are vectors

What makes it linear?

What makes it linear?

• Important! A linear function must be linear

in the unknowns

• Take sin(a1) x1 + cos(a2) x2
• Is it linear if a1, a2 are unknown?
• Is it linear if x1, x2 are unknown?

What makes it linear?

• Important! A linear function must be linear

in the unknowns

• Take sin(a1) x1 + cos(a2) x2
• Is it linear if a1, a2 are unknown?
• Is it linear if x1, x2 are unknown?
• Let b1 = sin(a1), b2 = cos(a2), then above

is b1 x1 + b2 x2

Our Example

• Recall we want to find x, y, z such that:

2x + 2z = 6 y - 3z = 4 2x + 4y = 2

• Rewrite as

2x + 0y + 2z = 6 0x + 1y + -3z = 4 2x + 4y + 0z = 2

Our Example

• Recall we want to find x, y, z such that:

2x + 2z = 6 y - 3z = 4 2x + 4y = 2

• Rewrite as

2x + 0y + 2z = 6 0x + 1y + -3z = 4 2x + 4y + 0z = 2 2 0 2 0 1 -3 2 4 0 x y z 6 4 2 =

More generally

More generally

• With n equations in n unknowns:

a11 a12 .... a1n a21 a22 .... a2n . . . . . . . . . . . . an1 an2 .... ann x1 x2 . . . xn b1 b2 . . . bn =

More generally

• With n equations in n unknowns:
• Ax = b

a11 a12 .... a1n a21 a22 .... a2n . . . . . . . . . . . . an1 an2 .... ann x1 x2 . . . xn b1 b2 . . . bn =

Solutions

• The system Ax = b can have one solution,

no solution, or infintely many solutions

• If A is nonsingular, guaranteed to have one

solution

• From now on, assume A is nonsingular

Examples / Applications

• Linear systems extremely common
• Partly due to convenience
• Used in one way or another in most of the
• ther topics we will discuss

• Given Rj and Vk, find i1, i2, i3

Circuit Diagrams

R1 V1 R2 R4 R5 R3 V2 i1 i2 i3

Circuit Diagrams

• Equations follow from Kirchoff’s Laws:
• current is conserved
• voltage losses = voltage gains around

loops in circuit

Linear Algebra

• Given Ax = b, solution is x = A-1b
• However, in practice we rarely, if ever,

form the inverse

• Would like to convert this into an easier

problem to solve without changing the answer

Solving Linear Systems

• Observation: Diagonal systems are easy to

solve

Solving Linear Systems

• Observation: Diagonal systems are easy to

solve a11 0 .... 0 0 a22 .... 0 . . . . . . . . . . . . 0 0 .... ann x1 x2 . . . xn b1 b2 . . . bn =

Linear Algebra Redux

• We can scale any equation, or add any

multiple of an equation to any other

• Matrix formulation: scale row, add

multiple of one row to another row

• Need to also perform operation on right

hand side

Gauss-Jordan Elimination

• Take first equation, scale and subtract from

all other equations in order to eliminate first variable in equations 2...n

• Repeat with second equation and second

variable...

• does this affect first variable?
• After n steps, have diagonal system

Our Example

2 0 2 0 1 -3 2 4 0 x1 x2 x3 6 4 2 = Scale by -1 and add to Eqn 3

Our Example

2 0 2 0 1 -3 0 4 -2 x1 x2 x3 6 4

• 4

= Scale by -1 and add to Eqn 3

Our Example

2 0 2 0 1 -3 0 4 -2 x1 x2 x3 6 4

• 4

= Scale by -4 and add to Eqn 3

Our Example

2 0 2 0 1 -3 0 0 10 x1 x2 x3 6 4

• 20

= Scale by -4 and add to Eqn 3

Our Example

2 0 2 0 1 -3 0 0 10 x1 x2 x3 6 4

• 20

= Scale by -2/10 and add to Eqn 1

Our Example

2 0 0 0 1 -3 0 0 10 x1 x2 x3 10 4

• 20

= Scale by -2/10 and add to Eqn 1

Our Example

2 0 0 0 1 -3 0 0 10 x1 x2 x3 10 4

• 20

= Scale by 3/10 and add to Eqn 2

Our Example

2 0 0 0 1 0 0 0 10 x1 x2 x3 10

• 2
• 20

= Scale by 3/10 and add to Eqn 2

Our Example

2 0 0 0 1 0 0 0 10 x1 x2 x3 10

• 2
• 20

= Scale each row so diagonal entry is 1

Our Example

1 0 0 0 1 0 0 0 1 x1 x2 x3 5

• 2
• 2

= Scale each row so diagonal entry is 1

Our Example

1 0 0 0 1 0 0 0 1 x1 x2 x3 5

• 2
• 2

= Scale each row so diagonal entry is 1 Answer is x1=5, x2=-2, x3=-2

Matrix Justification

• Want to make solving Ax=b easier
• Suppose solution to MAx=Mb for some

choice of M is x’

• Then if M-1 exists, x’ is obviously a solution

to Ax=b as well.

• Want to find non-singular M such that

MAx=Mb is easier to solve

Matrix Justification

• Matrix Mj adds multiple
• f jth row to every
• ther row
• What multiple?

Matrix Justification

• Matrix Mj adds multiple
• f jth row to every
• ther row
• What multiple?

1 0 0 ... -a1j/ajj ... 0 0 1 0 ... -a2j/ajj ... 0 . . . . . . . . . . . . . . 0 0 0 ... -1/ajj ... 0 . . . . . . . . . . . . . . 0 0 0 ... -anj/ajj ... 1

Matrix Justification

• Note: Never really need to form the

matrices M

• We know what happens when we

multiply by M

• Faster to just perform the scales and

additions

• How much faster?

Matrix Justification

• At the end,

Mn-1...M2M1Ax = Mn-1...M2M1b

• Left hand side is the identity matrix
• But that means Mn-1...M2M1 is the

inverse of A

• Again, dont need to form Mi
• Just apply row operations to identity

Multiple Right Hand Sides

• Suppose we have several sets of equations,

each with the same coefficients but different RHS

• Matrix notation: Ax=b1, Ax=b2,

Ax=b3, ... Ax=bm

• Can just stack bi into n x m B matrix
• Apply Mi to matrix B, get answer X

Triangular Matrices

• Don’t have to get it all the way diagonal to

solve easily

• Observation: once we know a variable’s

value, we can substitute it in all other equations

• If that substitution allows us to solve for

another variable...

Triangular Matrices

Triangular Matrices

• Upper Triangular Matrix

a11 a12 .... a1n 0 a22 .... a2n . . . . . . . . . . an-1,n-1 an-1,n 0 0 .... ann x1 x2 . . xn-1 xn b1 b2 . . bn-1 bn =

Triangular Matrices

• Upper Triangular Matrix

a11 a12 .... a1n 0 a22 .... a2n . . . . . . . . . . an-1,n-1 an-1,n 0 0 .... ann x1 x2 . . xn-1 xn b1 b2 . . bn-1 bn = Solve for xn

Triangular Matrices

• Upper Triangular Matrix

a11 a12 .... a1n 0 a22 .... a2n . . . . . . . . . . an-1,n-1 an-1,n 0 0 .... ann x1 x2 . . xn-1 xn b1 b2 . . bn-1 bn = Solve for xn Substitute xn, solve for xn-1

Triangular Matrices

• Upper Triangular Matrix

a11 a12 .... a1n 0 a22 .... a2n . . . . . . . . . . an-1,n-1 an-1,n 0 0 .... ann x1 x2 . . xn-1 xn b1 b2 . . bn-1 bn = Solve for xn Substitute xn, solve for xn-1 Repeat for each variable

Triangular Matrices

• Upper Triangular Matrix
• “Backward Substitution”

a11 a12 .... a1n 0 a22 .... a2n . . . . . . . . . . an-1,n-1 an-1,n 0 0 .... ann x1 x2 . . xn-1 xn b1 b2 . . bn-1 bn = Solve for xn Substitute xn, solve for xn-1 Repeat for each variable

Triangular Matrices

• Forward Substitution: Corresponding

method for lower triangular matrices

• MATLAB code for backward substitution

Gaussian Elimination

• Instead of reducing to diagonal matrix,

reduce to upper triangular system

• What do Mi matrices look like now?
• What do Mi-1 look like?
• What does (Mn-1...M2M1)-1 = M look

like?

LU Factorization

• Convert Ax=b into MAx=Mb, where

MA is upper triangular

• M is lower triangular, as is M-1
• Let MA=U, M-1 = L
• Then LU = M-1MA = A
• So Ax=b can be converted to LUx=b
• L lower triangular, U upper triangular

Solving with LU factorization

• Have LUx = b
• Solve Ly = b (forward substitution)
• Then solve Ux = y (backward

substitution)

• Note: Don’t need to have b in order to

find LU factorization

• Can factor and then solve for many RHS