Linear Algebra in A Nutshell Gilbert Strang Computational Science - - PowerPoint PPT Presentation

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Linear Algebra in A Nutshell

Gilbert Strang

Computational Science and Engineering Wellesley-Cambridge Press. 2007.

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Outline

1

Matrix Singularity

2

Matrix Multiplication by Columns or Rows Rank and nullspace Column space and solutions to linear equations

3

The Four Fundamental Subspaces

4

Dimension and Basis

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Outline

1

Matrix Singularity

2

Matrix Multiplication by Columns or Rows Rank and nullspace Column space and solutions to linear equations

3

The Four Fundamental Subspaces

4

Dimension and Basis

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Invertibility of an n-by-n matrix

A is invertible The columns are independent The rows are independent The determinant is not zero Ax = 0 has one solution x = 0 Ax = b has one solution A−1b A has n (nonzero) pivots A has full rank A is not invertible The columns are dependent The rows are dependent The determinant is zero Ax = 0 has infinitely many solutions Ax = b has no solution or infinitely many A has r < n pivots A has rank r < n

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Invertibility of an n-by-n matrix (cont.)

The reduced row echelon form is R = I The column space is all of Rn The row space is all of Rn All eigenvalues are nonzero ATA is symmetric positive definite A has n (positive) singular values R has at least one zero row The column space has dimension r < n The row space has dimension r < n Zero is an eigenvalue of A ATA is only semidefinite A has r < n nonzero (positive) singular values

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Outline

1

Matrix Singularity

2

Matrix Multiplication by Columns or Rows Rank and nullspace Column space and solutions to linear equations

3

The Four Fundamental Subspaces

4

Dimension and Basis

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Think of Ax a column at time

Instead of thinking of Ax inner products, think of Ax a linear combination of columns of A: 1 2 3 6 C D

• = C

1 3

• + D

2 6

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Think of Ax a column at time

Instead of thinking of Ax inner products, think of Ax a linear combination of columns of A: 1 2 3 6 C D

• = C

1 3

• + D

2 6

• In particular,

1 2 3 6 1

• = first column

1 2 3 6 1

• = last column

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

In general

matrix-vector multiplication: y = Ax

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

In general

matrix-vector multiplication: y = Ax column version y = zeros(m,1); for j=1:n y = y + x(j)*A(:,j); endfor

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

In general

matrix-vector multiplication: y = Ax column version y = zeros(m,1); for j=1:n y = y + x(j)*A(:,j); endfor matrix-matrix multiplication: C = AB

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

In general

matrix-vector multiplication: y = Ax column version y = zeros(m,1); for j=1:n y = y + x(j)*A(:,j); endfor matrix-matrix multiplication: C = AB column version (Fortran, step 1) C(:,j) = A*B(:,j)

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Row version

vector-matrix multiplication: vT = uTA

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Row version

vector-matrix multiplication: vT = uTA row version v = zeros(1,n); for i=1:m v = v + u(i)*A(i,:); endfor

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Row version

vector-matrix multiplication: vT = uTA row version v = zeros(1,n); for i=1:m v = v + u(i)*A(i,:); endfor matrix-matrix multiplication: C = AB

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Row version

vector-matrix multiplication: vT = uTA row version v = zeros(1,n); for i=1:m v = v + u(i)*A(i,:); endfor matrix-matrix multiplication: C = AB row version (C) C(i,:) = A(i,:)*B

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Rank and nullspace

Suppose A is an m-by-n matrix, Ax = 0 has at least one (trivial) solution, namely x = 0. There are other (nontrivial) solutions in case n > m. Even if m = n, there might be nonzero solutions to Ax = 0 when A is not invertible. It is the number r of independent rows or columns that counts.

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Rank and nullspace (cont.)

Rank The number r of independent rows or columns is the rank of A (r ≤ m and r ≤ n, that is, r ≤ min(m, n)).

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Rank and nullspace (cont.)

Rank The number r of independent rows or columns is the rank of A (r ≤ m and r ≤ n, that is, r ≤ min(m, n)). Null space The null space of A is the set of all solutions x to Ax = 0. x in nullspace x1(column 1) + · · · + xn(column n) = 0

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Rank and nullspace (cont.)

Rank The number r of independent rows or columns is the rank of A (r ≤ m and r ≤ n, that is, r ≤ min(m, n)). Null space The null space of A is the set of all solutions x to Ax = 0. x in nullspace x1(column 1) + · · · + xn(column n) = 0 This nullspace N(A) contains only x = 0 when the columns of A are independent. In that case A is of full column rank r = n.

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Rank and nullspace (cont.)

• Example. The nullspace of

1 2 3 6

• is a line.

Question Find the line.

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Rank and nullspace (cont.)

• Example. The nullspace of

1 2 3 6

• is a line.

Question Find the line. We often require that A is of full column rank. In that case, ATA, n-by-n, is invertible, and symmetric and positive definite.

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Column (range) space

Column (range) space The column (range) space contains all combinations of the columns.

• Example. The column space of

1 2 3 6

• is always through

1 3

• .

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Column (range) space (cont.)

In other words, the column space C(A) contains all possible products Ax, thus also called the range space R(A). For an m-by-n matrix, the column space is in m-dimensional space. The word “space” indicates: Any combination of vectors in the space stays in the space. The zero combination is allowed, so x = 0 is in every space.

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Solution to linear equations

A solution to Ax = b calls for a linear combination of the columns that equals b. Thus, if b is in R(A), there is a solution to Ax = b, otherwise, Ax = b has no solution.

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Solution to linear equations

A solution to Ax = b calls for a linear combination of the columns that equals b. Thus, if b is in R(A), there is a solution to Ax = b, otherwise, Ax = b has no solution. How do we write down all solutions, when b ∈ R(A)?

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Solution to linear equations

A solution to Ax = b calls for a linear combination of the columns that equals b. Thus, if b is in R(A), there is a solution to Ax = b, otherwise, Ax = b has no solution. How do we write down all solutions, when b ∈ R(A)? Suppose xp is a particular solution to Ax = b. Any vector xn in the nullspace solves Ax = 0. The complete solution to Ax = b has the form: x = (one xp) + (all xn).

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Solution to linear equations (cont.)

Questions Find the complete solution to 1 2 3 6

• x =

5 15

• .

Does the complete solution form a space?

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Comments

Suppose A is a square invertible matrix, then the nullspace

• nly contains xn = 0. The complete solution

x = A−1b + 0 = A−1b.

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Comments

Suppose A is a square invertible matrix, then the nullspace

• nly contains xn = 0. The complete solution

x = A−1b + 0 = A−1b. When Ax = b has infinitely many solutions, the shortest x always lies in the “row space” of A. A particular solution can be found by the pseudo-inverse pinv(A).

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Comments

Suppose A is a square invertible matrix, then the nullspace

• nly contains xn = 0. The complete solution

x = A−1b + 0 = A−1b. When Ax = b has infinitely many solutions, the shortest x always lies in the “row space” of A. A particular solution can be found by the pseudo-inverse pinv(A). Suppose A is tall and thin (m > n). The columns are likely to be independent. But if b is not in the column space, Ax = b has no solution. The least squares method minimizes Ax − b2

2 by solving ATA

x = ATb.

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Outline

1

Matrix Singularity

2

Matrix Multiplication by Columns or Rows Rank and nullspace Column space and solutions to linear equations

3

The Four Fundamental Subspaces

4

Dimension and Basis

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Four spaces of A

The column space R(A) of A is a subspace of Rm. The nullspace N(A) of A is a subspace of Rn. In addition, we consider N(AT), a subspace of Rm and R(AT), a subspace of Rn. Four spaces of A: R(A), N(A), N(AT), R(AT)

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Four spaces of A

The column space R(A) of A is a subspace of Rm. The nullspace N(A) of A is a subspace of Rn. In addition, we consider N(AT), a subspace of Rm and R(AT), a subspace of Rn. Four spaces of A: R(A), N(A), N(AT), R(AT) Question Let A = 1 2 3 6

• ,

Draw R(A) and N(AT) in the same figure, and draw N(A) and R(AT) in the another figure.

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Four spaces of A (cont.)

Four subspaces R(A) and N(AT) are perpendicular (in Rm). N(A) and R(AT) are perpendicular (in Rn).

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Four spaces of A (cont.)

Four subspaces R(A) and N(AT) are perpendicular (in Rm). N(A) and R(AT) are perpendicular (in Rn). Each subspace contains either infinitely many vectors or only the zero vector. If u is in a space, so are 10u and −100u (and most certainly 0u). We measure the dimension of a space not by the number of vector, but by the number of independent

• vectors. In the above example, a line has one independent

vector but not two.

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

Outline

1

Matrix Singularity

2

Matrix Multiplication by Columns or Rows Rank and nullspace Column space and solutions to linear equations

3

The Four Fundamental Subspaces

4

Dimension and Basis

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

A basis for a space

A full set of independent vectors is a basis for a space. Basis

1

The basis vectors are linearly independent.

2

Every vector in the space is a unique combination of those basis vectors.

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

A basis for a space

A full set of independent vectors is a basis for a space. Basis

1

The basis vectors are linearly independent.

2

Every vector in the space is a unique combination of those basis vectors. Some particular bases for Rn: standard basis = columns of the identity matrix general basis = columns of any invertible matrix

• rthogonal basis

= columns of any orthogonal matrix

Matrix Singularity Matrix Multiplication by Columns or Rows The Four Fundamental Subspaces Dimension and Basis

A basis for a space

A full set of independent vectors is a basis for a space. Basis

1

The basis vectors are linearly independent.

2

Every vector in the space is a unique combination of those basis vectors. Some particular bases for Rn: standard basis = columns of the identity matrix general basis = columns of any invertible matrix

• rthogonal basis

= columns of any orthogonal matrix The dimension of a space is the number of vectors in a basis for the space.