Singular Value Decomposition (matrix factorization) Singular Value - - PowerPoint PPT Presentation

Singular Value Decomposition (matrix factorization)

Singular Value Decomposition

The SVD is a factorization of a 𝑛×𝑜 matrix into 𝑩 = 𝑽 𝚻 𝑾𝑼 where 𝑽 is a 𝑛×𝑛 orthogonal matrix, 𝑾𝑼 is a 𝑜×𝑜 orthogonal matrix and 𝚻 is a 𝑛×𝑜 diagonal matrix.

For a square matrix (𝒏 = 𝒐):

𝑩 = ⋮ … ⋮ 𝒗& … 𝒗' ⋮ … ⋮ 𝜏& ⋱ 𝜏' … 𝐰&

(

… ⋮ ⋮ ⋮ … 𝐰'

(

… 𝑩 = ⋮ … ⋮ 𝒗& … 𝒗' ⋮ … ⋮ 𝜏& ⋱ 𝜏' ⋮ … ⋮ 𝒘& … 𝒘' ⋮ … ⋮

(

𝜏& ≥ 𝜏) ≥ 𝜏* …

Reduced SVD

𝑩 = 𝑽 𝚻 𝑾𝑼 = ⋮ … ⋮ … ⋮ 𝒗" … 𝒗# … 𝒗$ ⋮ … ⋮ … ⋮ 𝜏" ⋱ 𝜏# ⋮ … 𝐰"

%

… ⋮ ⋮ ⋮ … 𝐰#

%

…

𝑛×𝑜 𝑛×𝑛 𝑜×𝑜

𝑩 = 𝑽𝑺 𝚻𝑺𝑾𝑼

What happens when 𝑩 is not a square matrix? 1) 𝒏 > 𝒐 We can instead re-write the above as: Where 𝑽𝑺 is a 𝑛×𝑜 matrix and 𝚻𝑺 is a 𝑜×𝑜 matrix

Reduced SVD

𝑩 = 𝑽 𝚻 𝑾𝑼 =

⋮ … ⋮ 𝒗" … 𝒗# ⋮ … ⋮

𝜏& ⋱ 𝜏. ⋱

… 𝐰"

%

… ⋮ ⋮ ⋮ … 𝐰$

%

… ⋮ ⋮ ⋮ … 𝐰#

%

…

𝑛×𝑜 𝑛×𝑛 𝑜×𝑜

𝑩 = 𝑽 𝚻𝑺𝑾𝑺

𝑼 2) 𝒐 > 𝒏 We can instead re-write the above as: where 𝑾𝑺 is a 𝑜×𝑛 matrix and 𝚻𝑺 is a 𝑛×𝑛 matrix In general:

𝑩 = 𝑽𝑺𝚻𝑺𝑾𝑺

𝑼

𝑽𝑺 is a 𝑛×𝑙 matrix 𝚻𝑺 is a 𝑙 ×𝑙 matrix 𝑾𝑺 is a 𝑜×𝑙 matrix 𝑙 = min(𝑛, 𝑜)

Let’s take a look at the product 𝚻𝑼𝚻, where 𝚻 has the singular values of a 𝑩, a 𝑛×𝑜 matrix.

𝚻𝑼𝚻 = 𝜏" ⋱ 𝜏# ⋱ 𝜏" ⋱ 𝜏# ⋮ = 𝜏"$ ⋱ 𝜏#$

𝑛×𝑜 𝑜×𝑛 𝑜×𝑜

𝚻𝑼𝚻 = 𝜏" ⋱ 𝜏% ⋮ 𝜏" ⋱ 𝜏% ⋱ = 𝜏"$ ⋱ 𝜏%$ ⋱ ⋱ ⋱

𝑛×𝑜 𝑜×𝑛 𝑜×𝑜

𝑛 > 𝑜 𝑜 > 𝑛

Assume 𝑩 with the singular value decomposition 𝑩 = 𝑽 𝚻 𝑾𝑼. Let’s take a look at the eigenpairs corresponding to 𝑩𝑼𝑩: 𝑩𝑼𝑩 = 𝑽 𝚻 𝑾𝑼 𝑼 𝑽 𝚻 𝑾𝑼 𝑾𝑼 𝑼 𝚻

𝑼𝑽𝑼 𝑽 𝚻 𝑾𝑼 = 𝑾𝚻𝑼𝑽𝑼 𝑽 𝚻 𝑾𝑼 = 𝑾 𝚻𝑼𝚻 𝑾𝑼

Hence 𝑩𝑼𝑩 = 𝑾 𝚻𝟑 𝑾𝑼 Recall that columns of 𝑾 are all linear independent (orthogonal matrix), then from diagonalization (𝑪 = 𝒀𝑬𝒀1𝟐), we get:

• the columns of 𝑾 are the eigenvectors of the matrix 𝑩𝑼𝑩
• The diagonal entries of 𝚻𝟑 are the eigenvalues of 𝑩𝑼𝑩

Let’s call 𝜇 the eigenvalues of 𝑩𝑼𝑩, then 𝜏3) = 𝜇3

In a similar way, 𝑩𝑩𝑼 = 𝑽 𝚻 𝑾𝑼 𝑽 𝚻 𝑾𝑼 𝑼 𝑽 𝚻 𝑾𝑼 𝑾𝑼 𝑼 𝚻

𝑼𝑽𝑼 = 𝑽 𝚻 𝑾𝑼𝑾𝚻𝑼𝑽𝑼 = 𝑽𝚻 𝚻𝑼𝑽𝑼

Hence 𝑩𝑩𝑼 = 𝑽 𝚻𝟑 𝑽𝑼 Recall that columns of 𝑽 are all linear independent (orthogonal matrices), then from diagonalization (𝑪 = 𝒀𝑬𝒀1𝟐), we get:

• The columns of 𝑽 are the eigenvectors of the matrix 𝑩𝑩𝑼

How can we compute an SVD of a matrix A ?

• 1. Evaluate the 𝑜 eigenvectors 𝐰3 and eigenvalues 𝜇3 of 𝑩𝑼𝑩
• 2. Make a matrix 𝑾 from the normalized vectors 𝐰3. The columns are called

“right singular vectors”. 𝑾 = ⋮ … ⋮ 𝐰& … 𝐰' ⋮ … ⋮

• 3. Make a diagonal matrix from the square roots of the eigenvalues.

𝚻 = 𝜏& ⋱ 𝜏' 𝜏3= 𝜇3 and 𝜏&≥ 𝜏) ≥ 𝜏* …

• 4. Find 𝑽: 𝑩 = 𝑽 𝚻 𝑾𝑼 ⟹ 𝑽 𝚻 = 𝑩 𝑾 ⟹ 𝑽 = 𝑩 𝑾 𝚻1𝟐. The columns

are called the “left singular vectors”.

True or False?

𝑩 has the singular value decomposition 𝑩 = 𝑽 𝚻 𝑾𝑼.

• The matrices 𝑽 and 𝑾 are not singular
• The matrix 𝚻 can have zero diagonal entries
• 𝑽 ) = 1
• The SVD exists when the matrix 𝑩 is singular
• The algorithm to evaluate SVD will fail when taking the square root
• f a negative eigenvalue

Singular values cannot be negative since 𝑩𝑼𝑩 is a positive semi- definite matrix (for real matrices 𝑩)

• A matrix is positive definite if 𝒚𝑼𝑪𝒚 > 𝟏 for ∀𝒚 ≠ 𝟏
• A matrix is positive semi-definite if 𝒚𝑼𝑪𝒚 ≥ 𝟏 for ∀𝒚 ≠ 𝟏
• What do we know about the matrix 𝑩𝑼𝑩 ?

𝒚𝑼 𝑩𝑼𝑩 𝒚 = (𝑩𝒚)𝑼𝑩𝒚 = 𝑩𝒚 𝟑

𝟑 ≥ 0

• Hence we know that 𝑩𝑼𝑩 is a positive semi-definite matrix
• A positive semi-definite matrix has non-negative eigenvalues

𝑪𝒚 = 𝜇𝒚 ⟹ 𝒚𝑼𝑪𝒚 = 𝒚𝑼 𝜇 𝒚 = 𝜇 𝒚 𝟑

𝟑 ≥ 0 ⟹ 𝜇 ≥ 0

Singular values are always non-negative

Cost of SVD

The cost of an SVD is proportional to 𝒏 𝒐𝟑 + 𝒐𝟒where the constant of proportionality constant ranging from 4 to 10 (or more) depending on the algorithm.

𝐷456 = 𝛽 𝑛 𝑜) + 𝑜* = 𝑃 𝑜* 𝐷.78.78 = 𝑜*= 𝑃 𝑜* 𝐷9: = 2𝑜*/3 = 𝑃 𝑜*

SVD summary:

• The SVD is a factorization of a 𝑛×𝑜 matrix into 𝑩 = 𝑽 𝚻 𝑾𝑼 where 𝑽 is a 𝑛×𝑛
• rthogonal matrix, 𝑾𝑼 is a 𝑜×𝑜 orthogonal matrix and 𝚻 is a 𝑛×𝑜 diagonal matrix.
• In reduced form: 𝑩 = 𝑽𝑺𝚻𝑺𝑾𝑺

𝑼, where 𝑽𝑺 is a 𝑛×𝑙 matrix, 𝚻𝑺 is a 𝑙 ×𝑙 matrix,

and 𝑾𝑺 is a 𝑜×𝑙 matrix, and 𝑙 = min(𝑛, 𝑜).

• The columns of 𝑾 are the eigenvectors of the matrix 𝑩𝑼𝑩, denoted the right singular

vectors.

• The columns of 𝑽 are the eigenvectors of the matrix 𝑩𝑩𝑼, denoted the left singular

vectors.

• The diagonal entries of 𝚻𝟑 are the eigenvalues of 𝑩𝑼𝑩. 𝜏&=

𝜇& are called the singular values.

• The singular values are always non-negative (since 𝑩𝑼𝑩 is a positive semi-definite matrix,

the eigenvalues are always 𝜇 ≥ 0)

Singular Value Decomposition (applications)

1) Determining the rank of a matrix

𝑩 = ⋮ … ⋮ … ⋮ 𝒗" … 𝒗' … 𝒗# ⋮ … ⋮ … ⋮ 𝜏" ⋱ 𝜏' ⋮ … 𝐰"

(

… ⋮ ⋮ ⋮ … 𝐰'

(

…

Suppose 𝑩 is a 𝑛×𝑜 rectangular matrix where 𝑛 > 𝑜: 𝑩 = =

!"# $

𝜏!𝒗!𝐰!

%

𝑩# = 𝜏#𝒗#𝐰#

% what is rank 𝑩# = ?

In general, rank 𝑩& = 𝑙 A) 1 B) n C) depends on the matrix D) NOTA

𝑩 = ⋮ … ⋮ 𝒗" … 𝒗' ⋮ … ⋮ … 𝜏" 𝐰"

(

… ⋮ ⋮ ⋮ … 𝜏' 𝐰'

(

… = 𝜏"𝒗"𝐰"

( + 𝜏)𝒗)𝐰) ( + ⋯ + 𝜏'𝒗'𝐰' (

Rank of a matrix

For general rectangular matrix 𝑩 with dimensions 𝑛×𝑜, the reduced SVD is:

𝑩 = I

3G& H

𝜏3𝒗3𝐰3

(

𝑩 = 𝑽𝑺𝚻𝑺𝑾𝑺

𝑼

𝑛×𝑜 𝑛×𝑙 𝑙×𝑙 𝑙 ×𝑜 𝑙 = min(𝑛, 𝑜)

𝜯 = 𝜏# ⋱ 𝜏& ⋱ … 𝜯 = 𝜏# ⋱ 𝜏& ⋱ ⋮

If 𝜏& ≠ 0 ∀𝑗, then rank 𝑩 = 𝑙 (Full rank matrix) In general, rank 𝑩 = 𝒔, where 𝒔 is the number of non-zero singular values 𝜏& 𝑠 < 𝑙 (Rank deficient)

• The rank of A equals the number of non-zero singular values which is

the same as the number of non-zero diagonal elements in Σ.

• Rounding errors may lead to small but non-zero singular values in a

rank deficient matrix, hence the rank of a matrix determined by the number of non-zero singular values is sometimes called “effective rank”.

• The right-singular vectors (columns of 𝑾) corresponding to vanishing

singular values span the null space of A.

• The left-singular vectors (columns of 𝑽) corresponding to the non-zero

singular values of A span the range of A.

Rank of a matrix

2) Pseudo-inverse

• Problem: if A is rank-deficient, 𝚻 is not be invertible
• How to fix it: Define the Pseudo Inverse
• Pseudo-Inverse of a diagonal matrix:

𝚻N 3 = N

& O& , if 𝜏3 ≠ 0

0, if 𝜏3 = 0

• Pseudo-Inverse of a matrix 𝑩:

𝑩N = 𝑾𝚻N𝑽𝑼

3) Matrix norms

The Euclidean norm of an orthogonal matrix is equal to 1

𝑽 ) = max

𝒚 !+" 𝑽𝒚 ) = max 𝒚 !+"

𝑽𝒚 𝑼(𝑽𝒚) = max

𝒚 !+"

𝒚𝑼𝒚 = max

𝒚 !+" 𝒚 ) = 1

The Euclidean norm of a matrix is given by the largest singular value

𝑩 ) = max

𝒚 !+" 𝑩𝒚 ) = max 𝒚 !+" 𝑽 𝚻 𝑾𝑼𝒚 ) = max 𝒚 !+" 𝚻 𝑾𝑼𝒚 ) =

= max

𝑾𝑼𝒚 !+" 𝚻 𝑾𝑼𝒚 ) = max 𝒛 !+" 𝚻 𝒛 ) = max(𝜏&)

Where we used the fact that 𝑽 ) = 1, 𝑾 ) = 1 and 𝚻 is diagonal 𝑩 ) = max 𝜏& = 𝜏#./

𝜏'() is the largest singular value

4) Norm for the inverse of a matrix

The Euclidean norm of the inverse of a square-matrix is given by:

Assume here 𝑩 is full rank, so that 𝑩1& exists

𝑩0" ) = max

𝒚 !+" (𝑽 𝚻 𝑾𝑼)0"𝒚 ) = max 𝒚 !+" 𝑾 𝚻0𝟐𝑽𝑼𝒚 )

Since 𝑽 ) = 1, 𝑾 ) = 1 and 𝚻 is diagonal then 𝑩0" )=

" 2#$%

𝜏'!$ is the smallest singular value

5) Norm of the pseudo-inverse matrix

The norm of the pseudo-inverse of a 𝑛 × 𝑜 matrix is: 𝑩3 )= "

2&

where 𝜏4 is the smallest non-zero singular value. This is valid for any matrix, regardless

• f the shape or rank.

Note that for a full rank square matrix, 𝑩3 ) is the same as 𝑩0" ). Zero matrix: If 𝑩 is a zero matrix, then 𝑩3 is also the zero matrix, and 𝑩3 )= 0

The condition number of a matrix is given by 𝑑𝑝𝑜𝑒) 𝑩 = 𝑩 ) 𝑩3 ) If the matrix is full rank: 𝑠𝑏𝑜𝑙 𝑩 = 𝑛𝑗𝑜 𝑛, 𝑜 𝑑𝑝𝑜𝑒) 𝑩 = 𝜏#./ 𝜏#&' where 𝜏#./ is the largest singular value and 𝜏#&' is the smallest singular value If the matrix is rank deficient: 𝑠𝑏𝑜𝑙 𝑩 < 𝑛𝑗𝑜 𝑛, 𝑜 𝑑𝑝𝑜𝑒) 𝑩 = ∞

6) Condition number of a matrix

7) Low-Rank Approximation

Another way to write the SVD (assuming for now 𝑛 > 𝑜 for simplicity)

𝑩 = ⋮ … ⋮ 𝒗# … 𝒗' ⋮ … ⋮ 𝜏# ⋱ 𝜏$ ⋮ … 𝐰#

%

… ⋮ ⋮ ⋮ … 𝐰$

%

… = ⋮ … ⋮ 𝒗# … 𝒗$ ⋮ … ⋮ … 𝜏# 𝐰#

%

… ⋮ ⋮ ⋮ … 𝜏$ 𝐰$

%

… = 𝜏#𝒗#𝐰#

% + 𝜏*𝒗*𝐰* % + ⋯ + 𝜏$𝒗$𝐰$ %

The SVD writes the matrix A as a sum of outer products (of left and right singular vectors).

𝜏" ≥ 𝜏) ≥ 𝜏* … ≥ 0

𝑩H = 𝜏&𝒗&𝐰&

( + 𝜏)𝒗)𝐰) ( + ⋯ + 𝜏H𝒗H𝐰H (

Note that 𝑠𝑏𝑜𝑙 𝑩 = 𝑜 and 𝑠𝑏𝑜𝑙(𝑩H) = 𝑙 and the norm of the difference between the matrix and its approximation is The best rank-𝒍 approximation for a 𝑛×𝑜 matrix 𝑩, (where 𝑙 ≤ 𝑛𝑗𝑜(𝑛, 𝑜)) is the one that minimizes the following problem: When using the induced 2-norm, the best rank-𝒍 approximation is given by:

7) Low-Rank Approximation (cont.)

𝑩 − 𝑩&

* =

𝜏&+#𝒗&+#𝐰&+#

%

+ 𝜏&+*𝒗&+*𝐰&+*

%

+ ⋯ + 𝜏$𝒗$𝐰$

% * = 𝜏&+#

Example: Image compression

𝟔𝟏𝟏 𝟔𝟏𝟏 𝟔𝟏𝟏 𝟐𝟓𝟐𝟖 𝟐𝟓𝟐𝟖 𝟐𝟓𝟐𝟖 𝟔𝟏𝟏 𝟐𝟓𝟐𝟖

Example: Image compression

𝟔𝟏𝟏 𝟐𝟓𝟐𝟖 Image using rank-50 approximation

8) Using SVD to solve square system of linear equations

If 𝑩 is a 𝑜×𝑜 square matrix and we want to solve 𝑩 𝒚 = 𝒄, we can use the SVD for 𝑩 such that 𝑽 𝚻 𝑾𝑼𝒚 = 𝒄 𝚻 𝑾𝑼𝒚 = 𝑽𝑼𝒄 Solve: 𝚻 𝒛 = 𝑽𝑼𝒄 (diagonal matrix, easy to solve!) Evaluate: 𝒚 = 𝑾 𝒛 Cost of solve: 𝑃 𝑜( Cost of decomposition 𝑃 𝑜) (recall that SVD and LU have the same cost asymptotic behavior, however the number of operations - constant factor before 𝑜) - for the SVD is larger than LU)