Singular Value Decomposition (matrix factorization) Singular Value - PowerPoint PPT Presentation

Singular Value Decomposition (matrix factorization)

Singular Value Decomposition The SVD is a factorization of a 𝑛×𝑜 matrix into 𝑩 = 𝑽 𝚻 𝑾 𝑼 where 𝑽 is a 𝑛×𝑛 orthogonal matrix, 𝑾 𝑼 is a 𝑜×𝑜 orthogonal matrix and 𝚻 is a 𝑛×𝑜 diagonal matrix. For a square matrix ( 𝒏 = 𝒐): 𝜏 & ≥ 𝜏 ) ≥ 𝜏 * … ( 𝜏 & ⋮ … ⋮ … 𝐰 & … 𝑩 = 𝒗 & … 𝒗 ' ⋮ ⋮ ⋮ ⋱ ( ⋮ … ⋮ … 𝐰 ' … 𝜏 ' ( 𝜏 & ⋮ … ⋮ ⋮ … ⋮ 𝑩 = 𝒗 & … 𝒗 ' 𝒘 & … 𝒘 ' ⋱ ⋮ … ⋮ ⋮ … ⋮ 𝜏 '

Reduced SVD What happens when 𝑩 is not a square matrix? 1) 𝒏 > 𝒐 𝜏 " ⋱ % ⋮ … ⋮ … ⋮ … 𝐰 " … 𝜏 # 𝑩 = 𝑽 𝚻 𝑾 𝑼 = 𝒗 " … 𝒗 # … 𝒗 $ ⋮ ⋮ ⋮ 0 % ⋮ … ⋮ … ⋮ … 𝐰 # … ⋮ 0 𝑜×𝑜 𝑛×𝑛 𝑛×𝑜 We can instead re-write the above as: 𝑩 = 𝑽 𝑺 𝚻 𝑺 𝑾 𝑼 Where 𝑽 𝑺 is a 𝑛×𝑜 matrix and 𝚻 𝑺 is a 𝑜×𝑜 matrix

Reduced SVD 2) 𝒐 > 𝒏 % … 𝐰 " … 𝜏 & 0 ⋮ ⋮ ⋮ ⋮ … ⋮ 𝑩 = 𝑽 𝚻 𝑾 𝑼 = % ⋱ ⋱ … 𝐰 $ … 𝒗 " … 𝒗 # 𝜏 . 0 ⋮ ⋮ ⋮ ⋮ … ⋮ % … 𝐰 # … 𝑛×𝑛 𝑛×𝑜 𝑜×𝑜 We can instead re-write the above as: 𝑼 𝑩 = 𝑽 𝚻 𝑺 𝑾 𝑺 where 𝑾 𝑺 is a 𝑜×𝑛 matrix and 𝚻 𝑺 is a 𝑛×𝑛 matrix In general: 𝑽 𝑺 is a 𝑛×𝑙 matrix 𝑼 𝑩 = 𝑽 𝑺 𝚻 𝑺 𝑾 𝑺 𝑙 = min(𝑛, 𝑜) 𝚻 𝑺 is a 𝑙 ×𝑙 matrix 𝑾 𝑺 is a 𝑜×𝑙 matrix

Let’s take a look at the product 𝚻 𝑼 𝚻, where 𝚻 has the singular values of a 𝑩 , a 𝑛×𝑜 matrix. 𝜏 " ⋱ 𝜏 "$ 𝜏 " 0 𝜏 # 𝚻 𝑼 𝚻 = = ⋱ ⋱ ⋱ 0 𝜏 #$ 𝜏 # 0 ⋮ 0 𝑛 > 𝑜 𝑜×𝑜 𝑜×𝑛 𝑛×𝑜 𝜏 "$ 𝜏 " 0 ⋱ ⋱ ⋱ 𝜏 " 0 𝜏 %$ 𝜏 % 0 𝚻 𝑼 𝚻 = = ⋱ ⋱ 0 0 0 𝜏 % 0 ⋮ ⋱ ⋱ 0 0 0 𝑛×𝑜 𝑜 > 𝑛 𝑜×𝑛 𝑜×𝑜

Assume 𝑩 with the singular value decomposition 𝑩 = 𝑽 𝚻 𝑾 𝑼 . Let’s take a look at the eigenpairs corresponding to 𝑩 𝑼 𝑩: 𝑩 𝑼 𝑩 = 𝑽 𝚻 𝑾 𝑼 𝑼 𝑽 𝚻 𝑾 𝑼 𝑾 𝑼 𝑼 𝚻 𝑼 𝑽 𝑼 𝑽 𝚻 𝑾 𝑼 = 𝑾𝚻 𝑼 𝑽 𝑼 𝑽 𝚻 𝑾 𝑼 = 𝑾 𝚻 𝑼 𝚻 𝑾 𝑼 Hence 𝑩 𝑼 𝑩 = 𝑾 𝚻 𝟑 𝑾 𝑼 Recall that columns of 𝑾 are all linear independent (orthogonal matrix), then from diagonalization ( 𝑪 = 𝒀𝑬𝒀 1𝟐 ), we get: • the columns of 𝑾 are the eigenvectors of the matrix 𝑩 𝑼 𝑩 • The diagonal entries of 𝚻 𝟑 are the eigenvalues of 𝑩 𝑼 𝑩 Let’s call 𝜇 the eigenvalues of 𝑩 𝑼 𝑩, then 𝜏 3) = 𝜇 3

In a similar way, 𝑽 𝚻 𝑾 𝑼 𝑼 𝑩𝑩 𝑼 = 𝑽 𝚻 𝑾 𝑼 𝑾 𝑼 𝑼 𝚻 𝑼 𝑽 𝑼 = 𝑽 𝚻 𝑾 𝑼 𝑾𝚻 𝑼 𝑽 𝑼 = 𝑽𝚻 𝚻 𝑼 𝑽 𝑼 𝑽 𝚻 𝑾 𝑼 Hence 𝑩𝑩 𝑼 = 𝑽 𝚻 𝟑 𝑽 𝑼 Recall that columns of 𝑽 are all linear independent (orthogonal matrices), then from diagonalization ( 𝑪 = 𝒀𝑬𝒀 1𝟐 ), we get: • The columns of 𝑽 are the eigenvectors of the matrix 𝑩𝑩 𝑼

How can we compute an SVD of a matrix A ? 1. Evaluate the 𝑜 eigenvectors 𝐰 3 and eigenvalues 𝜇 3 of 𝑩 𝑼 𝑩 2. Make a matrix 𝑾 from the normalized vectors 𝐰 3 . The columns are called “right singular vectors”. ⋮ … ⋮ 𝑾 = 𝐰 & … 𝐰 ' ⋮ … ⋮ 3. Make a diagonal matrix from the square roots of the eigenvalues. 𝜏 & 𝚻 = 𝜏 3 = 𝜇 3 and 𝜏 & ≥ 𝜏 ) ≥ 𝜏 * … ⋱ 𝜏 ' 4. Find 𝑽: 𝑩 = 𝑽 𝚻 𝑾 𝑼 ⟹ 𝑽 𝚻 = 𝑩 𝑾 ⟹ 𝑽 = 𝑩 𝑾 𝚻 1𝟐 . The columns are called the “left singular vectors”.

True or False? 𝑩 has the singular value decomposition 𝑩 = 𝑽 𝚻 𝑾 𝑼 . • The matrices 𝑽 and 𝑾 are not singular • The matrix 𝚻 can have zero diagonal entries 𝑽 ) = 1 • • The SVD exists when the matrix 𝑩 is singular • The algorithm to evaluate SVD will fail when taking the square root of a negative eigenvalue

Singular values are always non-negative Singular values cannot be negative since 𝑩 𝑼 𝑩 is a positive semi- definite matrix (for real matrices 𝑩 ) • A matrix is positive definite if 𝒚 𝑼 𝑪𝒚 > 𝟏 for ∀𝒚 ≠ 𝟏 • A matrix is positive semi-definite if 𝒚 𝑼 𝑪𝒚 ≥ 𝟏 for ∀𝒚 ≠ 𝟏 • What do we know about the matrix 𝑩 𝑼 𝑩 ? 𝒚 𝑼 𝑩 𝑼 𝑩 𝒚 = (𝑩𝒚) 𝑼 𝑩𝒚 = 𝟑 ≥ 0 𝑩𝒚 𝟑 • Hence we know that 𝑩 𝑼 𝑩 is a positive semi-definite matrix • A positive semi-definite matrix has non-negative eigenvalues 𝑪𝒚 = 𝜇𝒚 ⟹ 𝒚 𝑼 𝑪𝒚 = 𝒚 𝑼 𝜇 𝒚 = 𝜇 𝒚 𝟑 𝟑 ≥ 0 ⟹ 𝜇 ≥ 0

Cost of SVD The cost of an SVD is proportional to 𝒏 𝒐 𝟑 + 𝒐 𝟒 where the constant of proportionality constant ranging from 4 to 10 (or more) depending on the algorithm. 𝐷 456 = 𝛽 𝑛 𝑜 ) + 𝑜 * = 𝑃 𝑜 * 𝐷 .78.78 = 𝑜 * = 𝑃 𝑜 * 𝐷 9: = 2𝑜 * /3 = 𝑃 𝑜 *

SVD summary: • The SVD is a factorization of a 𝑛×𝑜 matrix into 𝑩 = 𝑽 𝚻 𝑾 𝑼 where 𝑽 is a 𝑛×𝑛 orthogonal matrix, 𝑾 𝑼 is a 𝑜×𝑜 orthogonal matrix and 𝚻 is a 𝑛×𝑜 diagonal matrix. 𝑼 , where 𝑽 𝑺 is a 𝑛×𝑙 matrix, 𝚻 𝑺 is a 𝑙 ×𝑙 matrix, • In reduced form: 𝑩 = 𝑽 𝑺 𝚻 𝑺 𝑾 𝑺 and 𝑾 𝑺 is a 𝑜×𝑙 matrix, and 𝑙 = min(𝑛, 𝑜) . • The columns of 𝑾 are the eigenvectors of the matrix 𝑩 𝑼 𝑩 , denoted the right singular vectors. The columns of 𝑽 are the eigenvectors of the matrix 𝑩𝑩 𝑼 , denoted the left singular • vectors. • The diagonal entries of 𝚻 𝟑 are the eigenvalues of 𝑩 𝑼 𝑩. 𝜏 & = 𝜇 & are called the singular values. • The singular values are always non-negative (since 𝑩 𝑼 𝑩 is a positive semi-definite matrix, the eigenvalues are always 𝜇 ≥ 0 )

Singular Value Decomposition (applications)

1) Determining the rank of a matrix Suppose 𝑩 is a 𝑛×𝑜 rectangular matrix where 𝑛 > 𝑜 : 𝜏 " ⋱ ( ⋮ … ⋮ … ⋮ … 𝐰 " … 𝜏 ' 𝑩 = 𝒗 " … 𝒗 ' … 𝒗 # ⋮ ⋮ ⋮ 0 ( ⋮ … ⋮ … ⋮ … 𝐰 ' … ⋮ 0 ( ⋮ … ⋮ … 𝜏 " 𝐰 " … ( + 𝜏 ) 𝒗 ) 𝐰 ) ( + ⋯ + 𝜏 ' 𝒗 ' 𝐰 ' ( 𝑩 = = 𝜏 " 𝒗 " 𝐰 " 𝒗 " … 𝒗 ' ⋮ ⋮ ⋮ ( ⋮ … ⋮ … 𝜏 ' 𝐰 ' … $ % 𝑩 = = 𝜏 ! 𝒗 ! 𝐰 ! !"# % what is rank 𝑩 # = ? 𝑩 # = 𝜏 # 𝒗 # 𝐰 # A) 1 B) n C) depends on the matrix In general, rank 𝑩 & = 𝑙 D) NOTA

Rank of a matrix For general rectangular matrix 𝑩 with dimensions 𝑛×𝑜 , the reduced SVD is: 𝑼 𝑩 = 𝑽 𝑺 𝚻 𝑺 𝑾 𝑺 𝑙 = min(𝑛, 𝑜) H 𝑙 ×𝑜 𝑛×𝑜 𝑛×𝑙 ( 𝑩 = I 𝜏 3 𝒗 3 𝐰 3 𝑙×𝑙 3G& 𝜏 # 𝜏 # ⋱ 0 𝜏 & 𝜯 = ⋱ ⋱ 𝜯 = 𝜏 & 0 … 0 0 0 ⋱ ⋮ 0 If 𝜏 & ≠ 0 ∀𝑗 , then rank 𝑩 = 𝑙 (Full rank matrix) In general, rank 𝑩 = 𝒔 , where 𝒔 is the number of non-zero singular values 𝜏 & 𝑠 < 𝑙 (Rank deficient)

Rank of a matrix • The rank of A equals the number of non-zero singular values which is the same as the number of non-zero diagonal elements in Σ . • Rounding errors may lead to small but non-zero singular values in a rank deficient matrix, hence the rank of a matrix determined by the number of non-zero singular values is sometimes called “effective rank”. • The right-singular vectors (columns of 𝑾 ) corresponding to vanishing singular values span the null space of A . • The left-singular vectors (columns of 𝑽 ) corresponding to the non-zero singular values of A span the range of A .

2) Pseudo-inverse • Problem: if A is rank-deficient, 𝚻 is not be invertible • How to fix it: Define the Pseudo Inverse • Pseudo-Inverse of a diagonal matrix : & O & , if 𝜏 3 ≠ 0 𝚻 N 3 = N 0, if 𝜏 3 = 0 • Pseudo-Inverse of a matrix 𝑩 : 𝑩 N = 𝑾𝚻 N 𝑽 𝑼

3) Matrix norms The Euclidean norm of an orthogonal matrix is equal to 1 𝑽𝒚 𝑼 (𝑽𝒚) = max 𝒚 𝑼 𝒚 = max 𝑽 ) = max 𝒚 ! +" 𝑽𝒚 ) = max 𝒚 ! +" 𝒚 ) = 1 𝒚 ! +" 𝒚 ! +" The Euclidean norm of a matrix is given by the largest singular value 𝒚 ! +" 𝑽 𝚻 𝑾 𝑼 𝒚 ) = max 𝒚 ! +" 𝚻 𝑾 𝑼 𝒚 ) = 𝑩 ) = max 𝒚 ! +" 𝑩𝒚 ) = max 𝑾 𝑼 𝒚 ! +" 𝚻 𝑾 𝑼 𝒚 ) = max = max 𝒛 ! +" 𝚻 𝒛 ) = max(𝜏 & ) Where we used the fact that 𝑽 ) = 1 , 𝑾 ) = 1 and 𝚻 is diagonal 𝑩 ) = max 𝜏 & = 𝜏 #./ 𝜏 '() is the largest singular value

4) Norm for the inverse of a matrix The Euclidean norm of the inverse of a square-matrix is given by: Assume here 𝑩 is full rank, so that 𝑩 1& exists 𝑩 0" ) = max 𝒚 ! +" (𝑽 𝚻 𝑾 𝑼 ) 0" 𝒚 ) = max 𝒚 ! +" 𝑾 𝚻 0𝟐 𝑽 𝑼 𝒚 ) Since 𝑽 ) = 1 , 𝑾 ) = 1 and 𝚻 is diagonal then " 𝑩 0" ) = 𝜏 '!$ is the smallest singular value 2 #$%