Eigenvalues and Eigenvectors Few concepts to remember from linear - - PowerPoint PPT Presentation

Eigenvalues and Eigenvectors

Few concepts to remember from linear algebra

Let 𝑩 be an 𝑜×𝑛 matrix and the linear transformation 𝒛 = 𝑩𝒚

• Rank: maximum number of linearly independent columns or rows of 𝑩
• Range 𝑩 = 𝒛 = 𝑩𝒚 ∀𝒚}
• Null 𝑩 = 𝒚 𝑩𝒚 = 𝟏}

𝒚 ∈ ℛ𝒏 →

𝑩 𝒛 ∈ ℛ𝒐

Eigenvalue problem

Let 𝑩 be an 𝑜×𝑜 matrix: 𝒚 ≠ 𝟏 is an eigenvector of 𝑩 if there exists a scalar 𝜇 such that 𝑩 𝒚 = 𝜇 𝒚 where 𝜇 is called an eigenvalue. If 𝒚 is an eigenvector, then α𝒚 is also an eigenvector. Therefore, we will usually seek for normalized eigenvectors, so that 𝒚 = 1 Note: When using Python, numpy.linalg.eig will normalize using p=2 norm.

How do we find eigenvalues?

Linear algebra approach: 𝑩 𝒚 = 𝜇 𝒚 𝑩 − 𝜇 𝑱 𝒚 = 𝟏 Therefore the matrix 𝑩 − 𝜇 𝑱 is singular ⟹ 𝑒𝑓𝑢 𝑩 − 𝜇 𝑱 = 0 𝑞 𝜇 = 𝑒𝑓𝑢 𝑩 − 𝜇 𝑱 is the characteristic polynomial of degree 𝑜. In most cases, there is no analytical formula for the eigenvalues of a matrix (Abel proved in 1824 that there can be no formula for the roots

• f a polynomial of degree 5 or higher) ⟹ Approximate the

eigenvalues numerically!

Example

Notes: The matrix 𝑩 is singular (det(A)=0), and rank(𝑩)=1 The matrix has two distinct real eigenvalues The eigenvectors are linearly independent

𝑩 = 2 1 4 2 𝑒𝑓𝑢 2 − 𝜇 1 4 2 − 𝜇 = 0

Solution of characteristic polynomial gives: 𝜇. = 4, 𝜇/ = 0 To get the eigenvectors, we solve: 𝑩 𝒚 = 𝜇 𝒚

2 − (4) 1 4 2 − (4) 𝑦$ 𝑦% = 0 𝒚 = 1 2 2 − (0) 1 4 2 − (0) 𝑦$ 𝑦% = 0 𝒚 = −1 2

Diagonalizable Matrices

A 𝑜×𝑜 matrix 𝑩 with 𝑜 linearly independent eigenvectors 𝒗 is said to be diagonalizable. 𝑩 𝒗𝟐 = 𝜇. 𝒗𝟐, 𝑩 𝒗𝟑 = 𝜇/ 𝒗𝟑, … 𝑩 𝒗𝒐 = 𝜇; 𝒗𝒐, In matrix form:

𝑩 𝒗𝟐 … 𝒗𝒐 = 𝜇#𝒗𝟐 … 𝜇$𝒗𝒐 = 𝒗𝟐 … 𝒗𝒐 𝜇# ⋱ 𝜇$ This corresponds to a similarity transformation 𝑩𝑽 = 𝑽𝑬 ⟺ 𝑩 = 𝑽𝑬𝑽%𝟐

Example

𝑩 = 2 1 4 2 𝑒𝑓𝑢 2 − 𝜇 1 4 2 − 𝜇 = 0

Solution of characteristic polynomial gives: 𝜇. = 4, 𝜇/ = 0 To get the eigenvectors, we solve: 𝑩 𝒚 = 𝜇 𝒚

2 − (4) 1 4 2 − (4) 𝑦$ 𝑦% = 0 𝒚 = 1 2 2 − (0) 1 4 2 − (0) 𝑦$ 𝑦% = 0 𝒚 = −1 2

𝑩 = 𝑽𝑬𝑽<. 𝑽 = 0.447 0.894 −0.447 0.894

• r normalized

eigenvector (𝑞 = 2 norm) 𝒚 = 0.447 0.894 𝒚 = −0.447 0.894

𝑬 = 4 Notes: The matrix 𝑩 is singular (det(A)=0), and rank(𝑩)=1 Since 𝑩 has two linearly independent eigenvectors, the matrix 𝑽 is full rank, and hence, the matrix 𝑩 is diagonalizable.

Example

The eigenvalues of the matrix: 𝑩 = 3 −18 2 −9 are 𝜇. = 𝜇/ = −3. Select the incorrect statement: A) Matrix 𝑩 is diagonalizable B) The matrix 𝑩 has only one eigenvalue with multiplicity 2 C) Matrix 𝑩 has only one linearly independent eigenvector D) Matrix 𝑩 is not singular

Let’s look back at diagonalization…

1) If a 𝑜×𝑜 matrix 𝑩 has 𝑜 linearly independent eigenvectors 𝒚 then 𝑩 is diagonalizable, i.e., 𝑩 = 𝑽𝑬𝑽<𝟐 where the columns of 𝑽 are the linearly independent normalized eigenvectors 𝒚 of 𝑩 (which guarantees that 𝑽<𝟐 exists) and 𝑬 is a diagonal matrix with the eigenvalues of 𝑩. 2) If a 𝑜×𝑜 matrix 𝑩 has less then 𝑜 linearly independent eigenvectors, the matrix is called defective (and therefore not diagonalizable). 3) If a 𝑜×𝑜 symmetric matrix 𝑩 has 𝑜 distinct eigenvalues then 𝑩 is diagonalizable.

A 𝒐×𝒐 symmetric matrix 𝑩 with 𝒐 distinct eigenvalues is diagonalizable. Suppose 𝜇,𝒗 and 𝜈, 𝒘 are eigenpairs of 𝑩 𝜇 𝒗 = 𝑩𝒗 𝜈 𝒘 = 𝑩𝒘 𝜇 𝒗 = 𝑩𝒗 → 𝒘 1 𝜇 𝒗 = 𝒘 1 𝑩𝒗 𝜇 𝒘 1 𝒗 = 𝑩𝑼𝒘 1 𝒗 = 𝑩 𝒘 1 𝒗 = 𝜈 𝒘 1 𝒗 → 𝜈 − 𝜇 𝒘 1 𝒗 = 0 If all 𝑜 eigenvalues are distinct → 𝜈 − 𝜇 ≠ 0 Hence, 𝒘 1 𝒗 = 0, i.e., the eigenvectors are orthogonal (linearly independent), and consequently the matrix 𝑩 is diagonalizable. Note that a diagonalizable matrix 𝑩 does not guarantee 𝑜 distinct eigenvalues.

Some things to remember about eigenvalues:

• Eigenvalues can have zero value
• Eigenvalues can be negative
• Eigenvalues can be real or complex numbers
• A 𝑜×𝑜 real matrix can have complex eigenvalues
• The eigenvalues of a 𝑜×𝑜 matrix are not necessarily unique. In fact,

we can define the multiplicity of an eigenvalue.

• If a 𝑜×𝑜 matrix has 𝑜 linearly independent eigenvectors, then the

matrix is diagonalizable

How can we get eigenvalues numerically?

Assume that 𝑩 is diagonalizable (i.e., it has 𝑜 linearly independent eigenvectors 𝒗). We can propose a vector 𝒚 which is a linear combination of these eigenvectors: 𝒚 = 𝛽#𝒗# + 𝛽'𝒗' + ⋯ + 𝛽$𝒗$ Then we evaluate 𝑩 𝒚: 𝑩 𝒚 = 𝛽#𝑩𝒗# + 𝛽'𝑩𝒗' + ⋯ + 𝛽$𝑩𝒗$ And since 𝑩𝒗# = 𝜇#𝒗# we can also write: 𝑩 𝒚 = 𝛽#𝜇#𝒗# + 𝛽'𝜇'𝒗' + ⋯ + 𝛽$𝜇$𝒗$ where 𝜇( is the eigenvalue corresponding to eigenvector 𝒗( and we assume |𝜇#| > |𝜇'| ≥ |𝜇)| ≥ ⋯ ≥ |𝜇$|

Power Iteration

Our goal is to find an eigenvector 𝒗( of 𝑩. We will use an iterative process, where we start with an initial vector, where here we assume that it can be written as a linear combination of the eigenvectors of 𝑩. 𝒚* = 𝛽#𝒗# + 𝛽'𝒗' + ⋯ + 𝛽$𝒗$ And multiply by 𝑩 to get: 𝒚# = 𝑩 𝒚* = 𝛽#𝜇#𝒗# + 𝛽'𝜇'𝒗' + ⋯ + 𝛽$𝜇$𝒗$ 𝒚' = 𝑩 𝒚# = 𝛽# 𝜇# '𝒗# + 𝛽' 𝜇' '𝒗' + ⋯ + 𝛽$ 𝜇$ '𝒗$ ⋮ 𝒚+ = 𝑩 𝒚+%# = 𝛽# 𝜇# +𝒗# + 𝛽' 𝜇' +𝒗' + ⋯ + 𝛽$ 𝜇$ +𝒗$ Or rearranging… 𝒚+ = 𝜇# + 𝛽#𝒗# + 𝛽' 𝜇' 𝜇#

+

𝒗' + ⋯ + 𝛽$ 𝜇$ 𝜇#

+

𝒗$

Power Iteration

𝒚B = 𝜇. B 𝛽.𝒗. + 𝛽/ 𝜇/ 𝜇.

B

𝒗/ + ⋯ + 𝛽; 𝜇; 𝜇.

B

𝒗; Assume that 𝛽. ≠ 0, the term 𝛽.𝒗. dominates the others when 𝑙 is very large. Since |𝜇. > |𝜇/ , we have C!

C" B

≪ 1 when 𝑙 is large Hence, as 𝑙 increases, 𝒚B converges to a multiple of the first eigenvector 𝒗., i.e., lim

B→D 𝒚# C" # = 𝛽.𝒗. or 𝒚B → 𝛽. 𝜇. B 𝒗.

How can we now get the eigenvalues?

If 𝒚 is an eigenvector of 𝑩 such that 𝑩 𝒚 = 𝜇 𝒚 then how can we evaluate the corresponding eigenvalue 𝜇? 𝜇 = 𝒚𝑼𝑩𝒚 𝒚𝑼𝒚 Rayleigh coefficient

Normalized Power Iteration

𝒚𝟏 = arbitrary nonzero vector 𝒚𝟏 = 𝒚𝟏 𝒚𝟏 for 𝑙 = 1,2, … 𝒛B = 𝑩 𝒚B<. 𝒚B =

𝒛# 𝒛# 𝒚& = 𝜇$ & 𝛽$𝒗$ + 𝛽% 𝜇% 𝜇$

&

𝒗% + ⋯ + 𝛽' 𝜇' 𝜇$

&

𝒗'

Normalized Power Iteration

Demo “Power Iteration

𝒚B = 𝜇. B 𝛽.𝒗. + 𝛽/ 𝜇/ 𝜇.

B

𝒗/ + ⋯ + 𝛽; 𝜇; 𝜇.

B

𝒗;

What if the starting vector 𝒚𝟏 have no component in the dominant eigenvector 𝒗$ (𝛽$ = 0)?

Normalized Power Iteration

Demo “Power Iteration

𝒚B = 𝜇. B 𝛽.𝒗. + 𝛽/ 𝜇/ 𝜇.

B

𝒗/ + ⋯ + 𝛽; 𝜇; 𝜇.

B

𝒗;

What if the first two largest eigenvalues (in magnitude) are the same, |𝜇$ = |𝜇% ?

𝒚B = 𝜇. B𝛽.𝒗. + 𝜇. B 𝜇/ 𝜇.

B

𝛽/𝒗/ + 𝜇. B … + 𝛽; 𝜇; 𝜇.

B

𝒗;

Potential pitfalls

1. Starting vector 𝒚𝟏 may have no component in the dominant eigenvector 𝒗" (𝛽" = 0). This is usually unlikely to happen if 𝒚𝟏 is chosen randomly, and in practice not a problem because rounding will usually introduce such component. 2. Risk of eventual overflow (or underflow): in practice the approximated eigenvector is normalized at each iteration (Normalized Power Iteration) 3. First two largest eigenvalues (in magnitude) may be the same: |𝜇"| = |𝜇#|. In this case, power iteration will give a vector that is a linear combination of the corresponding eigenvectors:

• If signs are the same, the method will converge to correct magnitude of the
• eigenvalue. If the signs are different, the method will not converge.
• This is a “real” problem that cannot be discounted in practice.

Error

𝒚B = 𝜇. B 𝛽.𝒗. + 𝛽/ 𝜇/ 𝜇.

B

𝒗/ + ⋯ + 𝛽; 𝜇; 𝜇.

B

𝒗;

𝐹𝑠𝑠𝑝𝑠

We can see from the above that the rate of convergence depends on the ratio C!

C" , that is:

𝜇. <B 𝒚B − 𝛽.𝒗. = 𝑃 𝜇/ 𝜇.

B

Convergence and error

𝒚B = 𝒗. + 𝛽/ 𝛽. 𝜇/ 𝜇.

B

𝒗/ + ⋯

𝒇&

Power method has linear convergence, which is quite slow.

𝒇PQR 𝒇P ≈ ?S ?R

A) 0.1536 B) 0.192 C) 0.09 D) 0.027

Iclicker question

Iclicker question

Suppose we want to use the normalized power iteration, starting from 𝑦T = (−0.5,0). Select the correct statement A) Normalized power iteration will not converge B) Normalized power iteration will converge to the eigenvector corresponding to the eigenvalue 2. C) Normalized power iteration will converge to the eigenvector corresponding to the eigenvalue 4.

𝒗$ 𝒗%

The matrix 𝑩 = 3 1 1 3 has eigenvalues (4,2) and corresponding eigenvectors 𝒗. = (1,1) and 𝒗/ = (−1,1).

𝒚)

Iclicker question

Suppose 𝒚 is an eigenvector of 𝑩 such that 𝑩 𝒚 = 𝜇 𝒚 What is an eigenvalue of 𝑩<.? A) 𝜇 B) −𝜇 C) 1/𝜇 D) − .

C

E) Can’t tell without knowing 𝜇

Inverse Power Method

Previously we learned that we can use the Power Method to obtain the largest eigenvalue and corresponding eigenvector, by using the update 𝒚BU. = 𝑩 𝒚B Suppose there is a single smallest eigenvalue of 𝑩. With the previous

• rdering

|𝜇.| > |𝜇/| ≥ |𝜇V| ≥ ⋯ > |𝜇;| the smallest eigenvalue is 𝜇;. When computing the eigenvalues of the inverse matrix 𝑩<., we get the following ordering 1 𝜇; > 1 𝜇;<. ≥ ⋯ ≥ 1 𝜇. And hence we can use the Power Method update on the matrix 𝑩<. to compute the dominant eigenvalue .

C$, i.e.,

𝒚BU. = 𝑩<. 𝒚B

Iclicker question

Which code snippet is the best option to compute the smallest eigenvalue of the matrix 𝑩?

A) B) D) E) I have no idea! C)

Inverse Power Method

Note that the update 𝒚BU. = 𝑩<. 𝒚B can be instead written as 𝑩 𝒚BU. = 𝒚B Where 𝒚B is know and we need to solve for 𝒚BU. (we are just solving a linear system of equations!). Since the matrix 𝑩 does not change from iteration to the next, we can factorize the matrix once and then perform a series of backward and forward substitutions. Recall 𝑸𝑩 = 𝑴𝑽 and 𝑩 𝒚 = 𝒄 resulting in 𝑴𝑽 𝒚 = 𝑸𝒄 Hence we can efficiently solve 𝑴 𝒛 = 𝑸 𝒚B 𝑽 𝒚BU. = 𝒛

Cost of computing eigenvalues using inverse power iteration

Iclicker question

What is the approximated cost of computing the largest eigenvalue using Power Method? A) 𝑙 𝑜 B) 𝑜/ + 𝑙 𝑜 C) 𝑙 𝑜/ D) 𝑜V + 𝑙 𝑜/ E) 𝑂𝑃𝑈𝐵

Iclicker question

Suppose 𝒚 is an eigenvector of 𝑩 such that 𝑩 𝒚 = 𝜇𝟐 𝒚 and also 𝒚 is an eigenvector of 𝑪 such that 𝑪 𝒚 = 𝜇𝟑 𝒚. What is an eigenvalue of What is an eigenvalue of (𝑩 + 𝟐

𝟑 𝑪)<.?

A)

C𝟐 /C𝟐UC𝟑

B)

C𝟑 /C𝟐UC𝟑

C)

/ /C𝟐UC𝟑

D)

C𝟐 /C𝟑UC𝟐

E)

C𝟑 /C𝟑UC𝟐

Iclicker question

Suppose 𝒚 is an eigenvector of 𝑩 such that 𝑩 𝒚 = 𝜇 𝒚 , but 𝜇 is not the largest or smallest eigenvalue. We want to compute the eigenvalue 𝜇 that is close to a given number 𝜏. Which of the following modified matrices will give such eigenvalue? A) A) (𝑩 − 𝜏𝑱) B) (𝑩 − 𝜏𝑱) <. C) C) (1 − 𝜏) 𝑩 D) .

W 𝑩

E) I still have no clue how to answer to these iclicker questions…

Eigenvalues of a Shifted Inverse Matrix

Suppose the eigenpairs 𝒚, 𝜇 satisfy 𝑩𝒚 = 𝜇 𝒚. We can describe the eigenvalues for the shifted inverse matrix as (𝑩 − 𝜏𝑱)<.𝒚 = ̅ 𝜇 𝒚 𝑱𝒚 = ̅ 𝜇 𝜇𝑱 − 𝜏𝑱 𝒚 ̅ 𝜇 = 1 𝜇 − 𝜏 Hence the eigensystem problem is (𝑩 − 𝜏𝑱)<.𝒚 = 1 𝜇 − 𝜏 𝒚

Eigenvalues of a Shifted Inverse Matrix

We use the update 𝑩 − 𝜏𝑱 𝒚BU. = 𝒚B To obtain the eigenpair 𝒚, 𝜇 that satisfy 𝑩𝒚 = 𝜇 𝒚 such that 𝜇 is an eigenvalue close to the number 𝜏 We can factorize the matrix 𝑪 = 𝑩 − 𝜏𝑱 such that 𝑸𝑪 = 𝑴𝑽 and then efficiently solve 𝑴 𝒛 = 𝑸 𝒚B 𝑽 𝒚BU. = 𝒛

Convergence summary

Method Cost Convergence 𝒇𝒍+𝟐 / 𝒇𝒍 Power Method

𝒚𝑙+1 = 𝑩 𝒚𝑙 𝑙 𝑜' 𝜇' 𝜇#

Inverse Power Method

𝑩 𝒚𝑙+1 = 𝒚𝑙 𝑜) + 𝑙 𝑜' 𝜇$ 𝜇$%#

Shifted Inverse Power Method

(𝑩 − 𝜏𝑱)𝒚𝑙+1= 𝒚𝑙 𝑜) + 𝑙 𝑜' 𝜇/ − 𝜏 𝜇/' − 𝜏

𝜇$: largest eigenvalue (in magnitude) 𝜇%: second largest eigenvalue (in magnitude) 𝜇': smallest eigenvalue (in magnitude) 𝜇'-$: second smallest eigenvalue (in magnitude) 𝜇.: closest eigenvalue to 𝜏 𝜇.%: second closest eigenvalue to 𝜏