Math 221: LINEAR ALGEBRA 8-3. Positive Definite Matrices Le Chen 1 - PowerPoint PPT Presentation

Math 221: LINEAR ALGEBRA §8-3. Positive Definite Matrices Le Chen 1 Emory University, 2020 Fall (last updated on 08/27/2020) Creative Commons License (CC BY-NC-SA) 1 Slides are adapted from those by Karen Seyffarth from University of Calgary.

is invertible. diag , and therefore det ; it follows that for all , is positive defjnite, Since det det so . Similar matrices have the same determinant, where , is orthogonally diagonalizable. In particular, is symmetric, Since . denote the (not necessarily distinct) eigenvalues of Let det Positive Definite Matrices Definition An n × n matrix A is positive definite if it is symmetric and has positive eigenvalues, i.e., if λ is a eigenvalue of A, then λ > 0 .

is invertible. diag , and therefore det ; it follows that for all , is positive defjnite, Since det det so . Similar matrices have the same determinant, where , is orthogonally diagonalizable. In particular, is symmetric, Since . denote the (not necessarily distinct) eigenvalues of Let Positive Definite Matrices Definition An n × n matrix A is positive definite if it is symmetric and has positive eigenvalues, i.e., if λ is a eigenvalue of A, then λ > 0 . Theorem If A is a positive definite matrix, then det ( A ) > 0 and A is invertible.

so Positive Definite Matrices Definition An n × n matrix A is positive definite if it is symmetric and has positive eigenvalues, i.e., if λ is a eigenvalue of A, then λ > 0 . Theorem If A is a positive definite matrix, then det ( A ) > 0 and A is invertible. Proof. Let λ 1 , λ 2 , . . . , λ n denote the (not necessarily distinct) eigenvalues of A . Since A is symmetric, A is orthogonally diagonalizable. In particular, A ∼ D , where D = diag ( λ 1 , λ 2 , . . . , λ n ) . Similar matrices have the same determinant, det ( A ) = det ( D ) = λ 1 λ 2 · · · λ n . Since A is positive defjnite, λ i > 0 for all i , 1 ≤ i ≤ n ; it follows that det ( A ) > 0 , and therefore A is invertible. �

Theorem x T A � A symmetric matrix A is positive definite if and only if � x > 0 for all x ∈ R n , � x � = � 0 . �

Theorem x T A � A symmetric matrix A is positive definite if and only if � x > 0 for all x ∈ R n , � x � = � 0 . � Proof. Since A is symmetric, there exists an orthogonal matrix P so that P T AP = diag ( λ 1 , λ 2 , . . . , λ n ) = D , where λ 1 , λ 2 , . . . , λ n are the (not necessarily distinct) eigenvalues of A. Let x ∈ R n , � x � = � y = P T � 0 , and define � x. Then � x T A � x T ( PDP T ) � x T P ) D ( P T � x ) = ( P T � x ) T D ( P T � y T D � � x = � x = ( � x ) = � y . � � y T = Writing � y 1 y 2 · · · y n ,   y 1  y 2  � �   x T A � � x = y 1 y 2 · · · y n diag ( λ 1 , λ 2 , . . . , λ n )  .  .   . y n λ 1 y 2 1 + λ 2 y 2 2 + · · · λ n y 2 = n .

the th column of is positive defjnite. whenever . Therefore, i.e., , so when and Thus , and thus is invertible, . Since is , where , choose Conversely, if Proof (continued). x ∈ R n , � x � = � 0 . Since P T is invertible, ( ⇒ ) Suppose A is positive defjnite, and � y = P T � x � = � 0 , and thus y j � = 0 for some j , implying y 2 j > 0 for some j . � Furthermore, since all eigenvalues of A are positive, λ i y 2 i ≥ 0 for all i ; in particular λ j y 2 x T A � j > 0 . Therefore, � x > 0 .

Proof (continued). x ∈ R n , � x � = � 0 . Since P T is invertible, ( ⇒ ) Suppose A is positive defjnite, and � y = P T � x � = � 0 , and thus y j � = 0 for some j , implying y 2 j > 0 for some j . � Furthermore, since all eigenvalues of A are positive, λ i y 2 i ≥ 0 for all i ; in particular λ j y 2 x T A � j > 0 . Therefore, � x > 0 . x T A � x � = � ( ⇐ ) Conversely, if � x > 0 whenever � 0 , choose � x = P � e j , where � e j is x � = � the j th column of I n . Since P is invertible, � 0 , and thus y = P T � x = P T ( P � � e j ) = � e j . Thus y j = 1 and y i = 0 when i � = j , so λ 1 y 2 1 + λ 2 y 2 2 + · · · λ n y 2 n = λ j , x T A � i.e., λ j = � x > 0 . Therefore, A is positive defjnite. �

Example (Constructing Positive Definite Matrices) x � = � Let U be an n × n invertible matrix, and let A = U T U. Let � x ∈ R n , � 0 . Then x T A � x T ( U T U ) � � x = � x x T U T )( U � = ( � x ) x ) T ( U � = ( U � x ) x || 2 . = || U � x || 2 > 0 , i.e., x � = � x � = � Since U is invertible and � 0 , U � 0 , and hence || U � x || 2 > 0 . Therefore, A is positive definite. x T A � � x = || U �

Notation � � be an n × n matrix. For 1 ≤ r ≤ n , ( r ) A denotes the r × r Let A = a ij submatrix in the upper left corner of A , i.e., � � ( r ) A = a ij , 1 ≤ i , j ≤ r . (1) A , (2) A , . . . , ( n ) A are called the principal submatrices of A .

Notation � � be an n × n matrix. For 1 ≤ r ≤ n , ( r ) A denotes the r × r Let A = a ij submatrix in the upper left corner of A , i.e., � � ( r ) A = a ij , 1 ≤ i , j ≤ r . (1) A , (2) A , . . . , ( n ) A are called the principal submatrices of A . Lemma If A is an n × n positive definite matrix, then each principal submatrix of A is positive definite.

Proof. Suppose A is an n × n positive definite matrix. For any integer r, 1 ≤ r ≤ n, write A in block form as � � ( r ) A B A = , C D where B is an r × ( n − r ) matrix, C is an ( n − r ) × r matrix, and D is an   y 1  y 2      y 1 .   .   .   y 2      � = �   ( n − r ) × ( n − r ) matrix. Let � y = 0 and let � x = y r . Then  .    .  .   0   y r   .  .  . 0 x � = � x T A � 0 , and by the previous theorem, � x > 0 . �

. . previous theorem. . . . But . Proof (continued).   y 1     � � �   y T � � � ( r ) A   B y r x T A �   ( r ) A � x = y 1 · · · y r 0 · · · 0 = � � y ,   0 C D       0 y T � � ( r ) A y > 0 . Then ( r ) A is positive defjnite again by the and therefore � � �

The expression is called the Cholesky factorization of . Cholesky factorization Theorem Let A be an n × n symmetric matrix. Then the following conditions are equivalent. 1. A is positive definite. 2. det ( ( r ) A ) > 0 for r = 1 , 2 , . . . , n. 3. A = U T U where U is upper triangular and has positive entries on its main diagonal. Furthermore, U is unique.

Cholesky factorization Theorem Let A be an n × n symmetric matrix. Then the following conditions are equivalent. 1. A is positive definite. 2. det ( ( r ) A ) > 0 for r = 1 , 2 , . . . , n. 3. A = U T U where U is upper triangular and has positive entries on its main diagonal. Furthermore, U is unique. The expression A = U T U is called the Cholesky factorization of A .

be obtained as follows. Using only type 3 elementary row operations, with multiples of rows added to lower rows, put in upper triangular form. Call this matrix ; then has positive entries on its main diagonal (this can be proved by induction on ). Obtain from by dividing each row of by the square root of the diagonal entry in that row. Algorithm for Cholesky Factorization Let A be a positive defjnite matrix. The Cholesky factorization A = U T U can

be obtained as follows. Obtain from by dividing each row of by the square root of the diagonal entry in that row. Algorithm for Cholesky Factorization Let A be a positive defjnite matrix. The Cholesky factorization A = U T U can 1. Using only type 3 elementary row operations, with multiples of rows added to lower rows, put A in upper triangular form. Call this matrix � U ; then � U has positive entries on its main diagonal (this can be proved by induction on n ).

diagonal entry in that row. be obtained as follows. Algorithm for Cholesky Factorization Let A be a positive defjnite matrix. The Cholesky factorization A = U T U can 1. Using only type 3 elementary row operations, with multiples of rows added to lower rows, put A in upper triangular form. Call this matrix � U ; then � U has positive entries on its main diagonal (this can be proved by induction on n ). 2. Obtain U from � U by dividing each row of � U by the square root of the

and and that row, to give Now divide the entries in each row by the square root of the diagonal entry in positive defjnite. is , it follows that . Since det and det so det Problem   9 − 6 3   is positive definite, and find the Show that A = − 6 5 − 3 3 − 3 6 Cholesky factorization of A.

and and that row, to give Now divide the entries in each row by the square root of the diagonal entry in positive defjnite. Problem   9 − 6 3   is positive definite, and find the Show that A = − 6 5 − 3 3 − 3 6 Cholesky factorization of A. Solution � � � � 9 − 6 (1) A = (2) A = 9 , − 6 5 so det ( (1) A ) = 9 and det ( (2) A ) = 9 . Since det ( A ) = 36 , it follows that A is