Canonical Correlations for Group Symmetry Models Jesse Crawford - PowerPoint PPT Presentation

Canonical Correlations for Group Symmetry Models Jesse Crawford Department of Mathematics Tarleton State University jcrawford@tarleton.edu March 28, 2009 Jesse Crawford (Tarleton State University) March 28, 2009 1 / 29

Outline Canonical Correlations 1 Group Symmetry Models 2 Generalized Canonical Correlations 3 Jesse Crawford (Tarleton State University) March 28, 2009 2 / 29

Outline Canonical Correlations 1 Group Symmetry Models 2 Generalized Canonical Correlations 3 Jesse Crawford (Tarleton State University) March 28, 2009 3 / 29

Developed by Hotelling (1936). Theory covered in Chapter 12 of Anderson (1984). Jesse Crawford (Tarleton State University) March 28, 2009 4 / 29

Setting X 1 , . . . , X N are i.i.d. normally distributed random vectors with mean zero and covariance matrix Σ ∈ PD ( I ) . R I = R J 1 ⊕ R J 2 � X ( 1 ) � n X n = X ( 2 ) n Testing problem: are X ( 1 ) and X ( 2 ) independent? n n Jesse Crawford (Tarleton State University) March 28, 2009 5 / 29

Notation Suppose x ∈ R J 1 and y ∈ R J 2 Linear combinations: x t X ( 1 ) and y t X ( 2 ) n n Cov Σ ( x , y ) := x t Σ 12 y V Σ ( x ) := x t Σ 11 x V Σ ( y ) := y t Σ 22 y Jesse Crawford (Tarleton State University) March 28, 2009 6 / 29

Definitions The first canonical correlation is c 1 = max { Cov Σ ( x , y ) x ∈ R J 1 , y ∈ R J 2 , V Σ ( x ) = V Σ ( y ) = 1 } . Suppose the maximum is attained at ( x 1 , y 1 ) . ( x 1 , y 1 ) is the first pair of canonical covariates . Jesse Crawford (Tarleton State University) March 28, 2009 7 / 29

Definitions The second canonical correlation is max { Cov Σ ( x , y ) x ∈ R J 1 , y ∈ R J 2 , V Σ ( x ) = V Σ ( y ) = 1 , c 2 = Cov Σ ( x , x 1 ) = Cov Σ ( y , y 1 ) = 0 } , Max is attained at ( x 2 , y 2 ) , the second pair of canonical covariates . . . . The k th canonical correlation is max { Cov Σ ( x , y ) x ∈ R J 1 , y ∈ R J 2 , V Σ ( x ) = V Σ ( y ) = 1 , c k = Cov Σ ( x , x i ) = Cov Σ ( y , y i ) = 0 , i = 1 , . . . , k − 1 } , Max is attained at ( x k , y k ) , the k th pair of canonical covariates . Jesse Crawford (Tarleton State University) March 28, 2009 8 / 29

Results Canonical correlations: c 1 , . . . , c J 2 Canonical covariate pairs: ( x 1 , y 1 ) , . . . , ( x J 2 , y J 2 ) . Theorem c k is the kth largest root of � � − c Σ 11 Σ 12 � � � = 0 , � � Σ 21 − c Σ 22 � and the canonical covariates satisfy � − c k Σ 11 � � x k � Σ 12 = 0 . Σ 21 − c k Σ 22 y k Jesse Crawford (Tarleton State University) March 28, 2009 9 / 29

Proof. c 1 is the maximum value of x t Σ 12 y subject to the constraints x t Σ 11 x = 1 y t Σ 22 y = 1 Apply Lagrange multipliers to prove results for c 1 and ( x 1 , y 1 ) . Complete proof by inducting on J 2 . Jesse Crawford (Tarleton State University) March 28, 2009 10 / 29

Relation to the Maximal Invariant Statistic � A 1 � 0 A = , A 1 ∈ GL ( J 1 ) , A 2 ∈ GL ( J 2 ) 0 A 2 Group Actions: ◮ A · x = Ax , for x ∈ R I × N ◮ A · Σ = A Σ A t , for Σ ∈ PD ( I ) Testing problem is invariant under these actions The family of empirical canonical correlations is a maximal invariant statistic. Jesse Crawford (Tarleton State University) March 28, 2009 11 / 29

Empirical Canonical Correlations and Eigenvalues � S 11 S 12 � ˆ Σ = S = S 21 S 22 � S 11 � 0 ˆ Σ 0 = 0 S 22 Residual � � 0 S 12 R = ˆ Σ − ˆ Σ 0 = S 21 0 Empirical canonical correlations satisfy � � − c k S 11 S 12 � � � = 0 � � S 21 − c k S 22 � | R − c k ˆ Σ 0 | = 0 Jesse Crawford (Tarleton State University) March 28, 2009 12 / 29

Empirical Canonical Correlations and Eigenvalues | R − c k ˆ Σ 0 | = 0 Empirical canonical correlations are eigenvalues of R wrt. ˆ Σ 0 . Canonical covariates satisfy � x k � ( R − c k ˆ Σ 0 ) = 0 y k � x k u k = 1 � is an eigenvector of R wrt. ˆ Σ 0 corresponding to c k y k 2 v k = 1 � � x k is an eigenvector corresponding to − c k 2 − y k Jesse Crawford (Tarleton State University) March 28, 2009 13 / 29

Empirical Canonical Covariates and Eigenvectors x k = u k + v k y k = u k − v k Jesse Crawford (Tarleton State University) March 28, 2009 14 / 29

Outline Canonical Correlations 1 Group Symmetry Models 2 Generalized Canonical Correlations 3 Jesse Crawford (Tarleton State University) March 28, 2009 15 / 29

Theory: Andersson and Madsen (1998), Appendix A Ten Fundamental Testing Problems: Andersson, Brøns, and Jensen (1983) Jesse Crawford (Tarleton State University) March 28, 2009 16 / 29

Pattern Covariance Matrices Testing covariance structure of a multivariate normal distribution. Example (Testing Complex Structure) � A � − B H 0 : Σ = B A Example (Testing Independence) � Σ 11 � 0 H 0 : Σ = 0 Σ 22 Example (Bartlett’s Test) � Γ � Σ 11 � � 0 0 H 0 : Σ = vs. H : Σ = 0 Γ 0 Σ 22 Jesse Crawford (Tarleton State University) March 28, 2009 17 / 29

Notation G ≤ O ( I ) is a compact group GL G ( I ) = { Invertible matrices that commute with G } PD G ( I ) = { Positive definite matrices that commute with G } Group Symmetry Model X 1 , X 2 , . . . , X N i.i.d. Normal ( 0 , Σ) H G : Σ ∈ PD G ( I ) Jesse Crawford (Tarleton State University) March 28, 2009 18 / 29

Example (Testing Complex Structure) � 0 � �� − 1 J G 0 = ± 1 I , ± 1 J 0 Example (Testing Independence) � 1 J 1 � �� 0 G 0 = ± 1 I , ± 0 − 1 J 2 Example (Bartlett’s Test) �� 1 J � 0 0 � 1 J �� G 0 = , 0 − 1 J 1 J 0 Jesse Crawford (Tarleton State University) March 28, 2009 19 / 29

Estimation � gSg t dg ˆ Σ = Ψ G ( S ) = G Σ = 1 ˆ � gSg t | G | g ∈ G Example (Testing Complex Structure) � S 11 + S 22 Σ 0 = 1 � S 12 − S 21 ˆ S 21 − S 12 S 11 + S 22 2 ˆ Σ ∼ generalized Wishart distribution on PD G ( I ) . Jesse Crawford (Tarleton State University) March 28, 2009 20 / 29

Testing G ≤ G 0 PD G 0 ( I ) ⊆ PD G ( I ) Testing problem: H 0 : Σ ∈ PD G 0 ( I ) vs. H : Σ ∈ PD G ( I ) Actions of A ∈ GL G ( I ) ◮ A · x = Ax , for x ∈ R I × N ◮ A · Σ = A Σ A t , for Σ ∈ PD G ( I ) Maximal invariant: eigenvalues of R = ˆ Σ − ˆ Σ 0 wrt. ˆ Σ 0 . Jesse Crawford (Tarleton State University) March 28, 2009 21 / 29

Questions Can canonical correlations be generalized? Are these maximally invariant eigenvalues canonical correlations? Jesse Crawford (Tarleton State University) March 28, 2009 22 / 29

Outline Canonical Correlations 1 Group Symmetry Models 2 Generalized Canonical Correlations 3 Jesse Crawford (Tarleton State University) March 28, 2009 23 / 29

Setting X 1 , X 2 , . . . , X N i.i.d. Normal ( 0 , Σ) Θ 0 ⊆ Θ ⊆ PD ( I ) Testing problem: H 0 : Σ ∈ Θ 0 vs. H : Σ ∈ Θ t : Θ → Θ 0 Σ 0 = t (ˆ ˆ Σ) Jesse Crawford (Tarleton State University) March 28, 2009 24 / 29

Definitions c 1 = max { Cov Σ ( x , y ) Cov t (Σ) ( x , y ) = 0 , V Σ ( x ) = V Σ ( y ) = 1 } c k = max { Cov Σ ( x , y ) Cov t (Σ) ( x , y ) = 0 , V Σ ( x ) = V Σ ( y ) = 1 ◮ for i = 1 , . . . , k − 1, ◮ Cov t (Σ) ( x , x i ) = Cov t (Σ) ( x , y i ) = 0, ◮ Cov t (Σ) ( y , x i ) = Cov t (Σ) ( y , y i ) = 0 } Jesse Crawford (Tarleton State University) March 28, 2009 25 / 29

Results Eigenvalues of Σ − t (Σ) wrt. t (Σ) : λ 1 ≥ · · · ≥ λ I λ k − λ k + 1 − k λ k + λ I + 1 − k + 2, for k = 1 , . . . , ⌊ I c k = 2 ⌋ Covariate pairs: characterized in terms of eigenvectors Proof. WLOG, t (Σ) = 1 I and Σ − t (Σ) = Λ = Diag ( λ 1 , . . . , λ I ) Apply Lagrange multipliers to c 1 = max { x t Λ y x , y ∈ R I , x t ( 1 I + Λ) x = y t ( 1 I + Λ) y = 1 , x t y = 0 } Induct on I Jesse Crawford (Tarleton State University) March 28, 2009 26 / 29

Canonical Correlations for Group Symmetry Models Θ = PD G ( I ) Θ 0 = PD G 0 ( I ) t = Ψ G Eigenvalues of ˆ Σ − ˆ Σ 0 wrt. ˆ Σ 0 λ 1 , λ 2 , . . . , − λ 2 , − λ 1 c k = λ k , for k = 1 , . . . , ⌊ I 2 ⌋ Covariate pairs: characterized in terms of eigenvectors Jesse Crawford (Tarleton State University) March 28, 2009 27 / 29

Further Research Eigenvalues related to other testing problems, such as graphical models. Unbiasedness of likelihood ratio tests for group symmetry models. Jesse Crawford (Tarleton State University) March 28, 2009 28 / 29

References Andersson, S.A., Brøns, H.K., and Tolver Jensen, S. (1983). Distribution of Eigenvalues in multivariate statistical analysis. Ann. Statist. 11 392-415. Andersson, S.A. and Madsen, J. (1998). Symmetry and lattice conditional independence in a multivariate normal distribution. Ann. Statist. 26 525-572. Hotelling, Harold (1936). Relations Between Two Sets of Variates. Biometrika 28 321-377. Jesse Crawford (Tarleton State University) March 28, 2009 29 / 29