Latent Factor Analysis of Gaussian Distributions under Graphical - - PowerPoint PPT Presentation

Latent Factor Analysis of Gaussian Distributions under Graphical Constraints

Md Mahmudul Hasan, Shuangqing Wei, Ali Moharrer

School of Electrical Engineering and Computer Science, Louisiana State University

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 1 / 23

Outline

1

Motivation

2

Introduction

3

Main results

4

Building a Gaussian Tree

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 2 / 23

Motivation

Outline

1

Motivation

2

Introduction

3

Main results

4

Building a Gaussian Tree

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 3 / 23

Motivation

Motivation

• Here the primary aim is to do dimension reduction using rank of a matrix as the
• bject function, which is hard to achieve. Hence we use trace as the object function,

which is effectively almost as good as rank minimization.

• Unlike traditional ways to do factor analysis i.e. numerically, providing algorithms or

providing convergence proof, we are trying to do factor analysis under graphical constrainsts.

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 4 / 23

Introduction

Outline

1

Motivation

2

Introduction

3

Main results

4

Building a Gaussian Tree

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 5 / 23

Introduction

Definitions

The well-known factor analytic decomposition of an nxn population covariance matrix Σ, Σ = (Σ − D) + D (1)

• MTFA- matrix Σ − D is Gramian and matrix D is diagonal
• CMTFA- matrix Σ − D and matrix D is both are Gramian.

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 6 / 23

Introduction

A brief on CMTFA

CMTFA seeks a minimum trace Σt that solves the following decomposition problem. Σx = Σt + D (2) such that, Σt is low rank (rank < n ), D is diagonal, both Σt and D are Gramian matrices. It was shown in [1] that, the above decomposition problem is equivalent to solving the following convex optimization problem. min

D

−tr(D) s.t. − λmin(D) ≤ 0 and − Di,i ≤ 0, i = 1, . . . , n (3) where, λmin(D) is the minimum eigenvalue of the matrix D.

1Giacomo Della Riccia and Alexander Shapiro, ”Minimum Rank and Minimum Trace of Covariance

Matrices”, Psychometrika, vol. 47, No. 4, December, 1982.

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 7 / 23

Introduction

A brief on CMTFA (cont’d)

The following Theorem given in the same paper, sets the ground rules for a matrix D∗ to be the solution for the optimization problem given by (3). Theorem The matrix D∗ is a solution of the CMTFA problem if and only if D∗

i,i ≥ 0, 1 ≤ i ≤ n,

λmin(Σx − D∗) = 0, and there exists n × r matrix T such that

• t∗,i ∈ N(Σx − D∗),

i = 1, ...., r and the following holds,

• 1 =

r

• i=1
• t2

∗,i −

• j∈I(D∗)

µj ξj (4) where r ≤ n indicating the number of columns of the matrix T, I(D∗) = {i : D∗

i,i = 0, 1 ≤ i ≤ n}, {µj,

j ∈ I(D∗)} are non-negative numbers and { ξj, j ∈ I(D∗)} are column vectors in Rn with all the components equal to 0 except for the jth component which is equal to 1.

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 8 / 23

Introduction

Problem Statement

Our goal is to characterize the solution space of CMTFA, when CMTFA is applied to a specially generated Σx i.e. generated from the following model.    x1 . . . xn    =    α1 . . . αn   

• y
• +

   z1 . . . zn    (5) where, y ∼ N(0, 1), 0 ≤ |αj| ≤ 1, j = 1, 2, . . . , n. and {zi} are independent Gausian random varables with zi ∼ N(0, 1 − α2

i )

The above model makes the following star topology with y being the latent variable.

Figure: A star topology generative model.

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 9 / 23

Introduction

State of the Art

In [2], a necessary and sufficient condition was found on the subspace of Σx for MTFA solution of Σx to recover a star structure, when Σx is equipped with a latent star graphical constraint. The main differences between their work and ours are,

• We found the same condtion for CMTFA as they did for MTFA.
• Even more importantly, we also characterized the solution for the case when CMTFA

fails to recover a star structure.

• We characterized the solution for all possible situations.
• 2J. SAUNDERSON, V. CHANDRASEKARAN, P. A. PARRILO , AND A. S. WILLSKY, ”DIAGONAL AND

LOW-RANK MATRIX DECOMPOSITIONS, CORRELATION MATRICES, AND ELLIPSOID FITTING” SIAM

• J. MATRIX ANAL. APPL., vol. 33, no. 4, pp. 1395-1416, 2015.

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 10 / 23

Introduction

Our contributions

We define, vector α = [α1, α2, α3, ....., αn]′, and without the loss of generality we have assumed |α1| ≥ |α2| ≥ · · · ≥ |αn|. We apply CMTFA to the following Σx, aiming to characterize the solution space. Σx =    1 . . . α1αn . . . ... . . . αnα1 . . . 1    (6) Our contributions can be summarized by the following two theorems.

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 11 / 23

Main results

Outline

1

Motivation

2

Introduction

3

Main results

4

Building a Gaussian Tree

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 12 / 23

Main results

Theorem CMTFA solution of Σx is Σt,ND if and only if, α is non-dominant, i.e. |α1| ≤ n

i=2 |αi|.

Σt,ND =      α2

1

α1α2 . . . α1αn α2α1 α2

2

. . . α2αn . . . . . . ... . . . αnα1 αnα2 . . . α2

n

     (7) Theorem CMTFA solution of Σx is Σt,DM if and only if, α is dominant, i.e. |α1| > n

i=2 |αi|.

Σt,DM =      |α1|(n

i=2 |αi|)

α1α2 . . . α1αn α2α1 |α2|(|α1| −

i=1,2 |αi|)

. . . α2αn . . . ... . . . . . . αnα1 α1α2 . . . |αn|(|α1| −

i=1,n |αi|)

     (8)

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 13 / 23

Main results

Proof of the non-dominant case

The following Lemma will help us prove our first theorem. Lemma Non-dominance of vector α given by (9) is a necessary condition for the existence of such n × r matrix T that t∗,i ∈ N(Σt,ND), 1 ≤ i ≤ r and || tj,∗||2 = 1, 1 ≤ j ≤ n. |α1| ≤

n

• i=2

|αi| (9)

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 14 / 23

Main results

Proof of the non-dominant case (cont’d)

Referring to the necessary and sufficient condtion for CMTFA solution.

• Σt,ND is rank 1, its minimum eigenvalue is 0.
• We only need to show the existance of such n × r matrix T that
• t∗,i ∈ N(Σt,ND),

1 ≤ i ≤ r and || tj,∗||2 = 1, 1 ≤ j ≤ n where 1 ≤ r ≤ n.

• the Lemma has already stated that, for the existence of such T non-dominance of

α is a necessary condition.

• We next show that for the existence of such a T, non-dominance of

α is also a sufficient condition.

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 15 / 23

Main results

Proof of the non-dominant case (cont’d)

It is straightforward to find the following basis vectors for the null space of Σt,ND,

• v1 =

       − α2

α1

1 . . .        , v2 =        − α3

α1

1 . . .        , . . . ,

• vn−1 =

       − αn

α1

. . . 1        (10) We define, V = [ v1, . . . vn−1, n−1

i=1 ci+1

vi], where ci ∈ {1, −1}, i = 2, . . . , n . Let, Tn×n = Vn×n · Bn×n, where Bn×n is a diagonal matrix. We have, TT T = V BBT V T = V βV T (11) The diagonal matrix β = BBT can have only non-negative entries.

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 16 / 23

Main results

Proof of non-dominant case (cont’d)

Because of || tj,∗||2 = 1, 1 ≤ j ≤ n condition on T, we get the following n equations,

n−1

• i=1

α2

i+1

α2

1

βii + βnn n

• j=2

cj αj α1 2 = 1 (12) βii + c2

i+1βnn = 1,

i = 1, . . . , n − 1 (13) If we select ciαi = |αi|, i = 2, . . . , n and solve the above equations under such selections, we get βii ≥ 0, 1 ≤ i ≤ n. That completes the proof of our first Theorem .

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 17 / 23

Main results

Proof of the dominant case

The following two Lemmas play pivotal role in proving the second theorem we proposed. Lemma Σt,DM is a rank n − 1 matrix. Lemma There exists a column vector Φ = [Φ1, Φ2, ...., Φn]′ such that Σt,DMΦ = 0, where Φi ∈ {−1, 1}, 1 ≤ i ≤ n.

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 18 / 23

Main results

Dominant case proof (cont’d)

Proof. Again we refer to the necessary and sufficient condition for CMTFA solution.

• The first Lemma proves that the rank of Σt,DM is n − 1, so its minimum eigenvalue

is λmin(Σt,DM) = 0.

• Since 0 < |αi| < 1 and 0 < (Σt,DM)ii) < 1, i = 1, . . . , n, all the diagonal entries

D∗

i,i, 1 ≤ i ≤ n of the matrix D∗ are positive. As a result, the set I(D∗) is empty

and the second term in the right hand side of (4) vanishes. The dimension of the null space of Σt,DM is 1. It will suffice for us to prove the existence of a column vector Φn×1, Φi ∈ {1, −1}, 1 ≤ i ≤ n such that Σt,DMΦ = 0. The second Lemma gives that proof.

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 19 / 23

Building a Gaussian Tree

Outline

1

Motivation

2

Introduction

3

Main results

4

Building a Gaussian Tree

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 20 / 23

Building a Gaussian Tree

Building a Gaussian Tree

We are assuming the following few things

• We have a Σx generated with a latent Gaussian tree topology constrainst and these

Gaussian observable variables are further divided into m clusters.

• The ith cluster has ni observables i.e. Xi,1, · · · , Xi,ni.
• The individual edge weights of the observable variables αj are iid uniformly over

(−1, 1) or equivalently |αi| are iid uniformly over (0, 1).

• The correlation between any two observables Xi and Xj is the product of the

respective edge weights of the path connecting the two observed variables in the latent Gaussian tree [3]. The next Lemma and Theorem summarize our efforts towards building a Gaussian tree for any given set of observables under the special case of all clusters being non-dominant.

• 3A. Moharrer, Information theoretic study of Gaussian graphical models and their applications, Ph.D.

dissertation, Louisiana State University, 2014. [Online]. Available: https://digitalcommons.lsu.edu/gradschool dissertations/4092/

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 21 / 23

Building a Gaussian Tree

Building a Gaussian Tree (cont’d)

Lemma The probability that a particular cluster with n observables is non-dominant or equivalently the vector α = [α1, α2, . . . αn]′ is non-dominant, is 1 − 1

n!.

Theorem Equations (14) and (15) are respectively necessary and sufficient conditions for all of the m randomly generated clusters, equipped with a Gaussian latent tree structural interpretation, to be non-dominant with a probability at least equal to δ. 1 m

m

• i=1

ni! ≥ 1 1 − δ

1 m

(14) nmin! ≥ 1 1 − δ

1 m

(15) where nmin = min{n1, . . . , nm}.

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 22 / 23

Building a Gaussian Tree

Conclusion

We can summarize our contributions as follows.

• We proved that the CMTFA solution of Σx with a latent star interpretation is either

rank 1 or rank n − 1 and nothing in between.

• Found explicit condition under which the solution recovers a star structure.
• Also characterized the solution when the solution does not recover a star structure.

Md Mahmudul Hasan, S Wei, Ali Moharrer (LSU) ISIT 2020 June 2020 23 / 23