What kind of tensors are compressible? Tianyi Shi Cornell - PowerPoint PPT Presentation

What kind of tensors are compressible? Tianyi Shi Cornell University ts777@cornell.edu July 19, 2019 Work with: Alex Townsend (Cornell University) Tianyi Shi (Cornell) Compressible tensors July 19, 2019 1 / 14

Tensor decomposition CP [Hitchcock, 1927; Cattell, 1944; Carroll & Chang, 1970; Harshman, 1970] Tucker [Tucker, 1963] Tensor-train [Oseledets, 11] · · · Methodologies to understand the compressibility of tensors: Tianyi Shi (Cornell) Compressible tensors July 19, 2019 2 / 14

Tensor decomposition CP [Hitchcock, 1927; Cattell, 1944; Carroll & Chang, 1970; Harshman, 1970] Tucker [Tucker, 1963] Tensor-train [Oseledets, 11] · · · Methodologies to understand the compressibility of tensors: Algebraic structures: X i , j , k = f ( x i , y j , z k ) Tianyi Shi (Cornell) Compressible tensors July 19, 2019 2 / 14

Tensor decomposition CP [Hitchcock, 1927; Cattell, 1944; Carroll & Chang, 1970; Harshman, 1970] Tucker [Tucker, 1963] Tensor-train [Oseledets, 11] · · · Methodologies to understand the compressibility of tensors: Algebraic structures: X i , j , k = f ( x i , y j , z k ) Smoothness: f ( x , y , z ) ≈ p n ( x , y , z ) Tianyi Shi (Cornell) Compressible tensors July 19, 2019 2 / 14

Tensor decomposition CP [Hitchcock, 1927; Cattell, 1944; Carroll & Chang, 1970; Harshman, 1970] Tucker [Tucker, 1963] Tensor-train [Oseledets, 11] · · · Methodologies to understand the compressibility of tensors: Algebraic structures: X i , j , k = f ( x i , y j , z k ) Smoothness: f ( x , y , z ) ≈ p n ( x , y , z ) Displacement structure · · · Tianyi Shi (Cornell) Compressible tensors July 19, 2019 2 / 14

Rank bound of tensors with displacement structure Theorem (S. & Townsend, 19) Suppose X × 1 A (1) + X × 2 A (2) + X × 3 A (3) = G , where A (1) , A (2) , A (3) are Minkowski sum separated with disjoint sets E j and F j for j = 1 , 2 . Then, for a fixed 0 < ǫ < 1 , we have ( rank TT ( X )) j ≤ k j ν j , ν j = rank ( G j ) , j = 1 , 2 , ǫ √ where G j is the jth unfolding of G and k j is an integer so that Z k j ( E j , F j ) ≤ ǫ/ 3 . Tianyi Shi (Cornell) Compressible tensors July 19, 2019 3 / 14

Tensor-train decomposition 1 × s 1 s 1 × s 2 s 2 × 1 G 1 ( i 1 ) X i 1 , i 2 , i 3 = G 3 ( i 3 ) G 2 ( i 2 ) rank TT ( X ) = (1 , s 1 , s 2 , 1) . Storage: 3 � s k − 1 s k n k . k =1 Bound: 3 k � � s k ≤ rank ( X k ) , ( s s ǫ ) k ≤ rank ǫ ( X k ) , s X k = reshape ( X , n s , n s ) . s =1 s = k +1 Tianyi Shi (Cornell) Compressible tensors July 19, 2019 4 / 14

Zolotarev number [Zolotarev, 1877] sup z ∈ E | r ( z ) | Z k ( E , F ) := inf inf z ∈ F | r ( z ) | , k ≥ 0 , r ∈R k , k E and F are disjoint complex sets and R k , k is the set of irreducible rational functions of the form p ( x ) / q ( x ) with polynomials p and q of degree at most k . Tianyi Shi (Cornell) Compressible tensors July 19, 2019 5 / 14

Zolotarev number [Zolotarev, 1877] sup z ∈ E | r ( z ) | Z k ( E , F ) := inf inf z ∈ F | r ( z ) | , k ≥ 0 , r ∈R k , k E and F are disjoint complex sets and R k , k is the set of irreducible rational functions of the form p ( x ) / q ( x ) with polynomials p and q of degree at most k . Im E 1 Re F 1

Zolotarev number [Zolotarev, 1877] sup z ∈ E | r ( z ) | Z k ( E , F ) := inf inf z ∈ F | r ( z ) | , k ≥ 0 , r ∈R k , k E and F are disjoint complex sets and R k , k is the set of irreducible rational functions of the form p ( x ) / q ( x ) with polynomials p and q of degree at most k . Im E 1 F 2 E 2 Re F 1 Tianyi Shi (Cornell) Compressible tensors July 19, 2019 5 / 14

Minkowski sum separation Minkowski sum separated matrices For normal matrices A (1) , A (2) , A (3) , and disjoint sets E j and F j , Λ( A (1) ) ⊆ E 1 , − (Λ( A (2) ) + Λ( A (3) )) ⊆ F 1 , Λ( A (1) ) + Λ( A (2) ) ⊆ E 2 , − Λ( A (3) ) ⊆ F 2 . Tianyi Shi (Cornell) Compressible tensors July 19, 2019 6 / 14

Minkowski sum separated matrices Λ( A (1) ) ⊆ E 1 , − (Λ( A (2) ) + Λ( A (3) )) ⊆ F 1 , E 1 ∩ F 1 = ∅ , Λ( A (1) ) + Λ( A (2) ) ⊆ E 2 , − Λ( A (3) ) ⊆ F 2 , E 2 ∩ F 2 = ∅ . Tianyi Shi (Cornell) Compressible tensors July 19, 2019 7 / 14

Minkowski sum separated matrices Λ( A (1) ) ⊆ E 1 , − (Λ( A (2) ) + Λ( A (3) )) ⊆ F 1 , E 1 ∩ F 1 = ∅ , Λ( A (1) ) + Λ( A (2) ) ⊆ E 2 , − Λ( A (3) ) ⊆ F 2 , E 2 ∩ F 2 = ∅ . � A (3) � Λ Im × × × × A (1) � � Λ × × × × × × × × × × × � A (2) � Λ Re

Minkowski sum separated matrices Λ( A (1) ) ⊆ E 1 , − (Λ( A (2) ) + Λ( A (3) )) ⊆ F 1 , E 1 ∩ F 1 = ∅ , Λ( A (1) ) + Λ( A (2) ) ⊆ E 2 , − Λ( A (3) ) ⊆ F 2 , E 2 ∩ F 2 = ∅ . � A (3) � Λ Im × × × × A (1) � � Λ × × × × × × × E 1 × × × × × × × × × � A (2) � Λ Re

Minkowski sum separated matrices Λ( A (1) ) ⊆ E 1 , − (Λ( A (2) ) + Λ( A (3) )) ⊆ F 1 , E 1 ∩ F 1 = ∅ , Λ( A (1) ) + Λ( A (2) ) ⊆ E 2 , − Λ( A (3) ) ⊆ F 2 , E 2 ∩ F 2 = ∅ . � A (3) � Λ Im × × × × A (1) � � Λ × × × × × × × E 1 × × × × × × × × × � A (2) � Λ Re F 1

Minkowski sum separated matrices Λ( A (1) ) ⊆ E 1 , − (Λ( A (2) ) + Λ( A (3) )) ⊆ F 1 , E 1 ∩ F 1 = ∅ , Λ( A (1) ) + Λ( A (2) ) ⊆ E 2 , − Λ( A (3) ) ⊆ F 2 , E 2 ∩ F 2 = ∅ . � � A (3) � A (3) � Λ Λ Im × × × × × × × × A (1) � � Λ × × E 2 × × × × × × × × E 1 × × × × × × × × × × × × � � A (2) � A (2) � Λ Λ Re F 1

Minkowski sum separated matrices Λ( A (1) ) ⊆ E 1 , − (Λ( A (2) ) + Λ( A (3) )) ⊆ F 1 , E 1 ∩ F 1 = ∅ , Λ( A (1) ) + Λ( A (2) ) ⊆ E 2 , − Λ( A (3) ) ⊆ F 2 , E 2 ∩ F 2 = ∅ . � � A (3) � A (3) � Λ Λ Im × × × × × × × × A (1) � � Λ × × E 2 × × × × × × × × E 1 × × × × × × × × × × × × � � A (2) � A (2) � Λ Λ Re F 2 F 1 Tianyi Shi (Cornell) Compressible tensors July 19, 2019 7 / 14

Rank bound of tensors with displacement structure (ctd.) Theorem (S. & Townsend, 19) Suppose X × 1 A (1) + X × 2 A (2) + X × 3 A (3) = G , where A (1) , A (2) , A (3) are Minkowski sum separated with disjoint sets E j and F j for j = 1 , 2 . Then, for a fixed 0 < ǫ < 1 , we have ( rank TT ( X )) j ≤ k j ν j , ν j = rank ( G j ) , j = 1 , 2 , ǫ √ where G j is the jth unfolding of G and k j is an integer so that Z k j ( E j , F j ) ≤ ǫ/ 3 . Tianyi Shi (Cornell) Compressible tensors July 19, 2019 8 / 14

Rank bound of tensors with displacement structure (ctd.) Theorem (S. & Townsend, 19) Suppose X × 1 A (1) + X × 2 A (2) + X × 3 A (3) = G , where A (1) , A (2) , A (3) are Minkowski sum separated with disjoint sets E j and F j for j = 1 , 2 . Then, for a fixed 0 < ǫ < 1 , we have ( rank TT ( X )) j ≤ k j ν j , ν j = rank ( G j ) , j = 1 , 2 , ǫ √ where G j is the jth unfolding of G and k j is an integer so that Z k j ( E j , F j ) ≤ ǫ/ 3 . Special case If Λ( A ( j ) ) ⊆ [ a , b ] for 0 < a < b < ∞ , and γ j = (3 a + j ( b − a ))(3 b − j ( b − a )) , then 9 ab √ � � log(16 γ j ) log(4 3 /ǫ ) ( rank TT ( X )) j ≤ k j ν j , k j = . ǫ π 2 Tianyi Shi (Cornell) Compressible tensors July 19, 2019 8 / 14

Solving 3D Poisson equation − ( u xx + u yy + u zz ) = f on Ω = [ − 1 , 1] 3 , u | ∂ Ω = 0 . Ultraspherical spectral discretization [Fortunato & Townsend, 17]: p m n � � � X i , j , k ˜ C (3 / 2) ( x ) ˜ C (3 / 2) ( y ) ˜ C (3 / 2) u = (1 − x 2 )(1 − y 2 )(1 − z 2 ) ( z ) , i j k i =0 j =0 k =0 p m n � � � F i , j , k ˜ C (3 / 2) ( x ) ˜ C (3 / 2) ( y ) ˜ C (3 / 2) f = ( z ) , i j k i =0 j =0 k =0 Tianyi Shi (Cornell) Compressible tensors July 19, 2019 9 / 14

Solving 3D Poisson equation − ( u xx + u yy + u zz ) = f on Ω = [ − 1 , 1] 3 , u | ∂ Ω = 0 . Ultraspherical spectral discretization [Fortunato & Townsend, 17]: p m n � � � X i , j , k ˜ C (3 / 2) ( x ) ˜ C (3 / 2) ( y ) ˜ C (3 / 2) u = (1 − x 2 )(1 − y 2 )(1 − z 2 ) ( z ) , i j k i =0 j =0 k =0 p m n � � � F i , j , k ˜ C (3 / 2) ( x ) ˜ C (3 / 2) ( y ) ˜ C (3 / 2) f = ( z ) , i j k i =0 j =0 k =0 X × 1 A − 1 + X × 2 A − 1 + X × 3 A − 1 = G , Λ( A ) ⊆ [ − 1 , − 1 / (30 n 4 )] , G = F × 1 M − 1 × 2 M − 1 × 3 M − 1 , Tianyi Shi (Cornell) Compressible tensors July 19, 2019 9 / 14

Solving 3D Poisson equation − ( u xx + u yy + u zz ) = f on Ω = [ − 1 , 1] 3 , u | ∂ Ω = 0 . Ultraspherical spectral discretization [Fortunato & Townsend, 17]: p m n � � � X i , j , k ˜ C (3 / 2) ( x ) ˜ C (3 / 2) ( y ) ˜ C (3 / 2) u = (1 − x 2 )(1 − y 2 )(1 − z 2 ) ( z ) , i j k i =0 j =0 k =0 p m n � � � F i , j , k ˜ C (3 / 2) ( x ) ˜ C (3 / 2) ( y ) ˜ C (3 / 2) f = ( z ) , i j k i =0 j =0 k =0 X × 1 A − 1 + X × 2 A − 1 + X × 3 A − 1 = G , Λ( A ) ⊆ [ − 1 , − 1 / (30 n 4 )] , G = F × 1 M − 1 × 2 M − 1 × 3 M − 1 , ( rank TT ( X )) j ≤ s j , s j = O ( ν j log( n ) log(1 /ǫ )) . ǫ Tianyi Shi (Cornell) Compressible tensors July 19, 2019 9 / 14