What kind of tensors are compressible? Tianyi Shi Cornell - PowerPoint PPT Presentation

What kind of tensors are compressible? Tianyi Shi Cornell University ts777@cornell.edu June 28, 2019 Work with: Alex Townsend (Cornell University) Tianyi Shi (Cornell) Compressible tensors June 28, 2019 1 / 17

What is a tensor? X Why are low rank tensors important? Explicit storage (3D): 3 � n i . i =1 Tianyi Shi (Cornell) Compressible tensors June 28, 2019 2 / 17

What is a tensor? X Why are low rank tensors important? Explicit storage (3D): 3 � n i . i =1 Methodologies to understand the compressibility of tensors: Algebraic structures: X i , j , k = f ( x i , y j , z k ) Tianyi Shi (Cornell) Compressible tensors June 28, 2019 2 / 17

What is a tensor? X Why are low rank tensors important? Explicit storage (3D): 3 � n i . i =1 Methodologies to understand the compressibility of tensors: Algebraic structures: X i , j , k = f ( x i , y j , z k ) Smoothness: f ( x , y , z ) ≈ p n ( x , y , z ) Tianyi Shi (Cornell) Compressible tensors June 28, 2019 2 / 17

What is a tensor? X Why are low rank tensors important? Explicit storage (3D): 3 � n i . i =1 Methodologies to understand the compressibility of tensors: Algebraic structures: X i , j , k = f ( x i , y j , z k ) Smoothness: f ( x , y , z ) ≈ p n ( x , y , z ) Displacement structure Tianyi Shi (Cornell) Compressible tensors June 28, 2019 2 / 17

Tensor decompositions CP decomposition [Kolda & Bader, 09] w 1 w R z 1 z R + . . . + = X y 1 y R Tianyi Shi (Cornell) Compressible tensors June 28, 2019 3 / 17

Tensor decompositions Tucker decomposition [Tucker, 1963] C B G = X A Tianyi Shi (Cornell) Compressible tensors June 28, 2019 4 / 17

Tensor decompositions Tucker decomposition [Tucker, 1963] C B G = X A Tensor-train (TT) decomposition Tianyi Shi (Cornell) Compressible tensors June 28, 2019 4 / 17

Example: Hilbert tensor 1 H i , j , k = i + j + k − 2 , 1 ≤ i , j , k ≤ n . TT-rank Accuracy n TT-rank E.g. n = 100 , ǫ = 10 − 10 , instead of 100 3 , in tensor-train: 25500. Tianyi Shi (Cornell) Compressible tensors June 28, 2019 5 / 17

Tensor-train decomposition [Oseledets, 11] 1 × s 1 s 1 × s 2 s 2 × 1 G 1 ( i 1 ) X i 1 , i 2 , i 3 = G 3 ( i 3 ) G 2 ( i 2 ) rank TT ( X ) = (1 , s 1 , s 2 , 1) . Storage: 3 � s k − 1 s k n k . k =1 Tianyi Shi (Cornell) Compressible tensors June 28, 2019 6 / 17

Tensor-train decomposition [Oseledets, 11] 1 × s 1 s 1 × s 2 s 2 × 1 G 1 ( i 1 ) X i 1 , i 2 , i 3 = G 3 ( i 3 ) G 2 ( i 2 ) rank TT ( X ) = (1 , s 1 , s 2 , 1) . Storage: 3 � s k − 1 s k n k . k =1 Bound: 3 k � � s k ≤ rank ( X k ) , X k = reshape ( X , n s , n s ) . s =1 s = k +1 Tianyi Shi (Cornell) Compressible tensors June 28, 2019 6 / 17

Numerical tensor-train rank Numerical tensor-train rank s ǫ is the smallest vector such that rank TT ( ˜ rank TT ( X ) = s s s ǫ where s s X ) = s s s ǫ , ǫ i 1 , i 2 , i 3 ( X i 1 , i 2 , i 3 ) 2 � 1 / 2 �� �X − ˜ X� F ≤ ǫ �X� F , and ||X|| F = . Lexicographical ordering A vector x x = ( x 1 , . . . , x d ) is less than y x y y = ( y 1 , . . . , y d ), denoted by x x x < lex y y y , if in the first entry for which the vectors differ, x j < y j . In addition, x x ≤ lex y x y y if x x x < lex y y y or x j = y j for all j . Tianyi Shi (Cornell) Compressible tensors June 28, 2019 7 / 17

Displacement structure Matrix AX + XB T = G , A ∈ C m × m , B ∈ C n × n , Tianyi Shi (Cornell) Compressible tensors June 28, 2019 8 / 17

Displacement structure Matrix AX + XB T = G , A ∈ C m × m , B ∈ C n × n , 3D Tensor X × 1 A (1) + X × 2 A (2) + X × 3 A (3) = G , A ( k ) ∈ C n k × n k , The k -mode product For a tensor X ∈ C n 1 ×···× n d and a matrix A ∈ C n k × n k n k � ( X × k A ) i 1 ,..., i k − 1 , j , i k +1 ,..., i d = X i 1 ,..., i d A j , i k . i k =1 Tianyi Shi (Cornell) Compressible tensors June 28, 2019 8 / 17

Displacement structure Matrix AX + XB T = G , A ∈ C m × m , B ∈ C n × n , 3D Tensor X × 1 A (1) + X × 2 A (2) + X × 3 A (3) = G , A ( k ) ∈ C n k × n k , The k -mode product For a tensor X ∈ C n 1 ×···× n d and a matrix A ∈ C n k × n k n k � ( X × k A ) i 1 ,..., i k − 1 , j , i k +1 ,..., i d = X i 1 ,..., i d A j , i k . i k =1 Matrix (again) X × 1 A + X × 2 B = G Tianyi Shi (Cornell) Compressible tensors June 28, 2019 8 / 17

Rank bound of matrices with displacement structure If A and B are normal matrices, Λ( A ) ⊆ E and Λ( B ) ⊆ F , then AX − XB T = G , rank ( G ) = ν implies 2-norm [Beckermann & Townsend, 19] � X − X ν k � 2 ≤ Z k ( E , F ) � X � 2 . Frobenius norm [S. & Townsend] � X − X ν k � F ≤ Z k ( E , F ) � X � F . Tianyi Shi (Cornell) Compressible tensors June 28, 2019 9 / 17

Zolotarev number [Zolotarev, 1877] sup z ∈ E | r ( z ) | Z k ( E , F ) := inf inf z ∈ F | r ( z ) | , k ≥ 0 , r ∈R k , k E and F are disjoint complex sets and R k , k is the set of irreducible rational functions of the form p ( x ) / q ( x ) with polynomials p and q of degree at most k . Tianyi Shi (Cornell) Compressible tensors June 28, 2019 10 / 17

Zolotarev number [Zolotarev, 1877] sup z ∈ E | r ( z ) | Z k ( E , F ) := inf inf z ∈ F | r ( z ) | , k ≥ 0 , r ∈R k , k E and F are disjoint complex sets and R k , k is the set of irreducible rational functions of the form p ( x ) / q ( x ) with polynomials p and q of degree at most k . Im E 1 Re F 1

Zolotarev number [Zolotarev, 1877] sup z ∈ E | r ( z ) | Z k ( E , F ) := inf inf z ∈ F | r ( z ) | , k ≥ 0 , r ∈R k , k E and F are disjoint complex sets and R k , k is the set of irreducible rational functions of the form p ( x ) / q ( x ) with polynomials p and q of degree at most k . Im E 1 F 2 E 2 Re F 1 Tianyi Shi (Cornell) Compressible tensors June 28, 2019 10 / 17

Rank bound of tensors with displacement structure Minkowski sum separated For normal matrices A (1) , A (2) , A (3) , and disjoint sets E j and F j , Λ( A (1) ) ⊆ E 1 , − (Λ( A (2) ) + Λ( A (3) )) ⊆ F 1 , Λ( A (1) ) + Λ( A (2) ) ⊆ E 2 , − Λ( A (3) ) ⊆ F 2 . Tianyi Shi (Cornell) Compressible tensors June 28, 2019 11 / 17

Minkowski sum separated Λ( A (1) ) ⊆ E 1 , − (Λ( A (2) ) + Λ( A (3) )) ⊆ F 1 , E 1 ∩ F 1 = ∅ , Λ( A (1) ) + Λ( A (2) ) ⊆ E 2 , − Λ( A (3) ) ⊆ F 2 , E 2 ∩ F 2 = ∅ . Tianyi Shi (Cornell) Compressible tensors June 28, 2019 12 / 17

Minkowski sum separated Λ( A (1) ) ⊆ E 1 , − (Λ( A (2) ) + Λ( A (3) )) ⊆ F 1 , E 1 ∩ F 1 = ∅ , Λ( A (1) ) + Λ( A (2) ) ⊆ E 2 , − Λ( A (3) ) ⊆ F 2 , E 2 ∩ F 2 = ∅ . � A (3) � Λ Im × × × × A (1) � � Λ × × × × × × × × × × × � A (2) � Λ Re

Minkowski sum separated Λ( A (1) ) ⊆ E 1 , − (Λ( A (2) ) + Λ( A (3) )) ⊆ F 1 , E 1 ∩ F 1 = ∅ , Λ( A (1) ) + Λ( A (2) ) ⊆ E 2 , − Λ( A (3) ) ⊆ F 2 , E 2 ∩ F 2 = ∅ . � A (3) � Λ Im × × × × A (1) � � Λ × × × × × × × E 1 × × × × × × × × × � A (2) � Λ Re

Minkowski sum separated Λ( A (1) ) ⊆ E 1 , − (Λ( A (2) ) + Λ( A (3) )) ⊆ F 1 , E 1 ∩ F 1 = ∅ , Λ( A (1) ) + Λ( A (2) ) ⊆ E 2 , − Λ( A (3) ) ⊆ F 2 , E 2 ∩ F 2 = ∅ . � A (3) � Λ Im × × × × A (1) � � Λ × × × × × × × E 1 × × × × × × × × × � A (2) � Λ Re F 1

Minkowski sum separated Λ( A (1) ) ⊆ E 1 , − (Λ( A (2) ) + Λ( A (3) )) ⊆ F 1 , E 1 ∩ F 1 = ∅ , Λ( A (1) ) + Λ( A (2) ) ⊆ E 2 , − Λ( A (3) ) ⊆ F 2 , E 2 ∩ F 2 = ∅ . � � A (3) � A (3) � Λ Λ Im × × × × × × × × A (1) � � Λ × × E 2 × × × × × × × × E 1 × × × × × × × × × × × × � � A (2) � A (2) � Λ Λ Re F 1

Minkowski sum separated Λ( A (1) ) ⊆ E 1 , − (Λ( A (2) ) + Λ( A (3) )) ⊆ F 1 , E 1 ∩ F 1 = ∅ , Λ( A (1) ) + Λ( A (2) ) ⊆ E 2 , − Λ( A (3) ) ⊆ F 2 , E 2 ∩ F 2 = ∅ . � � A (3) � A (3) � Λ Λ Im × × × × × × × × A (1) � � Λ × × E 2 × × × × × × × × E 1 × × × × × × × × × × × × � � A (2) � A (2) � Λ Λ Re F 2 F 1 Tianyi Shi (Cornell) Compressible tensors June 28, 2019 12 / 17

Rank bound of tensors with displacement structure (ctd.) Theorem (S. & Townsend) Suppose X × 1 A (1) + X × 2 A (2) + X × 3 A (3) = G , where A (1) , A (2) , A (3) are Minkowski sum separated with disjoint sets E j and F j for j = 1 , 2 . Then, for a fixed 0 < ǫ < 1 , we have rank TT ( X ) ≤ lex (1 , k 1 ν 1 , k 2 ν 2 , 1) , ν j = rank ( G j ) , j = 1 , 2 , ǫ √ where G j is the jth unfolding of G and k j is an integer so that Z k j ( E j , F j ) ≤ ǫ/ 3 . Tianyi Shi (Cornell) Compressible tensors June 28, 2019 13 / 17