What kind of tensors are compressible? Tianyi Shi Cornell - - PowerPoint PPT Presentation

What kind of tensors are compressible?

Tianyi Shi

Cornell University ts777@cornell.edu

June 28, 2019 Work with: Alex Townsend (Cornell University)

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 1 / 17

What is a tensor? X Why are low rank tensors important? Explicit storage (3D):

3

• i=1

ni.

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 2 / 17

What is a tensor? X Why are low rank tensors important? Explicit storage (3D):

3

• i=1

ni. Methodologies to understand the compressibility of tensors: Algebraic structures: Xi,j,k = f (xi, yj, zk)

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 2 / 17

What is a tensor? X Why are low rank tensors important? Explicit storage (3D):

3

• i=1

ni. Methodologies to understand the compressibility of tensors: Algebraic structures: Xi,j,k = f (xi, yj, zk) Smoothness: f (x, y, z) ≈ pn(x, y, z)

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 2 / 17

What is a tensor? X Why are low rank tensors important? Explicit storage (3D):

3

• i=1

ni. Methodologies to understand the compressibility of tensors: Algebraic structures: Xi,j,k = f (xi, yj, zk) Smoothness: f (x, y, z) ≈ pn(x, y, z) Displacement structure

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 2 / 17

Tensor decompositions

CP decomposition [Kolda & Bader, 09] X = y1 z1 w1 + . . . + yR zR wR

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 3 / 17

Tensor decompositions

Tucker decomposition [Tucker, 1963] X = A G B C

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 4 / 17

Tensor decompositions

Tucker decomposition [Tucker, 1963] X = A G B C Tensor-train (TT) decomposition

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 4 / 17

Example: Hilbert tensor

Hi,j,k = 1 i + j + k − 2, 1 ≤ i, j, k ≤ n. n TT-rank TT-rank Accuracy E.g. n = 100, ǫ = 10−10, instead of 1003, in tensor-train: 25500.

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 5 / 17

Tensor-train decomposition [Oseledets, 11]

Xi1,i2,i3 = G1(i1) 1×s1 G2(i2) s1×s2 G3(i3) s2×1 rankTT(X) = (1, s1, s2, 1). Storage:

3

• k=1

sk−1sknk.

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 6 / 17

Tensor-train decomposition [Oseledets, 11]

Xi1,i2,i3 = G1(i1) 1×s1 G2(i2) s1×s2 G3(i3) s2×1 rankTT(X) = (1, s1, s2, 1). Storage:

3

• k=1

sk−1sknk. Bound: sk ≤ rank(Xk), Xk = reshape(X,

k

• s=1

ns,

3

• s=k+1

ns).

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 6 / 17

Numerical tensor-train rank

Numerical tensor-train rank rankTT

ǫ

(X) = s s sǫ where s s sǫ is the smallest vector such that rankTT( ˜ X) = s s sǫ, X − ˜ XF ≤ ǫXF, and ||X||F =

• i1,i2,i3(Xi1,i2,i3)21/2

. Lexicographical ordering A vector x x x = (x1, . . . , xd) is less than y y y = (y1, . . . , yd), denoted by x x x <lex y y y, if in the first entry for which the vectors differ, xj < yj. In addition, x x x ≤lex y y y if x x x <lex y y y or xj = yj for all j.

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 7 / 17

Displacement structure

Matrix AX + XBT = G, A ∈ Cm×m, B ∈ Cn×n,

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 8 / 17

Displacement structure

Matrix AX + XBT = G, A ∈ Cm×m, B ∈ Cn×n, 3D Tensor X ×1 A(1) + X ×2 A(2) + X ×3 A(3) = G, A(k) ∈ Cnk×nk, The k-mode product For a tensor X ∈ Cn1×···×nd and a matrix A ∈ Cnk×nk (X ×k A)i1,...,ik−1,j,ik+1,...,id =

nk

• ik=1

Xi1,...,idAj,ik.

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 8 / 17

Displacement structure

Matrix AX + XBT = G, A ∈ Cm×m, B ∈ Cn×n, 3D Tensor X ×1 A(1) + X ×2 A(2) + X ×3 A(3) = G, A(k) ∈ Cnk×nk, The k-mode product For a tensor X ∈ Cn1×···×nd and a matrix A ∈ Cnk×nk (X ×k A)i1,...,ik−1,j,ik+1,...,id =

nk

• ik=1

Xi1,...,idAj,ik. Matrix (again) X ×1 A + X ×2 B = G

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 8 / 17

Rank bound of matrices with displacement structure

If A and B are normal matrices, Λ(A) ⊆ E and Λ(B) ⊆ F, then AX − XBT = G, rank(G) = ν implies 2-norm [Beckermann & Townsend, 19] X − Xνk2 ≤ Zk(E, F)X2. Frobenius norm [S. & Townsend] X − XνkF ≤ Zk(E, F)XF.

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 9 / 17

Zolotarev number [Zolotarev, 1877]

Zk(E, F) := inf

r∈Rk,k

supz∈E |r(z)| infz∈F |r(z)| , k ≥ 0, E and F are disjoint complex sets and Rk,k is the set of irreducible rational functions of the form p(x)/q(x) with polynomials p and q of degree at most k.

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 10 / 17

Zolotarev number [Zolotarev, 1877]

Zk(E, F) := inf

r∈Rk,k

supz∈E |r(z)| infz∈F |r(z)| , k ≥ 0, E and F are disjoint complex sets and Rk,k is the set of irreducible rational functions of the form p(x)/q(x) with polynomials p and q of degree at most k. Re Im E1 F1

Zolotarev number [Zolotarev, 1877]

Zk(E, F) := inf

r∈Rk,k

supz∈E |r(z)| infz∈F |r(z)| , k ≥ 0, E and F are disjoint complex sets and Rk,k is the set of irreducible rational functions of the form p(x)/q(x) with polynomials p and q of degree at most k. Re Im E1 F1 E2 F2

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 10 / 17

Rank bound of tensors with displacement structure

Minkowski sum separated For normal matrices A(1), A(2), A(3), and disjoint sets Ej and Fj, Λ(A(1)) ⊆ E1, −(Λ(A(2)) + Λ(A(3))) ⊆ F1, Λ(A(1)) + Λ(A(2)) ⊆ E2, −Λ(A(3)) ⊆ F2.

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 11 / 17

Minkowski sum separated

Λ(A(1)) ⊆ E1, −(Λ(A(2)) + Λ(A(3))) ⊆ F1, E1 ∩ F1 = ∅, Λ(A(1)) + Λ(A(2)) ⊆ E2, −Λ(A(3)) ⊆ F2, E2 ∩ F2 = ∅.

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 12 / 17

Minkowski sum separated

Λ(A(1)) ⊆ E1, −(Λ(A(2)) + Λ(A(3))) ⊆ F1, E1 ∩ F1 = ∅, Λ(A(1)) + Λ(A(2)) ⊆ E2, −Λ(A(3)) ⊆ F2, E2 ∩ F2 = ∅. Re Im Λ

• A(1)

× × × × ×

Λ

• A(2)

× × × × ×

Λ

• A(3)

× × × × ×

Minkowski sum separated

Λ(A(1)) ⊆ E1, −(Λ(A(2)) + Λ(A(3))) ⊆ F1, E1 ∩ F1 = ∅, Λ(A(1)) + Λ(A(2)) ⊆ E2, −Λ(A(3)) ⊆ F2, E2 ∩ F2 = ∅. Re Im Λ

• A(1)

× × × × ×

Λ

• A(2)

× × × × ×

Λ

• A(3)

× × × × ×

E1

× × × × ×

Minkowski sum separated

Λ(A(1)) ⊆ E1, −(Λ(A(2)) + Λ(A(3))) ⊆ F1, E1 ∩ F1 = ∅, Λ(A(1)) + Λ(A(2)) ⊆ E2, −Λ(A(3)) ⊆ F2, E2 ∩ F2 = ∅. Re Im Λ

• A(1)

× × × × ×

Λ

• A(2)

× × × × ×

Λ

• A(3)

× × × × ×

E1

× × × × ×

F1

Minkowski sum separated

Λ(A(1)) ⊆ E1, −(Λ(A(2)) + Λ(A(3))) ⊆ F1, E1 ∩ F1 = ∅, Λ(A(1)) + Λ(A(2)) ⊆ E2, −Λ(A(3)) ⊆ F2, E2 ∩ F2 = ∅. Re Im Λ

• A(1)

× × × × ×

Λ

• A(2)

× × × × ×

Λ

• A(3)

× × × × ×

E1

× × × × ×

F1 E2 Λ

• A(2)

× × × × ×

Λ

• A(3)

× × × × ×

Minkowski sum separated

Λ(A(1)) ⊆ E1, −(Λ(A(2)) + Λ(A(3))) ⊆ F1, E1 ∩ F1 = ∅, Λ(A(1)) + Λ(A(2)) ⊆ E2, −Λ(A(3)) ⊆ F2, E2 ∩ F2 = ∅. Re Im Λ

• A(1)

× × × × ×

Λ

• A(2)

× × × × ×

Λ

• A(3)

× × × × ×

E1

× × × × ×

F1 E2 Λ

• A(2)

× × × × ×

Λ

• A(3)

× × × × ×

F2

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 12 / 17

Rank bound of tensors with displacement structure (ctd.)

Theorem (S. & Townsend) Suppose X ×1 A(1) + X ×2 A(2) + X ×3 A(3) = G, where A(1), A(2), A(3) are Minkowski sum separated with disjoint sets Ej and Fj for j = 1, 2. Then, for a fixed 0 < ǫ < 1, we have rankTT

ǫ

(X) ≤lex (1, k1ν1, k2ν2, 1), νj = rank(Gj), j = 1, 2, where Gj is the jth unfolding of G and kj is an integer so that Zkj(Ej, Fj) ≤ ǫ/ √ 3.

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 13 / 17

Rank bound of tensors with displacement structure (ctd.)

Theorem (S. & Townsend) Suppose X ×1 A(1) + X ×2 A(2) + X ×3 A(3) = G, where A(1), A(2), A(3) are Minkowski sum separated with disjoint sets Ej and Fj for j = 1, 2. Then, for a fixed 0 < ǫ < 1, we have rankTT

ǫ

(X) ≤lex (1, k1ν1, k2ν2, 1), νj = rank(Gj), j = 1, 2, where Gj is the jth unfolding of G and kj is an integer so that Zkj(Ej, Fj) ≤ ǫ/ √ 3. Special case If Λ(A(j)) ⊆ [a, b] for 0 < a < b < ∞, and γj = (3a+j(b−a))(3b−j(b−a))

9ab

, then rankTT

ǫ

(X) ≤lex (1, k1ν1, k2ν2, 1), kj =

• log(16γj) log(4

√ 3/ǫ) π2

• .

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 13 / 17

Hilbert tensor revisited

Hi,j,k = 1 i + j + k − 2, 1 ≤ i, j, k ≤ n.

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 14 / 17

Hilbert tensor revisited

Hi,j,k = 1 i + j + k − 2, 1 ≤ i, j, k ≤ n. H ×1 D + H ×2 D + H ×3 D = S, S is the tensor of all ones and D is a diagonal matrix with Dii = i − 2

3.

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 14 / 17

Hilbert tensor revisited

Hi,j,k = 1 i + j + k − 2, 1 ≤ i, j, k ≤ n. H ×1 D + H ×2 D + H ×3 D = S, S is the tensor of all ones and D is a diagonal matrix with Dii = i − 2

3.

rankTT

ǫ

(H) ≤lex (1, s1, s1, 1), s1 =

• 1

π2 log 16n(2n − 1) 3n − 2

• log
• 4

√ 3 ǫ

• .

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 14 / 17

Example: Solution to 3D Poisson equation

−(uxx + uyy + uzz) = f on Ω = [−1, 1]3, u|∂Ω = 0. Xi,j,k = u(xi, yj, zk), Fi,j,k = f (xi, yj, zk),

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 15 / 17

Example: Solution to 3D Poisson equation

−(uxx + uyy + uzz) = f on Ω = [−1, 1]3, u|∂Ω = 0. Xi,j,k = u(xi, yj, zk), Fi,j,k = f (xi, yj, zk), X ×1 K + X ×2 K + X ×3 K = F, K = − 1 h2       2 −1 −1 ... ... ... ... −1 −1 2       ,

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 15 / 17

Example: Solution to 3D Poisson equation

−(uxx + uyy + uzz) = f on Ω = [−1, 1]3, u|∂Ω = 0. Xi,j,k = u(xi, yj, zk), Fi,j,k = f (xi, yj, zk), X ×1 K + X ×2 K + X ×3 K = F, K = − 1 h2       2 −1 −1 ... ... ... ... −1 −1 2       , Consider f = 1, rankTT

ǫ

(X) ≤lex (1, s1, s1, 1), s1 =

• 1

π2 log 16(n2 + 2)(2n2 + 1) 9n2

• log
• 4

√ 3 ǫ

• .

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 15 / 17

Example: Solution of 3D Poisson equation

TT-rank Accuracy n = 1 n = 100 n = 500 3D Poisson solver Constructive bound proof Solve with ultraspherical spectral methods [Fortunato & Townsend, 17] Solve in TT format super fast with complexity O(n(log n)2(log 1/ǫ)2)

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 16 / 17

Takeaways Several methodologies guarantee compressibility of tensors. Numerical TT ranks of tensors with specific displacement structure is O(log n log(1/ǫ)). 3D Poisson solver with optimal complexity and spectral accuracy in tensor-train format. Ongoing work Make the fast Poisson solver open-source codes. Ranks of matrices/tensors with more general displacement structure. Fast solve other separable PDEs (e.g. generalized Sylvester equation).

Tianyi Shi (Cornell) Compressible tensors June 28, 2019 17 / 17