What kind of tensors are compressible? Tianyi Shi Cornell - - PowerPoint PPT Presentation

What kind of tensors are compressible?

Tianyi Shi

Cornell University ts777@cornell.edu

July 19, 2019 Work with: Alex Townsend (Cornell University)

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 1 / 14

Tensor decomposition CP [Hitchcock, 1927; Cattell, 1944; Carroll & Chang, 1970; Harshman, 1970] Tucker [Tucker, 1963] Tensor-train [Oseledets, 11] · · · Methodologies to understand the compressibility of tensors:

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 2 / 14

Tensor decomposition CP [Hitchcock, 1927; Cattell, 1944; Carroll & Chang, 1970; Harshman, 1970] Tucker [Tucker, 1963] Tensor-train [Oseledets, 11] · · · Methodologies to understand the compressibility of tensors: Algebraic structures: Xi,j,k = f (xi, yj, zk)

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 2 / 14

Tensor decomposition CP [Hitchcock, 1927; Cattell, 1944; Carroll & Chang, 1970; Harshman, 1970] Tucker [Tucker, 1963] Tensor-train [Oseledets, 11] · · · Methodologies to understand the compressibility of tensors: Algebraic structures: Xi,j,k = f (xi, yj, zk) Smoothness: f (x, y, z) ≈ pn(x, y, z)

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 2 / 14

Tensor decomposition CP [Hitchcock, 1927; Cattell, 1944; Carroll & Chang, 1970; Harshman, 1970] Tucker [Tucker, 1963] Tensor-train [Oseledets, 11] · · · Methodologies to understand the compressibility of tensors: Algebraic structures: Xi,j,k = f (xi, yj, zk) Smoothness: f (x, y, z) ≈ pn(x, y, z) Displacement structure · · ·

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 2 / 14

Rank bound of tensors with displacement structure

Theorem (S. & Townsend, 19) Suppose X ×1 A(1) + X ×2 A(2) + X ×3 A(3) = G, where A(1), A(2), A(3) are Minkowski sum separated with disjoint sets Ej and Fj for j = 1, 2. Then, for a fixed 0 < ǫ < 1, we have (rankTT

ǫ

(X))j ≤ kjνj, νj = rank(Gj), j = 1, 2, where Gj is the jth unfolding of G and kj is an integer so that Zkj(Ej, Fj) ≤ ǫ/ √ 3.

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 3 / 14

Tensor-train decomposition

Xi1,i2,i3 = G1(i1) 1×s1 G2(i2) s1×s2 G3(i3) s2×1 rankTT(X) = (1, s1, s2, 1). Storage:

3

• k=1

sk−1sknk. Bound: sk ≤ rank(Xk), (s s sǫ)k ≤ rankǫ(Xk), Xk = reshape(X,

k

• s=1

ns,

3

• s=k+1

ns).

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 4 / 14

Zolotarev number [Zolotarev, 1877]

Zk(E, F) := inf

r∈Rk,k

supz∈E |r(z)| infz∈F |r(z)| , k ≥ 0, E and F are disjoint complex sets and Rk,k is the set of irreducible rational functions of the form p(x)/q(x) with polynomials p and q of degree at most k.

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 5 / 14

Zolotarev number [Zolotarev, 1877]

Zk(E, F) := inf

r∈Rk,k

supz∈E |r(z)| infz∈F |r(z)| , k ≥ 0, E and F are disjoint complex sets and Rk,k is the set of irreducible rational functions of the form p(x)/q(x) with polynomials p and q of degree at most k. Re Im E1 F1

Zolotarev number [Zolotarev, 1877]

Zk(E, F) := inf

r∈Rk,k

supz∈E |r(z)| infz∈F |r(z)| , k ≥ 0, E and F are disjoint complex sets and Rk,k is the set of irreducible rational functions of the form p(x)/q(x) with polynomials p and q of degree at most k. Re Im E1 F1 E2 F2

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 5 / 14

Minkowski sum separation

Minkowski sum separated matrices For normal matrices A(1), A(2), A(3), and disjoint sets Ej and Fj, Λ(A(1)) ⊆ E1, −(Λ(A(2)) + Λ(A(3))) ⊆ F1, Λ(A(1)) + Λ(A(2)) ⊆ E2, −Λ(A(3)) ⊆ F2.

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 6 / 14

Minkowski sum separated matrices

Λ(A(1)) ⊆ E1, −(Λ(A(2)) + Λ(A(3))) ⊆ F1, E1 ∩ F1 = ∅, Λ(A(1)) + Λ(A(2)) ⊆ E2, −Λ(A(3)) ⊆ F2, E2 ∩ F2 = ∅.

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 7 / 14

Minkowski sum separated matrices

Λ(A(1)) ⊆ E1, −(Λ(A(2)) + Λ(A(3))) ⊆ F1, E1 ∩ F1 = ∅, Λ(A(1)) + Λ(A(2)) ⊆ E2, −Λ(A(3)) ⊆ F2, E2 ∩ F2 = ∅. Re Im Λ

• A(1)

× × × × ×

Λ

• A(2)

× × × × ×

Λ

• A(3)

× × × × ×

Minkowski sum separated matrices

Λ(A(1)) ⊆ E1, −(Λ(A(2)) + Λ(A(3))) ⊆ F1, E1 ∩ F1 = ∅, Λ(A(1)) + Λ(A(2)) ⊆ E2, −Λ(A(3)) ⊆ F2, E2 ∩ F2 = ∅. Re Im Λ

• A(1)

× × × × ×

Λ

• A(2)

× × × × ×

Λ

• A(3)

× × × × ×

E1

× × × × ×

Minkowski sum separated matrices

Λ(A(1)) ⊆ E1, −(Λ(A(2)) + Λ(A(3))) ⊆ F1, E1 ∩ F1 = ∅, Λ(A(1)) + Λ(A(2)) ⊆ E2, −Λ(A(3)) ⊆ F2, E2 ∩ F2 = ∅. Re Im Λ

• A(1)

× × × × ×

Λ

• A(2)

× × × × ×

Λ

• A(3)

× × × × ×

E1

× × × × ×

F1

Minkowski sum separated matrices

Λ(A(1)) ⊆ E1, −(Λ(A(2)) + Λ(A(3))) ⊆ F1, E1 ∩ F1 = ∅, Λ(A(1)) + Λ(A(2)) ⊆ E2, −Λ(A(3)) ⊆ F2, E2 ∩ F2 = ∅. Re Im Λ

• A(1)

× × × × ×

Λ

• A(2)

× × × × ×

Λ

• A(3)

× × × × ×

E1

× × × × ×

F1 E2 Λ

• A(2)

× × × × ×

Λ

• A(3)

× × × × ×

Minkowski sum separated matrices

Λ(A(1)) ⊆ E1, −(Λ(A(2)) + Λ(A(3))) ⊆ F1, E1 ∩ F1 = ∅, Λ(A(1)) + Λ(A(2)) ⊆ E2, −Λ(A(3)) ⊆ F2, E2 ∩ F2 = ∅. Re Im Λ

• A(1)

× × × × ×

Λ

• A(2)

× × × × ×

Λ

• A(3)

× × × × ×

E1

× × × × ×

F1 E2 Λ

• A(2)

× × × × ×

Λ

• A(3)

× × × × ×

F2

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 7 / 14

Rank bound of tensors with displacement structure (ctd.)

Theorem (S. & Townsend, 19) Suppose X ×1 A(1) + X ×2 A(2) + X ×3 A(3) = G, where A(1), A(2), A(3) are Minkowski sum separated with disjoint sets Ej and Fj for j = 1, 2. Then, for a fixed 0 < ǫ < 1, we have (rankTT

ǫ

(X))j ≤ kjνj, νj = rank(Gj), j = 1, 2, where Gj is the jth unfolding of G and kj is an integer so that Zkj(Ej, Fj) ≤ ǫ/ √ 3.

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 8 / 14

Rank bound of tensors with displacement structure (ctd.)

Theorem (S. & Townsend, 19) Suppose X ×1 A(1) + X ×2 A(2) + X ×3 A(3) = G, where A(1), A(2), A(3) are Minkowski sum separated with disjoint sets Ej and Fj for j = 1, 2. Then, for a fixed 0 < ǫ < 1, we have (rankTT

ǫ

(X))j ≤ kjνj, νj = rank(Gj), j = 1, 2, where Gj is the jth unfolding of G and kj is an integer so that Zkj(Ej, Fj) ≤ ǫ/ √ 3. Special case If Λ(A(j)) ⊆ [a, b] for 0 < a < b < ∞, and γj = (3a+j(b−a))(3b−j(b−a))

9ab

, then (rankTT

ǫ

(X))j ≤ kjνj, kj =

• log(16γj) log(4

√ 3/ǫ) π2

• .

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 8 / 14

Solving 3D Poisson equation

−(uxx + uyy + uzz) = f on Ω = [−1, 1]3, u|∂Ω = 0. Ultraspherical spectral discretization [Fortunato & Townsend, 17]: u = (1 − x2)(1 − y 2)(1 − z2)

m

• i=0

n

• j=0

p

• k=0

Xi,j,k ˜ C (3/2)

i

(x) ˜ C (3/2)

j

(y) ˜ C (3/2)

k

(z), f =

m

• i=0

n

• j=0

p

• k=0

Fi,j,k ˜ C (3/2)

i

(x) ˜ C (3/2)

j

(y) ˜ C (3/2)

k

(z),

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 9 / 14

Solving 3D Poisson equation

−(uxx + uyy + uzz) = f on Ω = [−1, 1]3, u|∂Ω = 0. Ultraspherical spectral discretization [Fortunato & Townsend, 17]: u = (1 − x2)(1 − y 2)(1 − z2)

m

• i=0

n

• j=0

p

• k=0

Xi,j,k ˜ C (3/2)

i

(x) ˜ C (3/2)

j

(y) ˜ C (3/2)

k

(z), f =

m

• i=0

n

• j=0

p

• k=0

Fi,j,k ˜ C (3/2)

i

(x) ˜ C (3/2)

j

(y) ˜ C (3/2)

k

(z), X ×1 A−1 + X ×2 A−1 + X ×3 A−1 = G, Λ(A) ⊆ [−1, −1/(30n4)], G = F ×1 M−1 ×2 M−1 ×3 M−1,

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 9 / 14

Solving 3D Poisson equation

−(uxx + uyy + uzz) = f on Ω = [−1, 1]3, u|∂Ω = 0. Ultraspherical spectral discretization [Fortunato & Townsend, 17]: u = (1 − x2)(1 − y 2)(1 − z2)

m

• i=0

n

• j=0

p

• k=0

Xi,j,k ˜ C (3/2)

i

(x) ˜ C (3/2)

j

(y) ˜ C (3/2)

k

(z), f =

m

• i=0

n

• j=0

p

• k=0

Fi,j,k ˜ C (3/2)

i

(x) ˜ C (3/2)

j

(y) ˜ C (3/2)

k

(z), X ×1 A−1 + X ×2 A−1 + X ×3 A−1 = G, Λ(A) ⊆ [−1, −1/(30n4)], G = F ×1 M−1 ×2 M−1 ×3 M−1, (rankTT

ǫ

(X))j ≤ sj, sj = O(νj log(n) log(1/ǫ)).

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 9 / 14

Solving 3D Poisson equation

When f = 1, TT-rank Accuracy n = 1 n = 100 n = 500 3D Poisson solver Constructive bound proof ADI-based algorithm Solve in TT format if unfoldings of F is low rank

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 10 / 14

Alternating Direction Implicit (ADI) method and factored ADI

Solving matrix Sylvester equation: AX − XB = F, A ∈ Cm×m, B ∈ Cn×n, ADI [Wachspress, 08] Find vectors of shift parameters p p p, and q q q. Solve (A − qiI)Xi+1/2 = F + Xi(B − qiI). Solve Xi+1(B − piI) = (A − piI)Xi+1/2 − F.

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 11 / 14

Alternating Direction Implicit (ADI) method and factored ADI

Solving matrix Sylvester equation: AX − XB = F, A ∈ Cm×m, B ∈ Cn×n, ADI [Wachspress, 08] Find vectors of shift parameters p p p, and q q q. Solve (A − qiI)Xi+1/2 = F + Xi(B − qiI). Solve Xi+1(B − piI) = (A − piI)Xi+1/2 − F. f-ADI [Benner, Li, & Truhar, 09] If F = MN∗, M ∈ Cm×ν, N ∈ Cn×ν, and ν smaller than m and n, solve for X = ZDY ∗.

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 11 / 14

TT-SVD and f-ADI based Poisson solver (rough sketch)

TT-SVD is an algorithm that calculates the TT decomposition of a given tensor ”train” by ”train”:

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 12 / 14

TT-SVD and f-ADI based Poisson solver (rough sketch)

TT-SVD is an algorithm that calculates the TT decomposition of a given tensor ”train” by ”train”: Compute SVD of X1 = U1D1V ∗

1 and use U1 as first ”train”.

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 12 / 14

TT-SVD and f-ADI based Poisson solver (rough sketch)

TT-SVD is an algorithm that calculates the TT decomposition of a given tensor ”train” by ”train”: Compute SVD of X1 = U1D1V ∗

1 and use U1 as first ”train”.

Use f-ADI to solve only for orthogonal row space U1 of X1, which satisfies A−1X1 + X1(I ⊗ A−1 + A−1 ⊗ I)T = G1 = M1N∗

1 .

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 12 / 14

TT-SVD and f-ADI based Poisson solver (rough sketch)

TT-SVD is an algorithm that calculates the TT decomposition of a given tensor ”train” by ”train”: Compute SVD of X1 = U1D1V ∗

1 and use U1 as first ”train”.

Use f-ADI to solve only for orthogonal row space U1 of X1, which satisfies A−1X1 + X1(I ⊗ A−1 + A−1 ⊗ I)T = G1 = M1N∗

1 .

Compute SVD of a reshaping of D1V ∗

1 to get second and third ”train”.

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 12 / 14

TT-SVD and f-ADI based Poisson solver (rough sketch)

TT-SVD is an algorithm that calculates the TT decomposition of a given tensor ”train” by ”train”: Compute SVD of X1 = U1D1V ∗

1 and use U1 as first ”train”.

Use f-ADI to solve only for orthogonal row space U1 of X1, which satisfies A−1X1 + X1(I ⊗ A−1 + A−1 ⊗ I)T = G1 = M1N∗

1 .

Compute SVD of a reshaping of D1V ∗

1 to get second and third ”train”.

Use f-ADI to solve for both orthogonal row and column spaces of C2, which satisfies (I ⊗ (U∗

1 A−1U1) + A−1 ⊗ I)C2 + C2(A−1)T = (I ⊗ U∗ 1 )G2 = (I ⊗ U∗ 1 )M2N∗ 2 .

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 12 / 14

f = −2(1 − y 2)(1 − z2) − 2(1 − x2)(1 − z2) − 2(1 − x2)(1 − y 2). n Time Naive Eigen Eigen & fADI fADI Overall complexity Converting between Chebyshev and ˜ C (3/2) takes O(n2(log n)2 log(1/ǫ)) complexity [Townsend, Webb, & Olver, 17]. Solving for trains using fADI and ADI takes O(n2(log n)2(log(1/ǫ))2) complexity.

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 13 / 14

Summary Several methodologies guarantee compressibility of certain tensors in various formats. Super fast spectrally accurate Poisson equation solver. Ongoing work Make the fast Poisson solver open-source codes. What about

d

• j=1

AjXBT

j

= F,

d

• j=1

X ×1 Aj ×2 Bj ×3 Cj = F.

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 14 / 14

Bonus: Hilbert tensor

Hi,j,k = 1 i + j + k − 2, 1 ≤ i, j, k ≤ n. H ×1 D + H ×2 D + H ×3 D = S, S is the tensor of all ones and D is a diagonal matrix with Dii = i − 2

3.

rankTT

ǫ

(H) ≤lex (1, s1, s1, 1), s1 =

• 1

π2 log 16n(2n − 1) 3n − 2

• log
• 4

√ 3 ǫ

• .

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 14 / 14

Bonus: Solving 3D Poisson in Orthogonal Tucker format

Tucker decomposition [Tucker, 1963] X = A G B C HOSVD [De Lathauwer, De Moor, & Vanderwalle, 00] and fADI-based Poisson solver Use fADI to solve for A, B, and C as orthogonal row spaces of matricizations

• f X.

Solve a smaller tensor Sylvester equation of G.

Tianyi Shi (Cornell) Compressible tensors July 19, 2019 14 / 14