Developing Tensor Operations with an Underlying Group Structure Carla D. Martin (with M. Kilmer, Tufts University) James Madison University NSF Workshop: Future Directions in Tensor-Based Computation and Modeling February 21, 2009 Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 1 / 25
Tensor Decompositions (Tucker/HOSVD, PARAFAC) Let A ∈ R M × N × P M N P � � � A = σ ijk ( u i ◦ v j ◦ w k ) i =1 j =1 k =1 = Σ × 1 U × 2 V × 3 W r � A = ( u i ◦ v i ◦ w i ) i =1 Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 2 / 25
What other factorizations are possible? Develop different notions of factorizations and projections based on different tensor operations Tie factorizations to fundamental concepts in linear algebra such as group structure, invertibility, existence, uniqueness New compression strategies that may be modified for tensors with special structure Investigate computational efficiencies with regard to sparse and dense tensors Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 3 / 25
Tensor-tensor Multiplication (contracted product) Contracted product in the first-mode : A ∈ R L × M 1 × N 1 AB ∈ R M 1 × N 1 × M 2 × N 2 ⇒ B ∈ R L × M 2 × N 2 L � ( AB ) m 1 n 1 m 2 n 2 = A ℓ m 1 n 1 B ℓ m 2 n 2 ℓ =1 m 1 = 1 , . . . , M 1 n 1 = 1 , . . . , N 1 = 1 , . . . , M 2 = 1 , . . . , N 2 m 2 n 2 Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 4 / 25
Tensor-tensor Multiplication Using contracted product... Set of all third-order tensors is not closed No notion of inverse possible What happens if we create an operation that is closed under “multiplication”? Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 5 / 25
New tensor-tensor operation A ∈ R L × M × N A ∗ B ∈ R L × P × N ⇒ B ∈ R M × P × N Operation defined in terms of the tensor “slices” Circulant matrices play a role Operation is associative Can define an inverse Set of N × N × N invertible tensors form a group under this operation Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 6 / 25
New tensor-tensor operation ∗ = A ∗ B = C A 1 B 1 + A 3 B 2 + A 2 B 3 C (: , : , 1) A 1 A 3 A 2 B 1 = = A 2 A 1 A 3 B 2 A 2 B 1 + A 1 B 2 + A 3 B 3 C (: , : , 2) A 3 B 1 + A 2 B 2 + A 1 B 3 C (: , : , 3) A 3 A 2 A 1 B 3 Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 7 / 25
Computation More efficient if performed in the Fourier domain. For example, if A ∈ R L × M × 4 , B ∈ R M × P × 4 : C = A ∗ B A 1 A 4 A 3 A 2 B 1 A 2 A 1 A 4 A 3 B 2 = ( F ∗ 4 ⊗ I L )( F 4 ⊗ I L ) ( F ∗ 4 ⊗ I M )( F 4 ⊗ I M ) A 3 A 2 A 1 A 4 B 3 A 4 A 3 A 2 A 1 B 4 ˜ D 1 B 1 ˜ D 2 B 2 = ( F ∗ 4 ⊗ I L ) ˜ D 3 B 3 ˜ D 4 B 4 Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 8 / 25
Higher Order Tensor Operations (recursive) Suppose A , B ∈ R 3 x 3 x 3 x 3 ( A ij ∈ R 3 x 3 ) A 1 = A (: , : , : , 1) = B 1 = B (: , : , : , 1) = A 2 = A (: , : , : , 2) = B 2 = B (: , : , : , 2) = A 3 = A (: , : , : , 3) = B 3 = B (: , : , : , 3) = Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 9 / 25
Higher Order Tensor Operations (recursive) Then, A ∗ B : A 1 A 3 A 2 B 1 ∗ A 2 A 1 A 3 B 2 A 3 A 2 A 1 B 3 ∗ 9x9x3 9x3x3 Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 10 / 25
Higher Order Tensor Operations (recursive) A 11 A 13 A 12 A 31 A 33 A 32 A 21 A 23 A 22 B 11 A 12 A 11 A 13 A 32 A 31 A 33 A 22 A 21 A 23 B 21 A 13 A 12 A 11 A 33 A 32 A 31 A 23 A 22 A 21 B 31 A 21 A 23 A 22 A 11 A 13 A 12 A 31 A 33 A 32 B 12 A 22 A 21 A 23 A 12 A 11 A 13 A 32 A 31 A 33 B 22 A 23 A 22 A 21 A 13 A 12 A 11 A 33 A 32 A 31 B 32 A 31 A 33 A 32 A 21 A 23 A 22 A 11 A 13 A 12 B 13 A 32 A 31 A 33 A 22 A 21 A 23 A 12 A 11 A 13 B 23 A 33 A 32 A 31 A 23 A 22 A 21 A 13 A 12 A 11 B 33 Matrix multiply → Leads to a recursive algorithm Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 11 / 25
Transpose Let C ∈ R L × M × P with faces C 1 , . . . , C P ∈ R L × M . Then It follows that ( B ∗ C ) T = C T ∗ B T Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 12 / 25
Higher Order Tensor Transpose (recursive) The higher order tensor transpose follows a recursive process. Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 13 / 25
Identity The N × N × P identity tensor , I , is the tensor whose frontal face is the N × N identity matrix and whose other faces are zeros. In general, A ∗ I = I ∗ A = A Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 14 / 25
Inverse Let A ∈ R N × N × N . Then the tensor inverse of A is any tensor B ∈ R N × N × N such that A ∗ B = B ∗ A = I We denote the inverse of A as A − 1 . It follows that ( A ∗ B ) − 1 = B − 1 ∗ A − 1 Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 15 / 25
Frobenius Norm and Orthogonality Let A = ( a ijk ) ∈ R L × M × N . Then the Frobenius norm of A is � L M N � � � � � a 2 ||A|| F = � ijk i =1 j =1 k =1 Let Q ∈ R N × N × P . Q is orthogonal if Q T ∗ Q = Q ∗ Q T = I If A is a tensor, then it follows that ||Q ∗ A|| F = ||A|| F Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 16 / 25
Tensor SVD Let A ∈ R L × M × N . Then A can be factored as A = U ∗ S ∗ V T where U ∈ R L × L × N and V ∈ R M × M × N are orthogonal tensors and S ∈ R L × M × N has diagonal matrix faces. If A ∈ R N × N × N , N � U (: , i , :) ∗ S ( i , i , :) ∗ V (: , i , :) T A = i =1 Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 17 / 25
Tensor SVD: computation Computation of the tensor SVD involves SVDs of block diagonal elements obtained from block diagonalizing the circulant matrix generated by A Using the SVDs of the blocks leads to algorithms for compression Decomposition extends recursively to order- p tensors when p > 3 Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 18 / 25
Tensor SVD: computation A 1 A 4 A 3 A 2 D 1 A 2 A 1 A 4 A 3 D 2 ( F ∗ ⊗ I ) = ( F ⊗ I ) A 3 A 2 A 1 A 4 D 3 A 4 A 3 A 2 A 1 D 4 U 1 Σ 1 V T 1 U 2 Σ 2 V T ( F ∗ ⊗ I ) 2 = ( F ⊗ I ) U 3 Σ 3 V T 3 U 4 Σ 4 V T 4 V T U 1 Σ 1 1 V T Σ 2 U 2 ( F ∗ ⊗ I ) = ( F ⊗ I ) 2 V T U 3 Σ 3 3 U 4 Σ 4 V T 4 Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 19 / 25
Tensor SVD: computation 2 3 V T 2 3 2 3 Σ 1 U 1 1 6 7 6 7 6 7 ... = ( F ∗ ⊗ I m ) ... ( F ∗ ⊗ I n ) ... ( F ∗ ⊗ I n ) 6 7 5 ( F ⊗ I m ) 5 ( F ⊗ I n ) 5 ( F ⊗ I n ) 6 7 6 7 6 7 4 4 4 V T U p Σ p p | {z } | {z } | {z } ˆ ˆ ˆ T T T ˆ ˆ ˆ ˆ ˆ ˆ V 1 V p V 2 U 1 U p U 2 S 1 S p S 2 . . . . . . . . . ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ T T T U 2 U 1 U 3 S 2 S 1 S 3 . . . . . . V 2 V 1 V 3 . . . = . . . . . . ... ... . . . ... . . . . . . . . . . . . . . . . . . ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ U p U p − 1 U 1 S p S p − 1 S 1 T T T . . . . . . V p V p − 1 V 1 . . . = U ∗ S ∗ V T Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 20 / 25
One Compression Strategy Suppose A ∈ R L × M × N Can prove that if A = U ∗ S ∗ V T then N N � � U (: , : , i ) , V (: , : , i ) are orthogonal. i =1 i =1 Therefore � N � � N � � N � T N � � � � A (: , : , i ) = U (: , : , i ) S (: , : , i ) V (: , : , i ) i =1 i =1 i =1 i =1 Carla Martin - martincd@jmu.edu (JMU) Tensor Operations with Group Structure NSF Tensor Workshop 21 / 25
Recommend
More recommend
