Parallel Numerical Algorithms Chapter 7 Differential Equations - PowerPoint PPT Presentation

Tensor Algebra Tensor Decompositions Fast Algorithms Parallel Numerical Algorithms Chapter 7 – Differential Equations Section 7.5 – Tensor Analysis Edgar Solomonik Department of Computer Science University of Illinois at Urbana-Champaign CS 554 / CSE 512 Edgar Solomonik Parallel Numerical Algorithms 1 / 26

Tensor Algebra Tensor Decompositions Fast Algorithms Outline Tensor Algebra 1 Tensors Tensor Transposition Tensor Contractions Tensor Decompositions 2 CP Decomposition Tucker Decomposition Tensor Train Decomposition Fast Algorithms 3 Strassen’s Algorithm Bilinear Algorithms Edgar Solomonik Parallel Numerical Algorithms 2 / 26

Tensor Algebra Tensors Tensor Decompositions Tensor Transposition Fast Algorithms Tensor Contractions Tensors A tensor T ∈ R n 1 ×···× n d has Order d (i.e. d modes / indices ) Dimensions n 1 -by- · · · -by- n d Elements t i 1 ...i d = t i where i ∈ � d i =1 { 1 , . . . , n i } Order d tensors represent d -dimensional arrays ( d ≥ 3) -dimensional arrays are prevalent in scientific computing Regular grids, collections of matrices, multilinear operators Experimental data, visual/graphic data Tensors analysis is the expression and study of numerical methods using tensor representations Edgar Solomonik Parallel Numerical Algorithms 3 / 26

Tensor Algebra Tensors Tensor Decompositions Tensor Transposition Fast Algorithms Tensor Contractions Reshaping Tensors When using tensors, it is often necessary to transition between high-order and low-order representations of the same object Recall for a matrix A ∈ R m × n its unfolding is given by v = vec ( A ) , ⇒ v ∈ R mn , v i + jm = a ij A tensor T ∈ R n 1 ×···× n d can be fully unfolded the same way v = vec ( T ) , ⇒ v ∈ R n 1 ··· n d , v i 1 + i 2 n 1 + i 3 n 1 n 2 + ... = t i 1 i 2 i 3 ... Often we also want to fold tensors into higher-order ones Generally, we can reshape (fold or unfold) any tensor U ∈ R n 1 ×···× n d , U = o n 1 ×···× n d ( V ) ⇒ vec ( U ) = vec ( V ) Edgar Solomonik Parallel Numerical Algorithms 4 / 26

Tensor Algebra Tensors Tensor Decompositions Tensor Transposition Fast Algorithms Tensor Contractions Tensor Transposition For tensors of order ≥ 3 , there is more than one way to transpose modes A tensor transposition is defined by a permutation p containing elements { 1 , . . . , d } Y = X � p � ⇒ y i p 1 ,...,i pd = x i 1 ,...,i d In this notation, a transposition of matrix A is defined as A T = A � [2 , 1] � Tensor transposition is a convenient primitive for manipulating multidimensional arrays and mapping tensor computations to linear algebra In tensor derivations, indices are often carried through to avoid transpositions Edgar Solomonik Parallel Numerical Algorithms 5 / 26

Tensor Algebra Tensors Tensor Decompositions Tensor Transposition Fast Algorithms Tensor Contractions Tensor Symmetry We say a tensor is symmetric if ∀ j, k ∈ { 1 , . . . , d } T = T � [1 ,...,j,...,k,...d ] � t i 1 ...i j ...i k ...i d = t i 1 ...i k ...i j ...i d or equivalently A tensor is antisymmetric (skew-symmetric) if ∀ j, k ∈ { 1 , . . . , d } t i 1 ...i j ...i k ...i d = ( − 1) t i 1 ...i k ...i j ...i d A tensor is partially-symmetric if such index interchanges are restricted to be within subsets of { 1 , . . . , d } , e.g. t ijkl = t jikl = t jilk = t ijlk Edgar Solomonik Parallel Numerical Algorithms 6 / 26

Tensor Algebra Tensors Tensor Decompositions Tensor Transposition Fast Algorithms Tensor Contractions Tensor Products and Kronecker Products Tensor products can be defined with respect to maps f : V f → W f and g : V g → W g h = f × g ⇒ g : ( V f × V g ) → ( W f × W g ) , h ( x, y ) = f ( x ) g ( y ) Tensors can be used to represent multilinear maps and have a corresponding definition for a tensor product T = X × Y ⇒ t i 1 ,...,i m ,j 1 ,...,j n = x i 1 ,...,i m y j 1 ,...,j n The Kronecker product between two matrices A ∈ R m 1 × m 2 , B ∈ R n 1 × n 2 C = A ⊗ B ⇒ c i 2 + i 1 m 2 ,j 2 + j 1 n 2 = a i 1 j 1 b i 2 j 2 corresponds to transposing and reshaping the tensor product A ⊗ B = o m 1 n 1 ,m 2 n 2 (( A × B ) � [3 , 1 , 4 , 2] � ) Edgar Solomonik Parallel Numerical Algorithms 7 / 26

Tensor Algebra Tensors Tensor Decompositions Tensor Transposition Fast Algorithms Tensor Contractions Tensor Partial Sum Y of order d − r is a partial sum of X of order d if for some q containing r elements of { 1 , . . . , d } � X = X � [ q 1 ,...,q r ,... ] � , ¯ Y = ( X ) ⇒ q � � y i 1 ...i d − r = · · · ¯ x j 1 ,...j r ,i 1 ,...i d − r j 1 j r Partial summations provide a powerful primitive operation when coupled with transposition and reshape Edgar Solomonik Parallel Numerical Algorithms 8 / 26

Tensor Algebra Tensors Tensor Decompositions Tensor Transposition Fast Algorithms Tensor Contractions Tensor Trace Z of order d − 2 r is a trace of X of order d ≥ r if for some p , q each containing a different set of r elements of { 1 , . . . , d } X = X � [ p 1 ,...,p r q 1 ,...,q r ,... ] � , ¯ Y = trace p , q ( X ) ⇒ � � y i 1 ...i d − 2 r = · · · x j 1 ,...j r ,j 1 ,...j r ,i 1 ,...i d − 2 r ¯ j 1 j r The trace of a matrix A in this notation is � trace( A ) = trace [0] , [1] ( A ) = a ii i Edgar Solomonik Parallel Numerical Algorithms 9 / 26

Tensor Algebra Tensors Tensor Decompositions Tensor Transposition Fast Algorithms Tensor Contractions Tensor Contraction Tensor contraction is a transpose of a trace of a tensor product � � r � � C = trace p , q ( A × B ) for some p , q , r Examples in linear algebra include: vector inner and outer products, matrix–vector product, matrix–matrix product The contracted modes of A appear in p and of B in q , while uncontracted modes appear in r Matrix multiplication would be given by p = [2] , q = [3] , r = [1 , 4] Edgar Solomonik Parallel Numerical Algorithms 10 / 26

Tensor Algebra Tensors Tensor Decompositions Tensor Transposition Fast Algorithms Tensor Contractions Tensor Times Matrix Tensor times matrix (TTM) is one of the most common tensor contractions involving tensors of order ≥ 3 Given an order 3 tensor T and matrix V , TTM computes order 3 tensor W , generalizes naturally to higher-order T TTM can contract one of three modes of T � � [1 , 2 , 5] � � � [1 , 3 , 5] � � � W = trace [3] , [4] ( T × V ) or W = trace [2] , [4] ( T × V ) � � [2 , 3 , 5] � � or W = trace [1] , [4] ( T × V ) In the first case, we have � w ijk = t ijl v lk l Edgar Solomonik Parallel Numerical Algorithms 11 / 26

Tensor Algebra Tensors Tensor Decompositions Tensor Transposition Fast Algorithms Tensor Contractions Tensor Contraction Diagrams Consider the tensor contraction � w ijk = t ijlm v mkl lm which we can also write in tensor notation � � [1 , 2 , 6] � � W = [3 , 4] , [7 , 5] ( T × V ) trace or in the following diagrammatic form Edgar Solomonik Parallel Numerical Algorithms 12 / 26

Tensor Algebra CP Decomposition Tensor Decompositions Tucker Decomposition Fast Algorithms Tensor Train Decomposition CP decomposition The SVD corresponds to a sum of outer products of the form r � σ k u k v T A = k k =1 so it is natural to seek to approximate a tensor as r σ k u (1) × · · · × u ( d ) � T = k k k =1 where each u ( i ) k is orthogonal to any other u ( i ) k ′ , yielding the canonical polyadic (CP) decomposition of T r is referred to as the canonical rank of the tensor Edgar Solomonik Parallel Numerical Algorithms 13 / 26

Tensor Algebra CP Decomposition Tensor Decompositions Tucker Decomposition Fast Algorithms Tensor Train Decomposition Computing the CP decomposition Computing the canonical rank is NP hard Approximation by CP decomposition is ill-posed Regularization (imposing bounds on the norm of the factor matrices) make the optimization problem feasible Alternating least squares (ALS) commonly used for computation Optimizes for one factor matrix at a time Least squares problem for each matrix Alternatives include coordinate and gradient descent methods, much like in numerical optimization for matrix completion Edgar Solomonik Parallel Numerical Algorithms 14 / 26

Tensor Algebra CP Decomposition Tensor Decompositions Tucker Decomposition Fast Algorithms Tensor Train Decomposition Tucker Decomposition The Tucker decomposition introduces an order d core tensor into the CP decomposition s k 1 ...k d w (1) i 1 k 1 · · · w ( d ) � t i 1 ...i d = i d k d k 1 ··· k d where the columns of each W ( i ) are orthonormal Unlike CP decomposition (given by ‘diagonal’ tensor S ), each index appears in no more than two tensors Tucker decomposition is not low-order since the order of T matches that of S Edgar Solomonik Parallel Numerical Algorithms 15 / 26