TENSOR LAYERS FOR COMPRESSION OF DEEP LEARNING NETWORKS Cris Cecka - - PowerPoint PPT Presentation

Cris Cecka Senior Research Scientist, NVIDIA GTC 2018

TENSOR LAYERS FOR COMPRESSION OF DEEP LEARNING NETWORKS

2

AGENDA

Tensors Computations and the GPU Tensor Networks and Decompositions Tensor Layers in Deep Learning

3

TENSOR COMPUTATIONS AND THE GPU

Modern data is inherently multi-dimensional.

X

TENSOR CONTRACTIONS

• Core primitive of multilinear algebra
• BLAS level 3 — unbounded compute intensity.

X

TENSOR LIBRARIES

Explicit permutation dominates.

Y . Shi, U. N. Niranjan, A. Anandkumar and C. Cecka, "Tensor Contractions with Extended BLAS Kernels on CPU and GPU," 2016 IEEE 23rd International Conference on High Performance Computing (HiPC), Hyderabad, 2016, pp. 193-202.

X

CONTRACTIONS

: Single GEMM (Provided compact layout)

X

BATCHED MATRIX-MATRIX MULTIPLY

cublas<T>gemmStridedBatched

X

CONTRACTIONS

: Single SB-GEMM

(Any layout)

X

APPLICATION: FFT

: Tensor/FFT — vendor optimized

cublas<T>gemmStridedBatched

: Custom kernel : FMM Communication

StridedBatchedGEMM: 75%+ of the runtime 1.5x over cuFFT on 2xV100 2.6x over cuFFT on 8xV100

Cris Cecka. “Low communication FMM-accelerated FFT on GPUs." In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '17). ACM, New York, NY, USA.

10

WHY TENSORS?

11

DENSITY AND SPARSITY

• H. Anzt, S. Tomov, J. Dongarra, “Energy Efficiency and Performance Frontiers for Sparse Computations on GPU

Supercomputers," PMAM 2015.

12

DENSITY AND SPARSITY

In general, need < 5% sparsity for a computational win. Solutions Block-sparse — Locally dense and globally sparse

13

TENSOR DECOMPOSITIONS

Decompositions for data sparse representations.

14

TENSOR NETWORKS

Notation and Visualization

Scalar Vector

ai = Ai =     · · · ·     =

α = A =

Matrix

Aij =   · · · · · · · · ·   =

Tensor

Aijk` = Aijk` = =

15

TENSOR NETWORKS

Notation and Visualization

Aijk`

A(ij)(k`) ≡ Apq

Aij

Aij ≡ A(pq)(mn)

“Matricize” “Tensorize”

A(ijk`) ≡ Am

“Vectorize”

16

TENSOR NETWORKS

Notation and Visualization

inner product

• uter product

SVD

17

CP DECOMPOSITION

Canonical Polyadic Decomposition

Ai1i2···in = λr C(1)

i1r C(2) i2r · · · C(n) inr

18

CP DECOMPOSITION

Ai1i2···in = λr C(1)

i1r C(2) i2r · · · C(n) inr

19

CP DECOMPOSITION

Properties

Analogous to “SVD for Tensors” “Rank” is the size of the diagonal “core tensor” in it’s CP Finding the minimal rank is NP-hard “Truncated SVD is the best rank-k approximation” is NOT true. Uniqueness of the factors Matrix decompositions are not

20

CP DECOMPOSITION

Properties

A vector’s CP Decomposition is itself. A matrix’s CP Decomposition is the SVD. CP-ALS to compute a CP Decomposition from a 3D+ Tensor. Choose rank. Fix all but one core tensor and solve linear least squares to fit. Continue.

=

21

TUCKER DECOMPOSITION

Ai1i2···in = Gr1r2···rn C(1)

i1r1 C(2) i2r2 · · · C(n) inrn

22

TUCKER DECOMPOSITION

Ai1i2···in = Gr1r2···rn C(1)

i1r1 C(2) i2r2 · · · C(n) inrn

CP is a special case of Tucker when core is superdiagonal and R1 = R2 = · · · = Rn

Called High-Order SVD (HOSVD/MLSVD) when core and factor tensors are orthogonal

Computing a Tucker Decomposition from a tensor

23

TUCKER DECOMPOSITION

Algorithms

24

TENSOR RING DECOMPOSITION

Ai1i2···in = C(1)

r1i1r2 C(2) r2i2r3 · · · C(n) rninr1

25

TENSOR RING DECOMPOSITION

Often found as the Tensor Train

Ai1i2···in = C(1)

i1r2 C(2) r2i2r3 · · · C(n) rnin

Similar algorithms for computation from a tensor: Direct HOSVD Iterative ALS Adaptive Rank ALS Block-wise adaptive rank ALS

26

TENSOR RING DECOMPOSITION

Algorithms

• Q. Zhao, G Zhou, S. Xie, L Zhang, A Cichocki. “Tensor Ring Decomposition.” arXiv:1606.05535 [cs.NA] Jun 2016

27

TENSOR RING DECOMPOSITION

Properties

Interpretation as a hierarchical method: Relation to Kernel methods Hierarchical (H-matrix) decomposition Translational invariance

• E. Corona, A. Rahimian, D. Zorin. “A Tensor Train Accelerated Solver For Integral Equation in Complex Geometries.” Journal of

Computational Physics, Volume 334, 2017.

28

KRONECKER DECOMPOSITION

Ai1i2···in = C(1)

a1a2···an ⊗ C(2) b1b2···bn ⊗ · · · ⊗ C(m) d1d2···dn

29

KRONECKER DECOMPOSITION

Algorithms and Properties

Similar algorithms for computation from a tensor: Direct SVD Iterative ALS (KPCA) Iterative Lanczos

Relation to Perfect shuffles + Z-order curves Butterfly algorithms and factorizations

30

KRONECKER DECOMPOSITION

Properties

31

EXOTIC DECOMPOSITIONS

Hierarchical Tucker Decomposition

Any tensor network without cycles where all nodes have degree 3 or less.

32

EXOTIC DECOMPOSITIONS

Other Compositions

Construct arbitrary rich structure that reflects any a-priori knowledge

• f the structure of inputs and outputs

33

TENSOR DECOMPOSITIONS IN DEEP LEARNING

Compress and accelerate layers in Deep Learning

34

DEEP NETWORKS

Fully connected layers take up a lot of space!

TOTAL PARAMS FC PARAMS % FC LAYER AlexNet

61,100,840 58,631,144 96%

VGG-19

143,667,240 123,642,856 86%

ResNet-50

25,557,032 2,049,000 8%

ResNet-101

44,549,160 2,049,000 4.6%

35

DEEP NETWORKS

A general tensor requires O(IN) storage

I is the maximum mode dimension N is the tensor order Exponential savings Storage Compute

Compression

36

DEEP NETWORKS

Observation: In CNNs (and other networks) Fully Connected Layers flatten data

37

CP DECOMPOSITIONS

Application in latent variable models Single topic models Gaussian mixture models (GMM) Latent Dirichlet allocation (LDA) Hidden Markov models (HMM) But in Deep Learning?

in machine learning

• E. Allman ,C. Matias, J. Rhodes. “Identifiability of parameters in latent structure models with many observed variables.” Ann.
• Stat. 37 (2009)
• A. Anandkumar, R. Ge, D. Hsu, S. Kakade, M. Talgarsky. “Tensor Decompositions for Learning Latent Variable Models.” Journal of

Machine Learning Research 15, Jan 2014.

38

CP DECOMPOSITIONS

Compact representation for matrices and tensors. Efficient application of linear algebra operations. Replace a fully connected layer with a CP decomposed layer Initialize from a trained network or randomly Fine-tune Match the modal structure of the input and output

in deep learning

“TCL” LAYER

Special case of CP Layer with R = 1 and input/output mode fusion

R = 1 Matricization TCL Application Fully Connected

40

TCL LAYER

• J. Kossaifi, A. Khanna, Z. Lipton, T

. Furanello, A. Anandkumar. “Tensor Contraction Layers for Parsimonious Deep Nets.” 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, 2017, pp. 1940-1946.

AlexNet, CIFAR-100

41

TCL LAYER

• J. Kossaifi, A. Khanna, Z. Lipton, T

. Furanello, A. Anandkumar. “Tensor Contraction Layers for Parsimonious Deep Nets.” 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, 2017, pp. 1940-1946.

VGG-19, CIFAR-100

42

CP LAYER

Resnet-32, CIFAR-10

• X. Cao, G. Rabusseau, J. Pineau. “Tensor Regression Networks with various Low-Rank Tensor Approximations.” arXiv:

1712.09520 [cs.LG] Dec 2017

43

CP LAYER

Training

ImageNet

44

CP LAYER

Initialization from Pretrained

ImageNet Demonstrates initialization from existing networks to gain a large head-start in training Fine-tuning still very important

45

TUCKER LAYERS

A nice geometric interpretation: Reveal latent features in each mode. Core tensor, G, yields relatives importance

• f all “combined” features.

Straight forward compression and application Core tensor is of the same order.

Ai1i2···in = Gr1r2···rn C(1)

i1r1 C(2) i2r2 · · · C(n) inrn

46

TUCKER LAYERS

Replace Fully Connected

• X. Cao, G. Rabusseau, J. Pineau. “Tensor Regression Networks with various Low-Rank Tensor Approximations.” arXiv:

1712.09520 [cs.LG] Dec 2017

Resnet-32, CIFAR-10

47

TUCKER LAYERS

Replace Fully Connected

• J. Kossaifi, Z. Lipton, A. Khanna, T

. Furanello, A. Anandkumar. “Tensor Contraction and Regression Networks.”. arXiv: 1707.08308 [cs.LG] Nov 2017

TCL (CP) TRL (Tucker)

48

TUCKER LAYERS

Resnet-101 on ImageNet: Compression of the FullyConnected Layer

Performance and Compression

• J. Kossaifi, Z. Lipton, A. Khanna, T

. Furanello, A. Anandkumar. “Tensor Contraction and Regression Networks.”. arXiv: 1707.08308 [cs.LG] Nov 2017

49

TENSOR RING LAYERS

Compression for fully connected layers and convolutional layers

50

TENSOR TRAIN LAYERS

Replace Fully Connected

Resnet-32, CIFAR-10

• X. Cao, G. Rabusseau, J. Pineau. “Tensor Regression Networks with various Low-Rank Tensor Approximations.” arXiv:

1712.09520 [cs.LG] Dec 2017

51

TENSOR TRAIN LAYERS

Replace Fully Connected

VGG-16/19, ImageNet

• A. Novikov, D. Podoprikhin, A. Osokin, D. Vetrov. “Tensorizing Neural Networks.” arXiv:1509.06569 [cs.LG] Dec 2015

52

TENSOR TRAIN LAYERS

Replace Fully Connected

ImageNet

53

TENSOR TRAIN LAYERS

Replace Convolutional

Resnet-like, CIFAR-10 VGG-like, CIFAR-10

T . Garipov, D. Podoprikhin, A. Novikov, D. Vetrov. “Ultimate Tensorization: Compressing Convolutional and FC Layers Alike.” arXiv: 1611.03214 [cs.LG] Nov 2016

54

KRONECKER LAYERS

Compact representation of advanced linear operators Preservation of properties of linear operators

55

KRONECKER LAYERS

Fully Connected Layer

• S. Zhou, J. Wu. “Compression of Fully-Connected Layer in Neural Network by Kronecker Product.” Advanced Computational

Intelligence (ICACI) IEEE 2016.

56

KRONECKER LAYERS

Kronecker product preserves unitarity Control RNN vanishing/exploding gradient problem on small Kronecker factors

RNN Layer

L(·) + λ kC(i) C(i)T Ik2

2

Ai1i2 = C(1)

a1a2 ⊗ C(2) b1b2 ⊗ · · · ⊗ C(m) d1d2

Complex-valued factors for compact unitary set!

57

KRONECKER LAYERS

RNN Layer

• C. Jose, M. Cisse, F. Fleuret. “Kronecker Recurrent Units.” arXiv:1705.10142 [cs.LG] Dec 2017

58

KRONECKER LAYERS

RNN Application

• C. Jose, M. Cisse, F. Fleuret. “Kronecker Recurrent Units.” arXiv:1705.10142 [cs.LG] Dec 2017

59

GUIDANCE

Many ML papers suffer selection bias — Non-trivial to get these layers working. Sufficient rank/structure. Batch-normalization, other regularization. Careful random initialization of tensor factors. Initialization from pre-trained layers + fine-tuning.

60

CHALLENGES

Determine good tensor networks for a given problem. Alternatively, adaptively adjust these layers to determine hyperparameters. E.g. Rank adjustment, hierarchical factorizations, optimizer regularizations. Speed-of-light tensor computations in machine learning libraries. Many rely on suboptimal transpose-transpose-GEMM-transpose patterns. Deployment in real-world applications.

61

SOFTWARE

TensorLy: Tensor Learning in Python Supports CP and Tucker tensors Decomposition algorithms Multiple backends PyTorch MxNet NumPy

62

SOFTWARE

https://github.com/tensorly/tensorly https://github.com/jacobgil/pytorch-tensor-decompositions https://github.com/ebigelow/tf-decompose https://github.com/xwcao/LowRankTRN https://github.com/Tuyki/TT_RNN

References

63

CONCLUSION

Fully connected layers can be significantly algebraically compressed. Without resorting to sparsity. Tensor networks to represent data sparse dense linear operators with custom structure. Training and fine-tuning demonstrate that New BLAS-like primitives to accelerate tensor computations. Globally sparse and locally dense computations for speed-of-light performance.