tensor matrix products with a compressed sparse tensor
play

Tensor-Matrix Products with a Compressed Sparse Tensor Shaden Smith - PowerPoint PPT Presentation

Feb 22, 2024 •601 likes •1.09k views

Tensor-Matrix Products with a Compressed Sparse Tensor Shaden Smith George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Tensor-Matrix Products with a Compressed Sparse Tensor 1 / 47

  1. Tensor-Matrix Products with a Compressed Sparse Tensor Shaden Smith George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Tensor-Matrix Products with a Compressed Sparse Tensor 1 / 47 http://cs.umn.edu/~splatt/

  2. Tensor Introduction Tensors are the generalization of matrices to ≥ 3 D Tensors have m dimensions (or modes ) and are I 1 × . . . × I m . users items words Tensor-Matrix Products with a Compressed Sparse Tensor 2 / 47 http://cs.umn.edu/~splatt/

  3. Canonical Polyadic Decomposition (CPD) The CPD is an extension of the SVD to tensors We compute matrices A 1 , . . . , A m , each with F columns and λ , a vector of weights λ 1 λ F + · · · + ≈ Usually computed via alternating least squares (ALS) Tensor-Matrix Products with a Compressed Sparse Tensor 3 / 47 http://cs.umn.edu/~splatt/

  4. MTTKRP Matricized Tensor Times Khatri-Rao Product (MTTKRP) MTTKRP is the core computation of each iteration A 1 = X (1) ( A m � · · · � A 2 ) I 1 I 2 F X (1) I 1 I 1 I 2 ( A 3 � A 2 ) Tensor-Matrix Products with a Compressed Sparse Tensor 4 / 47 http://cs.umn.edu/~splatt/

  5. Related Work Tensor-Matrix Products with a Compressed Sparse Tensor 5 / 47 http://cs.umn.edu/~splatt/

  6. Uncompressed Tensors Stored as a list of coordinates ( i , j , k ) = v represents one nonzero ← A 1 ( i , :) ← A 1 ( i , :) + X ( i , j , k ) [ A 2 ( j , :) ∗ A 3 ( k , :)] Tensor-Matrix Products with a Compressed Sparse Tensor 6 / 47 http://cs.umn.edu/~splatt/

  7. Compressed Tensors SPLATT SPLATT uses a hierarchical storage scheme for 3D tensors This allows for operation reduction and coarse-grained parallelism ← Tensor-Matrix Products with a Compressed Sparse Tensor 7 / 47 http://cs.umn.edu/~splatt/

  8. Contributions Tensor-Matrix Products with a Compressed Sparse Tensor 8 / 47 http://cs.umn.edu/~splatt/

  9. Compressed Sparse Fiber (CSF) 1 2  i j k l  i 1 1 1 2     1 1 1 3 j 1 2 2     1 2 1 3   →   1 2 2 1 k 1 1 2 1 2     2 2 1 1     2 2 1 3 2 3 3 1 1 3 2   l 2 2 2 2 Tensor-Matrix Products with a Compressed Sparse Tensor 9 / 47 http://cs.umn.edu/~splatt/

  10. MTTKRP with a CSF Tensor Objective We want to perform MTTKRP on each tensor mode with only one CSF representation There are three types of nodes in a tree: root , internal , and leaf ◮ Each will have a tailored algorithm Tensor-Matrix Products with a Compressed Sparse Tensor 10 / 47 http://cs.umn.edu/~splatt/

  11. CSF-ROOT We do a depth-first traversal on the CSF structure Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 11 / 47 http://cs.umn.edu/~splatt/

  12. CSF-ROOT Inner products are accumulated in a buffer Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 12 / 47 http://cs.umn.edu/~splatt/

  13. CSF-ROOT Inner products are accumulated in a buffer Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 13 / 47 http://cs.umn.edu/~splatt/

  14. CSF-ROOT Hadamard products are then propagated up the CSF tree Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 14 / 47 http://cs.umn.edu/~splatt/

  15. CSF-ROOT Hadamard products are then propagated up the CSF tree Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 15 / 47 http://cs.umn.edu/~splatt/

  16. CSF-ROOT Results are accumulated when we reach the top Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 16 / 47 http://cs.umn.edu/~splatt/

  17. CSF-ROOT The traversal continues... Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 17 / 47 http://cs.umn.edu/~splatt/

  18. CSF-ROOT The traversal continues... Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 18 / 47 http://cs.umn.edu/~splatt/

  19. CSF-ROOT Partial results are kept in buffer Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 19 / 47 http://cs.umn.edu/~splatt/

  20. CSF-ROOT Inner products are accumulated in a buffer Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 20 / 47 http://cs.umn.edu/~splatt/

  21. CSF-ROOT Inner products are accumulated in a buffer Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 21 / 47 http://cs.umn.edu/~splatt/

  22. CSF-LEAF This time, Hadamard products are pushed down the tree Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 22 / 47 http://cs.umn.edu/~splatt/

  23. CSF-LEAF This time, Hadamard products are pushed down the tree Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 23 / 47 http://cs.umn.edu/~splatt/

  24. CSF-LEAF This time, Hadamard products are pushed down the tree Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 24 / 47 http://cs.umn.edu/~splatt/

  25. CSF-LEAF This time, Hadamard products are pushed down the tree Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 25 / 47 http://cs.umn.edu/~splatt/

  26. CSF-LEAF Leaves designate write locations Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 26 / 47 http://cs.umn.edu/~splatt/

  27. CSF-LEAF Leaves designate write locations Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 27 / 47 http://cs.umn.edu/~splatt/

  28. CSF-LEAF The traversal continues... Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 28 / 47 http://cs.umn.edu/~splatt/

  29. CSF-LEAF The traversal continues... Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 29 / 47 http://cs.umn.edu/~splatt/

  30. CSF-LEAF The traversal continues... Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 30 / 47 http://cs.umn.edu/~splatt/

  31. CSF-LEAF The traversal continues... Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 31 / 47 http://cs.umn.edu/~splatt/

  32. CSF-LEAF The traversal continues... Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 32 / 47 http://cs.umn.edu/~splatt/

  33. CSF-INTERNAL Internal nodes use a combination of CSF-ROOT and CSF-LEAF Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 33 / 47 http://cs.umn.edu/~splatt/

  34. CSF-INTERNAL Hadamard products are pushed down to the output level Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 34 / 47 http://cs.umn.edu/~splatt/

  35. CSF-INTERNAL CSF-ROOT next pulls up to the output level Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 35 / 47 http://cs.umn.edu/~splatt/

  36. CSF-INTERNAL CSF-ROOT next pulls up to the output level Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 36 / 47 http://cs.umn.edu/~splatt/

  37. CSF-INTERNAL CSF-ROOT next pulls up to the output level Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 37 / 47 http://cs.umn.edu/~splatt/

  38. CSF-INTERNAL CSF-ROOT next pulls up to the output level Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 38 / 47 http://cs.umn.edu/~splatt/

  39. CSF-INTERNAL CSF-ROOT next pulls up to the output level Z 1 A 1 A 2 A 3 A 4 1 2 1 1 2 2 3 3 1 Tensor-Matrix Products with a Compressed Sparse Tensor 39 / 47 http://cs.umn.edu/~splatt/

  40. Parallelism – Tiling A 1 A 3 A 2 Tensor-Matrix Products with a Compressed Sparse Tensor 40 / 47 http://cs.umn.edu/~splatt/

  41. Datasets Dataset I 1 I 2 I 3 nnz NELL-2 12K 9K 28K 77M Beer 33K 66K 960K 94M Netflix 480K 18K 2K 100M Delicious 532K 17M 3M 140M NELL-1 3M 2M 25M 143M Amazon 5M 18M 2M 1.7B Tensor-Matrix Products with a Compressed Sparse Tensor 41 / 47 http://cs.umn.edu/~splatt/

  42. Storage Comparison SPLATT COORD CSF-M CSF-T 99.3 10 2 51.9 47.5 36.4 Tensor storage (GB) 10 1 8.8 8.2 5.0 4.5 4.2 4.3 4.0 3.7 3.6 3.0 2.8 2.6 2.4 2.3 2.1 1.8 1.6 1.4 1.2 1.1 10 0 NELL-2 Beer Netflix Delicious NELL-1 Amazon Dataset Tensor-Matrix Products with a Compressed Sparse Tensor 42 / 47 http://cs.umn.edu/~splatt/

  43. CSF-ROOT 90 SPLATT CSF-M CSF-T 80 Speedup over COORD with 16 threads 70 60 50 40 30 20 10 0 NELL-2 Beer Netflix Delicious NELL-1 Amazon Dataset Tensor-Matrix Products with a Compressed Sparse Tensor 43 / 47 http://cs.umn.edu/~splatt/

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Memory-Efficient Parallel Computation of Tensor and Matrix Products for Big Tensor Decomposition

Memory-Efficient Parallel Computation of Tensor and Matrix Products for Big Tensor Decomposition

Memory-Efficient Parallel Computation of Tensor and Matrix Products for Big Tensor Decomposition N. Ravindran , N.D. Sidiropoulos , S. Smith , and G. Karypis Dept. of ECE & DTC, Dept. of CSci & DTC University of

553 views • 19 slides
8. Tensor Field Visualization Tensor: extension of concept of scalar and vector Tensor data

8. Tensor Field Visualization Tensor: extension of concept of scalar and vector Tensor data

8. Tensor Field Visualization Tensor: extension of concept of scalar and vector Tensor data for a tensor of level k is given by t i1,i2,,ik (x 1 ,,x n ) Second-order tensor often represented by matrix Examples:

292 views • 15 slides
A Sparse Tensor Format and a Benchmark Suite Jiajia Li Pacific Northwest National Laboratory

A Sparse Tensor Format and a Benchmark Suite Jiajia Li Pacific Northwest National Laboratory

A Sparse Tensor Format and a Benchmark Suite Jiajia Li Pacific Northwest National Laboratory January 25, 2019 @ MIT Figure sources: A brief survey of tensors by Berton Earnshaw and NVIDIA Tensor Cores HiCOO: Hierarchical Storage of Sparse

272 views • 23 slides
and You Tensor network methods Matrix product states (MPS) Projected Entangled Pair States

and You Tensor network methods Matrix product states (MPS) Projected Entangled Pair States

Tensor Networks and You Nikko Pomata How do you network tensors? Tensors: a review The tensor-network notation T ENSOR N ETWORKS Tensor network examples and You Tensor network methods Matrix product states (MPS) Projected Entangled Pair

642 views • 24 slides
Tensor For F , F Sh( X ) we have F F Sh( X ). Since tensor commutes with

Tensor For F , F Sh( X ) we have F F Sh( X ). Since tensor commutes with

Tensor For F , F Sh( X ) we have F F Sh( X ). Since tensor commutes with taking stalk, the non-exactness comes from that F x might not torsion-free, and this can be taken care by a simple resolution 0 P P F

1.85k views • 7 slides
Renormalization of Tensor Network States II. RG of Tensor Network States Tao Xiang Institute of

Renormalization of Tensor Network States II. RG of Tensor Network States Tao Xiang Institute of

Renormalization of Tensor Network States II. RG of Tensor Network States Tao Xiang Institute of Physics Chinese Academy of Sciences txiang@iphy.ac.cn Tensor-Network Ansatz of the ground state wavefunction 1D: Matrix Product State (MPS) : m

603 views • 57 slides
Tensor Products of A -algebras with Homotopy Inner Products (Joint work with Thomas Tradler,

Tensor Products of A -algebras with Homotopy Inner Products (Joint work with Thomas Tradler,

Tensor Products of A -algebras with Homotopy Inner Products (Joint work with Thomas Tradler, CUNY) Ron Umble Millersville University Lehigh Geometry and Topology Conference May 25, 2013 Tensor Products of A -infinity Algebras Let K =

1.17k views • 105 slides
TENSOR ALGEBRA Continuum Mechanics Course (MMC) - ETSECCPB - UPC Introduction to Tensors Tensor

TENSOR ALGEBRA Continuum Mechanics Course (MMC) - ETSECCPB - UPC Introduction to Tensors Tensor

TENSOR ALGEBRA Continuum Mechanics Course (MMC) - ETSECCPB - UPC Introduction to Tensors Tensor Algebra 2 Introduction SCALAR , , ... v VECTOR v f , , ... , , ... MATRIX ? , ... C 3 Concept of Tensor A TENSOR

885 views • 88 slides
Tensor Field Visualization 9-1 Ronald Peikert SciVis 2007 - Tensor Fields Tensors

Tensor Field Visualization 9-1 Ronald Peikert SciVis 2007 - Tensor Fields Tensors

Tensor Field Visualization 9-1 Ronald Peikert SciVis 2007 - Tensor Fields Tensors "Tensors are the language of mechanics" T Tensor of order (rank) f d ( k) 0: scalar 1: vector 1: vector 2: matrix (example: stress tensor)

773 views • 28 slides
Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges Shaden Smith &

Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges Shaden Smith &

Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges Shaden Smith & George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Sparse Tensor Factorization: Algorithms,

844 views • 80 slides
(Some) Challenges in (Some) Challenges in Tensor Mining Tensor Mining Evrim Acar Sandia

(Some) Challenges in (Some) Challenges in Tensor Mining Tensor Mining Evrim Acar Sandia

(Some) Challenges in (Some) Challenges in Tensor Mining Tensor Mining Evrim Acar Sandia National Labs., Livermore, CA Tensor Mining Tensor Mining Parafac Parafac unsupervised unsupervised = + + X X E X X = + dense or sparse

340 views • 17 slides
SPLATT Efficient and Parallel Sparse Tensor-Matrix Multiplication Shaden Smith 1 Niranjay

SPLATT Efficient and Parallel Sparse Tensor-Matrix Multiplication Shaden Smith 1 Niranjay

SPLATT Efficient and Parallel Sparse Tensor-Matrix Multiplication Shaden Smith 1 Niranjay Ravindran Nicholas D. Sidiropoulos George Karypis University of Minnesota 1 shaden@cs.umn.edu Shaden Smith, shaden@cs.umn.edu (U. Minnesota) SPLATT 1

3.48k views • 24 slides
Tensor products of finitely cococomplete and abelian categories 1 Ignacio Lpez Franco

Tensor products of finitely cococomplete and abelian categories 1 Ignacio Lpez Franco

Delignes tensor product Questions we answer Existence of Delignes tensor Counterexample to the existence Tensor products of finitely cococomplete and abelian categories 1 Ignacio Lpez Franco University of Cambridge Gonville and Caius

554 views • 33 slides
Parallel Sparse Matrix-Vector and Matrix- Transpose-Vector Multiplication using Compressed Sparse

Parallel Sparse Matrix-Vector and Matrix- Transpose-Vector Multiplication using Compressed Sparse

Parallel Sparse Matrix-Vector and Matrix- Transpose-Vector Multiplication using Compressed Sparse Blocks Aydn Bulu, UCSB Jeremy T. Fineman (MIT) Matteo Frigo (Cilk Arts) John R. Gilbert (UCSB) Charles E. Leiserson (MIT & Cilk Arts) 1

523 views • 21 slides
A Medium-Grained Algorithm for Distributed Sparse Tensor Factorization Shaden Smith George

A Medium-Grained Algorithm for Distributed Sparse Tensor Factorization Shaden Smith George

A Medium-Grained Algorithm for Distributed Sparse Tensor Factorization Shaden Smith George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Medium-Grained Sparse Tensor Factorization 1 / 24

424 views • 25 slides
Scalable Tensor Computations with Cyclops and Faster Algorithms for Alternating Least Squares

Scalable Tensor Computations with Cyclops and Faster Algorithms for Alternating Least Squares

Scalable Tensor Computations with Cyclops and Faster Algorithms for Alternating Least Squares Edgar Solomonik Department of Computer Science University of Illinois at Urbana-Champaign Invited Workshop on Compiler Techniques for Sparse Tensor

222 views • 9 slides
The Way of Optimizing and Troubleshooting Our Lua Waf agentzh@gmail.com Yichun Zhang

The Way of Optimizing and Troubleshooting Our Lua Waf agentzh@gmail.com Yichun Zhang

pdf |<< < > >>| 1 / 25 The Way of Optimizing and Troubleshooting Our Lua Waf agentzh@gmail.com Yichun Zhang (agentzh) 2013.04.19 Dane now loves to open a conversation with me via a Flame Graph. One day, Dane

1.06k views • 26 slides
Week 6: An Introduction to Time Series Dependent data, autocorrelation, AR and periodic

Week 6: An Introduction to Time Series Dependent data, autocorrelation, AR and periodic

BUS41100 Applied Regression Analysis Week 6: An Introduction to Time Series Dependent data, autocorrelation, AR and periodic regression models Max H. Farrell The University of Chicago Booth School of Business Time series data and dependence

1.49k views • 60 slides
LOD2 Stack and the NLP2RDF project http://slideshare.net/kurzum http://nlp2rdf.org

LOD2 Stack and the NLP2RDF project http://slideshare.net/kurzum http://nlp2rdf.org

Creating Knowledge out of Interlinked Data Rome 2013/03/13 Page 1 http://lod2.eu LOD2 Stack and the NLP2RDF project http://slideshare.net/kurzum http://nlp2rdf.org http://lod2.eu Sebastian Hellmann AKSW, Universitt Leipzig

761 views • 32 slides
The Android security jungle: pitfalls, threats and survival tips Scott Alexander-Bown @scottyab

The Android security jungle: pitfalls, threats and survival tips Scott Alexander-Bown @scottyab

The Android security jungle: pitfalls, threats and survival tips Scott Alexander-Bown @scottyab The Jungle Ecosystem Googles protection Threats Risks Survival Network Data protection (encryption) App/device

888 views • 45 slides
CALL OF DUTY: DEV OPS STEPHEN BURTON, TECH EV ANGELIST , APPDYNAMICS my company Im Steve

CALL OF DUTY: DEV OPS STEPHEN BURTON, TECH EV ANGELIST , APPDYNAMICS my company Im Steve

CALL OF DUTY: DEV OPS STEPHEN BURTON, TECH EV ANGELIST , APPDYNAMICS my company Im Steve Burton my passion tech ev angelist @burtonsa ys THE GAME toda y GAME SELECT DEVELOPER DEVELOPER A OPERATIONS DEVOPS NOOPS Dev MISSION

811 views • 53 slides
A Support Engineer Walkthrough on pt-stalk Marcos Albe Marcelo Altmann Principal Support

A Support Engineer Walkthrough on pt-stalk Marcos Albe Marcelo Altmann Principal Support

A Support Engineer Walkthrough on pt-stalk Marcos Albe Marcelo Altmann Principal Support Engineer - Percona Senior Support Engineer - Percona Thank You Sponsors! 2 April 23-25, 2018 SAVE THE DATE! Santa Clara Convention Center CALL FOR

870 views • 30 slides
Security in Mobile Devices Hacking Mobiles for Fun and Profit Tobias Mueller Universit at

Security in Mobile Devices Hacking Mobiles for Fun and Profit Tobias Mueller Universit at

Intro Hardware Security Platform Security Hacking Q&A Security in Mobile Devices Hacking Mobiles for Fun and Profit Tobias Mueller Universit at Hamburg & Dublin City University 2010-12-16 1 / 54 Intro Hardware Security

974 views • 79 slides
E/R Diagrams Converting E/R Diagrams to Relations DB design is a serious and

E/R Diagrams Converting E/R Diagrams to Relations DB design is a serious and

E/R Diagrams Converting E/R Diagrams to Relations DB design is a serious and possibly complex business. A client may know they want a database, but they don t know what they want in it or how it should look. E/R

977 views • 75 slides

More recommend