sparse tensor factorization algorithms data structures
play

Sparse Tensor Factorization: Algorithms, Data Structures, and - PowerPoint PPT Presentation

Jul 20, 2023 •44 likes •844 views

Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges Shaden Smith & George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Sparse Tensor Factorization: Algorithms,

  1. Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges Shaden Smith & George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  2. Talk Outline Introduction 1 Compressed Sparse Fiber 2 Cache-Friendly Reordering & Tiling 3 Distributed-Memory MTTKRP 4 Conclusions 5 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  3. Table of Contents Introduction 1 Compressed Sparse Fiber 2 Cache-Friendly Reordering & Tiling 3 Distributed-Memory MTTKRP 4 Conclusions 5 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  4. Tensor Introduction Tensors are the generalization of matrices to ≥ 3 D Tensors have m dimensions (or modes ) and are I 1 × . . . × I m ◮ We’ll usually stick to I × J × K in this talk users contexts items Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  5. Applications Dataset I J K nnz NELL-2 12K 9K 28K 77M Beer 33K 66K 960K 94M Netflix 480K 18K 2K 100M Delicious 532K 17M 3M 140M NELL-1 3M 2M 25M 143M Amazon 5M 18M 2M 1.7B Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  6. Canonical Polyadic Decomposition (CPD) We compute matrices A , B , C , each with F columns ◮ We will use A (1) , . . . , A ( m ) when ≥ 3 modes ≈ + · · · + Usually computed via alternating least squares (ALS) Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  7. Matricized Tensor Times Khatri-Rao Product MTTKRP MTTKRP is the core computation of each iteration A = X (1) ( C � B ) J · K F X (1) I J · K ( C � B ) Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  8. Alternating Least Squares 1: while not converged do A ⊺ = ( C ⊺ C ∗ B ⊺ B ) − 1 � � ⊺ X (1) ( C � B ) 2: B ⊺ = ( C ⊺ C ∗ A ⊺ A ) − 1 � � ⊺ X (2) ( C � A ) 3: C ⊺ = ( B ⊺ B ∗ A ⊺ A ) − 1 � � ⊺ X (3) ( B � A ) 4: 5: end while Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  9. Tensor Storage – Coordinate Form   i j k l v 1 1 1 2 1 .     1 1 1 3 1 .     1 2 1 3 3 .     1 2 2 1 8 .     2 2 1 1 1 .     2 2 1 3 3 .   2 2 2 2 8 . Why don’t we unfold? We need a representation of X for each mode NELL has dimensions 3 M × 2 M × 25 M ◮ Add a fourth mode and we exceed 2 64 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  10. MTTKRP A C i ← k B j A ( i , :) ← A ( i , :) + X ( i , j , k ) [ B ( j , :) ∗ C ( k , :)] Limitations Memory bandwidth Parallelism Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  11. Table of Contents Introduction 1 Compressed Sparse Fiber 2 Cache-Friendly Reordering & Tiling 3 Distributed-Memory MTTKRP 4 Conclusions 5 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  12. Can we do better? Consider three nonzeros in the fiber X ( i , j , :) (a vector) A ( i , :) ← A ( i , :) + X ( i , j , k 1 ) [ B ( j , :) ∗ C ( k 1 , :)] A ( i , :) ← A ( i , :) + X ( i , j , k 2 ) [ B ( j , :) ∗ C ( k 2 , :)] A ( i , :) ← A ( i , :) + X ( i , j , k 3 ) [ B ( j , :) ∗ C ( k 3 , :)] Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  13. Can we do better? Consider three nonzeros in the fiber X ( i , j , :) (a vector) A ( i , :) ← A ( i , :) + X ( i , j , k 1 ) [ B ( j , :) ∗ C ( k 1 , :)] A ( i , :) ← A ( i , :) + X ( i , j , k 2 ) [ B ( j , :) ∗ C ( k 2 , :)] A ( i , :) ← A ( i , :) + X ( i , j , k 3 ) [ B ( j , :) ∗ C ( k 3 , :)] A little factoring... � � 3 � A ( i , :) ← A ( i , :) + B ( j , :) ∗ X ( i , j , k x ) C ( k x , :) x =1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  14. SPLATT: The S urprisingly P aralle L sp A rse T ensor T oolkit ← [Smith, Ravindran, Sidiropoulos, and Karypis 2015] Fibers are sparse vectors Slice X ( i , : , :) is almost a CSR matrix... But, we need m representations of X Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  15. Compressed Sparse Fiber (CSF)   1 2 i j k l i 1 1 1 2     1 1 1 3 j 1 2 2     1 2 1 3     → 1 2 2 1   1 1 2 1 2 k   2 2 1 1     2 2 1 3   2 3 3 1 1 3 2 l 2 2 2 2 [Smith and Karypis 2015] Modes are recursively compressed Values are stored in the leaves (not shown) Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  16. MTTKRP with a CSF Tensor Objective We want to perform MTTKRP on each tensor mode with only one CSF representation There are three types of nodes in a tree: root , internal , and leaf ◮ Each will have a tailored algorithm ◮ root and leaf are special cases of internal Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  17. CSF-LEAF The leaf nodes determine the output location Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  18. CSF-LEAF Hadamard products are pushed down the tree Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  19. CSF-LEAF Hadamard products are pushed down the tree Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  20. CSF-LEAF Hadamard products are pushed down the tree Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  21. CSF-LEAF Leaves designate write locations Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  22. CSF-LEAF Leaves designate write locations Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  23. CSF-LEAF The traversal continues... Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  24. CSF-LEAF The traversal continues... Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  25. CSF-LEAF The traversal continues... Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  26. CSF-LEAF The traversal continues... Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  27. CSF-LEAF The traversal continues... Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  28. CSF-ROOT Inner products are accumulated in a buffer Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  29. CSF-ROOT Inner products are accumulated in a buffer Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  30. CSF-ROOT Hadamard products are then propagated up the CSF tree Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  31. CSF-ROOT Hadamard products are then propagated up the CSF tree Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  32. CSF-ROOT Results are written to A (1) Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  33. CSF-ROOT The traversal continues... Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  34. CSF-ROOT The traversal continues... Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  35. CSF-ROOT Partial results are kept in buffer Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

  36. CSF-ROOT Inner products are accumulated in a buffer Z 1 A (1) A (2) A (3) A (4) 1 2 1 1 2 2 3 3 1 Sparse Tensor Factorization: Algorithms, Data Structures, and Challenges shaden@cs.umn.edu

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

A Medium-Grained Algorithm for Distributed Sparse Tensor Factorization Shaden Smith George

A Medium-Grained Algorithm for Distributed Sparse Tensor Factorization Shaden Smith George

A Medium-Grained Algorithm for Distributed Sparse Tensor Factorization Shaden Smith George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Medium-Grained Sparse Tensor Factorization 1 / 24

424 views • 25 slides
Sparse Tensor Factorization on Many-Core Processors with High-Bandwidth Memory Shaden Smith 1

Sparse Tensor Factorization on Many-Core Processors with High-Bandwidth Memory Shaden Smith 1

Sparse Tensor Factorization on Many-Core Processors with High-Bandwidth Memory Shaden Smith 1 , Jongsoo Park 2 , and George Karypis 1 1 Department of Computer Science & Engineering, University of Minnesota 2 Parallel Computing Lab, Intel

802 views • 37 slides
Parallel Numerical Algorithms Chapter 4 Sparse Linear Systems Section 4.1 Direct Methods

Parallel Numerical Algorithms Chapter 4 Sparse Linear Systems Section 4.1 Direct Methods

Sparse Matrices Sparse Triangular Solve Cholesky Factorization Sparse Cholesky Factorization Parallel Numerical Algorithms Chapter 4 Sparse Linear Systems Section 4.1 Direct Methods Michael T. Heath and Edgar Solomonik Department of

906 views • 63 slides
Algorithms and Data Structures: Overview Algorithms and data structures Data Abstraction,

Algorithms and Data Structures: Overview Algorithms and data structures Data Abstraction,

Algorithms and Data Structures: Overview Algorithms and data structures Data Abstraction, Ch. 3 Linked lists, Ch. 4 Recursion, Ch. 5 Stacks, Ch. 6 Queues, Ch. 7 Algorithm Efficiency and Sorting, Ch. 9 Trees, Ch. 10

729 views • 44 slides
Scalable Tensor Computations with Cyclops and Faster Algorithms for Alternating Least Squares

Scalable Tensor Computations with Cyclops and Faster Algorithms for Alternating Least Squares

Scalable Tensor Computations with Cyclops and Faster Algorithms for Alternating Least Squares Edgar Solomonik Department of Computer Science University of Illinois at Urbana-Champaign Invited Workshop on Compiler Techniques for Sparse Tensor

222 views • 9 slides
Basker : A Threaded Sparse LU factorization utilizing Hierarchical Parallelism and Data Layouts

Basker : A Threaded Sparse LU factorization utilizing Hierarchical Parallelism and Data Layouts

Basker : A Threaded Sparse LU factorization utilizing Hierarchical Parallelism and Data Layouts Siva Rajamanickam Joshua Booth, Heidi Thornquist Sixth International Workshop on Accelerators and Hybrid Exascale Systems (IPDPS 2016) Sandia

424 views • 15 slides
Introduction to Algorithms and Data Structures CSC 1051 Algorithms and Data Structures I Dr.

Introduction to Algorithms and Data Structures CSC 1051 Algorithms and Data Structures I Dr.

Introduction to Algorithms and Data Structures CSC 1051 Algorithms and Data Structures I Dr. Mary-Angela Papalaskari Department of Computing Sciences Villanova University Course website: www.csc.villanova.edu/~map/1051/s18 Some slides in

971 views • 64 slides
Invited Workshop on Compiler Techniques for Sparse Tensor Algebra Cambridge MA, January 26th

Invited Workshop on Compiler Techniques for Sparse Tensor Algebra Cambridge MA, January 26th

Invited Workshop on Compiler Techniques for Sparse Tensor Algebra Cambridge MA, January 26th Sparse Tensors are Everywhere Data Analytics Machine Learning Science and Engineering Movie ratings Convolutional Networks Product Reviews QCD

427 views • 29 slides
Tensor-Matrix Products with a Compressed Sparse Tensor Shaden Smith George Karypis University

Tensor-Matrix Products with a Compressed Sparse Tensor Shaden Smith George Karypis University

Tensor-Matrix Products with a Compressed Sparse Tensor Shaden Smith George Karypis University of Minnesota Department of Computer Science & Engineering shaden@cs.umn.edu Tensor-Matrix Products with a Compressed Sparse Tensor 1 / 47

1.09k views • 47 slides
A Sparse Tensor Format and a Benchmark Suite Jiajia Li Pacific Northwest National Laboratory

A Sparse Tensor Format and a Benchmark Suite Jiajia Li Pacific Northwest National Laboratory

A Sparse Tensor Format and a Benchmark Suite Jiajia Li Pacific Northwest National Laboratory January 25, 2019 @ MIT Figure sources: A brief survey of tensors by Berton Earnshaw and NVIDIA Tensor Cores HiCOO: Hierarchical Storage of Sparse

272 views • 23 slides
ACCELERATING SPARSE CHOLESKY FACTORIZATION ON THE GPU STEVE RENNICH, SR. ENGINEER, NVIDIA

ACCELERATING SPARSE CHOLESKY FACTORIZATION ON THE GPU STEVE RENNICH, SR. ENGINEER, NVIDIA

ACCELERATING SPARSE CHOLESKY FACTORIZATION ON THE GPU STEVE RENNICH, SR. ENGINEER, NVIDIA DEVELOPER TECHNOLOGY DARKO STOSIC, PHD CANDIDATE, UNIV. FEDERAL DE PERNAMBUCO TIM DAVIS, PROFESSOR, CSE, TEXAS A&M UNIVERSITY SPARSE MATRIX

425 views • 28 slides
Data Structures and What is a data structure? Algorithms Way of storing data in computer

Data Structures and What is a data structure? Algorithms Way of storing data in computer

Introduction: Data Structures Data Structures and What is a data structure? Algorithms Way of storing data in computer so can be used efficiently Set of operations that access the data in prescribed ways Prof. Nadeem Abdul Hamid

683 views • 10 slides
Data Structures and Java Collections Framework 1 Algorithms and Data Structures

Data Structures and Java Collections Framework 1 Algorithms and Data Structures

IS311 Data Structures and Java Collections Framework 1 Algorithms and Data Structures Algorithm Sequence of steps used to solve a problem Operates on collection of data Each element of collection - > data structure

793 views • 54 slides
Tensor Factorization via Matrix Factorization Volodymyr Kuleshov Arun Tejasvi Chaganty Percy

Tensor Factorization via Matrix Factorization Volodymyr Kuleshov Arun Tejasvi Chaganty Percy

Tensor Factorization via Matrix Factorization Volodymyr Kuleshov Arun Tejasvi Chaganty Percy Liang Stanford University May 11, 2015 Kuleshov, Chaganty, Liang (Stanford University) Tensor Factorization May 11, 2015 1 / 28 Introduction:

1.1k views • 79 slides
Structured sparse methods for matrix factorization Francis Bach Sierra team, INRIA - Ecole

Structured sparse methods for matrix factorization Francis Bach Sierra team, INRIA - Ecole

Structured sparse methods for matrix factorization Francis Bach Sierra team, INRIA - Ecole Normale Sup erieure March 2011 Joint work with R. Jenatton, J. Mairal, G. Obozinski Structured sparse methods for matrix factorization Outline

1.07k views • 73 slides
Parallel Numerical Algorithms Chapter 7 Differential Equations Section 7.5 Tensor

Parallel Numerical Algorithms Chapter 7 Differential Equations Section 7.5 Tensor

Tensor Algebra Tensor Decompositions Fast Algorithms Parallel Numerical Algorithms Chapter 7 Differential Equations Section 7.5 Tensor Analysis Edgar Solomonik Department of Computer Science University of Illinois at Urbana-Champaign

722 views • 26 slides
Factoring Polynomials over Local Fields II Sebastian Pauli Department of Mathematics and

Factoring Polynomials over Local Fields II Sebastian Pauli Department of Mathematics and

Factoring Polynomials over Local Fields II Sebastian Pauli Department of Mathematics and Statistics University of North Carolina at Greensboro Sebastian Pauli (UNC Greensboro) Factoring Polynomials over Local Fields II 1 / 20 Polynomial

765 views • 44 slides
Factor Theorem MHF4U: Advanced Functions Example Divide f ( x ) = x 3 + 4 x 2 + x 6 by x

Factor Theorem MHF4U: Advanced Functions Example Divide f ( x ) = x 3 + 4 x 2 + x 6 by x

p o l y n o m i a l e q u a t i o n s & i n e q u a l i t i e s p o l y n o m i a l e q u a t i o n s & i n e q u a l i t i e s Factor Theorem MHF4U: Advanced Functions Example Divide f ( x ) = x 3 + 4 x 2 + x 6 by x 1. 1 4 1

72 views • 3 slides
Direct Methods for Solving Linear Systems Pivoting Strategies Numerical Analysis (9th Edition) R

Direct Methods for Solving Linear Systems Pivoting Strategies Numerical Analysis (9th Edition) R

Direct Methods for Solving Linear Systems Pivoting Strategies Numerical Analysis (9th Edition) R L Burden & J D Faires Beamer Presentation Slides prepared by John Carroll Dublin City University c 2011 Brooks/Cole, Cengage Learning

1.28k views • 85 slides
Fast Mining of Massive Tabular Data via Approximate Distance Computations Graham Cormode, Piotr

Fast Mining of Massive Tabular Data via Approximate Distance Computations Graham Cormode, Piotr

Fast Mining of Massive Tabular Data via Approximate Distance Computations Graham Cormode, Piotr Indyk, Nick Koudas, S. Muthukrishnan Tabular Data Much data is stored in tables: Cellphone traffic IP traffic between source and

401 views • 20 slides
Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components The eigenvalues and

Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components The eigenvalues and

Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components The eigenvalues and eigenvectors of a square matrix play a key role in some important operations in statistics. In particular, they are intimately connected with the

376 views • 25 slides
Factoring the Characteristic Polynomial Joshua Hallam and Bruce Sagan Department of Mathematics

Factoring the Characteristic Polynomial Joshua Hallam and Bruce Sagan Department of Mathematics

Factoring the Characteristic Polynomial Joshua Hallam and Bruce Sagan Department of Mathematics Michigan State University East Lansing, MI 48824-1027 www.math.msu.edu/ sagan June 21, 2014 Motivating Example Quotient Posets The Standard

1.73k views • 147 slides
Computation of a 768-bit prime field discrete logarithm C. Diem, T. Kleinjung, A. K. Lenstra, C.

Computation of a 768-bit prime field discrete logarithm C. Diem, T. Kleinjung, A. K. Lenstra, C.

Computation of a 768-bit prime field discrete logarithm C. Diem, T. Kleinjung, A. K. Lenstra, C. Priplata, C. Stahlke EPF Lausanne (Switzerland), Universit at Leipzig (Germany) 1 / 11 Paris, 1st April 1776 2 / 11 Paris, 1st April 1776

694 views • 30 slides
A. Operations with algebraic Algebra practice part 1 expressions 3 4 A. Operations with

A. Operations with algebraic Algebra practice part 1 expressions 3 4 A. Operations with

1 2 A. Operations with algebraic Algebra practice part 1 expressions 3 4 A. Operations with algebraic expressions A. Operations with algebraic expressions What is an algebraic expression? What is an algebraic expression? Example: An

459 views • 5 slides

More recommend