SLID IDE : In In Defen ense e of Smart Algorithms over er Ha - - PowerPoint PPT Presentation

SLID IDE : In In Defen ense e of Smart Algorithms over er Ha Hardware e Accel eler eration for Large-Scale e Deep eep Lea earning System ems

Beidi Chen

Collaborators: Tharun Medini*, James Farwell† , Sameh Gobriel†, Charlie Tai †, Anshumali Shrivastava*

* Rice University, †Intel

MLSys 2020

Ou Our SL

SLIDE Sy System (C++ from scratch) on a 44 44 co core CPU be beats TF on n V100 (1 (1 ho hour urs vs 3. 3.5 ho hour urs). ). 100+ million parameter networks. TF on same CPU is 16 hours with all HPC optimization (Intel MKL-DNN).

3.5x faster on CPU than TF on V100 (Log Scale in Time)

2

Th The Age ge of Large ge Networks

• More Data
• Large Models
• Tons of Engineering
• Backpropagation

(Aka Simple Gradient Descent)

3

Giant Matrix Multiplication for every data point in each epoch (Forward + Backward)

Fully Fully Connec nnected ed NN

1 2 3 4 1 2 3 1 2 3

Input Hidden 1 Hidden 2

1 5

… … …

9

Output

… … …

𝑔(𝑋$𝑦)

4

Ch Challenges

Do we really need all the computations? No!! Good News: Only high activations are important

• Sampling few neurons in proportion of activations is enough (Adaptive Dropouts)

(Ba et al. Neurips 13 , Makhzani et al. Neurips 15)

• Relu filtered negative activations (50% sparsity by design)
• Softmax

Bad News: We need to compute all to identify (or sample) the high activation neurons. NO SAVINGS

5

Th The Fundamental Sampling Puzzle

Given N fixed sampling weights, 𝑥(, 𝑥*, … , 𝑥, .

• Task: Sample 𝑦- with probability 𝑥-
• Cost of 1 sample 𝑃(𝑂).
• Cost of K samples 𝑃(𝑂).

Given N time-varying sampling weights (activations) 𝑥(

0, 𝑥* 0, … , 𝑥, 0 .

• Task: At time t, sample 𝑦- with probability 𝑥-
• Cost of sampling O(N), at every time t.
• Last Few years of work in Locality Sensitive Hashing: If 𝑥-

0 = 𝑔(𝑡𝑗𝑛(𝜄0, 𝑦-)), for a

specific set of f and sim, then 𝑃(1) every time after and initial preprocessing cost of 𝑃(𝑂).

6

Te Textbook Hashing (Dictionary)

Hashing: Function h that maps a given data point (𝑦 ∈ 𝑆9) to an integer key ℎ ∶ 𝑆9 ↦ 0, 1, 2, … , 𝑂 . ℎ(𝑦) serves as a discrete fingerprint. Property (Ideal Hash Functions):

• If x = 𝑧, then ℎ 𝑦 = ℎ(𝑧)
• If x ≠ 𝑧, then ℎ 𝑦 ≠ ℎ(𝑧)

7

Pr Probabilistic Fingerprinting (Hashing) (late ate 90s)

Hashing: Function (Randomized) h that maps a given data point (𝑦 ∈ 𝑆9) to an integer key ℎ ∶ 𝑆9 ↦ 0, 1, 2, … , 𝑂 . ℎ(𝑦) serves as a discrete fingerprint. Locality Sensitive Property:

• If x = 𝑧 S𝑗𝑛(𝑦, 𝑧) is high, then ℎ 𝑦 = ℎ 𝑧

Pr(ℎ 𝑦 = ℎ 𝑧 ) is high

• If x ≠ 𝑧 S𝑗𝑛(𝑦, 𝑧) is low, then ℎ 𝑦 ≠ ℎ(𝑧) Pr(ℎ 𝑦 = ℎ 𝑧 ) is low

Likely Unlikely

8

Ex Exampl ple 1: Si : Signed R Random P m Project ction ( (SR SRP)

monotonic in 𝜄 +

• +

Pr(h(x) = h(y)) = 1 − 1 π cos−1(θ) X Y X Y H1 H2

A classical result from Goemans-Williamson (95) 9

Ex Exampl ple 2: ( : (De Densif ifie ied) ) Wi Winne nner Ta Take Al All

Original Vectors: WTA hash codes: (ICCV 2011) DWTA hash codes: (UAI 2018)

K=3

Yagnik (ICCV11), Chen and Shrivastava (UAI 18) 10

Pr Probabilisti tic Has Hash h Tables ables

Given: 𝑔 is monotonic. Prh ⇥ h(x) = h(y) ⇤ = f(sim(x, y)),

• Given query, if ℎ( 𝑟 = 11

and ℎ* 𝑟 = 01, then probe bucket with index 1101. It is a good bucket !!

• (Locality Sensitive) ℎ- 𝑟 =

ℎ-(𝑦) noisy indicator of high similarity.

• Doing better than random !!

11

LSH LSH f for Se

• r Search

ch ( (Known)

Theory

• Super-linear 𝑃(𝑂(FG) memory
• Sub-linear query time, O(𝑂G)
• 𝜍 < 1 but generally large (close to 1) and often hard to determine

Practical Issues

• Needs lot of hash tables and distance computations for good accuracy on

near-neighbors

• Buckets can be quite heavy. Poor randomness, or unfavorable data

distributions

12

New Vi View: Data Structures for Efficient Sampling!

Is LSH really a search algorithm?

• Given the query 𝜄0, LSH samples 𝑦- from the dataset, with probability

𝑥-

0 = 1 − 1 − p xL, 𝜄0 M N

• 𝑥-

0 is proportional to p xL, 𝜄0 M and the some similarity of xL, 𝜄0

• LSH is considered a black box for nearest-neighbor search. It is not!!

13

LSH LSH a as Sa Samp mplers

We can pre-process the dataset D, such that

• Given any query q, we can sample 𝑦 ∈ 𝐸 with probability

𝐷𝑝𝑜𝑡𝑢× 1 − 1 − 𝑞 𝑟, 𝑦 V W in KL hash computation and L bucket probes.

• Even K = 1, L =1 is adaptive. So O(1) time adaptive.
• Adaptive: x is sampled with higher probability than y
• if and only if sim(q,x) > sim(q,y)

We can exactly compute the sampling probability.

• Const = No of elements sampled/ No of elements in Buckets

(Chen et al. NeurIPS 2019)

Sufficient for Importance Sampling Estimations. Sampling cost O(1).

14

SLI SLIDE: Sub-LInear Deep learning Engine

Step 1 – Build the hash tables by processing the weights of the hidden layers (initialization). Subtlety: Neurons (vectors) in hash tables are not the data

• vectors. Reorganizing neurons.

1 2 3 4 5 1 2 3 4 1 2 3 4 1 5

Input Hidden 1 Hidden 2

… … …

9

Output

H2 1 |3 2 | 1,4 3 | 2

1 1

H1 1 |1 2 | 2,4 3 | 3

15

SLI SLIDE: Sub-LInear Deep learning Engine

Step 2 – Hash the input to any given layer using its randomized hash function. 1 2 3 4 5 1 2 3 4 1 2 3 4 1 5

Input Hidden 1 Hidden 2

… … …

9

Output

H1 1 |1 2 | 2,4 3 | 3 H2 1 |3 2 | 1,4 3 | 2

2

16

SLI SLIDE: Sub-LInear Deep learning Engine

Step 3 – Query the hidden layer's hash table(s) for the active set using integer fingerprint. Sample neurons in proportion to their activations. 1 2 3 4 5 1 2 3 4 1 2 3 4 1 5

Input Hidden 1 Hidden 2

… … …

9

Output

H1 1 |1 2 | 2,4 3 | 3 H2 1 |3 2 | 1,4 3 | 2

3

17

SLI SLIDE: Sub-LInear Deep learning Engine

Step 4 – Perform forward and back propagation only on the nodes in the active set. Computation is in the same order

• f active neurons.

1 2 3 4 5 1 2 3 4 1 2 3 4 1 5

Input Hidden 1 Hidden 2

… … …

9

Output

H1 1 |1 2 | 2,4 3 | 3 H2 1 |3 2 | 1,4 3 | 2

4

18

SLI SLIDE: Sub-LInear Deep learning Engine

Step 5 – Update hash tables by rehashing the updated node weights. Computation is in the same order

• f active neurons.

1 2 3 4 5 1 2 3 4 1 2 3 4 1 5

Input Hidden 1 Hidden 2

… … …

9

Output

H1 1 |1 2 | 2,4 3 | 3 H2 1 |3 2 | 1,4 3 | 2

5 5

19

We We can go very sparse if Adaptive

• 2 Hidden Layers
• 1000 Nodes Per Layer
• Reduce both training

and inference cost by 95%!

• Significantly more for

larger networks. (The wider the better)

20

Sp Sparsity + + R Randomn

• mness à As

Asynchronous Updates

• 3 Hidden Layers
• 1000 Nodes Per Layer

21

Les ess Computations + Asynchronous Parallel elism

• Each update is computationally very small (100x+ reduction in

computation and energy)

• Updates are near-independent, very low chance of conflict. Hence,

parallel SGD!

22

SLI SLIDE: Sub-LInear Deep learning Engine

1 2 3 4 5 1 2 3 4 1 2 3 4

Input Hidden 1 Hidden 2

1 5

… … …

9

Output

Layer

Active Inputs 1 1 …… Active Inputs

0.1 0.2 0.5

…… Activation for each Inputs

0.3 0.8 0.7

…… Accumulated Gradients

• 0.3

… Weights

0.8

• 0.5

BatchSize

Neuron Network

Hash Table 1 Hash Table L

00 00 00 … 11 … … … … … 00 01 10 … 11

ℎ"

"

ℎ#

"

…

Buckets

… …

Empty

… …

…

1 9 2

00 00 00 … 11 … … … … … 00 01 10 … 11

ℎ"

$

ℎ#

$

…

Buckets

… …

Empty

… …

1 9 5

1 2

…

Previous Layer Size

23

(Extreme sparsity and randomness in gradient updates) Thanks to the theory of HOGWILD!

(Recht et al. Neurips 11)

Pa Parallelism with OpenMP

Parallel across training samples in a batch

1 1 …… Active Inputs

0.1 0.2 0.5

…… Activation for each Inputs

0.3 0.8 0.7

…… Accumulated Gradients Weights

Batchsize

Node

• 0.3

0.8

• 0.5

……

24

Fle Flexible ible cho hoic ices es of Has Hash h Func Functio tions ns

SLIDE supports four different LSH hash functions

• Simhash (cosine similarity)
• Winner-take-all Hashing (order)
• Densified Winner-take-all Hashing (for sparse data)∗
• Minhash (jaccard similarity)

Easily add more!

25

• Vanilla sub-sampling:
• choose sub-samples uniformly
• Top K sub-sampling:
• rank samples and choose topk
• Hard Thresholding sub-sampling:
• choose sub-samples that occur > threshold times

De Desig ign Ch Choi

• ices fo

for Sp Speed

26

Mi Micro-Ar Architecture Op Optimization

Cache Optimization Transparent Hugepages Vector Processing Software Pipelining and Prefetching

27

Look Looks G Good

• od on
• n P
• Paper. D

Doe

• es i

it ch change a anything?

Baseline

State-of-the-art optimized Implementations

• TF on Intel Xeon E5-2699A v4 @ 2.40GHz CPU (FMA,AVX, AVX2, SSE4.2)
• TF on NVIDIA Tesla V100 (32GB)

VS.

SLIDE on Intel Xeon E5-2699A v4 @ 2.40GHz CPU (FMA,AVX, AVX2, SSE4.2)

• TF on NVIDIA Tesla V100 (32GB)

28

Da Datas asets

29

Pe Performance

30

Pe Performance compared to sampled so soft ftmax

31

Pe Performance @ Dif Different Ba Batchsizes

32

As Asynchronous Parallelism gets best scalability

33

Ine Ineffic icienc iency Diag Diagnosis is

34 0.1 0.2 0.3 0.4 0.5 0.6 8 Threads 16 Threads 32 Threads Tensorflow-CPU Inefficiencies Ratio in CPU Usage Front-End Bound Memory Bound Retiring Bound Core Bound 0.1 0.2 0.3 0.4 0.5 0.6 8 Threads 16 Threads 32 Threads SLIDE Inefficiencies Ratio in CPU Usage Front-End Bound Memory Bound Retiring Bound Core Bound

Im Impac pact of

• f Hug

HugeP ePag ages es

35

Con Conclusion

• n: F

From

• m M

Matrix M Multiplication

• n t

to (f

• (few) H

) Hash L Look

• okups
• Standard
• Operation
• Matrix Multiply
• Pros
• Hardware Support
• Cons
• Expensive O(N^3)
• Can only scale with hardware.
• Energy
• SLIDE
• Operations
• Compute Random Hashes of Data
• Hash lookups, Sample and Update.

(Decades of work in Databases)

• Very Few Multiplication (100x+ reduction)
• Pros
• Energy (IoT), Latency
• Asynchronous Parallel Gradient updates
• Simple Hash Tables
• Larger Network à More Savings
• Cons
• Random Memory Access (but parallel SGD)

36

Futur Future Wo Work

• Distributed SLIDE
• SLIDE on more complex architectures like CNN/RNN

37

Re References

[1] Beidi Chen, Tharun Medini, Anshumali Shrivastava “SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems”. Proceedings of the 3rd MLSys Conference (2020). [2] Ryan Spring, Anshumali Shrivastava. "Scalable and sustainable deep learning via randomized hashing". Proceedings of the 23rd ACM SIGKDD (2017). [3] Makhzani, A. and Frey, B. J. “Winner-take-all autoencoders”. In Advances in neural information processing systems (2015). [4] Beid Chen, Anshumali Shrivastava. “Densified Winner Take All (WTA) Hashing for Sparse Datasets”. In Uncertainty in artificial intelligence (2018). [5] Beidi Chen, Yingchen Xu, and Anshumali Shrivastava. "LGD: Fast and Accurate Stochastic Gradient Estimation”. In Neurips, Dec. 2019. Vancouver. [6] Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. “Hogwild: A lock-free approach to parallelizing stochastic gradient descent”. In Advances in neural information processing systems (2011).

38

Th Thanks!!! We Welcome to to st stop by by Po Poster #7 #7

39