Covariance Matrices & All-pairs Similarity Reza Zadeh - - PowerPoint PPT Presentation

▶
covariance matrices all pairs similarity
SMART_READER_LITE
LIVE PREVIEW

Covariance Matrices & All-pairs Similarity Reza Zadeh - - PowerPoint PPT Presentation

Jun 11, 2023 46 likes •420 views

Covariance Matrices and All-pairs similarity Covariance Matrices & All-pairs Similarity Reza Zadeh Introduction Reza Zadeh First Pass DIMSUM Analysis Experiments Spark More Results April 2015, Stanford DAO Reza Zadeh (Stanford)

slide-1
SLIDE 1

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Covariance Matrices & All-pairs Similarity

Reza Zadeh April 2015, Stanford DAO

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 1 / 34

slide-2
SLIDE 2

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Notation for matrix A Given m × n matrix A, with m ≫ n. A =      a1,1 a1,2 · · · a1,n a2,1 a2,2 · · · a2,n . . . . . . ... . . . am,1 am,2 · · · am,n      A is tall and skinny, example values m = 1012, n = {104, 106}. A has sparse rows, each row has at most L nonzeros. A is stored across hundreds of machines and cannot be streamed through a single machine.

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 2 / 34

slide-3
SLIDE 3

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Computing ATA We compute ATA. ATA is n × n, considerably smaller than A. ATA is dense. Holds dot products between all pairs of columns of A.

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 3 / 34

slide-4
SLIDE 4

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Guarantees There is a knob γ which can be turned to preserve similarities and singular values. Paying O(nLγ) communication cost and O(γ) computation cost. With a low setting of γ, preserve similar entries of ATA (via Cosine, Dice, Overlap, and Jaccard similarity). With a high setting of γ, preserve singular values of ATA.

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 4 / 34

slide-5
SLIDE 5

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Computing All Pairs of Cosine Similarities We have to find dot products between all pairs of columns of A We prove results for general matrices, but can do better for those entries with cos(i, j) ≥ s Cosine similarity: a widely used definition for “similarity" between two vectors cos(i, j) = cT

i cj

||ci||||cj|| ci is the i′th column of A

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 5 / 34

slide-6
SLIDE 6

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Example matrix Rows: users. Columns: movies.

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 6 / 34

slide-7
SLIDE 7

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Distributed Computing Environment With such large datasets, we must use many machines. Algorithm code available in Spark and Scalding. Described with Maps and Reduces so that the framework takes care of distributing the computation.

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 7 / 34

slide-8
SLIDE 8

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Naive Implementation

1

Given row ri, Map with NaiveMapper (Algorithm 1)

2

Reduce using the NaiveReducer (Algorithm 2) Algorithm 1 NaiveMapper(ri) for all pairs (aij, aik) in ri do Emit ((j, k) → aijaik) end for Algorithm 2 NaiveReducer((i, j), v1, . . . , vR)

  • utput cT

i cj → R i=1 vi

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 8 / 34

slide-9
SLIDE 9

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Analysis for First Pass Very easy analysis 1) Shuffle size: O(mL2) 2) Largest reduce-key: O(m) Both depend on m, the larger dimension, and are intractable for m = 1012, L = 100. We’ll bring both down via clever sampling Assuming column norms are known or estimates available

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 9 / 34

slide-10
SLIDE 10

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Dimension Independent Matrix Square using MapReduce Algorithm 3 DIMSUMMapper(ri) for all pairs (aij, aik) in ri do With probability min

  • 1, γ

1 ||cj||||ck||

  • emit ((j, k) → aijaik)

end for Algorithm 4 DIMSUMReducer((i, j), v1, . . . , vR) if

γ ||ci||||cj|| > 1 then

  • utput bij →

1 ||ci||||cj||

R

i=1 vi

else

  • utput bij → 1

γ

R

i=1 vi

end if

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 10 / 34

slide-11
SLIDE 11

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Analysis for DIMSUM The algorithm outputs bij, which is a matrix of cosine similarities, call it B. Four things to prove:

1

Shuffle size: O(nLγ)

2

Largest reduce-key: O(γ)

3

The sampling scheme preserves similarities when γ = Ω(log(n)/s)

4

The sampling scheme preserves singular values when γ = Ω(n/ǫ2)

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 11 / 34

slide-12
SLIDE 12

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Shuffle size for DIMSUM Theorem For {0, 1} matrices, the expected shuffle size for DIMSUMMapper is O(nLγ). Proof. The expected contribution from each pair of columns will constitute the shuffle size:

n

  • i=1

n

  • j=i+1

#(ci,cj)

  • k=1

Pr[DIMSUMEmit(ci, cj)] =

n

  • i=1

n

  • j=i+1

#(ci, cj)Pr[DIMSUMEmit(ci, cj)]

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 12 / 34

slide-13
SLIDE 13

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Shuffle size for DIMSUM Proof. ≤

n

  • i=1

n

  • j=i+1

γ #(ci, cj)

  • #(ci)
  • #(cj)

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 13 / 34

slide-14
SLIDE 14

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Shuffle size for DIMSUM Proof. ≤

n

  • i=1

n

  • j=i+1

γ #(ci, cj)

  • #(ci)
  • #(cj)

(by AM-GM) ≤ γ 2

n

  • i=1

n

  • j=i+1

#(ci, cj)( 1 #(ci) + 1 #(cj))

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 13 / 34

slide-15
SLIDE 15

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Shuffle size for DIMSUM Proof. ≤

n

  • i=1

n

  • j=i+1

γ #(ci, cj)

  • #(ci)
  • #(cj)

(by AM-GM) ≤ γ 2

n

  • i=1

n

  • j=i+1

#(ci, cj)( 1 #(ci) + 1 #(cj)) ≤ γ

n

  • i=1

1 #(ci)

n

  • j=1

#(ci, cj)

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 13 / 34

slide-16
SLIDE 16

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Shuffle size for DIMSUM Proof. ≤

n

  • i=1

n

  • j=i+1

γ #(ci, cj)

  • #(ci)
  • #(cj)

(by AM-GM) ≤ γ 2

n

  • i=1

n

  • j=i+1

#(ci, cj)( 1 #(ci) + 1 #(cj)) ≤ γ

n

  • i=1

1 #(ci)

n

  • j=1

#(ci, cj) ≤ γ

n

  • i=1

1 #(ci)L#(ci) = γLn

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 13 / 34

slide-17
SLIDE 17

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Shuffle size for DIMSUM O(nLγ) has no dependence on the dimension m, this is the heart of DIMSUM. Happens because higher magnitude columns are sampled with lower probability: γ 1 ||c1||||c2||

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 14 / 34

slide-18
SLIDE 18

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Shuffle size for DIMSUM For matrices with real entries, we can still get a bound Let H be the smallest nonzero entry in magnitude, after all entries of A have been scaled to be in [−1, 1] E.g. for {0, 1} matrices, we have H = 1 Shuffle size is bounded by O(nLγ/H2)

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 15 / 34

slide-19
SLIDE 19

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Largest reduce key for DIMSUM Each reduce key receives at most γ values (the

  • versampling parameter)

Immediately get that reduce-key complexity is O(γ) Also independent of dimension m. Happens because high magnitude columns are sampled with lower probability.

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 16 / 34

slide-20
SLIDE 20

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Correctness Since higher magnitude columns are sampled with lower probability, are we guaranteed to obtain correct results w.h.p.?

  • Yes. By setting γ correctly.

Preserve similarities when γ = Ω(log(n)/s) Preserve singular values when γ = Ω(n/ǫ2)

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 17 / 34

slide-21
SLIDE 21

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Correctness Theorem Let A be an m × n tall and skinny (m > n) matrix. If γ = Ω(n/ǫ2) and D a diagonal matrix with entries dii = ||ci||, then the matrix B output by DIMSUM satisfies, ||DBD − ATA||2 ||ATA||2 ≤ ǫ with probability at least 1/2. Relative error guaranteed to be low with constant probability.

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 18 / 34

slide-22
SLIDE 22

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Proof Uses Latala’s theorem, bounds 2nd and 4th central moments of entries of B. Really need extra power of moments.

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 19 / 34

slide-23
SLIDE 23

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Latala’s Theorem Theorem (Latala’s theorem). Let X be a random matrix whose entries xij are independent centered random variables with finite fourth moment. Denoting ||X||2 as the matrix spectral norm, we have E ||X||2 ≤ C[max

i

 

j

E x2

ij

 

1/2

+ max

j

  • i

E x2

ij

1/2 +  

i,j

E x4

ij

 

1/4

].

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 20 / 34

slide-24
SLIDE 24

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Proof Prove two things E[(bij − Ebij)2] ≤ 1

γ (easy)

E[(bij − Ebij)4] ≤ 2

γ2 (not easy)

Details in paper.

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 21 / 34

slide-25
SLIDE 25

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Correctness Theorem For any two columns ci and cj having cos(ci, cj) ≥ s, let B be the output of DIMSUM with entries bij = 1

γ

m

k=1 Xijk with

Xijk as the indicator for the k’th coin in the call to

  • DIMSUMMapper. Now if γ = Ω(α/s), then we have,

Pr

  • ||ci||||cj||bij > (1 + δ)[ATA]ij
  • ≤
  • eδ

(1 + δ)(1+δ) α and Pr

  • ||ci||||cj||bi,j < (1 − δ)[ATA]ij
  • < exp(−αδ2/2)

Relative error guaranteed to be low with high probability.

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 22 / 34

slide-26
SLIDE 26

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Correctness Proof. In the paper Uses standard concentration inequality for sums of indicator random variables. Ends up requiring that the oversampling parameter γ be set to γ = log(n2)/s = 2 log(n)/s.

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 23 / 34

slide-27
SLIDE 27

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Observations DIMSUM helpful when there are some popular columns e.g. The Netflix Matrix (some columns way more popular than others) Power-law columns are effectively neutralized

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 24 / 34

slide-28
SLIDE 28

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

In practice Forget about theoretical settings for γ Crank up γ until application works Estimates for ||ci|| can be used, expectations still hold, but concentration isn’t guaranteed If using for singular values, watch for ill-conditioned matrices

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 25 / 34

slide-29
SLIDE 29

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Experiments Large scale production live at twitter.com

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 26 / 34

slide-30
SLIDE 30

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Experiments

Figure : Y-axis ranges from 0 to 100s of terabytes

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 27 / 34

slide-31
SLIDE 31

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Implementation

Figure : Widely distributed with Spark as of version 1.2

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 28 / 34

slide-32
SLIDE 32

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Other Similarity Measures Picking out similar columns work for some other similarity measures.

Similarity Definition Shuffle Size Reduce-key size Cosine

#(x,y)

√

#(x)√ #(y)

O(nL log(n)/s) O(log(n)/s) Jaccard

#(x,y) #(x)+#(y)−#(x,y)

O((n/s) log(n/s)) O(log(n/s)/s) Overlap

#(x,y) min(#(x),#(y))

O(nL log(n)/s) O(log(n)/s) Dice

2#(x,y) #(x)+#(y)

O(nL log(n)/s) O(log(n)/s)

Table : All sizes are independent of m, the dimension.

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 29 / 34

slide-33
SLIDE 33

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Locality Sensitive Hashing MinHash from the Locality-Sensitive-Hashing family can have its vanilla implementation greatly improved by DIMSUM. Another set of theorems for shuffle size and correctness in DISCO paper. stanford.edu/~rezab/papers/disco.pdf

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 30 / 34

slide-34
SLIDE 34

Covariance Matrices and All-pairs similarity Reza Zadeh Introduction First Pass DIMSUM Analysis Experiments Spark More Results

Conclusion Consider DIMSUM if you ever need to compute ATA for large sparse A Many more experiments and results in paper at stanford.edu/~rezab

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 31 / 34

slide-35
SLIDE 35

Covariance Matrices and All-pairs similarity Reza Zadeh

Combiners All bounds are without combining: can only get better with combining For similarities, DIMSUM (without combiners) beats naive with combining outright For singular values, DIMSUM (without combiners) beats naive with combining if the number of machines is large (e.g. 1000) DIMSUM with combining empirically beats naive with combining Difficult to analyze combiners since they happen at many levels. Optimizations break models. DIMSUM with combiners is best of both.

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 32 / 34

slide-36
SLIDE 36

Covariance Matrices and All-pairs similarity Reza Zadeh

Combiners With k machines DIMSUM shuffle with combiner: O(min(nLγ, kn2)) DIMSUM reduce-key with combiner: O(min(γ, k)) Naive shuffle with combiner: O(kn2) Naive reduce-key with combiner: O(k) DIMSUM with combiners is best of both.

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 33 / 34

slide-37
SLIDE 37

Covariance Matrices and All-pairs similarity Reza Zadeh

Experiments

2 4 6 8 10 12 14 16 18 0.2 0.4 0.6 0.8 1

log(p / ε)

DISCO Cosine shuffle size vs accuracy tradeoff DISCO Shuffle / Naive Shuffle 2 4 6 8 10 12 14 16 18 0.5 1 1.5 2 2.5 avg relative err DISCO Shuffle / Naive Shuffle avg relative err

Figure : As γ = p/ǫ increases, shuffle size increases and error

  • decreases. There is no thresholding for highly similar pairs here.

Reza Zadeh (Stanford) Covariance Matrices and All-pairs similarity April 2014 34 / 34