COMP9313: Big Data Management High Dimensional Similarity Search - - PowerPoint PPT Presentation

COMP9313: Big Data Management

High Dimensional Similarity Search

• Problem Definition:
• Given a query 𝑟 and dataset 𝐸, find o ∈ 𝐸, where

𝑝 is similar to 𝑟

• Two types of similarity search
• Range search:
• 𝑒𝑗𝑡𝑢 𝑝, 𝑟 ≤ τ
• Nearest neighbor search
• 𝑒𝑗𝑡𝑢 𝑝∗, 𝑟 ≤ 𝑒𝑗𝑡𝑢 𝑝, 𝑟 , ∀𝑝 ∈ 𝐸
• Top-k version
• Distance/similarity function varies
• Euclidean, Jaccard, inner product, …
• Classic problem, with mutual solutions

2

Similarity Search

𝑟 𝑝∗ 𝜐

• Applications and relationship to Big Data
• Almost every object can be and has been

represented by a high dimensional vector

• Words, documents
• Image, audio, video
• …
• Similarity search is a fundamental process in

information retrieval

• E.g., Google search engine, face recognition system, …
• High Dimension makes a huge difference!
• Traditional solutions are no longer feasible
• This lecture is about why and how
• We focus on high dimensional vectors in Euclidean space

3

High Dimensional Similarity Search

Similarity Search in Low Dimensional Space

4

Similarity Search in One Dimensional Space

• Just numbers, use binary search, binary

search tree, B+ Tree…

• The essential idea behind: objects can be

sorted

Similarity Search in Two Dimensional Space

• Why binary search no longer works?
• No order!
• Voronoi diagram

Euclidean distance Manhattan distance

• Partition based algorithms
• Partition data into “cells”
• Nearest neighbors are in the same cell with query or

adjacent cells

• How many “cells” to probe on 3-dimensional

space?

7

Similarity Search in Two Dimensional Space

• Triangle inequality
• 𝑒𝑗𝑡𝑢 𝑦, 𝑟 ≤ 𝑒𝑗𝑡𝑢 𝑦, 𝑧 + 𝑒𝑗𝑡𝑢 𝑧, 𝑟
• Orchard’s Algorithm
• for each 𝑦 ∈ 𝐸, create a list of points in increasing
• rder of distance to 𝑦
• given query 𝑟, randomly pick a point 𝑦 as the

initial candidate (i.e., pivot 𝑞), compute 𝑒𝑗𝑡𝑢 𝑞, 𝑟

• walk along the list of 𝑞, and compute the

distances to 𝑟. If found 𝑧 closer to 𝑟 than 𝑞, then use 𝑧 as the new pivot (e.g., 𝑞 ← 𝑧).

• repeat the procedure, and stop when
• 𝑒𝑗𝑡𝑢 𝑞, 𝑧 > 2 ⋅ 𝑒𝑗𝑡𝑢 𝑞, 𝑟

8

Similarity Search in Metric Space

• Orchard’s Algorithm, stop when 𝑒𝑗𝑡𝑢 𝑞, 𝑧 >

2 ⋅ 𝑒𝑗𝑡𝑢 𝑞, 𝑟 2 ⋅ 𝑒𝑗𝑡𝑢 𝑞, 𝑟 < 𝑒𝑗𝑡𝑢 𝑞, 𝑧 and 𝑒𝑗𝑡𝑢 𝑞, 𝑧 ≤ 𝑒𝑗𝑡𝑢 𝑞, 𝑟 + 𝑒𝑗𝑡𝑢 𝑧, 𝑟 ⇒ 2 ⋅ 𝑒𝑗𝑡𝑢 𝑞, 𝑟 < 𝑒𝑗𝑡𝑢 𝑞, 𝑟 + 𝑒𝑗𝑡𝑢 𝑧, 𝑟 ⇔ 𝑒𝑗𝑡𝑢 𝑞, 𝑟 < 𝑒𝑗𝑡𝑢 𝑧, 𝑟

• Since the list of 𝑞 is in increasing order of

distance to 𝑞, 𝑒𝑗𝑡𝑢 𝑞, 𝑧 > 2 ⋅ 𝑒𝑗𝑡𝑢 𝑞, 𝑟 hold for all the rest 𝑧’s.

9

Similarity Search in Metric Space

None of the Above Works in High Dimensional Space!

10

• Refers to various phenomena that arise in

high dimensional spaces that do not occur in low dimensional settings.

• Triangle inequality
• The pruning power reduces heavily
• What is the volume of a high dimensional

“ring” (i.e., hyperspherical shell)?

• 𝑊𝑠𝑗𝑜𝑕 𝑥=1,𝑒=2

𝑊𝑐𝑏𝑚𝑚 𝑠=10,𝑒=2 = 29%

• 𝑊𝑠𝑗𝑜𝑕 𝑥=1,𝑒=100

𝑊𝑐𝑏𝑚𝑚 𝑠=10,𝑒=100 = 99.997%

11

Curse of Dimensionality

• There is no sub-linear solution to find the

exact result of a nearest neighbor query

• So we relax the condition
• approximate nearest neighbor search (ANNS)
• allow returned points to be not the NN of query
• Success: returns the true NN
• use success rate (e.g., percentage of succeed

queries) to evaluate the method

• Hard to bound the success rate

12

Approximate Nearest Neighbor Search in High Dimensional Space

• Success: returns 𝑝 such that
• 𝑒𝑗𝑡𝑢 𝑝, 𝑟 ≤ 𝑑 ⋅ 𝑒𝑗𝑡𝑢(𝑝∗, 𝑟)
• Then we can bound the success

probability

• Usually noted as 1 − δ
• Solution: Locality Sensitive Hashing

(LSH)

13

c-approximate NN Search

𝑟 𝑝∗ 𝑑𝑠 𝑠

• Hash function
• Index: Map data/objects to values (e.g., hash key)
• Same data ⇒ same hash key (with 100% probability)
• Different data ⇒ different hash keys (with high probability)
• Retrieval: Easy to retrieve identical objects (as

they have the same hash key)

• Applications: hash map, hash join
• Low cost
• Space: 𝑃(𝑜)
• Time: 𝑃(1)
• Why it cannot be used in nearest neighbor search?
• Even a minor difference leads to totally different hash keys

14

Locality Sensitive Hashing

• Index: make the hash functions error tolerant
• Similar data ⇒ same hash key (with high probability)
• Dissimilar data ⇒ different hash keys (with high probability)
• Retrieval:
• Compute the hash key for the query
• Obtain all the data has the same key with query (i.e.,

candidates)

• Find the nearest one to the query
• Cost:
• Space: 𝑃(𝑜)
• Time: 𝑃 1 + 𝑃(|𝑑𝑏𝑜𝑒|)
• It is not the real Locality Sensitive Hashing!
• We still have several unsolved issues…

15

Locality Sensitive Hashing

• Formal definition:
• Given point 𝑝1, 𝑝2, distance 𝑠

1, 𝑠 2, probability

𝑞1, 𝑞2

• An LSH function ℎ(⋅) should satisfy
• Pr ℎ 𝑝1 = ℎ 𝑝2

≥ 𝑞1, if 𝑒𝑗𝑡𝑢 𝑝1, 𝑝2 ≤ 𝑠

1

• Pr ℎ 𝑝1 = ℎ 𝑝2

≤ 𝑞2, if 𝑒𝑗𝑡𝑢 𝑝1, 𝑝2 > 𝑠

2

• What is ℎ ⋅ for a given distance/similarity

function?

• Jaccard similarity
• Angular distance
• Euclidean distance

16

LSH Functions

• Each data object is a set
• 𝐾𝑏𝑑𝑑𝑏𝑠𝑒 𝑇1, 𝑇2 =

|𝑇𝑗∩𝑇𝑘| |𝑇𝑗∪𝑇𝑘|

• Randomly generate a global order for all the

elements in C =ڂ1

𝑜 𝑇𝑗

• Let ℎ(𝑇) be the minimal member of 𝑇 with

respect to the global order

• For example, 𝑇 = {𝑐, 𝑑, 𝑓, ℎ, 𝑗}, we use inversed

alphabet order, then re-ordered 𝑇 = {𝑗, ℎ, 𝑓, 𝑑, 𝑐}, hence ℎ 𝑇 = 𝑗.

17

MinHash - LSH Function for Jaccard Similarity

• Now we compute Pr ℎ 𝑇1 = ℎ 𝑇2
• Every element 𝑓 ∈ 𝑇1 ∪ 𝑇2 has equal chance

to be the first element among 𝑇1 ∪ 𝑇2 after re-

• rdering
• 𝑓 ∈ 𝑇1 ∩ 𝑇2 if and only if ℎ 𝑇1 = ℎ 𝑇2
• 𝑓 ∉ 𝑇1 ∩ 𝑇2 if and only if ℎ 𝑇1 ≠ ℎ 𝑇2
• Pr ℎ 𝑇1 = ℎ 𝑇2

=

|{𝑓𝑗|ℎ𝑗 𝑇1 =ℎ𝑗 𝑇2 }| |{𝑓𝑗}|

=

|𝑇𝑗∩𝑇𝑘| |𝑇𝑗∪𝑇𝑘| =

𝐾𝑏𝑑𝑑𝑏𝑠𝑒 𝑇1, 𝑇2

18

MinHash

• Each data object is a d dimensional vector
• 𝜄(𝑦, 𝑧) is the angle between 𝑦 and 𝑧
• Randomly generate a normal vector 𝑏, where

𝑏𝑗~𝑂(0,1)

• Let ℎ 𝑦; 𝑏 = sgn(𝑏𝑈𝑦)
• sgn o = ቊ 1; 𝑗𝑔 𝑝 ≥ 0

−1; 𝑗𝑔 𝑝 < 0

• 𝑦 lies on which side of 𝑏’s

corresponding hyperplane

19

SimHash – LSH Function for Angular Distance

• Now we compute Pr ℎ 𝑝1 = ℎ 𝑝2
• ℎ 𝑝1 ≠ ℎ 𝑝2 iff 𝑝1 and 𝑝2 are on different

sides of the hyperplane with 𝑏 as its normal vector

• Pr ℎ 𝑝1 = ℎ 𝑝2

= 1 − 𝜄

𝜌

20

SimHash

𝑝1 𝑝2 θ 𝑏 𝜄 𝜌 =

• Each data object is a d dimensional vector
• 𝑒𝑗𝑡𝑢 𝑦, 𝑧 =

σ1

𝑒 𝑦𝑗 − 𝑧𝑗 2

• Randomly generate a normal vector 𝑏, where

𝑏𝑗~𝑂(0,1)

• Normal distribution is 2-stable, i.e., if 𝑏𝑗~𝑂(0,1),

then σ1

𝑒 𝑏𝑗 ⋅ 𝑦𝑗 ~𝑂(0, 𝑦 2 2)

• Let ℎ 𝑦; 𝑏, 𝑐 =

𝑏𝑈𝑦+𝑐 𝑥

, where 𝑐~𝑉(0,1) and 𝑥 is user specified parameter

• Pr ℎ 𝑝1; 𝑏, 𝑐 = ℎ 𝑝2; 𝑏, 𝑐

= ׬

𝑥 1 𝑝1,𝑝2 𝑔 𝑞 𝑢 𝑝1,𝑝2

1 −

𝑢 𝑥 𝑒𝑢

• 𝑔

𝑞 ⋅ is the pdf of the absolute value of normal variable

21

p-stable LSH - LSH function for Euclidean distance

• Intuition of p-stable LSH
• Similar points have higher chance to be hashed

together

22

p-stable LSH

23

Pr ℎ 𝑦 = ℎ 𝑧 for different Hash Functions

MinHash SimHash p-stable LSH

• Hard to distinguish if two pairs have distances

close to each other

• Pr ℎ 𝑝1 = ℎ 𝑝2

≥ 𝑞1, if 𝑒𝑗𝑡𝑢 𝑝1, 𝑝2 ≤ 𝑠

1

• Pr ℎ 𝑝1 = ℎ 𝑝2

≤ 𝑞2, if 𝑒𝑗𝑡𝑢 𝑝1, 𝑝2 > 𝑠

2

• We also want to control where the drastic

change happens…

• Close to 𝑒𝑗𝑡𝑢(𝑝∗, 𝑟)
• Given range

24

Problem of Single Hash Function

• Recall for a single hash function, we have
• Pr ℎ 𝑝1 = ℎ 𝑝2

= 𝑞(𝑒𝑗𝑡𝑢(𝑝1, 𝑝2)), denoted as 𝑞𝑝1,𝑝2

• Now we consider two scenarios:
• Combine 𝑙 hashes together, using AND operation
• One must match all the hashes
• Pr 𝐼𝐵𝑂𝐸 𝑝1 = 𝐼𝐵𝑂𝐸 𝑝2

= 𝑞𝑝1,𝑝2

𝑙

• Combine 𝑚 hashes together, using OR operation
• One need to match at least one of the hashes
• Pr 𝐼𝑃𝑆 𝑝1 = 𝐼𝑃𝑆 𝑝2

= 1 − (1 − 𝑞𝑝1,𝑝2)𝑚

• Not match only when all the hashes don’t match

25

AND-OR Composition

• Example with minHash, 𝑙 = 5, 𝑚 = 5

26

AND-OR Composition

Pr 𝐼𝐵𝑂𝐸 𝑝1 = 𝐼𝐵𝑂𝐸 𝑝2 Pr 𝐼𝑃𝑆 𝑝1 = 𝐼𝑃𝑆 𝑝2 Pr ℎ 𝑝1 = ℎ 𝑝2

• Let ℎ𝑗,𝑘 be LSH functions, where 𝑗 ∈

1,2, … , 𝑚 , 𝑘 ∈ {1,2, … , 𝑙}

• Let 𝐼𝑗 𝑝 = [ℎ𝑗,1 𝑝 , ℎ𝑗,2 𝑝 , … , ℎ𝑗,𝑙 𝑝 ]
• super-hash
• 𝐼𝑗 𝑝1 = 𝐼𝑗 𝑝2 ⇔ ∀𝑘 ∈ 1,2, … , 𝑙 , ℎ𝑗,𝑘 𝑝1 = ℎ𝑗,𝑘 𝑝2
• Consider query 𝑟 and any data point 𝑝, 𝑝 is a

nearest neighbor candidate of 𝑟 if

• ∃𝑗 ∈ 1,2, … , 𝑚 , 𝐼𝑗 𝑝 = 𝐼𝑗 𝑟
• The probability of 𝑝 is a nearest neighbor

candidate of 𝑟 is

• 1 − (1 − 𝑞𝑟,𝑝

𝑙 )𝑚

27

AND-OR Composition in LSH

• 1 − (1 − 𝑞𝑟,𝑝

𝑙 )𝑚 changes with 𝑞𝑟,𝑝 , (𝑙 = 20, 𝑚 =

5)

• E.g., we are expected to retrieve 98.8% of the

data with Jaccard > 0.9

28

The Effectiveness of LSH

𝑞𝑟,𝑝 1 − (1 − 𝑞𝑟,𝑝

𝑙 )𝑚

0.2 0.002 0.4 0.050 0.6 0.333 0.7 0.601 0.8 0.863 0.9 0.988

• False Positive:
• returned data with dist o, q > 𝑠

2

• False Negative
• not returned data with dist o, q < 𝑠

1

• They can be controlled by carefully chosen 𝑙

and 𝑚

• It’s a trade-off between space/time and accuracy

29

False Positives and False Negatives

• Pre-processing
• Generate LSH functions
• minHash: random permutations
• simHash: random normal vectors
• p-stable: random normal vectors and random uniform values
• Index
• Compute 𝐼𝑗 𝑝 for each data object 𝑝, 𝑗 ∈ {1, … , 𝑚}
• Index 𝑝 using 𝐼𝑗 𝑝 as key in the 𝑗-th hash table
• Query
• Compute 𝐼𝑗 𝑟 for query 𝑟, 𝑗 ∈ {1, … , 𝑚}
• Generate candidate set 𝑝 ∃𝑗 ∈ 1, … , 𝑚 , 𝐼𝑗 𝑟 = 𝐼𝑗 𝑝
• Compute the actual distance for all the candidates and

return the nearest one to the query

30

The Framework of NNS using LSH

• Concatenate k hashes is too “strong”
• ℎ𝑗,𝑘 𝑝1 ≠ ℎ𝑗,𝑘 𝑝2 ⇒ 𝐼𝑗 𝑟 ≠ 𝐼𝑗 𝑝 for any 𝑘
• Not adaptive to the distribution of the

distances

• What if not enough candidates?
• Need to tune w (or build indexes different w’s) to

handle different cases

31

The Drawback of LSH

• Observation:
• If 𝑟’s nearest neighbor does not falls into 𝑟’s hash

bucket, then most likely it will fall into the adjacent bucket to 𝑟’s

• Why? σ1

𝑒 𝑏𝑗 ⋅ 𝑦𝑗 − σ1 𝑒 𝑏𝑗 ⋅ 𝑟𝑗 ~𝑂(0, 𝑦, 𝑟 2 2)

• Idea:
• Not only look at the hash bucket where 𝑟 falls

into, but also those adjacent to it

• Problem:
• How many such bucket? 2𝑙
• And they are not equally important!

32

Multi-Probe LSH

• Consider the case when 𝑙 = 2:
• Note that 𝐼𝑗(𝑝) = (ℎ𝑗,1 𝑝 , ℎ𝑗,2(𝑝))
• The ideal probe order would be:
• ℎ𝑗,1 𝑟 , ℎ𝑗,2 𝑟 : 0.315
• ℎ𝑗,1 𝑟 , ℎ𝑗,2 𝑟 − 1: 0.284
• ℎ𝑗,1 𝑟 + 1, ℎ𝑗,2 𝑟 : 0.150
• ℎ𝑗,1 𝑟 − 1, ℎ𝑗,2 𝑟 : 0.035
• ℎ𝑗,1 𝑟 , ℎ𝑗,2 𝑟 + 1: 0.019

33

Multi-Probe LSH

ℎ𝑗,1 ℎ𝑗,2 We don’t have to compute the integration, but use the offset between f 𝑟 and the boundaries.

• Pros:
• Requires less 𝑚
• Because we use hash tables more efficiently
• More robust against the unlucky points
• Cons:
• Lose the theoretical guarantee about the results
• Not parallel-friendly

34

Multi Probe LSH

• C2LSH (SIGMOG’12 paper)
• Which one is closer to 𝑟?
• We will omit the theoretical parts hence leads to a

slightly different version to the paper.

• But the essential ideas are the same
• Project 1 is to implement C2LSH using PySpark!

35

Collision Counting LSH (C2LSH)

𝒓

1 1 1 1

𝑝1

1 1 1 2

𝑝2

1 1 1 1 1 1 1 1 1 2 1 1 2 2 3 4 1 1 1 1 2 1 1 1 1 2 3 4

• Collision: match on a single hash function
• Use number of collisions to determine the

candidates

• Match one of the super hash with 𝑟 → collides at

least 𝛽𝑛 hash values with 𝑟

• Recall in LSH, The probability of 𝑝 with

dist 𝑝, 𝑟 ≤ 𝑠

1 is a nearest neighbor

candidate of 𝑟 is 1 − (1 − 𝑞1

𝑙)𝑚

• Now we compute the case with collision

counting…

36

Counting the Collisions

• ∀𝑝 with dist 𝑝, 𝑟 ≤ 𝑠

1, we have

• Pr #𝑑𝑝𝑚𝑚𝑗𝑡𝑗𝑝𝑜 𝑝 ≥ 𝛽𝑛 = σ𝑗=𝛽𝑛

𝑛 𝑛 𝑗

𝑞𝑗 1 − 𝑞 𝑛−𝑗

• 𝑞 = Pr ℎ𝑘 𝑝 = ℎ𝑘 𝑟

≥ 𝑞1

• We define 𝑛 Bernoulli random variables 𝑌𝑗 ∼

𝐶(𝑛, 1 − 𝑞) with 1 ≤ 𝑗 ≤ 𝑛.

• Let 𝑌𝑗 equal 1 if 𝑝 does not collide with 𝑟
• i.e., Pr 𝑌𝑗 = 1 = 1 − 𝑞
• Let 𝑌𝑗 equal 0 if 𝑝 collides with 𝑟
• i.e., Pr 𝑌𝑗 = 0 = 𝑞
• Hence E 𝑌𝑗 = 1 − 𝑞
• Thus 𝐹( ത

𝑌) = 1 − 𝑞 , where ത 𝑌 =

σ𝑗=1

𝑛

𝑌𝑗 𝑛

.

• Let 𝑢 = 𝑞 − 𝛽 > 0, we have:
• Pr ത

𝑌 − 𝐹 ത 𝑌 ≥ 𝑢 = Pr

σ𝑗=1

𝑛

𝑌𝑗 𝑛

− 1 − 𝑞 ≥ 𝑢 = Pr[σ𝑌𝑗 ≥ (1 − 𝛽)𝑛]

37

The Collision Probability

• From Hoeffding’s Inequality, we have
• Pr ത

𝑌 − 𝐹 ത 𝑌 ≥ 𝑢 = Pr σ𝑌𝑗 ≥ 1 − 𝛽 𝑛 ≤ exp −

2 𝑞−𝛽 2𝑛2 σ𝑗=1

𝑛

1−0 2

= exp −2 𝑞 − 𝛽 2𝑛 ≤ exp −2 𝑞1 − 𝛽 2𝑛

• Since the event “#collision(𝑝) ≥ 𝛽𝑛” is equivalent to the

event “𝑝 misses the collision with 𝑟 less than (1 − 𝛽)𝑛 times”,

• Pr #𝑑𝑝𝑚𝑚𝑗𝑡𝑗𝑝𝑜 𝑝 ≥ 𝛽𝑛 = Pr σ𝑌𝑗 < 1 − 𝛽 𝑛 ≥ 1 −

exp −2 𝑞1 − 𝛽 2𝑛

• Now you can compute the case for 𝑝 with dist 𝑝, 𝑟 ≥ 𝑠

2 in

a similar way…

• Then we can accordingly set 𝛽 to control false positives and

false negatives

38

The Collision Probability

• When we are not getting enough candidates…
• E.g., # of candidates < top-k
• Observation:
• A close point 𝑝 usually falls into adjacent hash

buckets of 𝑟’s if it does not collide with 𝑟

• Why?
• σ1

𝑒 𝑏𝑗 ⋅ 𝑦𝑗 − σ1 𝑒 𝑏𝑗 ⋅ 𝑟𝑗 ~𝑂(0, 𝑦, 𝑟 2 2)

• Idea:
• Include the adjacent hash buckets into

consideration

• So you don’t need to re-hash them again…

39

Virtual Rehashing

• At first consider ℎ 𝑝 = ℎ 𝑟
• Consider ℎ 𝑝 = ℎ 𝑟 ± 1 if not enough

candidates

• Then ℎ 𝑝 = ℎ 𝑟 ± 2 and so on…

40

Virtual Rehashing

𝒓

1 1 1 1 1 1 1 1 1 1

𝑝1

1 2

• 1

1 2 4

• 3
• 1

𝒓

1 1 1 1 1 1 1 1 1 1

𝑝1

1 2

• 1

1 2 4

• 3
• 1

𝒓

1 1 1 1 1 1 1 1 1 1

𝑝1

1 2

• 1

1 2 4

• 3
• 1

• Pre-processing
• Generate LSH functions
• Random normal vectors and random uniform values
• Index
• Compute and store ℎ𝑗 𝑝 for each data object 𝑝,

𝑗 ∈ {1, … , 𝑛}

• Query
• Compute ℎ𝑗 𝑟 for query 𝑟, 𝑗 ∈ {1, … , 𝑛}
• Take those 𝑝 that shares at least 𝛽𝑛 hashes with 𝑟 as

candidates

• Relax the collision condition (e.g., virtual rehashing) and

repeat the above step, until we got enough candidates

41

The Framework of NNS using C2LSH

candGen(data_hashes, query_hashes, 𝛽𝑛, 𝛾𝑜):

• ffset ← 0

cand ← ∅ while true: for each (id, hashes) in data_hashes: if count(hashes, query_hashes, offset) ≥ 𝛽𝑛: cand ← cand ∪ {id} if 𝑑𝑏𝑜𝑒 < 𝛾𝑜:

• ffset ← offset + 1

else: break return cand

42

Pseudo code of Candidate Generation in C2LSH

count(hashes_1, hashes_2, offset): counter ← 0 for each ℎ𝑏𝑡ℎ1, ℎ𝑏𝑡ℎ2 in hashes_1, hashes_2: if ℎ𝑏𝑡ℎ1 − ℎ𝑏𝑡ℎ2 ≤ offset: counter ← counter + 1 return counter

43

Pseudo code of Candidate Generation in C2LSH

• Spec has been released, deadline: 18 Jul, 2020
• Late Penalty: 10% on day 1 and 30% on each

subsequent day.

• Implement a light version of C2LSH (i.e., the
• ne we introduced in the lecture)
• Start working ASAP
• Evaluation: Correctness and Efficiency
• Must use PySpark, some python modules and

PySpark functions are banned.

• E.g., numpy, pandas, collect(), take(), …
• Use transformations!

44

Project 1

• There will be a bonus part (max 20 points) to

encourage efficient implementations.

• Details in the spec
• Make sure you have valid output
• Make your own test cases, a real dataset would

be more desirable

• Toy example in the spec is a real “toy” (e.g., for

babies…)

• Won’t accept excuses like “it works on my own

computer”

• Don’t violate the Student Conduct!!!

45

Project 1

Product Quantization

and K-Means Clustering

• Naïve (but exact) solution:
• Linear scan: compute 𝑒𝑗𝑡𝑢(𝑝, 𝑟) for all 𝑝 ∈ 𝐸
• 𝑒𝑗𝑡𝑢 𝑝, 𝑟 =

σ𝑗=1

𝑒

𝑝𝑗 − 𝑟𝑗 2

• 𝑃 𝑜𝑒
• 𝑜 times (𝑒 subtractions + 𝑒 − 1 additions + 𝑒 multiplications)
• Storage is also costly: 𝑃 𝑜𝑒
• Could be problematic in DBMS and distributed systems
• This motivates the idea of compression

47

Recall: NNS in High Dimensional Euclidean Space

• Idea: compressed representation of vectors
• Each vector 𝑝 is represented by a representative
• Denoted as 𝑅𝑎(𝑝)
• We will discuss how to get the representatives later
• We control the total number of representatives for

the dataset (denoted as 𝑙)

• One representative represents multiple vectors
• Instead of store 𝑝, we store its representative id
• 𝑒 floats => 1 integer
• Instead of compute 𝑒𝑗𝑡𝑢(𝑝, 𝑟), we compute

𝑒𝑗𝑡𝑢(𝑅𝑎(𝑝), 𝑟)

• We only need k computations of distance!

48

Vector Quantization

• Assigning representatives is essentially a

partition problem

• Construct a “good” partition of a database of 𝑜
• bjects into a set of 𝑙 clusters
• How to measure the “goodness” of a given

partitioning scheme?

• Cost of a cluster
• 𝐷𝑝𝑡𝑢 𝐷𝑗 = σ𝑝𝑘∈𝐷𝑗 𝑝

𝑘 − 𝑑𝑓𝑜𝑢𝑓𝑠 𝐷𝑗 2 2

• Cost of 𝑙 clusters: sum of 𝐷𝑝𝑡𝑢 𝐷𝑗

49

How to Generate Representatives

• It’s an optimization problem!
• Global optimal:
• NP-hard (for a wide range of cost functions)
• Requires exhaustively enumerate all 𝑜

𝑙 partitions

• Stirling numbers of the second kind
• 𝑜

𝑙 ~ 𝑙𝑜 𝑙! when 𝑜 → ∞

• Heuristic methods:
• k-means
• Many variants

50

Partitioning Problem: Basic Concept

• Given 𝑙, the k-means algorithm is implemented in

four steps:

• 1. Partition objects into k nonempty subsets (randomly)
• 2. Compute seed points as the centroids of the clusters of

the current partitioning (the centroid is the center, i.e., mean point, of the cluster)

• 3. Assign each object to the cluster with the nearest seed

point

• 4. Go back to Step 2, stop when the assignment does not

change

51

The k-Means Clustering Method

An Example of k-Means Clustering

K=2 Arbitrarily partition

• bjects into

k groups Update the cluster centroids Update the cluster centroids Reassign objects Loop if needed

52

The initial data set

◼

Partition objects into k nonempty subsets

◼

Repeat

◼

Compute centroid (i.e., mean point) for each partition

◼

Assign each object to the cluster of its nearest centroid

◼

Until no change

• Encode the vectors
• Generate a codebook 𝑋 = {𝑑1, … , 𝑑𝑙} via k-

means

• Assign 𝑝 to its nearest codeword in 𝑋
• E.g., 𝑅𝑎 𝑝 = 𝑑𝑗 𝑗 ∈ 1 … 𝑙 such that 𝑒𝑗𝑡𝑢 𝑝, 𝑑𝑗 ≤ 𝑒𝑗𝑡𝑢 𝑝, 𝑑

𝑘 ∀𝑘

• Represent each vector 𝑝 by its assigned codeword
• Assume 𝑒 = 256, 𝑙 = 216
• Before: 4 bytes * 256 = 1024 bytes for each

vector

• Now:
• data: 16 bits = 2 bytes
• codebook: 4 * 256 * 216

53

Vector Quantization

• Given query 𝑟, how to find a

point close to 𝑟?

• Algorithm:
• Compute 𝑅𝑎(𝑟)
• Candidate set 𝐷 = all data vectors

associated with 𝑅𝑎(𝑟)

• Verification: compute distance

between 𝑟 and 𝑝𝑗 ∈ 𝐷

• Requires loading the vectors in C
• Any problem/improvement?

54

Vector Quantization – Query Processing

Inverted index: a hash table that maps 𝑑

𝑘 to a list of oi that

are associated with 𝑑

𝑘

• To achieve better accuracy, fine-grained

quantizer with large 𝑙 is needed

• Large 𝑙
• Costly to run K-means
• Computing 𝑅𝑎(𝑟) is expensive: 𝑃(𝑙𝑒)
• May need to look beyond 𝑅𝑎(𝑟) cell
• Solution:
• Product Quantization

55

Limitations of VQ

• Idea
• Partition the dimension into m partitions
• Accordingly a vector => subvectors
• Use separate VQ with 𝑙 codewords for each

chunk

• Example:
• 8-dim vector decomposed into 𝑛 = 2 subvectors
• Each codebook has 𝑙 = 4 codewords, (i.e., 𝑑𝑗,𝑘)
• Total space in bits:
• Data: n ⋅ 𝑛 ⋅ 𝑚𝑝𝑕(𝑙)
• Codebook: 𝑛 ⋅

𝑒 𝑛 ⋅ 𝑙 ⋅ 32

56

Product Quantization

57

Example of PQ

2 4 6 5

• 2

6 4 1 1 2 1 4 9

• 1

2 … … … … … … … … … … … … … … … … … … … … … … … … 3.3 4.1 2.7 1.4 … … … … … … … … … … … … 00 00 01 11 … … … … … … 2.1 3.6 5.3 6.6 1.2 1.5 2.4 3.3 … … … … … … … …

𝑑1,0 𝑑1,2 𝑑1,3 𝑑1,1 𝑑2,1 𝑑2,3 𝑑2,0 𝑑2,2

• Euclidean distance between

a query point q and a data point encoded as t

• Restore the virtual joint center

by looking up each partition

• f t in the corresponding

codebooks => p

• 𝑒2 𝑟, 𝑢 = σ𝑗=1

𝑒

𝑟𝑗 − 𝑞𝑗 2

• Known as Asymmetric

Distance Computation (ADC)

• 𝑒2 𝑟, 𝑢 = σ𝑗=1

𝑛

𝑟(𝑗) − 𝑑𝑗,𝑢(𝑗)

2

58

Distance Estimation

• 3

7 3 2 3.3 4.1 2.7 1.4 … … … … … … … … … … … …

𝑑1,0 𝑑1,2 𝑑1,3 𝑑1,1 𝑑2,1 𝑑2,3 𝑑2,0 𝑑2,2

2.1 3.6 5.3 6.6 1.2 1.5 2.4 3.3 … … … … … … … … 01 00 1.2 1.5 2.4 3.3 3.3 4.1 2.7 1.4

q t p

1 3 5 4

• Compute ADC for every point in the database
• How?
• Candidate = those with the 𝑚 smallest AD
• [Optional] Reranking (if 𝑚 > 1):
• Load the data vectors and compute the actual

Euclidean distance

• Return the one with the smallest distance

59

Query Processing

60

Query Processing

• 3

7 3 2 01 00

q t1

1 3 5 4 11 10

t2

… … … …

𝑟(1) − 𝑑1,𝑢1(1)

2 + 𝑟(2) − 𝑑2,𝑢1(2) 2

𝑟(1) − 𝑑1,𝑢2(1)

2 + 𝑟(2) − 𝑑2,𝑢2(2) 2 3.3 4.1 2.7 1.4 … … … … … … … … … … … … 2.1 3.6 5.3 6.6 1.2 1.5 2.4 3.3 … … … … … … … …

𝑑1,0 𝑑1,2 𝑑1,3 𝑑1,1 𝑑2,1 𝑑2,3 𝑑2,0 𝑑2,2

• Pre-processing:
• Step 1: partition data vectors
• Step 2: generate codebooks (e.g., k-means)
• Step 3: encode data
• Query
• Step 1: compute distance between q and

codewords

• Step 2: compute AD for each point and return the

candidates

• Step 3: re-ranking (optional)

61

Framework of PQ