Diverse Near Neighbor Problem Sofiane Abbar (QCRI) Sihem Amer-Yahia - - PowerPoint PPT Presentation

Diverse Near Neighbor Problem

Sofiane Abbar (QCRI) Sihem Amer-Yahia (CNRS) Piotr Indyk (MIT) Sepideh Mahabadi (MIT) Kasturi R. Varadarajan (UIowa)

Near Neighbor Problem

• Definition

– Set of 𝑜 points 𝑸 in 𝑒-dimensional space – Query point 𝒓 – Report one neighbor of 𝒓 if there is any

• Neighbor: A point within distance 𝑠 of

query

• Application

– Major importance in databases (document, image, video), information retrieval, pattern recognition

• Object of interest as point
• Similarity is measured as distance.

Motivation

Search: How many answers?

• Small output size, e.g. 10

– Reporting 𝑙 Nearest Neighbors may not be informative (could be identical texts)

• Large output size

– Time to retrieve them is high

Small output size which is

• Relevant and Diverse
• Good to have result from each

cluster, i.e. should be diverse

Diverse Near Neighbor Problem

• Definition

– Set of 𝑜 points 𝑸 in 𝑒-dimensional space – Query point 𝒓 – Report the k most diverse neighbors of 𝑟

• Neighbor:

– Points within distance 𝑠 of query – We use Hamming distance

• Diversity:

– div S = m𝑗𝑜

𝑞,𝑟∈𝑇 |𝑞 − 𝑟|

• Goal: report Q (green points), s.t.

– 𝑅 ⊆ 𝑄 ∩ 𝐶 𝑟, 𝑠 – |Q| = k – 𝑒𝑗𝑒 𝑅 is maximized

Approximation

• Want sublinear query time, so need to approximate
• Approximate NN:

– 𝐶 𝑟, 𝑠 → 𝐶 𝑟, 𝑑𝑠 for some value of 𝑑 > 1 – Result: query time of 𝑃(𝑒𝑜

1 𝑑)

• Approximate Diverse NN:

– Bi-criterion approximation: distance and diversity – (𝐝, 𝜷)-Approximate 𝑙-diverse Near Neighbor – Let 𝑅∗ (green points) be the optimum solution for 𝐶 𝑟, 𝑠

• Report approximate neighbors 𝑅 (purple points)

𝑅 ⊆ 𝐶 𝑟, 𝑑𝑠

• Diversity approximates the optimum diversity

𝑒𝑗𝑒 𝑅 ≥

1 𝛽 𝑒𝑗𝑒 (𝑅∗) , 𝛽 ≥ 1

Results

Algorithm A Algorithm B Distance Apx. Factor c > 2 c >1 Diversity Apx. Factor α 6 6 Space (𝑜 log 𝑙)1+1/(𝑑−1)+𝑜𝑒 log 𝑙 ∗ 𝑜1+1/𝑑 + 𝑜𝑒 Query Time

𝑙2 + log 𝑜 𝑠 𝑒 (log 𝑙)𝑑/(𝑑−1)𝑜1/(𝑑−1) 𝑙2 + log 𝑜 𝑠 𝑒 ∗ log 𝑙 ∗ 𝑜1/𝑑

• Algorithm A was earlier introduced in [Abbar, Amer-yahia, Indyk, Mahabadi,

WWW’13]

Techniques

Compute k-diversity: GMM

• Have n points, compute the subset with

maximum diversity.

• Exact : NP-hard to approximate better

than 2 [Ravi et al.]

• GMM Algorithm [Ravi et al.] [Gonzales]

– Choose an arbitrary point – Repeat k-1 times

• Add the point whose minimum distance to the

currently chosen points is maximized

• Achieves approximation factor 2
• Running time of the algorithm is O(kn)

Locality Sensitive Hashing (LSH)

• LSH

– close points have higher probability of collision than far points – Hash functions: 𝑕1 , … , 𝑕𝑀

• 𝑕𝑗 = < ℎ𝑗,1, … , ℎ𝑗,𝑢 >
• ℎ𝑗,𝑘 ∈ ℋ is chosen randomly
• ℋ is a family of hash functions which is

𝑄

1, 𝑄2, 𝑠, 𝑑𝑠 -sensitive:

– If 𝑞 − 𝑞𝑞 ≤ 𝑠 then Pr ℎ 𝑞 = ℎ 𝑞𝑞 ≥ 𝑄

1

– If 𝑞 − 𝑞𝑞 ≥ 𝑑𝑠 then Pr ℎ 𝑞 = ℎ 𝑞𝑞 ≤ 𝑄2

• Example: Hamming distance:

– ℎ 𝑞 = 𝑞𝑗 , i.e., the ith bit of 𝑞 – Is (1 − 𝑠

𝑒 , 1 − 𝑠𝑑 𝑒 , 𝑠, 𝑠𝑑)-sensitive

– 𝑴 and 𝒖 are parameters of LSH

LSH-based Naïve Algorithm

• [Indyk, Motwani] Parameters 𝑀 and 𝑢 can be set s.t.

With constant probability

– Any neighbor of 𝑟 falls into the same bucket as 𝑟 in at least

• ne hash function

– Total number of outliers is at most 3𝑀 – Outlier : point farther than 𝑑𝑠 from the query point

Algorithm

• Arrays for each hash function 𝐵1, … , 𝐵𝑀
• For a query 𝒓 compute

– Retrieve the possible neighbors S = ⋃ 𝑩[𝑕𝑗(𝑟)]

𝑀 𝑗=1

– Remove the outliers S = S ∩ B q, cr – Report the approximate k most diverse points of S, or GMM(S)

• Achieves (c,2)-approximation
• Running time may be linear in 𝑜 

– Should prune the buckets before collecting them

Core-sets

• Core-sets [Agarwal, Har-Peled, Varadarajan]: subset of a point set S that

represents it.

• Approximately determines the solution to an optimization

problem

• Composes: A union of coresets is a coreset of the union
• β– core-set: Approximates the cost up-to a factor of β
• Our Optimization problem:

– Finding the k-diversity of S. – Instead we consider finding K-Center Cost of S

• 𝐿𝐿 𝑇, 𝑇′ = max

𝑞∈𝑇 min 𝑞′∈𝑇′ 𝑞 − 𝑞′

• 𝐿𝐿𝑙 𝑇 =

min

𝑇′⊆𝑇 , 𝑇′ =𝑙 𝐿𝐿(𝑇, 𝑇′)

– KC cost 2-approximates diversity

• 𝐿𝐿𝑙−1 𝑇 ≤ 𝑒𝑗𝑒𝑙 𝑇 ≤ 2. 𝐿𝐿𝑙−1 𝑇
• GMM computes a 1/3-Coreset for KC-cost

Algorithms

Algorithm A

• Parameters 𝑀 and 𝑢 can be set s.t. with constant probability

– Any neighbor of 𝑟 falls into the same bucket as 𝑟 in at least one hash function – There is no outlier

• No need to keep all the points in each bucket,
• just keep a coreset!

– 𝑩𝑞𝒋 𝒌 = 𝑯𝑯𝑯 𝑩𝒋 𝒌 – Keep a 1/3 coreset of 𝑩𝒋 𝒌

• Given query 𝒓

– Retrieve the coresets from buckets S = ⋃ 𝑩𝑞[𝑕𝑗(𝑟)]

𝑀 𝑗=1

– Run GMM(S) – Report the result

Analysis

• Achieves (c,6)-Approx

– Union of 1/3 coresets is a 1/3 coreset for the union – The last GMM call, adds a 2 approximation factor

• Only works if we set 𝑀 and 𝑢 s.t. there is no outlier in 𝑇

with constant probability

– Space: 𝑃 𝑜𝑀 = 𝑃((𝑜 log 𝑙)1+1/(𝑑−1) + 𝑜𝑒) – Time: 𝑃 𝑀𝑙2 = 𝑃( 𝑙2 + log 𝑜

𝑠

𝑒 (log 𝑙)𝑑/(𝑑−1)𝑜1/(𝑑−1)) – Only makes sense for 𝑑 > 2

• Not optimal:

– ANN query time is 𝑃(𝑒𝑜

1 𝑑)

– So if we want to improve over these we should be able to deal with outliers.

Robust Core-sets

• 𝑇′is an 𝑚-robust β-coreset for S if

– for any set 𝑃 of outliers of size at most 𝑚 – (𝑇𝑞\O) is a β-coreset for 𝑇

• Peeling Algorithm [Agarwal, Har-peled,

Yu,’06][Varadarajan, Xiao, ‘12]: – Repeat (𝑚 + 1) times

• Compute a β-coreset for 𝑇
• Add them to the coreset 𝑇𝑞
• Remove them from the set 𝑇

Note: if we order the points in 𝑇𝑞 as we find them, then the first 𝑚′ + 1 𝑙 points also form an 𝑚𝑞-robust β-coreset.

2 robust coreset: S’= {3, 5; 2, 9; 1, 6}

1 robust coreset

Algorithm B

• Parameters 𝑀 and 𝑢 can be set s.t. With constant

probability

– Any neighbor of 𝑟 falls into the same bucket as 𝑟 in at least one hash function – Total number of outliers is at most 3𝑀

• For each bucket 𝐵𝑗 𝑘 keep an 3𝑀-robust 1/3-coreset in

𝐵𝑞𝑗 𝑘 which has size 3𝑀 + 1 𝑙

• For query 𝑟

– For each bucket 𝐵𝑞[𝑕𝑗(𝑟)]

• Find smallest 𝑚 s.t. the first (𝑙𝑚) points contains less than 𝑚 outliers
• Add those 𝑙𝑚 points to 𝑇

– Remove outliers from 𝑇 – Return 𝐻𝐻𝐻(𝑇)

Example and Analysis

• Total # outliers ≤ 3𝑀 , 𝑇 < 𝑃(𝑀𝑙)
• Time: O(𝑀𝑙2) = O( 𝑙2 +

log 𝑜 𝑠

𝑒 ∗ log 𝑙 ∗ 𝑜

1 𝑑)

• Space: 𝑃 𝑜𝑀 = 𝑃(log 𝑙 ∗ 𝑜1+1/𝑑 + 𝑜𝑒)
• Achieves (c,6)-Approx for the same reason

Conclusion

Further Work

• Improve diversity factor α
• Consider other definitions of diversity , e.g., sum of distances

Algorithm A Algorithm B ANN

Distance Apx. Factor c > 2 c >1 c >1 Diversity Apx. Factor α 6 6

• Space

~𝑜1+ 1

𝑑−1

~𝑜1+1

𝑑

𝑜1+1

𝑑

Query Time

~𝑒 𝑜

1 𝑑−1

~𝑒 𝑜

1 𝑑

𝑒 𝑜

1 𝑑

Thank You!