Approximate Nearest Line Search in High Dimensions Sepideh Mahabadi - - PowerPoint PPT Presentation

Approximate Nearest Line Search in High Dimensions

Sepideh Mahabadi

The NLS Problem

• Given: a set of 𝑂 lines 𝑀 in ℝ𝑒

The NLS Problem

• Given: a set of 𝑂 lines 𝑀 in ℝ𝑒
• Goal: build a data structure s.t.

– given a query 𝑟, find the closest line ℓ∗ to 𝑟

The NLS Problem

• Given: a set of 𝑂 lines 𝑀 in ℝ𝑒
• Goal: build a data structure s.t.

– given a query 𝑟, find the closest line ℓ∗ to 𝑟 – polynomial space – sub-linear query time

The NLS Problem

• Given: a set of 𝑂 lines 𝑀 in ℝ𝑒
• Goal: build a data structure s.t.

– given a query 𝑟, find the closest line ℓ∗ to 𝑟 – polynomial space – sub-linear query time

Approximation

• Finds an approximate closest line ℓ

𝑒𝑒𝑒𝑒 𝑟,ℓ ≤ 𝑒𝑒𝑒𝑒(𝑟, ℓ∗)(1 + 𝜗)

BACKGROUND

Nearest Neighbor Problems Motivation Previous Work Our result Notation

Nearest Neighbor Problem

NN: Given a set of 𝑂 points 𝑄, build a data structure s.t. given a query point 𝑟, finds the closest point 𝑞∗ to 𝑟.

Nearest Neighbor Problem

NN: Given a set of 𝑂 points 𝑄, build a data structure s.t. given a query point 𝑟, finds the closest point 𝑞∗ to 𝑟.

• Applications: database, information retrieval,

pattern recognition, computer vision

– Features: dimensions – Objects: points – Similarity: distance between points

Nearest Neighbor Problem

NN: Given a set of 𝑂 points 𝑄, build a data structure s.t. given a query point 𝑟, finds the closest point 𝑞∗ to 𝑟.

• Applications: database, information retrieval,

pattern recognition, computer vision

– Features: dimensions – Objects: points – Similarity: distance between points

• Current solutions suffer from “curse of

dimensionality”:

– Either space or query time is exponential in 𝑒 – Little improvement over linear search

Approximate Nearest Neighbor(ANN)

• ANN: Given a set of 𝑂 points 𝑄, build a data

structure s.t. given a query point 𝑟, finds an approximate closest point 𝑞 to 𝑟, i.e., 𝑒𝑒𝑒𝑒 𝑟,𝑞 ≤ 𝑒𝑒𝑒𝑒 𝑟, 𝑞∗ 1 + 𝜗

Approximate Nearest Neighbor(ANN)

• ANN: Given a set of 𝑂 points 𝑄, build a data

structure s.t. given a query point 𝑟, finds an approximate closest point 𝑞 to 𝑟, i.e., 𝑒𝑒𝑒𝑒 𝑟,𝑞 ≤ 𝑒𝑒𝑒𝑒 𝑟, 𝑞∗ 1 + 𝜗

• There exist data structures with different
• tradeoffs. Example:

– Space: 𝑒𝑂 𝑃

1 𝜗2

– Query time:

𝑒 log 𝑂 𝜗 𝑃 1

Motivation for NLS

One of the simplest generalizations of ANN: data items are represented by 𝑙- flats (affine subspace) instead of points

Motivation for NLS

One of the simplest generalizations of ANN: data items are represented by 𝑙- flats (affine subspace) instead of points

• Model data under linear variations
• Unknown or unimportant parameters in

database

Motivation for NLS

One of the simplest generalizations of ANN: data items are represented by 𝑙- flats (affine subspace) instead of points

• Model data under linear variations
• Unknown or unimportant parameters in

database

• Example:

– Varying light gain parameter of images – Each image/point becomes a line – Search for the closest line to the query image

Previous and Related Work

• Magen[02]: Nearest Subspace Search for constant 𝑙

– Query time is fast : 𝑒 + log 𝑂 +

1 𝜗 𝑃 1

– Space is super-polynomial : 2 log 𝑂 𝑃 1

Previous and Related Work

• Magen[02]: Nearest Subspace Search for constant 𝑙

– Query time is fast : 𝑒 + log 𝑂 +

1 𝜗 𝑃 1

– Space is super-polynomial : 2 log 𝑂 𝑃 1

Dual Problem: Database is a set of points, query is a 𝑙-flat

• [AIKN] for 1-flat: for any 𝑒 > 0

– Query time: 𝑃 𝑒3𝑂0.5+𝑢 – Space: 𝑒2𝑂𝑃

1 𝜗2+ 1 𝑢2

Previous and Related Work

• Magen[02]: Nearest Subspace Search for constant 𝑙

– Query time is fast : 𝑒 + log 𝑂 +

1 𝜗 𝑃 1

– Space is super-polynomial : 2 log 𝑂 𝑃 1

Dual Problem: Database is a set of points, query is a 𝑙-flat

• [AIKN] for 1-flat: for any 𝑒 > 0

– Query time: 𝑃 𝑒3𝑂0.5+𝑢 – Space: 𝑒2𝑂𝑃

1 𝜗2+ 1 𝑢2

• Very recently [MNSS] extended it for 𝑙-flats

– Query time 𝑃 𝑜

𝑙 𝑙+1−𝜍+𝑢

– Space: 𝑃(𝑜

1+

𝜏𝑙 𝑙+1−𝜍 + 𝑜 log𝑃 1 𝑢 𝑜)

Our Result

We give a randomized algorithm that for any sufficiently small 𝜗 reports a 1 + 𝜗 -approximate solution with high probability

• Space: 𝑂 + 𝑒 𝑃

1 𝜗2

• Time : 𝑒 + log 𝑂 +

1 𝜗 𝑃 1

Our Result

We give a randomized algorithm that for any sufficiently small 𝜗 reports a 1 + 𝜗 -approximate solution with high probability

• Space: 𝑂 + 𝑒 𝑃

1 𝜗2

• Time : 𝑒 + log 𝑂 +

1 𝜗 𝑃 1

• Matches up to polynomials, the performance of best

algorithm for ANN. No exponential dependence on 𝑒

Our Result

We give a randomized algorithm that for any sufficiently small 𝜗 reports a 1 + 𝜗 -approximate solution with high probability

• Space: 𝑂 + 𝑒 𝑃

1 𝜗2

• Time : 𝑒 + log 𝑂 +

1 𝜗 𝑃 1

• Matches up to polynomials, the performance of best

algorithm for ANN. No exponential dependence on 𝑒

• The first algorithm with poly log query time and

polynomial space for objects other than points

Our Result

We give a randomized algorithm that for any sufficiently small 𝜗 reports a 1 + 𝜗 -approximate solution with high probability

• Space: 𝑂 + 𝑒 𝑃

1 𝜗2

• Time : 𝑒 + log 𝑂 +

1 𝜗 𝑃 1

• Matches up to polynomials, the performance of best

algorithm for ANN. No exponential dependence on 𝑒

• The first algorithm with poly log query time and

polynomial space for objects other than points

• Only uses reductions to ANN

Notation

• 𝑀 : the set of lines with size 𝑂
• q : the query point

Notation

• 𝑀 : the set of lines with size 𝑂
• q : the query point
• 𝐶(𝑑, 𝑠): ball of radius 𝑠 around 𝑑

Notation

• 𝑀 : the set of lines with size 𝑂
• q : the query point
• 𝐶(𝑑, 𝑠): ball of radius 𝑠 around 𝑑
• 𝑒𝑒𝑒𝑒: the Euclidean distance

between objects

Notation

• 𝑀 : the set of lines with size 𝑂
• q : the query point
• 𝐶(𝑑, 𝑠): ball of radius 𝑠 around 𝑑
• 𝑒𝑒𝑒𝑒: the Euclidean distance

between objects

• 𝑏𝑜𝑏𝑏𝑏: defined between lines

Notation

• 𝑀 : the set of lines with size 𝑂
• q : the query point
• 𝐶(𝑑, 𝑠): ball of radius 𝑠 around 𝑑
• 𝑒𝑒𝑒𝑒: the Euclidean distance

between objects

• 𝑏𝑜𝑏𝑏𝑏: defined between lines
• 𝜀-close: two lines ℓ , ℓ′ are 𝜀-close

if sin(𝑏𝑜𝑏𝑏𝑏 ℓ, ℓ′ ) ≤ 𝜀

MODULES

Net Module Unbounded Module Parallel Module

Net Module

• Intuition: sampling points from each line

finely enough to get a set of points 𝑄, and building an 𝐵𝑂𝑂(𝑄, 𝜗) should suffice to find the approximate closest line.

Net Module

• Intuition: sampling points from each line

finely enough to get a set of points 𝑄, and building an 𝐵𝑂𝑂(𝑄, 𝜗) should suffice to find the approximate closest line. Lemma:

• Let 𝑦 be the separation parameter:

distance between two adjacent samples

• n a line, Then

– Either the returned line ℓ𝑞 is an approximate closest line – Or 𝑒𝑒𝑒𝑒 𝑟, ℓ𝑞 ≤ 𝑦/𝜗

Net Module

• Intuition: sampling points from each line

finely enough to get a set of points 𝑄, and building an 𝐵𝑂𝑂(𝑄, 𝜗) should suffice to find the approximate closest line. Lemma:

• Let 𝑦 be the separation parameter:

distance between two adjacent samples

• n a line, Then

– Either the returned line ℓ𝑞 is an approximate closest line – Or 𝑒𝑒𝑒𝑒 𝑟, ℓ𝑞 ≤ 𝑦/𝜗

Issue: It should be used inside a bounded region

Unbounded Module - Intuition

• All lines in 𝑀 pass through the origin

𝑝

Unbounded Module - Intuition

• All lines in 𝑀 pass through the origin

𝑝

• Data structure:

– Project all lines onto any sphere 𝑇 𝑝,𝑠 to get point set 𝑄 – Build ANN data structure 𝐵𝑂𝑂(𝑄, 𝜗)

Unbounded Module - Intuition

• All lines in 𝑀 pass through the origin

𝑝

• Data structure:

– Project all lines onto any sphere 𝑇 𝑝,𝑠 to get point set 𝑄 – Build ANN data structure 𝐵𝑂𝑂(𝑄, 𝜗)

• Query Algorithm:

– Project the query on 𝑇(𝑝, 𝑠) to get 𝑟′ – Find the approximate closest point to 𝑟′, i.e., 𝑞 = 𝐵𝑂𝑂𝑄 𝑟′ – Return the corresponding line of 𝑞

Unbounded Module

• All lines in 𝑀 pass through a small ball

𝐶 𝑝, 𝑠

• Query is far enough, outside of 𝐶(𝑝, 𝑆)
• Use the same data structure and

query algorithm

Unbounded Module

• All lines in 𝑀 pass through a small ball

𝐶 𝑝, 𝑠

• Query is far enough, outside of 𝐶(𝑝, 𝑆)
• Use the same data structure and

query algorithm Lemma: if 𝑆 ≥ 𝑠

𝜗𝜗 , the returned line ℓ𝑞 is

• Either an approximate closest line
• Or is 𝜀-close to the closest line ℓ∗

Unbounded Module

• All lines in 𝑀 pass through a small ball

𝐶 𝑝, 𝑠

• Query is far enough, outside of 𝐶(𝑝, 𝑆)
• Use the same data structure and

query algorithm Lemma: if 𝑆 ≥ 𝑠

𝜗𝜗 , the returned line ℓ𝑞 is

• Either an approximate closest line
• Or is 𝜀-close to the closest line ℓ∗

This helps us in two ways

• Bound the region for the net module
• Restrict search to almost parallel lines

Parallel Module - Intuition

• All lines in 𝑀 are parallel

Parallel Module - Intuition

• All lines in 𝑀 are parallel
• Data structure:

– Project all lines onto any hyper-plane 𝑏 which is perpendicular to all the lines to get point set 𝑄 – Build ANN data structure 𝐵𝑂𝑂(𝑄, 𝜗)

Parallel Module - Intuition

• All lines in 𝑀 are parallel
• Data structure:

– Project all lines onto any hyper-plane 𝑏 which is perpendicular to all the lines to get point set 𝑄 – Build ANN data structure 𝐵𝑂𝑂(𝑄, 𝜗)

• Query algorithm:

– Project the query on 𝑏 to get 𝑟′ – Find the approximate closest point to 𝑟′, i.e., 𝑞 = 𝐵𝑂𝑂𝑄 𝑟′ – Return the corresponding line to 𝑞

Parallel Module

• All lines in 𝑀 are 𝜀-close to a base line ℓ𝑐
• Project the lines onto a hyper-plane 𝑏 which is

perpendicular to ℓ𝑐

• Query is close enough to 𝑏
• Use the same data structure and query algorithm

Parallel Module

• All lines in 𝑀 are 𝜀-close to a base line ℓ𝑐
• Project the lines onto a hyper-plane 𝑏 which is

perpendicular to ℓ𝑐

• Query is close enough to 𝑏
• Use the same data structure and query algorithm

Lemma: if 𝑒𝑒𝑒𝑒 𝑟, 𝑏 ≤

𝐸𝜗 𝜗 , then

• Either the returned line ℓ𝑞 is an approximate closest

line

• Or 𝑒𝑒𝑒𝑒 𝑟, ℓ𝑞

≤ 𝐸

Parallel Module

• All lines in 𝑀 are 𝜀-close to a base line ℓ𝑐
• Project the lines onto a hyper-plane 𝑏 which is

perpendicular to ℓ𝑐

• Query is close enough to 𝑏
• Use the same data structure and query algorithm

Lemma: if 𝑒𝑒𝑒𝑒 𝑟, 𝑏 ≤

𝐸𝜗 𝜗 , then

• Either the returned line ℓ𝑞 is an approximate closest

line

• Or 𝑒𝑒𝑒𝑒 𝑟, ℓ𝑞

≤ 𝐸 Thus, for a set of almost parallel lines, we can use a set

• f parallel modules to cover a bounded region.

How the Modules Work Together

Given a set of lines, we come up with a polynomial number of balls.

How the Modules Work Together

Given a set of lines, we come up with a polynomial number of balls.

• If 𝑟 is inside the ball

– Use net module

q

How the Modules Work Together

Given a set of lines, we come up with a polynomial number of balls.

• If 𝑟 is inside the ball

– Use net module

• If 𝑟 is outside the ball

– First use unbounded module to find a line ℓ

q

How the Modules Work Together

Given a set of lines, we come up with a polynomial number of balls.

• If 𝑟 is inside the ball

– Use net module

• If 𝑟 is outside the ball

– First use unbounded module to find a line ℓ

q ℓ

How the Modules Work Together

Given a set of lines, we come up with a polynomial number of balls.

• If 𝑟 is inside the ball

– Use net module

• If 𝑟 is outside the ball

– First use unbounded module to find a line ℓ – Then use parallel module to search among parallel lines to ℓ

q ℓ

Outline of the Algorithms

• Input: a set of 𝑜 lines 𝑇

Outline of the Algorithms

• Input: a set of 𝑜 lines 𝑇
• Randomly choose a subset of 𝑜/2 lines 𝑈

Outline of the Algorithms

• Input: a set of 𝑜 lines 𝑇
• Randomly choose a subset of 𝑜/2 lines 𝑈
• Solve the problem over 𝑈 to get a line ℓ𝑞

Outline of the Algorithms

• Input: a set of 𝑜 lines 𝑇
• Randomly choose a subset of 𝑜/2 lines 𝑈
• Solve the problem over 𝑈 to get a line ℓ𝑞
• For log𝑜 iterations

– Use ℓ𝑞 to find a much closer line ℓ𝑞′ – Update ℓ𝑞 with ℓ𝑞

′

Improvement step

Outline of the Algorithms

• Input: a set of 𝑜 lines 𝑇
• Randomly choose a subset of 𝑜/2 lines 𝑈
• Solve the problem over 𝑈 to get a line ℓ𝑞
• For log𝑜 iterations

– Use ℓ𝑞 to find a much closer line ℓ𝑞′ – Update ℓ𝑞 with ℓ𝑞

′

Improvement step

Outline of the Algorithms

• Input: a set of 𝑜 lines 𝑇
• Randomly choose a subset of 𝑜/2 lines 𝑈
• Solve the problem over 𝑈 to get a line ℓ𝑞
• For log𝑜 iterations

– Use ℓ𝑞 to find a much closer line ℓ𝑞′ – Update ℓ𝑞 with ℓ𝑞

′

Why?

Improvement step

Outline of the Algorithms

• Input: a set of 𝑜 lines 𝑇
• Randomly choose a subset of 𝑜/2 lines 𝑈
• Solve the problem over 𝑈 to get a line ℓ𝑞
• For log𝑜 iterations

– Use ℓ𝑞 to find a much closer line ℓ𝑞′ – Update ℓ𝑞 with ℓ𝑞

′

Let 𝑒1, … , 𝑒log 𝑜 be the log 𝑜 closest lines to 𝑟 in the set 𝑇

Improvement step

Outline of the Algorithms

• Input: a set of 𝑜 lines 𝑇
• Randomly choose a subset of 𝑜/2 lines 𝑈
• Solve the problem over 𝑈 to get a line ℓ𝑞
• For log𝑜 iterations

– Use ℓ𝑞 to find a much closer line ℓ𝑞′ – Update ℓ𝑞 with ℓ𝑞

′

Let 𝑒1, … , 𝑒log 𝑜 be the log 𝑜 closest lines to 𝑟 in the set 𝑇 With high probability at least one of {𝑒1, … , 𝑒log 𝑜} is sampled in 𝑈

– 𝑒𝑒𝑒𝑒 𝑟, ℓ𝑞 ≤ 𝑒𝑒𝑒𝑒 𝑟, 𝑒log 𝑜 (1 + 𝜗) – log 𝑜 improvement steps suffices to find an approximate closest line

Improvement step

Improvement Step

Given a line ℓ, how to improve it, i.e., find a closer line?

Improvement Step

Given a line ℓ, how to improve it, i.e., find a closer line?

• Data structure
• Query Processing Algorithm

Improvement Step

Given a line ℓ, how to improve it, i.e., find a closer line?

• Data structure
• Query Processing Algorithm

Use the three modules here

Conclusion

Bounds we get for NLS problem

– Polynomial Space: 𝑃 𝑂 + 𝑒 𝑃

1 𝜗2

– Poly-logarithmic query time : 𝑒 + log 𝑂 +

1 𝜗 𝑃 1

Conclusion

Bounds we get for NLS problem

– Polynomial Space: 𝑃 𝑂 + 𝑒 𝑃

1 𝜗2

– Poly-logarithmic query time : 𝑒 + log 𝑂 +

1 𝜗 𝑃 1

Future Work

• The current result is not efficient in practice

– Large exponents – Algorithm is complicated

Conclusion

Bounds we get for NLS problem

– Polynomial Space: 𝑃 𝑂 + 𝑒 𝑃

1 𝜗2

– Poly-logarithmic query time : 𝑒 + log 𝑂 +

1 𝜗 𝑃 1

Future Work

• The current result is not efficient in practice

– Large exponents – Algorithm is complicated

• Can we get a simpler algorithms?

Conclusion

Bounds we get for NLS problem

– Polynomial Space: 𝑃 𝑂 + 𝑒 𝑃

1 𝜗2

– Poly-logarithmic query time : 𝑒 + log 𝑂 +

1 𝜗 𝑃 1

Future Work

• The current result is not efficient in practice

– Large exponents – Algorithm is complicated

• Can we get a simpler algorithms?
• Generalization to higher dimensional flats

Conclusion

Bounds we get for NLS problem

– Polynomial Space: 𝑃 𝑂 + 𝑒 𝑃

1 𝜗2

– Poly-logarithmic query time : 𝑒 + log 𝑂 +

1 𝜗 𝑃 1

Future Work

• The current result is not efficient in practice

– Large exponents – Algorithm is complicated

• Can we get a simpler algorithm?
• Generalization to higher dimensional flats
• Generalization to other objects, e.g. balls

THANK YOU!