Graph-based timespace trade-offs for approximate near neighbors - - PowerPoint PPT Presentation

▶
graph based time space trade offs for approximate near
SMART_READER_LITE
LIVE PREVIEW

Graph-based timespace trade-offs for approximate near neighbors - - PowerPoint PPT Presentation

Sep 16, 2022 389 likes •1.07k views

Graph-based timespace trade-offs for approximate near neighbors Thijs Laarhoven mail@thijs.com http://thijs.com/ SoCG 2018 , Budapest, Hungary (June 13, 2018) Nearest neighbor searching Nearest neighbor problem Problem description

slide-1
SLIDE 1

Graph-based time–space trade-offs for approximate near neighbors

Thijs Laarhoven

mail@thijs.com http://thijs.com/

SoCG 2018, Budapest, Hungary (June 13, 2018)

slide-2
SLIDE 2

Nearest neighbor searching

slide-3
SLIDE 3

Nearest neighbor problem – Problem description

slide-4
SLIDE 4

Nearest neighbor problem – Problem description

slide-5
SLIDE 5

Nearest neighbor problem – Problem description

slide-6
SLIDE 6

r

Nearest neighbor problem – Approximate solutions

slide-7
SLIDE 7

r c · r

Nearest neighbor problem – Approximate solutions

slide-8
SLIDE 8

Nearest neighbor problem – Example: Voronoi cells

slide-9
SLIDE 9

Nearest neighbor problem – Example: Voronoi cells

slide-10
SLIDE 10

Nearest neighbor problem – Example: Voronoi cells

slide-11
SLIDE 11

Partition-based methods

slide-12
SLIDE 12

Partition-based methods – Data structure

slide-13
SLIDE 13

Partition-based methods – Data structure

slide-14
SLIDE 14

Partition-based methods – Hash table lookups

slide-15
SLIDE 15

Partition-based methods – Hash table lookups

slide-16
SLIDE 16

Partition-based methods – Hash table lookups

slide-17
SLIDE 17

Partition-based methods – Hash table lookups

slide-18
SLIDE 18

Partition-based methods – Near the boundaries

slide-19
SLIDE 19

Partition-based methods – Near the boundaries

slide-20
SLIDE 20

Partition-based methods – Near the boundaries

slide-21
SLIDE 21

Partition-based methods – Randomizations

slide-22
SLIDE 22

Partition-based methods – Randomizations

slide-23
SLIDE 23

Partition-based methods – Randomizations

slide-24
SLIDE 24

Partition-based methods – Challenges

Main problem: choosing the best types of space partitions.

  • Requires an effjcient decoding algorithm;
  • Space partitions should have nice shapes.

Utopia: disjoint spheres lying on an effjciently decodable code or lattice. Real world: approximate ideal solution as best as we can.

  • Product of bisections; [Cha03]
  • Voronoi cells induced by hypercube; [TT07, Laa16]
  • Random (overlapping) spheres; [AI06, AINR14]
  • Voronoi cells induced by cross-polytopes; [TT07, AIL+15, KW17]
  • Voronoi cells induced by (pseudo)random points. [BDGL16, ALRW17, Chr17]

Best techniques are theoretically optimal as well as practical.

slide-25
SLIDE 25

Partition-based methods – Challenges

Main problem: choosing the best types of space partitions.

  • Requires an effjcient decoding algorithm;
  • Space partitions should have nice shapes.

Utopia: disjoint spheres lying on an effjciently decodable code or lattice. Real world: approximate ideal solution as best as we can.

  • Product of bisections; [Cha03]
  • Voronoi cells induced by hypercube; [TT07, Laa16]
  • Random (overlapping) spheres; [AI06, AINR14]
  • Voronoi cells induced by cross-polytopes; [TT07, AIL+15, KW17]
  • Voronoi cells induced by (pseudo)random points. [BDGL16, ALRW17, Chr17]

Best techniques are theoretically optimal as well as practical.

slide-26
SLIDE 26

Partition-based methods – Challenges

Main problem: choosing the best types of space partitions.

  • Requires an effjcient decoding algorithm;
  • Space partitions should have nice shapes.

Utopia: disjoint spheres lying on an effjciently decodable code or lattice. Real world: approximate ideal solution as best as we can.

  • Product of bisections; [Cha03]
  • Voronoi cells induced by hypercube; [TT07, Laa16]
  • Random (overlapping) spheres; [AI06, AINR14]
  • Voronoi cells induced by cross-polytopes; [TT07, AIL+15, KW17]
  • Voronoi cells induced by (pseudo)random points. [BDGL16, ALRW17, Chr17]

Best techniques are theoretically optimal as well as practical.

slide-27
SLIDE 27

Partition-based methods – Challenges

Main problem: choosing the best types of space partitions.

  • Requires an effjcient decoding algorithm;
  • Space partitions should have nice shapes.

Utopia: disjoint spheres lying on an effjciently decodable code or lattice. Real world: approximate ideal solution as best as we can.

  • Product of bisections; [Cha03]
  • Voronoi cells induced by hypercube; [TT07, Laa16]
  • Random (overlapping) spheres; [AI06, AINR14]
  • Voronoi cells induced by cross-polytopes; [TT07, AIL+15, KW17]
  • Voronoi cells induced by (pseudo)random points. [BDGL16, ALRW17, Chr17]

Best techniques are theoretically optimal as well as practical.

slide-28
SLIDE 28

Nearest neighbor methods – Practice (ANN Benchmarks [ABF17])

0.2 0.4 0.6 0.8 1 101 102 103 104 Recall rate Queries per second

annoy BallTree(nmslib) bruteforce-blas bruteforce0(nmslib) dolphinn DolphinnPy faiss-gpu faiss-ivf faiss-lsh falconn fmann hnsw(nmslib) nearpy rpforest SW-graph(nmslib)

slide-29
SLIDE 29

Nearest neighbor methods – Practice (ANN Benchmarks [ABF17])

0.2 0.4 0.6 0.8 1 101 102 103 104 Recall rate Queries per second

annoy BallTree(nmslib) bruteforce-blas bruteforce0(nmslib) dolphinn DolphinnPy faiss-gpu faiss-ivf faiss-lsh falconn fmann hnsw(nmslib) nearpy rpforest SW-graph(nmslib)

slide-30
SLIDE 30

Graph-based methods

slide-31
SLIDE 31

Graph-based methods – Data structure

slide-32
SLIDE 32

Graph-based methods – Data structure

slide-33
SLIDE 33

Graph-based methods – Greedy algorithm

slide-34
SLIDE 34

Graph-based methods – Greedy algorithm

slide-35
SLIDE 35

Graph-based methods – Greedy algorithm

slide-36
SLIDE 36

Graph-based methods – Greedy algorithm

slide-37
SLIDE 37

Graph-based methods – Greedy algorithm

slide-38
SLIDE 38

Graph-based methods – Greedy algorithm

slide-39
SLIDE 39

Graph-based methods – Greedy algorithm

slide-40
SLIDE 40

Graph-based methods – Greedy algorithm

slide-41
SLIDE 41

Graph-based methods – Greedy algorithm

slide-42
SLIDE 42

Graph-based methods – Greedy algorithm

slide-43
SLIDE 43

Graph-based methods – Greedy algorithm

slide-44
SLIDE 44

Graph-based methods – Greedy algorithm

slide-45
SLIDE 45

Graph-based methods – Local solutions

slide-46
SLIDE 46

Graph-based methods – Local solutions

slide-47
SLIDE 47

Graph-based methods – Local solutions

slide-48
SLIDE 48

Graph-based methods – Local solutions

slide-49
SLIDE 49

Graph-based methods – Local solutions

slide-50
SLIDE 50

Graph-based methods – Local solutions

slide-51
SLIDE 51

Graph-based methods – Local solutions

slide-52
SLIDE 52

Graph-based methods – Randomizations

slide-53
SLIDE 53

Graph-based methods – Randomizations

slide-54
SLIDE 54

Graph-based methods – Randomizations

slide-55
SLIDE 55

Graph-based methods – Randomizations

slide-56
SLIDE 56

Graph-based methods – Randomizations

slide-57
SLIDE 57

Graph-based methods – Randomizations

slide-58
SLIDE 58

Graph-based methods – Challenges

Main problem: designing the graph.

  • Intuitively: connect near neighbors for gradual progress;
  • Avoiding local minima: add a few long edges;
  • Hierarchical graphs: long edges in upper layers, short edges in bottom layers.

Practically, graph-based methods are very effjcient as well. Theoretically, little is known about the performance of these methods.

slide-59
SLIDE 59

Graph-based methods – Challenges

Main problem: designing the graph.

  • Intuitively: connect near neighbors for gradual progress;
  • Avoiding local minima: add a few long edges;
  • Hierarchical graphs: long edges in upper layers, short edges in bottom layers.

Practically, graph-based methods are very effjcient as well. Theoretically, little is known about the performance of these methods.

slide-60
SLIDE 60

Graph-based methods – Challenges

Main problem: designing the graph.

  • Intuitively: connect near neighbors for gradual progress;
  • Avoiding local minima: add a few long edges;
  • Hierarchical graphs: long edges in upper layers, short edges in bottom layers.

Practically, graph-based methods are very effjcient as well. Theoretically, little is known about the performance of these methods.

slide-61
SLIDE 61

Graph-based methods – Challenges

Main problem: designing the graph.

  • Intuitively: connect near neighbors for gradual progress;
  • Avoiding local minima: add a few long edges;
  • Hierarchical graphs: long edges in upper layers, short edges in bottom layers.

Practically, graph-based methods are very effjcient as well. Theoretically, little is known about the performance of these methods.

slide-62
SLIDE 62

Graph-based methods – Contributions

Theorem (Main result, informal) For randomized greedy walks on the near neighbor graph and for “random” data sets, we can solve the approximate nearest neighbor problem on n points with query time O(nρq) and space O(n1+ρs) with ρq, ρs ≥ 0 satisfying (2c2 − 1)ρq + 2c2(c2 − 1)

  • ρs(1 − ρs) ≥ c4.
slide-63
SLIDE 63

Graph-based methods – Contributions

In the most common regime of c ≈ 1 (high recall rate) and ρs ≈ 0 (near-linear space), this scales equivalently as the best partition-based trade-offs: [ALRW17] ρq = 1 − 4(c − 1)√ρs · (1 + o(1)). (1) Positive result: greedy algorithm already “optimal” for c 1 and

s

0. Negative result: (analysis of) this algorithm is not competitive for c 1 or

s

0.

slide-64
SLIDE 64

Graph-based methods – Contributions

In the most common regime of c ≈ 1 (high recall rate) and ρs ≈ 0 (near-linear space), this scales equivalently as the best partition-based trade-offs: [ALRW17] ρq = 1 − 4(c − 1)√ρs · (1 + o(1)). (1) Positive result: greedy algorithm already “optimal” for c ≈ 1 and ρs ≈ 0. Negative result: (analysis of) this algorithm is not competitive for c 1 or

s

0.

slide-65
SLIDE 65

Graph-based methods – Contributions

In the most common regime of c ≈ 1 (high recall rate) and ρs ≈ 0 (near-linear space), this scales equivalently as the best partition-based trade-offs: [ALRW17] ρq = 1 − 4(c − 1)√ρs · (1 + o(1)). (1) Positive result: greedy algorithm already “optimal” for c ≈ 1 and ρs ≈ 0. Negative result: (analysis of) this algorithm is not competitive for c ≫ 1 or ρs ≫ 0.

slide-66
SLIDE 66

Graph-based methods – Open problems

Various open problems remain:

  • Current analysis may not be sharp – can it be tightened?
  • Does adding long edges lead to better theoretical guarantees?
  • What can theoretically be said about hierarchical approaches?
  • Can we obtain lower bounds showing limitations of graph-based methods?
slide-67
SLIDE 67

Thank you!