Locality-Sensitive Hashing CS 395T: Visual Recognition and Search Marc Alban Feb 22, 2008 1
Nearest Neighbor Given a query any point , return the point q closest to . q Useful for finding similar objects in a database. Brute force linear search is not practical for massive databases. ? Feb 22, 2008 2
The “Curse of Dimensionality” For , data structures exist that d < 10 to 20 require sublinear time and near linear space to perform a NN search. Time or space requirements grow exponentially in the dimension. The dimensionality of images or documents is usually in the order of several hundred or more. Brute force linear search is the best we can do. Feb 22, 2008 3
(r, )-Nearest Neighbor ² An approximate nearest neighbor should suffice in most cases. Definition: If for any query point , there exists q p 0 a point such that , w.h.p return jj q ¡ p jj · r p jj q ¡ p 0 jj · (1 + ² ) r such that . ? Feb 22, 2008 4
Locality-sensative Hash Families Definition: A LSH family , , has the H ( c; r; P 1 ; P 2 ) following properties for any : q; p 2 S 1. If then jj p ¡ q jj · r Pr H [ h ( p ) = h ( q )] ¸ P 1 2. If then jj p ¡ q jj ¸ cr Pr H [ h ( p ) = h ( p )] · P 2 Feb 22, 2008 5
Hamming Space Definition: Hamming space is the set of all 2 N binary strings of length . N Definition: The Hamming distance between two equal length binary strings is the number of positions for which the bits are different. k 1011101 ; 1001001 k H = 2 k 1110101 ; 1111101 k H = 1 Feb 22, 2008 6
Hamming Space Let a hashing family be defined as h i ( p ) = p i where is the bit of . i th p i p Pr H [ h ( p ) 6 = h ( q )] = k p; q k H d Pr H [ h ( p ) = h ( q )] = 1 ¡ k p; q k H d Clearly, this family is locality sensative. Feb 22, 2008 7
k-bit LSH Functions A k-bit locality-sensitive hash function (LSHF) is defined as: g ( p ) = [ h 1 ( p ) ; h 2 ( p ) ; : : : ; h k ( p )] T Each is chosen randomly from . H h i Each results in a single bit. h i µ ¶ k 1 ¡ 1 Pr(similar points collide) ¸ 1 ¡ P 1 Pr(dissimilar points collide) · P k 2 Feb 22, 2008 8
LSH Preprocessing Each training example is entered into hash l tables indexed by independantly constructed . g 1 ; : : : ; g l Preprocessing Space: O ( lN ) 1 2 l ... Feb 22, 2008 9
LSH Querying For each hash table i , 1 · i · l Return the bin indexed by g i ( q ) Perform a linear search on the union of the bins. q ... Feb 22, 2008 10
Parameter Selection Suppose we want to search at most B examples. Then setting ¶ log (1 =P 1 ) µ N ¶ µ N log (1 =P 2 ) k = log 1 =P 2 ; l = B B ensures that it will succeed with high probability. Feb 22, 2008 11
Experiment 1 Compare LSH accuracy and performance to exact NN search. Examine the influence of: k, the number of hash bits. l, the number of hash tables. B, the maximum search length. Dataset 59500 20x20 patches taken from motorcycle images. Represented as 400-dimensional column vectors Feb 22, 2008 12
Hash Function Convert the feature vectors into binary strings and use the Hamming hash functions. Given a vector we can create a unary x 2 N d representation for each element . x i = 1's followed by 0's, ( C ¡ x i ) x i Unary C ( x i ) where is the max coordinate for all points. C u ( x ) = Unary C ( x 1 ) ; : : : ; Unary C ( x d ) Note that for any two points : p; q k p; q k = k u ( p ) ; u ( q ) k H Feb 22, 2008 13
Example Query l = 20, k = 24, B = 1 Query = Examples searched: 7,722 of 59,500 Result = Actual NNs = Feb 22, 2008 14
Average Search Length Let B = 1 24 22 20 30 18 16 14 25 12 10 20 8 6 4 l 15 2 x1000 10 5 10 15 20 25 30 5 k Feb 22, 2008 15
Average Search Length Let B = 1 24 22 20 30 More hash bits, 18 (k), result in 16 shorter 14 25 searches. 12 10 More hash 20 8 tables (l), result 6 in longer 4 l 15 searches. 2 x1000 10 5 10 15 20 25 30 5 k Feb 22, 2008 16
Average Approximation Error Let B = 1 1.11 1.1 1.09 30 1.08 1.07 1.06 25 1.05 1.04 20 l 15 10 5 5 10 15 20 25 30 k Feb 22, 2008 17
Average Approximation Error Let B = 1 1.11 1.1 1.09 30 Over hashing 1.08 can result in too 1.07 few candidates 1.06 25 to return a good 1.05 approximation. 1.04 20 Over hashing can cause l 15 algorithm to fail. 10 5 5 10 15 20 25 30 k Feb 22, 2008 18
Average Approximation Error Let B = 1 1.11 1.1 1.09 30 Over hashing 1.08 can result in too 1.07 few candidates 1.06 25 to return a good 1.05 approximation. 1.04 20 Over hashing can cause l Average search 15 algorithm to fail. length = 8000 10 5 10 15 20 25 30 5 k Feb 22, 2008 19
Average Approximation Error N Let B = 5500 ¼ 1.15 ln N 1.14 1.13 30 1.12 1.11 1.1 25 1.09 1.08 20 l 15 10 5 5 10 15 20 25 30 k Feb 22, 2008 20
Average Approximation Error p Let B = 250 ¼ N 1.6 1.55 1.5 30 1.45 1.4 1.35 25 1.3 1.25 20 l 15 10 5 5 10 15 20 25 30 k Feb 22, 2008 21
Experiment 2 Examine the effect of the approximation on the subjective quality of the results. Dataset D. Nistér and H. Stewénius. Scalable recognition with a vocabulary tree 2550 sets of 4 images represented as document-term matrix of the visual words. Feb 22, 2008 22
Experiment 2: Issues LSH requires a vector representation. Not clear how to easily convert a bag of words representation into a vector one. A binary vector where the presence of each word is a bit does not provide a good distance measure. Each image has roughly the same number of different words from any other image. Boostmap? Feb 22, 2008 23
Conclusions Approximate Nearest Neighbors is neccessary for very large high dimensional datasets. LSH is a simple approach to aNN. LSH requires a vector representation. Clear relationship between search length and approximation error. Feb 22, 2008 24
Tools Octave (MATLAB) LSH Matlab Toolbox - http://www.cs.brown.edu/~gregory/code/lsh/ Python Gnuplot Feb 22, 2008 25
References 'Fast Pose Estimation with Parameter Senative Hashing' – Shakhnarovich et al. 'Similarity Search in High Dimensions via Hashing' – Gionis et al. 'Object Recognition Using Locality-Sensitive Hashing of Shape Contexts' - Andrea Frome and Jitendra Malik 'Nearest neighbors in high-dimensional spaces', Handbook of Discrete and Computational Geometry – Piotr Indyk Algorithms for Nearest Neighbor Search - http://simsearch.yury.name/tutorial.html LSH Matlab Toolbox - http://www.cs.brown.edu/~gregory/code/lsh/ Feb 22, 2008 26
Recommend
More recommend
