Composite Quantization for Approximate Nearest Neighbor Search - - PowerPoint PPT Presentation

Jingdong Wang Lead Researcher Microsoft Research

http://research.microsoft.com/~jingdw ICML 2104, joint work with my interns Ting Zhang from USTC and Chao Du from Tsinghua University

Composite Quantization for Approximate Nearest Neighbor Search

5/6/2015

Outline

• Introduction
• Problem
• Product quantization
• Cartesian k-means
• Composite quantization
• Experiments

5/6/2015

Nearest neighbor search

• Application to similar image search

query

5/6/2015

Nearest neighbor search

• Application to particular object retrieval

query

5/6/2015

Nearest neighbor search

• Application to duplicate image search

5/6/2015

Nearest neighbor search

• Similar image search

Application to K-NN annotation: Annotate the query image using the similar images

5/6/2015

Nearest neighbor search

• Definition
• Database:
• Query:
• Nearest neighbor:

𝑦 − 𝑟 2

∈ 𝑆𝑒

5/6/2015

Nearest neighbor search

• Exact nearest neighbor search
• linear scan:

5/6/2015

Nearest neighbor search - speedup

• K-dimensional tree (Kd tree)
• Generalized binary search tree
• Metric tree
• Ball tree
• VP tree
• BD tree
• Cover tree
• …

5/6/2015

Nearest neighbor search

• Exact nearest neighbor search
• linear scan:
• Costly and impractical for large scale high-dimensional cases
• Approximate nearest neighbor (ANN) search
• Efficient
• Acceptable accuracy
• Practically used

5/6/2015

Two principles for ANN search

Recall the complexity of linear scan:

• 1. Reduce the number of distance computations
• Time complexity:
• Tree structure, neighborhood graph search and inverted index

5/6/2015

Our work: TP Tree + NG Search

• TP Tree
• Jingdong Wang, Naiyan Wang, You Jia, Jian Li, Gang Zeng, Hongbin Zha, Xian-Sheng Hua:Trinary-

Projection Trees for Approximate Nearest Neighbor Search. IEEE Trans. Pattern Anal. Mach. Intell. 36(2): 388-403 (2014)

• You Jia, Jingdong Wang, Gang Zeng, Hongbin Zha, Xian-Sheng Hua: Optimizing kd-trees for scalable

visual descriptor indexing. CVPR 2010: 3392-3399

• Neighborhood graph search
• Jingdong Wang, Shipeng Li:Query-driven iterated neighborhood graph search for large scale
• indexing. ACM Multimedia 2012: 179-188
• Jing Wang, Jingdong Wang, Gang Zeng, Rui Gan, Shipeng Li, Baining Guo: Fast Neighborhood

Graph Search Using Cartesian Concatenation. ICCV 2013: 2128-2135

• Neighborhood graph construction
• Jing Wang, Jingdong Wang, Gang Zeng, Zhuowen Tu, Rui Gan, Shipeng Li: Scalable k-NN graph

construction for visual descriptors. CVPR 2012: 1106-1113

5/6/2015

Comparison over SIFT 1M

13

ICCV13 ACMMM12 CVPR10 1 NN

5/6/2015

Comparison over GIST 1M

14

ICCV13 ACMMM12 CVPR10 1 NN

5/6/2015

Comparison over HOG 10M

15

ICCV13 ACMMM12 CVPR10 1 NN

5/6/2015

Neighborhood Graph Search

• Shipped to Bing ClusterBed
• Number
• Index building time on 40M documents is only 2 hours and 10 minutes.
• Search DPS on each NNS machine stable at 950 without retry and errors
• Five times faster improved FLANN

5/6/2015

Two principles for ANN search

Recall the complexity of linear scan:

• 1. Reduce the number of distance computations
• Time complexity:
• Tree structure, neighborhood graph search and inverted index
• 2. Reduce the cost of each distance computation
• Time complexity:
• Hashing (compact codes)
• 1. High efficiency 
• 2. Large memory cost 
• 1. Small memory cost 
• 2. Low efficiency 

5/6/2015

Approximate nearest neighbor search

• Binary embedding methods (hashing)
• Produce a few distinct distances
• Limited ability and flexibility of distance approximation
• Vector quantization (compact codes)
• K-means
• Impossible to use for medium and large code length
• Impossible to learn the codebook
• Impossible to compute a code for a vector
• Product quantization
• Cartesian k-means

5/6/2015

Combined solution for very large scale search

Retrieve candidates with an index structure using compact codes Load raw features for retrieved candidates from disk Reranking using the true distances Efficient and small memory consumption IO cost is small

5/6/2015

Outline

• Introduction
• Problem
• Product quantization
• Cartesian k-means
• Composite quantization
• Experiments

5/6/2015

Product quantization

• Approximate x by the concatenation of M subvectors

5/6/2015

x = x1𝑗1 x2𝑗2 ⋮ x𝑁𝑗𝑁 ≈ x = p1𝑗1 p2𝑗2 ⋮ p𝑁𝑗𝑁

Product quantization

• Approximate x by the concatenation of M subvectors

{p11, p12, ⋯ , p1𝐿}

Codebook in the 1st subspace

{p21, p22, ⋯ , p2𝐿}

Codebook in the 2nd subspace

{p𝑁1, p𝑁2, ⋯ , p𝑁𝐿}

Codebook in the Mth subspace

5/6/2015

x = x1 x2 ⋮ x𝑁 ≈ x = p1𝑗1 p2𝑗2 ⋮ p𝑁𝑗𝑁

Product quantization

• Approximate x by the concatenation of M subvectors

{p11, p12, ⋯ , p1𝐿}

Codebook in the 1st subspace

{p21, p22, ⋯ , p2𝐿}

Codebook in the 2nd subspace

{p𝑁1, p𝑁2, ⋯ , p𝑁𝐿}

Codebook in the Mth subspace

5/6/2015

Product quantization

• Approximate x by the concatenation of M subvectors

{p11, p12, ⋯ , p1𝐿} {p21, p22, ⋯ , p2𝐿}

Codebook in the 2nd subspace

{p𝑁1, p𝑁2, ⋯ , p𝑁𝐿}

Codebook in the Mth subspace

x = x1 x2 ⋮ x𝑁 ≈ x = p1𝑗1 p2𝑗2 ⋮ p𝑁𝑗𝑁

Codebook in the 1st subspace

5/6/2015

Product quantization

• Approximate x by the concatenation of M subvectors

{p11, p12, ⋯ , p1𝐿}

Codebook in the 1st subspace

{p21, p22, ⋯ , p2𝐿}

Codebook in the 2nd subspace

{p𝑁1, p𝑁2, ⋯ , p𝑁𝐿}

Codebook in the Mth subspace

x = x1 x2 ⋮ x𝑁 ≈ x = p1𝑗1 p2𝑗2 ⋮ p𝑁𝑗𝑁

5/6/2015

Product quantization

• Approximate x by the concatenation of M subvectors
• Code presentation: (𝑗1, 𝑗2, …, 𝑗𝑁)
• Distance computation:
• 𝑒 q,

x 2 = 𝑒 q1, p1𝑗1

2 + 𝑒 q2, p2𝑗2 2 + ⋯ + 𝑒 qM, p𝑁𝑗𝑁 2

• M additions using a pre-computed distance table

{p11, p12, ⋯ , p1𝐿}

Codebook in the 1st subspace

{p21, p22, ⋯ , p2𝐿}

Codebook in the 2nd subspace

{p𝑁1, p𝑁2, ⋯ , p𝑁𝐿}

Codebook in the Mth subspace

x = x1 x2 ⋮ x𝑁 ≈ x = p1𝑗1 p2𝑗2 ⋮ p𝑁𝑗𝑁

q1

𝑒( 𝑒( 𝑒(

, q1 ) , q2 ) , q𝑁 ) ≜ {𝑒 q1, p11 , 𝑒 q1, p12 , ⋯ , 𝑒(q1, p1𝐿)}

5/6/2015

𝑁 = 2

Product quantization

• Approximate x by the concatenation of M subvectors
• Codebook generation
• Do k-means for each subspace

5/6/2015

Product quantization

• Approximate x by the concatenation of M subvectors
• Codebook generation
• Do k-means for each subspace

𝑁 = 2, 𝐿 = 3

𝑞11 𝑞12 𝑞13 𝑞21 𝑞22 𝑞23

5/6/2015

Product quantization

• Approximate x by the concatenation of M subvectors
• Codebook generation
• Do k-means for each subspace
• Result in 𝐿𝑁 groups
• The center of each is the concatenation of M subvectors

𝑁 = 2, 𝐿 = 3

𝑞11 𝑞12 𝑞13 𝑞21 𝑞22 𝑞23 = 𝑞11 𝑞21

5/6/2015

Product quantization

• Approximate x by the concatenation of M subvectors
• Codebook generation
• Do k-means for each subspace
• Result in 𝐿𝑁 groups
• The center of each is the concatenation of M subvectors

𝑁 = 2, 𝐿 = 3

5/6/2015

Product quantization

• Approximate x by the concatenation of M subvectors
• Codebook generation
• Do k-means for each subspace
• Result in 𝐿𝑁 groups
• The center of each is the concatenation of M subvectors

𝑁 = 2, 𝐿 = 3

5/6/2015

Product quantization

• Approximate x by the concatenation of M subvectors
• Codebook generation
• Do k-means for each subspace
• Result in 𝐿𝑁 groups
• The center of each is the concatenation of M subvectors

𝑁 = 2, 𝐿 = 3

x 𝑞13 𝑞22 x ≈ x = 𝑞13 𝑞22

5/6/2015

Outline

• Introduction
• Problem
• Product quantization
• Cartesian k-means
• Composite quantization
• Experiments

5/6/2015

Cartesian K-means

• Extended product quantization
• Optimal space rotation
• Perform PQ over the rotated space

x ≈ x = R p1𝑗1 p2𝑗2 ⋮ p𝑁𝑗𝑁

5/6/2015

Cartesian K-means

• Extended product quantization
• Optimal space rotation
• Perform PQ over the rotated space

𝑞11 𝑞21 𝑞12 𝑞13 𝑞23 𝑞22

x ≈ x = R p1𝑗1 p2𝑗2 ⋮ p𝑁𝑗𝑁

5/6/2015

Cartesian K-means

• Extended product quantization
• Optimal space rotation
• Perform PQ over the rotated space

x

x ≈ x = R 𝑞13 𝑞22

𝑞11 𝑞21 𝑞12 𝑞13 𝑞23 𝑞22

x ≈ x = R p1𝑗1 p2𝑗2 ⋮ p𝑁𝑗𝑁

5/6/2015

Outline

• Introduction
• Composite quantization
• Experiments

5/6/2015

Composite quantization

x ≈ x = c1𝑗1 + c2𝑗2 + ⋯ + c𝑁𝑗𝑁

{c11, c12, ⋯ , c1𝐿}

Source codebook 1

{c21, c22, ⋯ , c2𝐿}

Source codebook 2

{c𝑁1, c𝑁2, ⋯ , c𝑁𝐿}

Source codebook M

• Approximate x by the addition of M vectors

⋯

Each source codebook is composed of K d-dimensional vectors

5/6/2015

Composite quantization

c12 c11 c13 {c11, c12, c13} {c21, c22, c23}

2 source codebooks:

c22 c23 c21

5/6/2015

Composite quantization

{c11, c12, c13} {c21, c22, c23}

2 source codebooks: Composite center:

= c11 + c21 c11 c21

5/6/2015

Composite quantization

{c11, c12, c13} {c21, c22, c23}

2 source codebooks: Composite center:

= c11 + c22 c11 c22

5/6/2015

Composite quantization

{c11, c12, c13} {c21, c22, c23}

2 source codebooks: Composite center:

= c11 + c23 c11 c23

5/6/2015

Composite quantization

{c11, c12, c13} {c21, c22, c23}

2 source codebooks: More composite centers

5/6/2015

Composite quantization

{c11, c12, c13} {c21, c22, c23}

Source codebook: Composite codebook: 9 composite centers

5/6/2015

Composite quantization

{c11, c12, c13} {c21, c22, c23}

Source codebook:

x

c11 𝑑22 x ≈ x = c11+ c22

9 groups Space partition:

5/6/2015

Composite quantization

x ≈ x = c1𝑗1 + c2𝑗2 + ⋯ + c𝑁𝑗𝑁

{c11, c12, ⋯ , c1𝐿}

Source codebook 1

{c21, c22, ⋯ , c2𝐿}

Source codebook 2

{c𝑁1, c𝑁2, ⋯ , c𝑁𝐿}

Source codebook M

• Approximate x by the addition of M vectors

⋯

5/6/2015

Composite quantization

x ≈ x = c1𝑗1 + c2𝑗2 + ⋯ + c𝑁𝑗𝑁

{c11, c12, ⋯ , c1𝐿} {c21, c22, ⋯ , c2𝐿} {c𝑁1, c𝑁2, ⋯ , c𝑁𝐿}

Source codebook 1 Source codebook 2 Source codebook M

• Approximate x by the addition of M vectors
• Code representation: 𝑗1𝑗2 ⋯ 𝑗𝑁
• length: 𝑁log𝐿

⋯

5/6/2015

Connection to product quantization

• Concatenation in product quantization

x ≈ x = p1𝑗1 p2𝑗2 ⋮ p𝑁𝑗𝑁 = p1𝑗1 ⋮ + p2𝑗2 ⋮ + ⋯ + ⋮ p𝑁𝑗𝑁

5/6/2015

Connection to product quantization

• Concatenation in product quantization = addition

x ≈ x = p1𝑗1 p2𝑗2 ⋮ p𝑁𝑗𝑁 = p1𝑗1 ⋮ + p2𝑗2 ⋮ + ⋯ + ⋮ p𝑁𝑗𝑁

5/6/2015

Connection to product quantization

• Concatenation in product quantization = addition

x ≈ x = p1𝑗1 p2𝑗2 ⋮ p𝑁𝑗𝑁 = p1𝑗1 ⋮ + p2𝑗2 ⋮ + ⋯ + ⋮ p𝑁𝑗𝑁

5/6/2015

Connection to product quantization

• Concatenation in product quantization = addition

x ≈ x = p1𝑗1 p2𝑗2 ⋮ p𝑁𝑗𝑁 = p1𝑗1 ⋮ + p2𝑗2 ⋮ + ⋯ + ⋮ p𝑁𝑗𝑁

5/6/2015

Connection to product quantization

• Concatenation in product quantization = addition
• Composite quantization

x ≈ x = p1𝑗1 p2𝑗2 ⋮ p𝑁𝑗𝑁 = p1𝑗1 ⋮ + p2𝑗2 ⋮ + ⋯ + ⋮ p𝑁𝑗𝑁 x ≈ x = c1𝑗1 + c2𝑗2 + ⋯ + c𝑁𝑗𝑁

Product quantization is constrained composite quantization

5/6/2015

Connection to Cartesian k-means

• Concatenation in Cartesian k-means = addition

x ≈ x = R p1𝑗1 p2𝑗2 ⋮ p𝑁𝑗𝑁 = R p1𝑗1 ⋮ + R p2𝑗2 ⋮ + ⋯ + R ⋮ p𝑁𝑗𝑁

5/6/2015

Connection to Cartesian k-means

• Concatenation in Cartesian k-means = addition

x ≈ x = R p1𝑗1 p2𝑗2 ⋮ p𝑁𝑗𝑁 = R p1𝑗1 ⋮ + R p2𝑗2 ⋮ + ⋯ + R ⋮ p𝑁𝑗𝑁

5/6/2015

Connection to Cartesian k-means

• Concatenation in Cartesian k-means = addition
• Composite quantization

x ≈ x = R p1𝑗1 p2𝑗2 ⋮ p𝑁𝑗𝑁 = R p1𝑗1 ⋮ + R p2𝑗2 ⋮ + ⋯ + R ⋮ p𝑁𝑗𝑁 x ≈ x = c1𝑗1 + c2𝑗2 + ⋯ + c𝑁𝑗𝑁

Cartesian k-means is constrained composite quantization

5/6/2015

• Composite quantization
• Generalizing product quantization and Cartesian k-means
• Advantages
• More flexible codebooks
• More accurate data approximation
• Higher search accuracy
• Search efficiency?
• Depend on the efficiency of the distance computation

Composite quantization

Product quantization: Coordinate aligned space partition Cartesian k-means: Rotated coordinate aligned space partition Composite quantization: Flexible space partition

5/6/2015

Approximate distance computation

• Recall the data approximation
• x ≈

x = 𝑛=1

𝑁

c𝑛𝑗𝑛 x

• Approximate distance to the query q
• q − x 2

2 ≈

q − 𝑛=1

𝑁

c𝑛𝑗𝑛 x

2 2

Time-consuming

5/6/2015

Approximate distance computation

q − 𝑛=1

𝑁

c𝑛𝑗𝑛 x

2 2

= 𝑛=1

𝑁

q − c𝑛𝑗𝑛(x) 2

2 − 𝑁 − 1

q 2

2 + 𝑛≠𝑚 c𝑛𝑗𝑛(x) 𝑈

c𝑚𝑗𝑚(x)

Expanded into three terms

5/6/2015

Approximate distance computation

q − 𝑛=1

𝑁

c𝑛𝑗𝑛 x

2 2

= 𝑛=1

𝑁

q − c𝑛𝑗𝑛(x) 2

2 − 𝑁 − 1

q 2

2 + 𝑛≠𝑚 c𝑛𝑗𝑛(x) 𝑈

c𝑚𝑗𝑚(x)

𝑃(𝑁) additions Implemented with a pre-computed distance lookup table Distance lookup table: Store the distances from source codebook elements to q

5/6/2015

Approximate distance computation

q − 𝑛=1

𝑁

c𝑛𝑗𝑛 x

2 2

= 𝑛=1

𝑁

q − c𝑛𝑗𝑛(x) 2

2 − 𝑁 − 1

q 2

2 + 𝑛≠𝑚 c𝑛𝑗𝑛(x) 𝑈

c𝑚𝑗𝑚(x)

Constant 𝑃(𝑁) additions

5/6/2015

Using a pre-computed dot product lookup table

Approximate distance computation

q − 𝑛=1

𝑁

c𝑛𝑗𝑛 x

2 2

= 𝑛=1

𝑁

q − c𝑛𝑗𝑛(x) 2

2 − 𝑁 − 1

q 2

2 + 𝑛≠𝑚 c𝑛𝑗𝑛(x) 𝑈

c𝑚𝑗𝑚(x)

O(𝑁2) additions 𝑃(𝑁) additions Dot product lookup table: Store the dot products between codebook elements Constant

5/6/2015

Approximate distance computation

q − 𝑛=1

𝑁

c𝑛𝑗𝑛 x

2 2

= 𝑛=1

𝑁

q − c𝑛𝑗𝑛(x) 2

2 − 𝑁 − 1

q 2

2 + 𝑛≠𝑚 c𝑛𝑗𝑛(x) 𝑈

c𝑚𝑗𝑚(x)

𝑃(𝑁2): still expensive

𝑃(𝑁) additions O(𝑁2) additions Constant

5/6/2015

Approximate distance computation

q − 𝑛=1

𝑁

c𝑛𝑗𝑛 x

2 2

= 𝑛=1

𝑁

q − c𝑛𝑗𝑛(x) 2

2 − 𝑁 − 1

q 2

2 + 𝑛≠𝑚 c𝑛𝑗𝑛(x) 𝑈

c𝑚𝑗𝑚(x)

If constant

𝑃(𝑁) additions Constant

5/6/2015

Approximate distance computation

q − 𝑛=1

𝑁

c𝑛𝑗𝑛 x

2 2

= 𝑛=1

𝑁

q − c𝑛𝑗𝑛(x) 2

2 − 𝑁 − 1

q 2

2 + 𝑛≠𝑚 c𝑛𝑗𝑛(x) 𝑈

c𝑚𝑗𝑚(x)

If constant

Computing this is enough for search 𝑃(𝑁) additions Constant

5/6/2015

Formulation

q − 𝑛=1

𝑁

c𝑛𝑗𝑛 x

2 2

= 𝑛=1

𝑁

q − c𝑛𝑗𝑛(x) 2

2 − 𝑁 − 1

q 2

2 + 𝑛≠𝑚 c𝑛𝑗𝑛(x) 𝑈

c𝑚𝑗𝑚(x)

Subject to the third term is a constant Minimize quantization error: x − 𝑛=1

𝑁

c𝑛𝑗𝑛 x

2 2

Constant

5/6/2015

Formulation

• Constrained formulation

min

C𝑛 , 𝑗𝑛 x ,𝜗 x x − 𝑛=1 𝑁

c𝑛𝑗𝑛 x

2 2

𝑡. 𝑢. 𝑛≠𝑚 c𝑛𝑗𝑛(x)

𝑈

c𝑚𝑗𝑚(x) = 𝜗

Minimize quantization error for search accuracy Constant constraint for search efficiency

5/6/2015

Connection to PQ and CKM

• Constrained formulation

min

C𝑛 , 𝑗𝑛 x ,𝜗 x x − 𝑛=1 𝑁

c𝑛𝑗𝑛 x

2 2

𝑡. 𝑢. 𝑛≠𝑚 c𝑛𝑗𝑛(x)

𝑈

c𝑚𝑗𝑚(x) = 𝜗

Minimize quantization error for search accuracy Constant constraint for search efficiency

5/6/2015

Connection to PQ and CKM

• Constrained formulation

min

C𝑛 , 𝑗𝑛 x ,𝜗 x x − 𝑛=1 𝑁

c𝑛𝑗𝑛 x

2 2

𝑡. 𝑢. 𝑛≠𝑚 c𝑛𝑗𝑛(x)

𝑈

c𝑚𝑗𝑚(x) = 𝜗

• Product quantization and Cartesian k-means: suboptimal solutions of
• ur approach

Non-overlapped space partitioning Codebooks are mutually orthogonal

𝑛≠𝑚

c𝑛𝑗𝑛(x)

𝑈

c𝑚𝑗𝑚(x) = 𝜗 Product quantization and Cartesian k-means Minimize quantization error for search accuracy Constant constraint for search efficiency

5/6/2015

Formulation

• Constrained formulation

min

C𝑛 , 𝑗𝑛 x ,𝜗 x x − 𝑛=1 𝑁

c𝑛𝑗𝑛 x

2 2

𝑡. 𝑢. 𝑛≠𝑚 c𝑛𝑗𝑛(x)

𝑈

c𝑚𝑗𝑚(x) = 𝜗

• Transformed to unconstrained formulation
• Add the constraints into the objective function

𝜚 {C𝑛 , 𝑗𝑛(x) , 𝜗) = x x − 𝑛=1

𝑁

c𝑛𝑗𝑛 x

2 2 + 𝜈 x 𝑛≠𝑚 c𝑛𝑗𝑛(x) 𝑈

c𝑚𝑗𝑚(x) − 𝜗

2 Minimize quantization error for search accuracy Constant constraint for search efficiency

5/6/2015

Optimization

• Unconstrained formulation

𝜚 {C𝑛 , 𝑗𝑛(x) , 𝜗) = x x − 𝑛=1

𝑁

c𝑛𝑗𝑛 x

2 2 + 𝜈 x 𝑛≠𝑚 c𝑛𝑗𝑛(x) 𝑈

c𝑚𝑗𝑚(x) − 𝜗

2

• Alternative optimization between C𝑛 , 𝑗𝑛(x) , 𝜗

Selected by validation Distortion error Constraints violation

5/6/2015

Alternative optimization

• Update {𝑗𝑛(x)}
• Iteratively alternative optimization, fixing 𝑗𝑚(x) l≠𝑛, update 𝑗𝑛(x)
• Update 𝜗
• Closed form solution,𝜗 =

1 #{x} x 𝑛≠𝑚 c𝑛𝑗𝑛 x 𝑈

c𝑚𝑗𝑚(x)

• Update {C𝑛}
• L-BFGS algorithm

5/6/2015

Complexity

• Update {𝑗𝑛(x)}
• Iteratively alternative optimization, fixing 𝑗𝑚(x) l≠𝑛, update 𝑗𝑛(x)
• 𝑃(𝑁𝐿𝑒𝑈𝑒)
• Update 𝜗
• Closed form solution,𝜗 =

1 #{x} x 𝑛≠𝑚 c𝑛𝑗𝑛 x 𝑈

c𝑚𝑗𝑚(x)

• 𝑃(𝑂𝑁2)
• Update {C𝑛}
• L-BFGS algorithm
• 𝑃(𝑂𝑁𝑒𝑈𝑚𝑈

𝑑)

Convergency?

5/6/2015

Convergence

1MSIFT, 64 bits Converge about 10~15 iterations

5/6/2015

Outline

• Introduction
• Composite quantization
• Experiments

5/6/2015

Experiments on ANN search

• Datasets
• 1 million of 128D SIFT vectors, 10000 queries
• 1 million of 960D GIST vectors, 1000 queries
• 1 billion of 128D SIFT vectors, 1000 queries
• Evaluation
• Recall@R
• the fraction of queries for which the ground-truth Euclidean nearest neighbor is in the R

retrieved items

5/6/2015

Search pipeline

Query q Code of database vector x Distance between q and x Output the nearest vectors Repeated for n database vectors Source codebooks Distance tables

(between query and codebook elements)

5/6/2015

Comparison on 1M SIFT and 1M GIST

5/6/2015

Comparison on 1M SIFT and 1M GIST

Our:71.59% CKM:63.83% Recall@10: 64 btis

5/6/2015

Comparison on 1M SIFT and 1M GIST

Our:71.59% CKM:63.83% Relatively small improvement on 1M GIST might be that CKM has already achieved large improvement

5/6/2015

Comparison on 1M SIFT and 1M GIST

Our:71.59% 64 bits ITQ: 53.95% 128 bits ITQ without asymmetric distance underperformed ITQ with asymmetric distance Our approach with 64 bits outperforms (A) ITQ with 128 bits, with slightly smaller search cost

5/6/2015

Comparison on 1B SIFT

5/6/2015

Comparison on 1B SIFT

Our:70.12% CKM:64.57% Recall@100:

5/6/2015

Average query time

Average query time

5/6/2015

Application to object retrieval

• Two datasets
• INRIA Holidays contains 500 queries and 991 corresponding relevant images
• UKBench contains 2550 groups of 4 images each
• Results

MAP on the holiday dataset Scores on the UKBench dataset

5/6/2015

Discussion

• The effect of 𝜗
• min 𝑦 𝑦 −

𝑦 2

2 subject to 𝑛≠𝑚 c𝑛𝑗𝑛(𝑦) 𝑈

c𝑚𝑗𝑚(𝑦) = 𝜗

• Indicate the dictionaries are mutual orthogonal like splitting the space

𝑛≠𝑚 c𝑛𝑗𝑛(𝑦)

𝑈

c𝑚𝑗𝑚(𝑦) = 0

Search performance with learnt 𝜗 is better, since learning 𝜗 is more flexible

(R,T) recall@R

5/6/2015

Discussion

• The effect of translation
• min 𝑦 𝑦 −

𝑦 2

2 subject to 𝑛≠𝑚 c𝑛𝑗𝑛(𝑦) 𝑈

c𝑚𝑗𝑚(𝑦) = 𝜗

• search performance doesn’t change too much

𝑦 𝑦 − 𝑢 − 𝑦 2

2

recall@R (R,T)

Contribution of the offset is relatively small compared with the composite quantization

5/6/2015

Extension (CVPR15)

• Constrained formulation

min

C𝑛 , 𝑗𝑛 x ,𝜗 x x − 𝑛=1 𝑁

c𝑛𝑗𝑛 x

2 2

𝑡. 𝑢. 𝑛≠𝑚 c𝑛𝑗𝑛(x)

𝑈

c𝑚𝑗𝑚(x) = 𝜗 𝑛 C𝑛 1 ≤ 𝑈

Minimize quantization error for search accuracy Constant constraint for search efficiency Sparsity constraint for precomputation efficiency

5/6/2015

Extension (CVPR15)

• Constrained formulation

min

C𝑛 , 𝑗𝑛 x ,𝜗 x x − P 𝑛=1 𝑁

c𝑛𝑗𝑛 x

2 2

𝑡. 𝑢. 𝑛≠𝑚 c𝑛𝑗𝑛(x)

𝑈

c𝑚𝑗𝑚(x) = 𝜗 𝑛 C𝑛 1 ≤ 𝑈

Minimize quantization error for search accuracy Constant constraint for search efficiency Sparsity constraint for precomputation efficiency Dimension reduction

5/6/2015

Multi-stage vector quantization

Vector quantization (1) Vector quantization (2) Residuals Vector quantization (M) Residuals Residuals …… Database 𝑌

5/6/2015

Conclusion

• Composite quantization
• A compact coding approach for approximate nearest neighbor search
• Joint optimization of search accuracy and search efficiency
• State-of-the-art performance

5/6/2015

A Survey on Learning to Hash

http://research.microsoft.com/~jingdw/pubs/lthsurvey.pdf

5/6/2015

Call for papers

http://research.microsoft.com/~jingdw/cfp/CFP_TBDSI_BMD.pdf

5/6/2015

Call for papers

http://research.microsoft.com/~jingdw/cfp/CFP_ICDM15WORKSHOP_BMD.pdf

5/6/2015

Thanks Q&A

5/6/2015

A distance preserving view of quantization

• Quantization
• Data approximation:

x ≈ x

• Better search
• If better distance preserving: q −

x 2 ≈ q − x 2

• Distance preserving view
• Triangle inequality:

q − x 2 − q − x 2 ≤ x − x 2

• Minimize the upper bound: x x −

x 2

2

q x x

5/6/2015

A joint minimization view

q − x 2 − q − x 2 ≤ x − x 2

Triangle inequality

𝑒 q, x = (Σ𝑛=1

𝑁

q − c𝑛𝑗𝑛 x

2 2)1/2

𝑒 q, x = ( q − x 2

2 + 𝑁 − 1

q 2

2)1/2

𝜀 = Σ𝑛≠𝑚c𝑛𝑗𝑛(x)

𝑈

c𝑚𝑗𝑚(x) x = Σ𝑛=1

𝑁

c𝑛𝑗𝑛(𝑦)

Distortion Efficiency

𝑒 q, x − 𝑒(q, x) ≤ x − x 2 + |𝜀|1/2

Generalized triangle inequality

5/6/2015

A joint minimization view

min

C𝑛 , 𝑗𝑛 x ,𝜗 Σx x −

x 2

2

𝑡. 𝑢. δ = 𝜗

Our formulation

Minimize distortion for search accuracy Constant constraint for search efficiency

𝑒 q, x − 𝑒(q, x) ≤ x − x 2 + |𝜀|1/2

Generalized triangle inequality

Distortion Efficiency