Today. Cuckoo hashing. Today. Cuckoo hashing. - - PowerPoint PPT Presentation

Today.

Cuckoo hashing.

Today.

Cuckoo hashing. Johnson-Lindenstrass.

Cuckoo hashing.

Hashing with two choices: max load O(loglogn).

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing: Array.

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

• Array. Two hash functions h1, h2.

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

• Array. Two hash functions h1, h2.

Insert x: place in h1(x) or h2(x) if space.

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

• Array. Two hash functions h1, h2.

Insert x: place in h1(x) or h2(x) if space. Else bump elt y in hi(x) u.a.r. for i ∈ [1,2].

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

• Array. Two hash functions h1, h2.

Insert x: place in h1(x) or h2(x) if space. Else bump elt y in hi(x) u.a.r. for i ∈ [1,2]. Bump y,x: place y in hj(y) where j = i if space.

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

• Array. Two hash functions h1, h2.

Insert x: place in h1(x) or h2(x) if space. Else bump elt y in hi(x) u.a.r. for i ∈ [1,2]. Bump y,x: place y in hj(y) where j = i if space. Else bump y′ in hj(y).

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

• Array. Two hash functions h1, h2.

Insert x: place in h1(x) or h2(x) if space. Else bump elt y in hi(x) u.a.r. for i ∈ [1,2]. Bump y,x: place y in hj(y) where j = i if space. Else bump y′ in hj(y). And so on.

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

• Array. Two hash functions h1, h2.

Insert x: place in h1(x) or h2(x) if space. Else bump elt y in hi(x) u.a.r. for i ∈ [1,2]. Bump y,x: place y in hj(y) where j = i if space. Else bump y′ in hj(y). And so on.

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

• Array. Two hash functions h1, h2.

Insert x: place in h1(x) or h2(x) if space. Else bump elt y in hi(x) u.a.r. for i ∈ [1,2]. Bump y,x: place y in hj(y) where j = i if space. Else bump y′ in hj(y). And so on. If go too long.

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

• Array. Two hash functions h1, h2.

Insert x: place in h1(x) or h2(x) if space. Else bump elt y in hi(x) u.a.r. for i ∈ [1,2]. Bump y,x: place y in hj(y) where j = i if space. Else bump y′ in hj(y). And so on. If go too long. Fail.

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

• Array. Two hash functions h1, h2.

Insert x: place in h1(x) or h2(x) if space. Else bump elt y in hi(x) u.a.r. for i ∈ [1,2]. Bump y,x: place y in hj(y) where j = i if space. Else bump y′ in hj(y). And so on. If go too long. Fail. Rehash entire hash table.

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

• Array. Two hash functions h1, h2.

Insert x: place in h1(x) or h2(x) if space. Else bump elt y in hi(x) u.a.r. for i ∈ [1,2]. Bump y,x: place y in hj(y) where j = i if space. Else bump y′ in hj(y). And so on. If go too long. Fail. Rehash entire hash table. Fails if cycle for insert.

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

• Array. Two hash functions h1, h2.

Insert x: place in h1(x) or h2(x) if space. Else bump elt y in hi(x) u.a.r. for i ∈ [1,2]. Bump y,x: place y in hj(y) where j = i if space. Else bump y′ in hj(y). And so on. If go too long. Fail. Rehash entire hash table. Fails if cycle for insert. Cℓ - event of cycle of length ℓ at a vertex.

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

• Array. Two hash functions h1, h2.

Insert x: place in h1(x) or h2(x) if space. Else bump elt y in hi(x) u.a.r. for i ∈ [1,2]. Bump y,x: place y in hj(y) where j = i if space. Else bump y′ in hj(y). And so on. If go too long. Fail. Rehash entire hash table. Fails if cycle for insert. Cℓ - event of cycle of length ℓ at a vertex. Pr[Cℓ] ≤ m ℓ n ℓ ℓ n 2(ℓ) ≤ e2 8 ℓ (1)

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

• Array. Two hash functions h1, h2.

Insert x: place in h1(x) or h2(x) if space. Else bump elt y in hi(x) u.a.r. for i ∈ [1,2]. Bump y,x: place y in hj(y) where j = i if space. Else bump y′ in hj(y). And so on. If go too long. Fail. Rehash entire hash table. Fails if cycle for insert. Cℓ - event of cycle of length ℓ at a vertex. Pr[Cℓ] ≤ m ℓ n ℓ ℓ n 2(ℓ) ≤ e2 8 ℓ (1) Probability that an insert hits a cycle of length ℓ ≤ ℓ

n

• e2

8

ℓ

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

• Array. Two hash functions h1, h2.

Insert x: place in h1(x) or h2(x) if space. Else bump elt y in hi(x) u.a.r. for i ∈ [1,2]. Bump y,x: place y in hj(y) where j = i if space. Else bump y′ in hj(y). And so on. If go too long. Fail. Rehash entire hash table. Fails if cycle for insert. Cℓ - event of cycle of length ℓ at a vertex. Pr[Cℓ] ≤ m ℓ n ℓ ℓ n 2(ℓ) ≤ e2 8 ℓ (1) Probability that an insert hits a cycle of length ℓ ≤ ℓ

n

• e2

8

ℓ Rehash every Ω(n) inserts (if ≤ n/8 items in table.)

Cuckoo hashing.

Hashing with two choices: max load O(loglogn). Cuckoo hashing:

• Array. Two hash functions h1, h2.

Insert x: place in h1(x) or h2(x) if space. Else bump elt y in hi(x) u.a.r. for i ∈ [1,2]. Bump y,x: place y in hj(y) where j = i if space. Else bump y′ in hj(y). And so on. If go too long. Fail. Rehash entire hash table. Fails if cycle for insert. Cℓ - event of cycle of length ℓ at a vertex. Pr[Cℓ] ≤ m ℓ n ℓ ℓ n 2(ℓ) ≤ e2 8 ℓ (1) Probability that an insert hits a cycle of length ℓ ≤ ℓ

n

• e2

8

ℓ Rehash every Ω(n) inserts (if ≤ n/8 items in table.) O(1) time on average.

Johnson-Lindenstrass

Points: x1,...,xn ∈ Rd.

Johnson-Lindenstrass

Points: x1,...,xn ∈ Rd. Random k = c logn

ε2

dimensional subspace.

Johnson-Lindenstrass

Points: x1,...,xn ∈ Rd. Random k = c logn

ε2

dimensional subspace. Claim: with probability 1−

1 nc−2 ,

(1−ε)

• k

d |xi −xj| ≤ |yi −yj| ≤ (1+ε)

• k

d |xi −xj|

Johnson-Lindenstrass

Points: x1,...,xn ∈ Rd. Random k = c logn

ε2

dimensional subspace. Claim: with probability 1−

1 nc−2 ,

(1−ε)

• k

d |xi −xj| ≤ |yi −yj| ≤ (1+ε)

• k

d |xi −xj| “Projecting and scaling by

• d

k preserves all pairwise distances w/in

factor of 1±ε.”

Random subspace.

Method 1:

Random subspace.

Method 1: Pick unit v1

Random subspace.

Method 1: Pick unit v1 ,

Random subspace.

Method 1: Pick unit v1 , v2 orthogonal to v1,

Random subspace.

Method 1: Pick unit v1 , v2 orthogonal to v1, ...

Random subspace.

Method 1: Pick unit v1 , v2 orthogonal to v1, ... vk orthogonal to previous vectors...

Random subspace.

Method 1: Pick unit v1 , v2 orthogonal to v1, ... vk orthogonal to previous vectors... Method 2:

Random subspace.

Method 1: Pick unit v1 , v2 orthogonal to v1, ... vk orthogonal to previous vectors... Method 2: Choose k vectors v1,...,vk

Random subspace.

Method 1: Pick unit v1 , v2 orthogonal to v1, ... vk orthogonal to previous vectors... Method 2: Choose k vectors v1,...,vk Gram Schmidt orthonormalization of k ×d matrix where rows are vi.

Random subspace.

Method 1: Pick unit v1 , v2 orthogonal to v1, ... vk orthogonal to previous vectors... Method 2: Choose k vectors v1,...,vk Gram Schmidt orthonormalization of k ×d matrix where rows are vi. remove projection onto previous subspace.

Projections.

Project x into subspace spanned by v1,v2,··· ,vk.

Projections.

Project x into subspace spanned by v1,v2,··· ,vk. y1 = x ·v1,y2 = x·,v2,··· ,yk = x ·vk

Projections.

Project x into subspace spanned by v1,v2,··· ,vk. y1 = x ·v1,y2 = x·,v2,··· ,yk = x ·vk Projection: (y1,...,yk).

Projections.

Project x into subspace spanned by v1,v2,··· ,vk. y1 = x ·v1,y2 = x·,v2,··· ,yk = x ·vk Projection: (y1,...,yk). Have: Arbitrary vector, random k-dimensional subspace.

Projections.

Project x into subspace spanned by v1,v2,··· ,vk. y1 = x ·v1,y2 = x·,v2,··· ,yk = x ·vk Projection: (y1,...,yk). Have: Arbitrary vector, random k-dimensional subspace. View As: Random vector, standard basis for k dimensions.

Projections.

Project x into subspace spanned by v1,v2,··· ,vk. y1 = x ·v1,y2 = x·,v2,··· ,yk = x ·vk Projection: (y1,...,yk). Have: Arbitrary vector, random k-dimensional subspace. View As: Random vector, standard basis for k dimensions. Orthogonal U - rotates v1,...,vk onto e1,...,ek

Projections.

Project x into subspace spanned by v1,v2,··· ,vk. y1 = x ·v1,y2 = x·,v2,··· ,yk = x ·vk Projection: (y1,...,yk). Have: Arbitrary vector, random k-dimensional subspace. View As: Random vector, standard basis for k dimensions. Orthogonal U - rotates v1,...,vk onto e1,...,ek yi

Projections.

Project x into subspace spanned by v1,v2,··· ,vk. y1 = x ·v1,y2 = x·,v2,··· ,yk = x ·vk Projection: (y1,...,yk). Have: Arbitrary vector, random k-dimensional subspace. View As: Random vector, standard basis for k dimensions. Orthogonal U - rotates v1,...,vk onto e1,...,ek yi = vi|x

Projections.

Project x into subspace spanned by v1,v2,··· ,vk. y1 = x ·v1,y2 = x·,v2,··· ,yk = x ·vk Projection: (y1,...,yk). Have: Arbitrary vector, random k-dimensional subspace. View As: Random vector, standard basis for k dimensions. Orthogonal U - rotates v1,...,vk onto e1,...,ek yi = vi|x = Uvi|Ux

Projections.

Project x into subspace spanned by v1,v2,··· ,vk. y1 = x ·v1,y2 = x·,v2,··· ,yk = x ·vk Projection: (y1,...,yk). Have: Arbitrary vector, random k-dimensional subspace. View As: Random vector, standard basis for k dimensions. Orthogonal U - rotates v1,...,vk onto e1,...,ek yi = vi|x = Uvi|Ux = ei|Ux

Projections.

Project x into subspace spanned by v1,v2,··· ,vk. y1 = x ·v1,y2 = x·,v2,··· ,yk = x ·vk Projection: (y1,...,yk). Have: Arbitrary vector, random k-dimensional subspace. View As: Random vector, standard basis for k dimensions. Orthogonal U - rotates v1,...,vk onto e1,...,ek yi = vi|x = Uvi|Ux = ei|Ux = ei|z

Projections.

Project x into subspace spanned by v1,v2,··· ,vk. y1 = x ·v1,y2 = x·,v2,··· ,yk = x ·vk Projection: (y1,...,yk). Have: Arbitrary vector, random k-dimensional subspace. View As: Random vector, standard basis for k dimensions. Orthogonal U - rotates v1,...,vk onto e1,...,ek yi = vi|x = Uvi|Ux = ei|Ux = ei|z Inverse of U maps ei to random vector vi

Projections.

Project x into subspace spanned by v1,v2,··· ,vk. y1 = x ·v1,y2 = x·,v2,··· ,yk = x ·vk Projection: (y1,...,yk). Have: Arbitrary vector, random k-dimensional subspace. View As: Random vector, standard basis for k dimensions. Orthogonal U - rotates v1,...,vk onto e1,...,ek yi = vi|x = Uvi|Ux = ei|Ux = ei|z Inverse of U maps ei to random vector vi

Projections.

Project x into subspace spanned by v1,v2,··· ,vk. y1 = x ·v1,y2 = x·,v2,··· ,yk = x ·vk Projection: (y1,...,yk). Have: Arbitrary vector, random k-dimensional subspace. View As: Random vector, standard basis for k dimensions. Orthogonal U - rotates v1,...,vk onto e1,...,ek yi = vi|x = Uvi|Ux = ei|Ux = ei|z Inverse of U maps ei to random vector vi z = Ux is uniformly distributed on d sphere for unit x ∈ Rd.

Projections.

Project x into subspace spanned by v1,v2,··· ,vk. y1 = x ·v1,y2 = x·,v2,··· ,yk = x ·vk Projection: (y1,...,yk). Have: Arbitrary vector, random k-dimensional subspace. View As: Random vector, standard basis for k dimensions. Orthogonal U - rotates v1,...,vk onto e1,...,ek yi = vi|x = Uvi|Ux = ei|Ux = ei|z Inverse of U maps ei to random vector vi z = Ux is uniformly distributed on d sphere for unit x ∈ Rd. yi is ith coordinate of random vector z.

Expected value of yi.

Random projection: first k coordinates of random unit vector, zi.

Expected value of yi.

Random projection: first k coordinates of random unit vector, zi. E[∑i∈[d] z2

i ] = 1.

Expected value of yi.

Random projection: first k coordinates of random unit vector, zi. E[∑i∈[d] z2

i ] = 1. Linearity of Expectation.

Expected value of yi.

Random projection: first k coordinates of random unit vector, zi. E[∑i∈[d] z2

i ] = 1. Linearity of Expectation.

By symmetry, each zi is identically distributed.

Expected value of yi.

Random projection: first k coordinates of random unit vector, zi. E[∑i∈[d] z2

i ] = 1. Linearity of Expectation.

By symmetry, each zi is identically distributed. E[∑i∈[k] z2

i ] = k d .

Expected value of yi.

Random projection: first k coordinates of random unit vector, zi. E[∑i∈[d] z2

i ] = 1. Linearity of Expectation.

By symmetry, each zi is identically distributed. E[∑i∈[k] z2

i ] = k d . Linearity of Expectation.

Expected value of yi.

Random projection: first k coordinates of random unit vector, zi. E[∑i∈[d] z2

i ] = 1. Linearity of Expectation.

By symmetry, each zi is identically distributed. E[∑i∈[k] z2

i ] = k d . Linearity of Expectation.

Expected length is

• k

d .

Expected value of yi.

Random projection: first k coordinates of random unit vector, zi. E[∑i∈[d] z2

i ] = 1. Linearity of Expectation.

By symmetry, each zi is identically distributed. E[∑i∈[k] z2

i ] = k d . Linearity of Expectation.

Expected length is

• k

d .

Johnson-Lindenstrass: close to expectation.

Expected value of yi.

Random projection: first k coordinates of random unit vector, zi. E[∑i∈[d] z2

i ] = 1. Linearity of Expectation.

By symmetry, each zi is identically distributed. E[∑i∈[k] z2

i ] = k d . Linearity of Expectation.

Expected length is

• k

d .

Johnson-Lindenstrass: close to expectation. k is large enough →

Expected value of yi.

Random projection: first k coordinates of random unit vector, zi. E[∑i∈[d] z2

i ] = 1. Linearity of Expectation.

By symmetry, each zi is identically distributed. E[∑i∈[k] z2

i ] = k d . Linearity of Expectation.

Expected length is

• k

d .

Johnson-Lindenstrass: close to expectation. k is large enough → ≈ (1±ε)

• k

d with decent probability.

Concentration Bounds.

z is uniformly random unit vector.

Concentration Bounds.

z is uniformly random unit vector. Random point on the unit sphere. E[∑i∈[k] z2

i ] = k d .

Concentration Bounds.

z is uniformly random unit vector. Random point on the unit sphere. E[∑i∈[k] z2

i ] = k d .

Claim: Pr[|z1| >

t √ d ] ≤ e−t2/2

Concentration Bounds.

z is uniformly random unit vector. Random point on the unit sphere. E[∑i∈[k] z2

i ] = k d .

Claim: Pr[|z1| >

t √ d ] ≤ e−t2/2

Sphere view: surface “far” from equator defined by e1.

Concentration Bounds.

z is uniformly random unit vector. Random point on the unit sphere. E[∑i∈[k] z2

i ] = k d .

Claim: Pr[|z1| >

t √ d ] ≤ e−t2/2

Sphere view: surface “far” from equator defined by e1. ∆

Concentration Bounds.

z is uniformly random unit vector. Random point on the unit sphere. E[∑i∈[k] z2

i ] = k d .

Claim: Pr[|z1| >

t √ d ] ≤ e−t2/2

Sphere view: surface “far” from equator defined by e1. ∆ |z1| ≥ ∆ if

Concentration Bounds.

z is uniformly random unit vector. Random point on the unit sphere. E[∑i∈[k] z2

i ] = k d .

Claim: Pr[|z1| >

t √ d ] ≤ e−t2/2

Sphere view: surface “far” from equator defined by e1. ∆ |z1| ≥ ∆ if z ≥ ∆ from equator of sphere.

Concentration Bounds.

z is uniformly random unit vector. Random point on the unit sphere. E[∑i∈[k] z2

i ] = k d .

Claim: Pr[|z1| >

t √ d ] ≤ e−t2/2

Sphere view: surface “far” from equator defined by e1. ∆ |z1| ≥ ∆ if z ≥ ∆ from equator of sphere. Point on “∆-spherical cap”.

Concentration Bounds.

z is uniformly random unit vector. Random point on the unit sphere. E[∑i∈[k] z2

i ] = k d .

Claim: Pr[|z1| >

t √ d ] ≤ e−t2/2

Sphere view: surface “far” from equator defined by e1. ∆ |z1| ≥ ∆ if z ≥ ∆ from equator of sphere. Point on “∆-spherical cap”. Area of caps

Concentration Bounds.

z is uniformly random unit vector. Random point on the unit sphere. E[∑i∈[k] z2

i ] = k d .

Claim: Pr[|z1| >

t √ d ] ≤ e−t2/2

Sphere view: surface “far” from equator defined by e1. ∆ |z1| ≥ ∆ if z ≥ ∆ from equator of sphere. Point on “∆-spherical cap”. Area of caps ≤ S.A. of sphere of radius √ 1−∆2

Concentration Bounds.

z is uniformly random unit vector. Random point on the unit sphere. E[∑i∈[k] z2

i ] = k d .

Claim: Pr[|z1| >

t √ d ] ≤ e−t2/2

Sphere view: surface “far” from equator defined by e1. ∆ |z1| ≥ ∆ if z ≥ ∆ from equator of sphere. Point on “∆-spherical cap”. Area of caps ≤ S.A. of sphere of radius √ 1−∆2 ∝ r d =

• 1−∆2d/2

Concentration Bounds.

z is uniformly random unit vector. Random point on the unit sphere. E[∑i∈[k] z2

i ] = k d .

Claim: Pr[|z1| >

t √ d ] ≤ e−t2/2

Sphere view: surface “far” from equator defined by e1. ∆ |z1| ≥ ∆ if z ≥ ∆ from equator of sphere. Point on “∆-spherical cap”. Area of caps ≤ S.A. of sphere of radius √ 1−∆2 ∝ r d =

• 1−∆2d/2

∝

• 1− t2

d

d/2

Concentration Bounds.

z is uniformly random unit vector. Random point on the unit sphere. E[∑i∈[k] z2

i ] = k d .

Claim: Pr[|z1| >

t √ d ] ≤ e−t2/2

Sphere view: surface “far” from equator defined by e1. ∆ |z1| ≥ ∆ if z ≥ ∆ from equator of sphere. Point on “∆-spherical cap”. Area of caps ≤ S.A. of sphere of radius √ 1−∆2 ∝ r d =

• 1−∆2d/2

∝

• 1− t2

d

d/2 ≈ e

−t2 2

Concentration Bounds.

z is uniformly random unit vector. Random point on the unit sphere. E[∑i∈[k] z2

i ] = k d .

Claim: Pr[|z1| >

t √ d ] ≤ e−t2/2

Sphere view: surface “far” from equator defined by e1. ∆ |z1| ≥ ∆ if z ≥ ∆ from equator of sphere. Point on “∆-spherical cap”. Area of caps ≤ S.A. of sphere of radius √ 1−∆2 ∝ r d =

• 1−∆2d/2

∝

• 1− t2

d

d/2 ≈ e

−t2 2

Constant of ∝ is unit sphere area.

Concentration Bounds.

z is uniformly random unit vector. Random point on the unit sphere. E[∑i∈[k] z2

i ] = k d .

Claim: Pr[|z1| >

t √ d ] ≤ e−t2/2

Sphere view: surface “far” from equator defined by e1. ∆ |z1| ≥ ∆ if z ≥ ∆ from equator of sphere. Point on “∆-spherical cap”. Area of caps ≤ S.A. of sphere of radius √ 1−∆2 ∝ r d =

• 1−∆2d/2

∝

• 1− t2

d

d/2 ≈ e

−t2 2

Constant of ∝ is unit sphere area. Pr[any z2

i > (2logd)E[z2 i ]] is small.

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length?

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length? z =

• z2

1 +z2 2 +···z2 k .

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length? z =

• z2

1 +z2 2 +···z2 k .

Pr[

• (z2

1 +z2 2 +···+z2 k )−

• k

d

• > t] ≤ e−t2d/2

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length? z =

• z2

1 +z2 2 +···z2 k .

Pr[

• (z2

1 +z2 2 +···+z2 k )−

• k

d

• > t] ≤ e−t2d/2

Substituting t = ε

• k

d , k = c logn ε2 .

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length? z =

• z2

1 +z2 2 +···z2 k .

Pr[

• (z2

1 +z2 2 +···+z2 k )−

• k

d

• > t] ≤ e−t2d/2

Substituting t = ε

• k

d , k = c logn ε2 .

Pr[

• z2

1 +z2 2 +···+z2 k −

• k

d

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length? z =

• z2

1 +z2 2 +···z2 k .

Pr[

• (z2

1 +z2 2 +···+z2 k )−

• k

d

• > t] ≤ e−t2d/2

Substituting t = ε

• k

d , k = c logn ε2 .

Pr[

• z2

1 +z2 2 +···+z2 k −

• k

d

• > ε
• k

d ]

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length? z =

• z2

1 +z2 2 +···z2 k .

Pr[

• (z2

1 +z2 2 +···+z2 k )−

• k

d

• > t] ≤ e−t2d/2

Substituting t = ε

• k

d , k = c logn ε2 .

Pr[

• z2

1 +z2 2 +···+z2 k −

• k

d

• > ε
• k

d ] ≤ e−ε2k

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length? z =

• z2

1 +z2 2 +···z2 k .

Pr[

• (z2

1 +z2 2 +···+z2 k )−

• k

d

• > t] ≤ e−t2d/2

Substituting t = ε

• k

d , k = c logn ε2 .

Pr[

• z2

1 +z2 2 +···+z2 k −

• k

d

• > ε
• k

d ] ≤ e−ε2k = e−c logn = 1 nc

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length? z =

• z2

1 +z2 2 +···z2 k .

Pr[

• (z2

1 +z2 2 +···+z2 k )−

• k

d

• > t] ≤ e−t2d/2

Substituting t = ε

• k

d , k = c logn ε2 .

Pr[

• z2

1 +z2 2 +···+z2 k −

• k

d

• > ε
• k

d ] ≤ e−ε2k = e−c logn = 1 nc

Johnson-Lindenstraus: For n points, x1,...,xn, all distances preserved to within 1±ε under

• k

d -scaled projection above.

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length? z =

• z2

1 +z2 2 +···z2 k .

Pr[

• (z2

1 +z2 2 +···+z2 k )−

• k

d

• > t] ≤ e−t2d/2

Substituting t = ε

• k

d , k = c logn ε2 .

Pr[

• z2

1 +z2 2 +···+z2 k −

• k

d

• > ε
• k

d ] ≤ e−ε2k = e−c logn = 1 nc

Johnson-Lindenstraus: For n points, x1,...,xn, all distances preserved to within 1±ε under

• k

d -scaled projection above.

View one pair xi −xj as vector.

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length? z =

• z2

1 +z2 2 +···z2 k .

Pr[

• (z2

1 +z2 2 +···+z2 k )−

• k

d

• > t] ≤ e−t2d/2

Substituting t = ε

• k

d , k = c logn ε2 .

Pr[

• z2

1 +z2 2 +···+z2 k −

• k

d

• > ε
• k

d ] ≤ e−ε2k = e−c logn = 1 nc

Johnson-Lindenstraus: For n points, x1,...,xn, all distances preserved to within 1±ε under

• k

d -scaled projection above.

View one pair xi −xj as vector. Scale to unit.

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length? z =

• z2

1 +z2 2 +···z2 k .

Pr[

• (z2

1 +z2 2 +···+z2 k )−

• k

d

• > t] ≤ e−t2d/2

Substituting t = ε

• k

d , k = c logn ε2 .

Pr[

• z2

1 +z2 2 +···+z2 k −

• k

d

• > ε
• k

d ] ≤ e−ε2k = e−c logn = 1 nc

Johnson-Lindenstraus: For n points, x1,...,xn, all distances preserved to within 1±ε under

• k

d -scaled projection above.

View one pair xi −xj as vector. Scale to unit. Projection fails to preserve |xi −xj|

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length? z =

• z2

1 +z2 2 +···z2 k .

Pr[

• (z2

1 +z2 2 +···+z2 k )−

• k

d

• > t] ≤ e−t2d/2

Substituting t = ε

• k

d , k = c logn ε2 .

Pr[

• z2

1 +z2 2 +···+z2 k −

• k

d

• > ε
• k

d ] ≤ e−ε2k = e−c logn = 1 nc

Johnson-Lindenstraus: For n points, x1,...,xn, all distances preserved to within 1±ε under

• k

d -scaled projection above.

View one pair xi −xj as vector. Scale to unit. Projection fails to preserve |xi −xj| with probability ≤ 1

nc

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length? z =

• z2

1 +z2 2 +···z2 k .

Pr[

• (z2

1 +z2 2 +···+z2 k )−

• k

d

• > t] ≤ e−t2d/2

Substituting t = ε

• k

d , k = c logn ε2 .

Pr[

• z2

1 +z2 2 +···+z2 k −

• k

d

• > ε
• k

d ] ≤ e−ε2k = e−c logn = 1 nc

Johnson-Lindenstraus: For n points, x1,...,xn, all distances preserved to within 1±ε under

• k

d -scaled projection above.

View one pair xi −xj as vector. Scale to unit. Projection fails to preserve |xi −xj| with probability ≤ 1

nc

Scaled vector length also preserved.

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length? z =

• z2

1 +z2 2 +···z2 k .

Pr[

• (z2

1 +z2 2 +···+z2 k )−

• k

d

• > t] ≤ e−t2d/2

Substituting t = ε

• k

d , k = c logn ε2 .

Pr[

• z2

1 +z2 2 +···+z2 k −

• k

d

• > ε
• k

d ] ≤ e−ε2k = e−c logn = 1 nc

Johnson-Lindenstraus: For n points, x1,...,xn, all distances preserved to within 1±ε under

• k

d -scaled projection above.

View one pair xi −xj as vector. Scale to unit. Projection fails to preserve |xi −xj| with probability ≤ 1

nc

Scaled vector length also preserved. ≤ n2 pairs

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length? z =

• z2

1 +z2 2 +···z2 k .

Pr[

• (z2

1 +z2 2 +···+z2 k )−

• k

d

• > t] ≤ e−t2d/2

Substituting t = ε

• k

d , k = c logn ε2 .

Pr[

• z2

1 +z2 2 +···+z2 k −

• k

d

• > ε
• k

d ] ≤ e−ε2k = e−c logn = 1 nc

Johnson-Lindenstraus: For n points, x1,...,xn, all distances preserved to within 1±ε under

• k

d -scaled projection above.

View one pair xi −xj as vector. Scale to unit. Projection fails to preserve |xi −xj| with probability ≤ 1

nc

Scaled vector length also preserved. ≤ n2 pairs plus union bound

Many coordinates.

Argued Pr[any z2

i > (2logd)E[z2 i ]] is small.

Total Length? z =

• z2

1 +z2 2 +···z2 k .

Pr[

• (z2

1 +z2 2 +···+z2 k )−

• k

d

• > t] ≤ e−t2d/2

Substituting t = ε

• k

d , k = c logn ε2 .

Pr[

• z2

1 +z2 2 +···+z2 k −

• k

d

• > ε
• k

d ] ≤ e−ε2k = e−c logn = 1 nc

Johnson-Lindenstraus: For n points, x1,...,xn, all distances preserved to within 1±ε under

• k

d -scaled projection above.

View one pair xi −xj as vector. Scale to unit. Projection fails to preserve |xi −xj| with probability ≤ 1

nc

Scaled vector length also preserved. ≤ n2 pairs plus union bound → prob any pair fails to be preserved with ≤

1 nc−2 .

Locality Preserving Hashing

Find nearby points in high dimensional space.

Locality Preserving Hashing

Find nearby points in high dimensional space. Points could be images!

Locality Preserving Hashing

Find nearby points in high dimensional space. Points could be images! Hash function h(·) s.t. h(xi) = h(xj) if d(xi,xj) ≤ δ.

Locality Preserving Hashing

Find nearby points in high dimensional space. Points could be images! Hash function h(·) s.t. h(xi) = h(xj) if d(xi,xj) ≤ δ. Low dimensions: grid cells give √ d-approximation.

Locality Preserving Hashing

Find nearby points in high dimensional space. Points could be images! Hash function h(·) s.t. h(xi) = h(xj) if d(xi,xj) ≤ δ. Low dimensions: grid cells give √ d-approximation. Not quite a solution.

Locality Preserving Hashing

Find nearby points in high dimensional space. Points could be images! Hash function h(·) s.t. h(xi) = h(xj) if d(xi,xj) ≤ δ. Low dimensions: grid cells give √ d-approximation. Not quite a solution. Why?

Locality Preserving Hashing

Find nearby points in high dimensional space. Points could be images! Hash function h(·) s.t. h(xi) = h(xj) if d(xi,xj) ≤ δ. Low dimensions: grid cells give √ d-approximation. Not quite a solution. Why? Close to grid boundary.

Locality Preserving Hashing

Find nearby points in high dimensional space. Points could be images! Hash function h(·) s.t. h(xi) = h(xj) if d(xi,xj) ≤ δ. Low dimensions: grid cells give √ d-approximation. Not quite a solution. Why? Close to grid boundary. Find close points to x:

Locality Preserving Hashing

Find nearby points in high dimensional space. Points could be images! Hash function h(·) s.t. h(xi) = h(xj) if d(xi,xj) ≤ δ. Low dimensions: grid cells give √ d-approximation. Not quite a solution. Why? Close to grid boundary. Find close points to x: Check grid cell and neighboring grid cells.

Locality Preserving Hashing

Find nearby points in high dimensional space. Points could be images! Hash function h(·) s.t. h(xi) = h(xj) if d(xi,xj) ≤ δ. Low dimensions: grid cells give √ d-approximation. Not quite a solution. Why? Close to grid boundary. Find close points to x: Check grid cell and neighboring grid cells.

Locality Preserving Hashing

Find nearby points in high dimensional space. Points could be images! Hash function h(·) s.t. h(xi) = h(xj) if d(xi,xj) ≤ δ. Low dimensions: grid cells give √ d-approximation. Not quite a solution. Why? Close to grid boundary. Find close points to x: Check grid cell and neighboring grid cells. Project high dimensional points into low dimensions.

Locality Preserving Hashing

Find nearby points in high dimensional space. Points could be images! Hash function h(·) s.t. h(xi) = h(xj) if d(xi,xj) ≤ δ. Low dimensions: grid cells give √ d-approximation. Not quite a solution. Why? Close to grid boundary. Find close points to x: Check grid cell and neighboring grid cells. Project high dimensional points into low dimensions. Use grid hash function.

Implementing Johnson-Lindenstraus

Random vectors

Implementing Johnson-Lindenstraus

Random vectors have many bits

Implementing Johnson-Lindenstraus

Random vectors have many bits Use random bit vectors: {−1,+1}d instead.

Implementing Johnson-Lindenstraus

Random vectors have many bits Use random bit vectors: {−1,+1}d instead. Almost orthogonal.

Implementing Johnson-Lindenstraus

Random vectors have many bits Use random bit vectors: {−1,+1}d instead. Almost orthogonal. Project z.

Implementing Johnson-Lindenstraus

Random vectors have many bits Use random bit vectors: {−1,+1}d instead. Almost orthogonal. Project z. Coordinate for bit vector b.

Implementing Johnson-Lindenstraus

Random vectors have many bits Use random bit vectors: {−1,+1}d instead. Almost orthogonal. Project z. Coordinate for bit vector b. Cl =

1 √ d ∑i bizi

Implementing Johnson-Lindenstraus

Random vectors have many bits Use random bit vectors: {−1,+1}d instead. Almost orthogonal. Project z. Coordinate for bit vector b. Cl =

1 √ d ∑i bizi

E[C2

l ] =

Implementing Johnson-Lindenstraus

Random vectors have many bits Use random bit vectors: {−1,+1}d instead. Almost orthogonal. Project z. Coordinate for bit vector b. Cl =

1 √ d ∑i bizi

E[C2

l ] = E[ 1 d ∑i,j bibjzizj]

Implementing Johnson-Lindenstraus

Random vectors have many bits Use random bit vectors: {−1,+1}d instead. Almost orthogonal. Project z. Coordinate for bit vector b. Cl =

1 √ d ∑i bizi

E[C2

l ] = E[ 1 d ∑i,j bibjzizj] = 1 d ∑i,j E[bibj]zizj = 1 d ∑i z2 i

Implementing Johnson-Lindenstraus

Random vectors have many bits Use random bit vectors: {−1,+1}d instead. Almost orthogonal. Project z. Coordinate for bit vector b. Cl =

1 √ d ∑i bizi

E[C2

l ] = E[ 1 d ∑i,j bibjzizj] = 1 d ∑i,j E[bibj]zizj = 1 d ∑i z2 i = 1 d

Implementing Johnson-Lindenstraus

Random vectors have many bits Use random bit vectors: {−1,+1}d instead. Almost orthogonal. Project z. Coordinate for bit vector b. Cl =

1 √ d ∑i bizi

E[C2

l ] = E[ 1 d ∑i,j bibjzizj] = 1 d ∑i,j E[bibj]zizj = 1 d ∑i z2 i = 1 d

Implementing Johnson-Lindenstraus

Random vectors have many bits Use random bit vectors: {−1,+1}d instead. Almost orthogonal. Project z. Coordinate for bit vector b. Cl =

1 √ d ∑i bizi

E[C2

l ] = E[ 1 d ∑i,j bibjzizj] = 1 d ∑i,j E[bibj]zizj = 1 d ∑i z2 i = 1 d

E[∑l C2

l ] = k d

Binary Johnson-Lindenstrass

Project onto [−1,+1] vectors.

Binary Johnson-Lindenstrass

Project onto [−1,+1] vectors. E[C] = E[∑l C2

l ] = k d

Binary Johnson-Lindenstrass

Project onto [−1,+1] vectors. E[C] = E[∑l C2

l ] = k d

Concentration?

Binary Johnson-Lindenstrass

Project onto [−1,+1] vectors. E[C] = E[∑l C2

l ] = k d

Concentration? Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Binary Johnson-Lindenstrass

Project onto [−1,+1] vectors. E[C] = E[∑l C2

l ] = k d

Concentration? Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Choose k = c logn

ε2 .

Binary Johnson-Lindenstrass

Project onto [−1,+1] vectors. E[C] = E[∑l C2

l ] = k d

Concentration? Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Choose k = c logn

ε2 .

→ failure probability ≤ 1/nc.

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Variance of C2?

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Variance of C2? Recall Cl =

1 √ d ∑i bizi

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Variance of C2? Recall Cl =

1 √ d ∑i bizi

Var(C) ≤

• k

d2

• (∑i zi 4 +4∑i,j z2

i z2 j )

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Variance of C2? Recall Cl =

1 √ d ∑i bizi

Var(C) ≤

• k

d2

• (∑i zi 4 +4∑i,j z2

i z2 j ) ≤

• k

d2

• 2(∑i z2

i )2

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Variance of C2? Recall Cl =

1 √ d ∑i bizi

Var(C) ≤

• k

d2

• (∑i zi 4 +4∑i,j z2

i z2 j ) ≤

• k

d2

• 2(∑i z2

i )2 ≤ 2k d2 .

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Variance of C2? Recall Cl =

1 √ d ∑i bizi

Var(C) ≤

• k

d2

• (∑i zi 4 +4∑i,j z2

i z2 j ) ≤

• k

d2

• 2(∑i z2

i )2 ≤ 2k d2 .

Roughly normal (gaussian):

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Variance of C2? Recall Cl =

1 √ d ∑i bizi

Var(C) ≤

• k

d2

• (∑i zi 4 +4∑i,j z2

i z2 j ) ≤

• k

d2

• 2(∑i z2

i )2 ≤ 2k d2 .

Roughly normal (gaussian): Density ∝ e−t2/2 for t std deviations away.

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Variance of C2? Recall Cl =

1 √ d ∑i bizi

Var(C) ≤

• k

d2

• (∑i zi 4 +4∑i,j z2

i z2 j ) ≤

• k

d2

• 2(∑i z2

i )2 ≤ 2k d2 .

Roughly normal (gaussian): Density ∝ e−t2/2 for t std deviations away. So, assuming normality

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Variance of C2? Recall Cl =

1 √ d ∑i bizi

Var(C) ≤

• k

d2

• (∑i zi 4 +4∑i,j z2

i z2 j ) ≤

• k

d2

• 2(∑i z2

i )2 ≤ 2k d2 .

Roughly normal (gaussian): Density ∝ e−t2/2 for t std deviations away. So, assuming normality σ =

√ 2k d ,

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Variance of C2? Recall Cl =

1 √ d ∑i bizi

Var(C) ≤

• k

d2

• (∑i zi 4 +4∑i,j z2

i z2 j ) ≤

• k

d2

• 2(∑i z2

i )2 ≤ 2k d2 .

Roughly normal (gaussian): Density ∝ e−t2/2 for t std deviations away. So, assuming normality σ =

√ 2k d , t = ε k

d √ 2k d

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Variance of C2? Recall Cl =

1 √ d ∑i bizi

Var(C) ≤

• k

d2

• (∑i zi 4 +4∑i,j z2

i z2 j ) ≤

• k

d2

• 2(∑i z2

i )2 ≤ 2k d2 .

Roughly normal (gaussian): Density ∝ e−t2/2 for t std deviations away. So, assuming normality σ =

√ 2k d , t = ε k

d √ 2k d

= ε √ k/ √ 2.

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Variance of C2? Recall Cl =

1 √ d ∑i bizi

Var(C) ≤

• k

d2

• (∑i zi 4 +4∑i,j z2

i z2 j ) ≤

• k

d2

• 2(∑i z2

i )2 ≤ 2k d2 .

Roughly normal (gaussian): Density ∝ e−t2/2 for t std deviations away. So, assuming normality σ =

√ 2k d , t = ε k

d √ 2k d

= ε √ k/ √ 2. Probability of failure roughly ≤ e−t2/2

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Variance of C2? Recall Cl =

1 √ d ∑i bizi

Var(C) ≤

• k

d2

• (∑i zi 4 +4∑i,j z2

i z2 j ) ≤

• k

d2

• 2(∑i z2

i )2 ≤ 2k d2 .

Roughly normal (gaussian): Density ∝ e−t2/2 for t std deviations away. So, assuming normality σ =

√ 2k d , t = ε k

d √ 2k d

= ε √ k/ √ 2. Probability of failure roughly ≤ e−t2/2 → eε2k/4

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Variance of C2? Recall Cl =

1 √ d ∑i bizi

Var(C) ≤

• k

d2

• (∑i zi 4 +4∑i,j z2

i z2 j ) ≤

• k

d2

• 2(∑i z2

i )2 ≤ 2k d2 .

Roughly normal (gaussian): Density ∝ e−t2/2 for t std deviations away. So, assuming normality σ =

√ 2k d , t = ε k

d √ 2k d

= ε √ k/ √ 2. Probability of failure roughly ≤ e−t2/2 → eε2k/4 “Roughly normal.”

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Variance of C2? Recall Cl =

1 √ d ∑i bizi

Var(C) ≤

• k

d2

• (∑i zi 4 +4∑i,j z2

i z2 j ) ≤

• k

d2

• 2(∑i z2

i )2 ≤ 2k d2 .

Roughly normal (gaussian): Density ∝ e−t2/2 for t std deviations away. So, assuming normality σ =

√ 2k d , t = ε k

d √ 2k d

= ε √ k/ √ 2. Probability of failure roughly ≤ e−t2/2 → eε2k/4 “Roughly normal.” Chernoff, Berry-Esseen, Central Limit Theorems.

Analysis Idea.

Pr

• |C − k

d | ≥ ε k d

• ≤ e−ε2k

Variance of C2? Recall Cl =

1 √ d ∑i bizi

Var(C) ≤

• k

d2

• (∑i zi 4 +4∑i,j z2

i z2 j ) ≤

• k

d2

• 2(∑i z2

i )2 ≤ 2k d2 .

Roughly normal (gaussian): Density ∝ e−t2/2 for t std deviations away. So, assuming normality σ =

√ 2k d , t = ε k

d √ 2k d

= ε √ k/ √ 2. Probability of failure roughly ≤ e−t2/2 → eε2k/4 “Roughly normal.” Chernoff, Berry-Esseen, Central Limit Theorems.

Summary

Cuckoo hashing.

Summary

Cuckoo hashing. Two hash functions.

Summary

Cuckoo hashing. Two hash functions. Few cycles in random sparse graph.

Summary

Cuckoo hashing. Two hash functions. Few cycles in random sparse graph. Chaining works!

Summary

Cuckoo hashing. Two hash functions. Few cycles in random sparse graph. Chaining works! Johnson-Lindenstrass.

Summary

Cuckoo hashing. Two hash functions. Few cycles in random sparse graph. Chaining works! Johnson-Lindenstrass. O(logn) dimensions give good approximation of distances.