Lecture #2: Advanced hashing and concentration bounds o Bloom - - PowerPoint PPT Presentation

• utline

Lecture #2: Advanced hashing and concentration bounds

• Bloom filters
• Cuckoo hashing
• Load balancing
• Tail bounds

Bloom filters

Idea: For the sake of efficiency, sometime we allow our data structure to make mistakes Bloom filter: Hash table that has only false positives (may report that a key is present when it is not, but always reports a key that is present) Very simple and fast Example: Google Chrome uses a Bloom filter to maintain its list of potentially malicious web sites.

• Most queried keys are not in the table
• If a key is in the table, can check against a slower (errorless) hash table

Many applications in networking (see survey by Broder and Mitzenmacher)

Bloom filters

Data structure: Universe 𝒱. Parameters 𝑙, 𝑁 ≥ 1 Maintain an array 𝐵 of 𝑁 bits; initially 𝐵 0 = 𝐵 1 = ⋯ = 𝐵 𝑁 − 1 = 0 Choose 𝑙 hash functions ℎ1, ℎ2, … , ℎ𝑙: 𝒱 → 𝑁 (assume completely random functions for sake of analysis)

Bloom filters

Data structure: Universe 𝒱. Parameters 𝑙, 𝑁 ≥ 1 Maintain an array 𝐵 of 𝑁 bits; initially 𝐵 0 = 𝐵 1 = ⋯ = 𝐵 𝑁 − 1 = 0 Choose 𝑙 hash functions ℎ1, ℎ2, … , ℎ𝑙: 𝒱 → 𝑁 (assume completely random functions for sake of analysis) To add a key 𝑦 ∈ 𝒱 to the dictionary 𝑇 ⊆ 𝒱, set bits 𝐵 ℎ1 𝑦 ≔ 1, 𝐵 ℎ2 𝑦 ≔ 1, … , 𝐵 ℎ𝑙 𝑦 ≔ 1

Bloom filters

Data structure: Universe 𝒱. Parameters 𝑙, 𝑁 ≥ 1 Maintain an array 𝐵 of 𝑁 bits; initially 𝐵 0 = 𝐵 1 = ⋯ = 𝐵 𝑁 − 1 = 0 Choose 𝑙 hash functions ℎ1, ℎ2, … , ℎ𝑙: 𝒱 → 𝑁 (assume completely random functions for sake of analysis) To add a key 𝑦 ∈ 𝒱 to the dictionary 𝑇 ⊆ 𝒱, set bits 𝐵 ℎ1 𝑦 ≔ 1, 𝐵 ℎ2 𝑦 ≔ 1, … , 𝐵 ℎ𝑙 𝑦 ≔ 1 To answer a query: 𝑟 ∈ 𝑇 ? Check whether 𝐵 ℎ𝑗 𝑦 = 1 for all 𝑗 = 1,2, … , 𝑙 If yes, answer Yes. If no, answer No.

Bloom filters

No false negatives: Clearly if 𝑦 ∈ 𝑇, we return Yes. But there is some chance that other keys have caused the bits in positions ℎ1 𝑦 , … , ℎ𝑙(𝑦) to be set even if 𝑦 ∉ 𝑇.

Bloom filters

No false negatives: Clearly if 𝑦 ∈ 𝑇, we return Yes. Let us assume that 𝑇 = 𝑜. Compute ℙ[𝐵 ℓ = 0] for some location ℓ ∈ [𝑁]: 𝑞 𝑙, 𝑂 = 1 − 1 𝑁

𝑙𝑂

≈ 𝑓−𝑙𝑂

𝑁

But there is some chance that other keys have caused the bits in positions ℎ1 𝑦 , … , ℎ𝑙(𝑦) to be set even if 𝑦 ∉ 𝑇. Heuristic analysis:

(Here we use the approximation 1 −

1 𝑁 𝑁

≈ 𝑓−1 for 𝑁 large enough.)

Bloom filters

No false negatives: Clearly if 𝑦 ∈ 𝑇, we return Yes. Let us assume that 𝑇 = 𝑜. Compute ℙ[𝐵 ℓ = 0] for some location ℓ ∈ [𝑁]: 𝑞 𝑙, 𝑂 = 1 − 1 𝑁

𝑙𝑂

≈ 𝑓−𝑙𝑂

𝑁

But there is some chance that other keys have caused the bits in positions ℎ1 𝑦 , … , ℎ𝑙(𝑦) to be set even if 𝑦 ∉ 𝑇. Heuristic analysis:

(Here we use the approximation 1 −

1 𝑁 𝑁

≈ 𝑓−1 for 𝑁 large enough.)

If each location in 𝐵 is 0 with probability 𝑞(𝑙, 𝑂), then a false positive for 𝑦 ∉ 𝑇 should happen with probability at most 1 − 𝑞 𝑙, 𝑂

𝑙 ≈ 1 − 𝑓−𝑙𝑂 𝑁 𝑙

Bloom filters

Heuristic analysis: If each location in 𝐵 is 0 with probability 𝑞(𝑙, 𝑂), then a false positive for 𝑦 ∉ 𝑇 should happen with probability at most 1 − 𝑞 𝑙, 𝑂

𝑙 ≈ 1 − 𝑓−𝑙𝑂 𝑁 𝑙

Bloom filters

Heuristic analysis: If each location in 𝐵 is 0 with probability 𝑞(𝑙, 𝑂), then a false positive for 𝑦 ∉ 𝑇 should happen with probability at most 1 − 𝑞 𝑙, 𝑂

𝑙 ≈ 1 − 𝑓−𝑙𝑂 𝑁 𝑙

But the actual fraction of 0′𝑡 in the hash table is a random variable 𝑌𝑙,𝑂 with expectation 𝔽 𝑌𝑙,𝑂 = 𝑞 𝑙, 𝑂 To get the analysis right, we need a concentration bound: Want to say that 𝑌𝑙,𝑂 is close to its expected value with high probability. [We will return to this in the 2nd half of the lecture]

Bloom filters

Heuristic analysis: If each location in 𝐵 is 0 with probability 𝑞(𝑙, 𝑂), then a false positive for 𝑦 ∉ 𝑇 should happen with probability at most 1 − 𝑞 𝑙, 𝑂

𝑙 ≈ 1 − 𝑓−𝑙𝑂 𝑁 𝑙

But the actual fraction of 0′𝑡 in the hash table is a random variable 𝑌𝑙,𝑂 with expectation 𝔽 𝑌𝑙,𝑂 = 𝑞 𝑙, 𝑂 To get the analysis right, we need a concentration bound: Want to say that 𝑌𝑙,𝑂 is close to its expected value with high probability. [We will return to this in the 2nd half of the lecture] If the heuristic analysis is correct, it gives nice estimates: For instance, if 𝑁 = 8𝑂, then choosing the optimal value of 𝑙 = 7 gives false positive rate about 2%.

• utline

Lecture #2: Advanced hashing and concentration bounds

• Bloom filters
• Cuckoo hashing
• Load balancing
• Tail bounds

Cuckoo hashing is a hash scheme with worst-case constant lookup time. The name derives from the behavior of some species of cuckoo, where the cuckoo chick pushes the other eggs or young out of the nest when it hatches; analogously, inserting a new key into a cuckoo hashing table may push an older key to a different location in the table.

Cuckoo hashing

Idea: Simple hashing without errors Lookups are worst case 𝑃(1) time Deletions are worst case 𝑃(1) time Insertions are expected 𝑃(1) time Insertion time is 𝑃(1) with good probability [will require a concentration bound]

Cuckoo hashing

Data structure: Two tables 𝐵1 and 𝐵2 both of size 𝑁 = 𝑃 𝑂 Two hash functions ℎ1, ℎ2 ∶ 𝒱 → [𝑁] (will assume hash functions are fully random) When an element 𝑦 ∈ 𝑇 is inserted, if either 𝐵1 ℎ1 𝑦

• r 𝐵2[ℎ2 𝑦 ]

is empty, store 𝑦 there.

Cuckoo hashing

Data structure: Two tables 𝐵1 and 𝐵2 both of size 𝑁 = 𝑃 𝑂 Two hash functions ℎ1, ℎ2 ∶ 𝒱 → [𝑁] (will assume hash functions are fully random) When an element 𝑦 ∈ 𝑇 is inserted, if either 𝐵1 ℎ1 𝑦

• r 𝐵2[ℎ2 𝑦 ]

is empty, store 𝑦 there. Bump: Whenever an element 𝑨 is bumped from 𝐵𝑗 ℎ𝑗 𝑨 , attempt to store it in the other location 𝐵𝑘 ℎ𝑘 𝑨

(here 𝑗, 𝑘 = 1,2 or 2,1 )

If both locations are occupied, then place 𝑦 in 𝐵1 ℎ1 𝑦 and bump the current occupant.

Cuckoo hashing

Data structure: Two tables 𝐵1 and 𝐵2 both of size 𝑁 = 𝑃 𝑂 Two hash functions ℎ1, ℎ2 ∶ 𝒱 → [𝑁] (will assume hash functions are fully random) When an element 𝑦 ∈ 𝑇 is inserted, if either 𝐵1 ℎ1 𝑦

• r 𝐵2[ℎ2 𝑦 ]

is empty, store 𝑦 there. Bump: Whenever an element 𝑨 is bumped from 𝐵𝑗 ℎ𝑗 𝑨 , attempt to store it in the other location 𝐵𝑘 ℎ𝑘 𝑨

(here 𝑗, 𝑘 = 1,2 or 2,1 )

Abort: After 6 log 𝑂 consecutive bumps, stop the process and build a fresh hash table using new random hash functions ℎ1, ℎ2. If both locations are occupied, then place 𝑦 in 𝐵1 ℎ1 𝑦 and bump the current occupant.

Cuckoo hashing

Arrows represent the alternate location for each key. If we insert an item at the location of 𝐵, it will get bumped, thereby bumping 𝐶, and then we are done. Cycles are possible (where the insertion process never completes). What’s an example? Alternately (as in the picture), we can use a single table with 2𝑁 entries and two hash functions ℎ1, ℎ2: 𝒱 → 2𝑁 (with the same “bumping” algorithm)

Cuckoo hashing

Data structure: Two tables 𝐵1 and 𝐵2 both of size 𝑁 = 𝑃 𝑂 Two hash functions ℎ1, ℎ2 ∶ 𝒱 → [𝑁] (will assume hash functions are fully random) Theorem: Expected time to perform an insert operation is 𝑃(1) if 𝑁 ≥ 4𝑂.

Cuckoo hashing

Data structure: Two tables 𝐵1 and 𝐵2 both of size 𝑁 = 𝑃 𝑂 Two hash functions ℎ1, ℎ2 ∶ 𝒱 → [𝑁] (will assume hash functions are fully random) Theorem: Expected time to perform an insert operation is 𝑃(1) if 𝑁 ≥ 4𝑂. Pretty good… but only 25% memory utilization. Can actually get about 50% memory utilization. Experimentally, with 3 hash functions instead of 2, can get ≈ 90% utilization, but it is an open question to provide tight analyses for 𝑒 hash functions when 𝑒 ≥ 3.

• utline

Lecture #2: Advanced hashing and concentration bounds

• Bloom filters
• Cuckoo hashing
• Load balancing
• Tail bounds

Load balancing

Suppose we have 𝑂 jobs to assign to 𝑂 servers. Clearly we could achieve a load of one job/server, but this might result in an expensive/hard-to-parallelize allocation rule.

Load balancing

Suppose we have 𝑂 jobs to assign to 𝑂 servers. Clearly we could achieve a load of one job/server, but this might result in an expensive/hard-to-parallelize allocation rule. We could hash the balls into bins. Let’s again consider the case of a uniformly random hash function ℎ ∶ 𝑂 → 𝑂

Load balancing

Suppose we have 𝑂 jobs to assign to 𝑂 servers. Clearly we could achieve a load of one job/server, but this might result in an expensive/hard-to-parallelize allocation rule. We could hash the balls into bins. Let’s again consider the case of a uniformly random hash function ℎ ∶ 𝑂 → 𝑂 Claim: The max-loaded server has < 8 log 𝑂 / log log 𝑂 jobs with probability at least 1 − 1/𝑂

Load balancing

Suppose we have 𝑂 jobs to assign to 𝑂 servers. Clearly we could achieve a load of one job/server, but this might result in an expensive/hard-to-parallelize allocation rule. We could hash the balls into bins. Let’s again consider the case of a uniformly random hash function ℎ ∶ 𝑂 → 𝑂 Claim: The max-loaded server has < 8 log 𝑂 / log log 𝑂 jobs with probability at least 1 − 1/𝑂 Proof: Probability that a fixed server 𝑗 ∈ {1,2, … , 𝑂} gets at least 𝑙 jobs is at most 𝑂 𝑙 1 𝑂

𝑙

≤ 𝑂𝑙 𝑙! ⋅ 1 𝑂𝑙 ≤ 1 𝑙! ≤ 𝑙−𝑙

2

Load balancing

Suppose we have 𝑂 jobs to assign to 𝑂 servers. Clearly we could achieve a load of one job/server, but this might result in an expensive/hard-to-parallelize allocation rule. We could hash the balls into bins. Let’s again consider the case of a uniformly random hash function ℎ ∶ 𝑂 → 𝑂 Claim: The max-loaded server has < 8 log 𝑂 / log log 𝑂 jobs with probability at least 1 − 1/𝑂 Proof: Probability that a fixed server 𝑗 ∈ {1,2, … , 𝑂} gets at least 𝑙 jobs is at most 𝑂 𝑙 1 𝑂

𝑙

≤ 𝑂𝑙 𝑙! ⋅ 1 𝑂𝑙 ≤ 1 𝑙! ≤ 𝑙−𝑙

2

If we choose 𝑙 =

8 log 𝑂 log log 𝑂 this is at most 1/𝑂2

Explanation: 𝑙

𝑙 2 ≥

log 𝑂

4 log 𝑂 log log 𝑂 ≥ 22 log 𝑂 = 𝑂2

Load balancing

Suppose we have 𝑂 jobs to assign to 𝑂 servers. Clearly we could achieve a load of one job/server, but this might result in an expensive/hard-to-parallelize allocation rule. We could hash the balls into bins. Let’s again consider the case of a uniformly random hash function ℎ ∶ 𝑂 → 𝑂 Claim: The max-loaded server has < 8 log 𝑂 / log log 𝑂 jobs with probability at least 1 − 1/𝑂 Proof: Probability that a fixed server 𝑗 ∈ {1,2, … , 𝑂} gets at least 𝑙 jobs is at most 𝑂 𝑙 1 𝑂

𝑙

≤ 𝑂𝑙 𝑙! ⋅ 1 𝑂𝑙 ≤ 1 𝑙! ≤ 𝑙−𝑙

2

If we choose 𝑙 =

8 log 𝑂 log log 𝑂 this is at most 1/𝑂2

Now a union bound shows that the probability of any server getting at least 𝑙 jobs is at most 1/𝑂.

Concentration bounds

Claim: The max-loaded server has < 8 log 𝑂 / log log 𝑂 jobs with probability at least 1 − 1/𝑂 Proof: Probability that a fixed server 𝑗 ∈ {1,2, … , 𝑂} gets at least 𝑙 jobs is at most 𝑂 𝑙 1 𝑂

𝑙

≤ 𝑂𝑙 𝑙! ⋅ 1 𝑂𝑙 ≤ 1 𝑙! ≤ 𝑙−𝑙

2

If we choose 𝑙 =

8 log 𝑂 log log 𝑂 this is at most 1/𝑂2

Now a union bound shows that the probability of any server getting at least 𝑙 jobs is at most 1/𝑂. This is an example of a concentration bound. Let 𝑌𝑗 be the number of jobs assigned to the 𝑗th server. By linearity of expectation, 𝔽 𝑌𝑗 = σ𝑘=1

𝑂

ℙ job 𝑘 → server 𝑗 = 𝑂 ⋅ 1/𝑂 = 1.

Concentration bounds

Claim: The max-loaded server has < 8 log 𝑂 / log log 𝑂 jobs with probability at least 1 − 1/𝑂 Proof: Probability that a fixed server 𝑗 ∈ {1,2, … , 𝑂} gets at least 𝑙 jobs is at most 𝑂 𝑙 1 𝑂

𝑙

≤ 𝑂𝑙 𝑙! ⋅ 1 𝑂𝑙 ≤ 1 𝑙! ≤ 𝑙−𝑙

2

If we choose 𝑙 =

8 log 𝑂 log log 𝑂 this is at most 1/𝑂2

Now a union bound shows that the probability of any server getting at least 𝑙 jobs is at most 1/𝑂. This is an example of a concentration bound. Let 𝑌𝑗 be the number of jobs assigned to the 𝑗th server. By linearity of expectation, 𝔽 𝑌𝑗 = σ𝑘=1

𝑂

ℙ job 𝑘 → server 𝑗 = 𝑂 ⋅ 1/𝑂 = 1. We showed that ℙ 𝑌𝑗 ≥

8 log 𝑂 log log 𝑂 ≤ 1 𝑂2 and then took a union bound over all 𝑂 servers.

Concentration bounds

This is an example of a concentration bound. Let 𝑌𝑗 be the number of jobs assigned to the 𝑗th server. By linearity of expectation, 𝔽 𝑌𝑗 = σ𝑘=1

𝑂

ℙ job 𝑘 → server 𝑗 = 𝑂 ⋅ 1/𝑂 = 1. We showed that ℙ 𝑌𝑗 ≥

8 log 𝑂 log log 𝑂 ≤ 1 𝑂2 and then took a union bound over all 𝑂 servers.

This is a common analysis technique: If a random variable (like 𝑌𝑗) depends in a “smooth” way on the outcome of many independent events, then it is likely not too far from its expectation.

Concentration bounds

Let 𝑌𝑗 be the number of jobs assigned to the 𝑗th server. By linearity of expectation, 𝔽 𝑌𝑗 = σ𝑘=1

𝑂

ℙ job 𝑘 → server 𝑗 = 𝑂 ⋅ 1/𝑂 = 1. We showed that ℙ 𝑌𝑗 ≥

8 log 𝑂 log log 𝑂 ≤ 1 𝑂2 and then took a union bound over all 𝑂 servers.

This is a common analysis technique: If a random variable (like 𝑌𝑗) depends in a “smooth” way on the outcome of many independent events, then it is likely not too far from its expectation. “Smooth” in this case means that the outcome of any decision (where to put job 𝑘) does not affect the value of 𝑌𝑗 by too much (only by 1). This is an example of a concentration bound.

EXERCISE

Is it concentrated? [why or why not?] #1: Choose a uniformly random vector 𝑌 ∈ ℝ𝑜 with 𝑌 = 𝑌1

2 + 𝑌2 2 + ⋯ + 𝑌𝑜 2 = 1

What is 𝔽[𝑌1

2] ?

What is the typical value of the maximum: max 𝑌1 , 𝑌2 , … , 𝑌𝑜 ? #2 Rich get richer: Suppose we have 𝑂 people. Everyone starts with 1 dollar.

We assign 𝑂2 more dollars in rounds. 𝒋th round: If person 𝑘 already has 𝑜𝑘 dollars, we give them the 𝑗th dollar with probability 𝑜𝑘 𝑗 − 1 i.e., with probability to the proportional the amount of money they already have. Let 𝑌𝑗 be the amount of money person 𝑗 ends up with. What is the typical value of 𝑌1? Is 𝑌1 concentrated? What is the typical value of max(𝑌1, 𝑌2, … , 𝑌𝑜)? Is it concentrated?

• utline

Lecture #2: Advanced hashing and concentration bounds

• Bloom filters
• Cuckoo hashing
• Load balancing
• Tail bounds

Markov’s inequality

The more you know: The more information we have about a random variable, the stronger the concentration we can prove.

Markov’s inequality

The more you know: The more information we have about a random variable, the stronger the concentration we can prove. The most basic concentration bound is Markov’s inequality. It requires knowing only the expected value: If 𝑌 is a non-negative random variable, then for any 𝜇 ≥ 1, ℙ 𝑌 ≥ 𝜇 ≤ 𝔽 𝑌 𝜇 Proof? (it’s written there)

Markov’s inequality

The more you know: The more information we have about a random variable, the stronger the concentration we can prove. The most basic concentration bound is Markov’s inequality. It requires knowing only the expected value: If 𝑌 is a non-negative random variable, then for any 𝜇 ≥ 1, ℙ 𝑌 ≥ 𝜇 ≤ 𝔽 𝑌 𝜇 Proof? (it’s written there) Example: If your expected revenue is $10,000, then the probability to make $1,000 is at most 1/10.

EXERCISE

Markov’s inequality: If 𝑌 is a non-negative random variable, then for any 𝜇 ≥ 1, ℙ 𝑌 ≥ 𝜇 ≤ 𝔽 𝑌 𝜇 A permutation is an invertible mapping 𝜌 ∶ 1,2, … , 𝑜 → {1,2, … , 𝑜} A number 𝑘 is called a fixed point of 𝜌 if 𝜌 𝑘 = 𝑘. Exercise: Prove that if 𝜌 is a uniformly random permutation, then ℙ 𝜌 has more than 𝑙 fixed points ≤ 1 𝑙

Chebyshev’s inequality

Recall that the variance of a random variable 𝑌 is the value var 𝑌 = 𝜏2 = 𝔽 𝑌 − 𝔽𝑌 2

Chebyshev’s inequality

Recall that the variance of a random variable 𝑌 is the value var 𝑌 = 𝜏2 = 𝔽 𝑌 − 𝔽𝑌 2 Chebyshev’s inequality: If 𝑌 is a random variable with var 𝑌 = 𝜏2, then for any 𝜇 > 0, ℙ 𝑌 − 𝔽𝑌 ≥ 𝜇𝜏 ≤ 1 𝜇2

Chebyshev’s inequality

Recall that the variance of a random variable 𝑌 is the value var 𝑌 = 𝜏2 = 𝔽 𝑌 − 𝔽𝑌 2 Chebyshev’s inequality: If 𝑌 is a random variable with var 𝑌 = 𝜏2, then for any 𝜇 > 0, ℙ 𝑌 − 𝔽𝑌 ≥ 𝜇𝜏 ≤ 1 𝜇2 Proof: Apply Markov’s inequality to the random variable 𝑍 = 𝑌 − 𝔽𝑌 2

Chebyshev’s inequality

Recall that the variance of a random variable 𝑌 is the value var 𝑌 = 𝜏2 = 𝔽 𝑌 − 𝔽𝑌 2 Chebyshev’s inequality: If 𝑌 is a random variable with var 𝑌 = 𝜏2, then for any 𝜇 > 0, ℙ 𝑌 − 𝔽𝑌 ≥ 𝜇𝜏 ≤ 1 𝜇2 Proof: Apply Markov’s inequality to the random variable 𝑍 = 𝑌 − 𝔽𝑌 2 Application: Suppose we map 𝑂 balls into 𝑂 bins using a 2-universal hash family ℋ. Then with probability at least 1/2, the maximum load is at most 𝑃 𝑂 .

Chebyshev’s inequality

Application: Suppose we map 𝑂 balls into 𝑂 bins using a 2-universal hash family ℋ. Then with probability at least 1/2, the maximum load is at most 𝑃 𝑂 . Let 𝑀𝑗 be the load of bin 𝑗. Let 𝑌𝑗𝑘 be the indicator random variable such that 𝑌𝑗𝑘 = 1 ↔ 𝑗th bin gets the 𝑘th ball. Note that 𝔽 𝑌𝑗𝑘 = 1/𝑂 for each 𝑗 = 1, … , 𝑂.

Chebyshev’s inequality

Application: Suppose we map 𝑂 balls into 𝑂 bins using a 2-universal hash family ℋ. Then with probability at least 1/2, the maximum load is at most 𝑃 𝑂 . Let 𝑀𝑗 be the load of bin 𝑗. Let 𝑌𝑗𝑘 be the indicator random variable such that 𝑌𝑗𝑘 = 1 ↔ 𝑗th bin gets the 𝑘th ball. Note that 𝔽 𝑌𝑗𝑘 = 1/𝑂 for each 𝑗 = 1, … , 𝑂. Exercise: For any random variable 𝑍, var 𝑍 = 𝔽 𝑍2 − 𝔽𝑍 2

Chebyshev’s inequality

Application: Suppose we map 𝑂 balls into 𝑂 bins using a 2-universal hash family ℋ. Then with probability at least 1/2, the maximum load is at most 𝑃 𝑂 . Let 𝑀𝑗 be the load of bin 𝑗. Let 𝑌𝑗𝑘 be the indicator random variable such that 𝑌𝑗𝑘 = 1 ↔ 𝑗th bin gets the 𝑘th ball. Note that 𝔽 𝑌𝑗𝑘 = 1/𝑂 for each 𝑗 = 1, … , 𝑂. Exercise: For any random variable 𝑍, var 𝑍 = 𝔽 𝑍2 − 𝔽𝑍 2 So write: var 𝑀𝑗 = 𝔽

𝑌𝑗1 + ⋯ + 𝑌𝑗𝑂 2 − 1 We have 𝔽 𝑌𝑗𝑘

2 = 𝔽 𝑌𝑗𝑘 = 1/𝑂 and 𝔽 𝑌𝑗𝑘𝑌𝑗𝑙 = ℙ ℎ 𝑘 = ℎ 𝑙 = 𝑗 ≤ 1/𝑂2

using the 2-universal property, so var 𝑀𝑗 ≤ 𝑂 ⋅ 1 𝑂 + 𝑂 𝑂 − 1 𝑂2 − 1 = 1 − 1 𝑂 ≤ 1

Chebyshev’s inequality var 𝑀𝑗 ≤ 𝑂 ⋅ 1 𝑂 + 𝑂 𝑂 − 1 𝑂2 − 1 = 1 − 1 𝑂 ≤ 1

Chebyshev’s inequality: If 𝑌 is a random variable with var 𝑌 = 𝜏2, then for any 𝜇 > 0, ℙ 𝑌 − 𝔽𝑌 ≥ 𝜇𝜏 ≤ 1 𝜇2

Chebyshev’s inequality var 𝑀𝑗 ≤ 𝑂 ⋅ 1 𝑂 + 𝑂 𝑂 − 1 𝑂2 − 1 = 1 − 1 𝑂 ≤ 1

Chebyshev’s inequality: If 𝑌 is a random variable with var 𝑌 = 𝜏2, then for any 𝜇 > 0, ℙ 𝑌 − 𝔽𝑌 ≥ 𝜇𝜏 ≤ 1 𝜇2 Apply Chebyshev’s inequality to 𝑀𝑗, yielding ℙ 𝑀𝑗 − 1 ≥ 𝜇 ≤ 1 𝜇2

Chebyshev’s inequality var 𝑀𝑗 ≤ 𝑂 ⋅ 1 𝑂 + 𝑂 𝑂 − 1 𝑂2 − 1 = 1 − 1 𝑂 ≤ 1

Chebyshev’s inequality: If 𝑌 is a random variable with var 𝑌 = 𝜏2, then for any 𝜇 > 0, ℙ 𝑌 − 𝔽𝑌 ≥ 𝜇𝜏 ≤ 1 𝜇2 Apply Chebyshev’s inequality to 𝑀𝑗, yielding ℙ 𝑀𝑗 − 1 ≥ 𝜇 ≤ 1 𝜇2 Thus ℙ 𝑀𝑗 − 1 ≥ 2𝑂 ≤ 1

2𝑂, so a union bound yields

ℙ max 𝑀1, … , 𝑀𝑂 ≥ 2𝑂 + 1 ≤ 1 2

EXERCISE

Let 𝑞 be the actual percentage of the population the prefers candidate #1 and let Ƹ 𝑞 = 𝑌1 + ⋯ + 𝑌𝑜 /𝑜 denote the empirical mean. Exercise: Prove that if we want 𝑞 − Ƹ 𝑞 ≤ 𝜗 to hold with 99% probability, then we need only sample 𝑜 = 𝑃(1/𝜗2) voters. Chebyshev’s inequality: If 𝑌 is a random variable with var 𝑌 = 𝜏2, then for any 𝜇 > 0, ℙ 𝑌 − 𝔽𝑌 ≥ 𝜇𝜏 ≤ 1 𝜇2 Suppose we choose 𝑜 independent random voters and ask them whether they prefer candidate #1 over candidate #2. We see outcomes 𝑌1, 𝑌2, … , 𝑌𝑜 ∈ 0,1 .

Sums of independent random variables

Hoeffding’s inequality: Let 𝑌1, … , 𝑌𝑜 be a sequence of independent random variables where, for each 1 ≤ 𝑗 ≤ 𝑜, we have 𝑏𝑗 ≤ 𝑌𝑗 ≤ 𝑐𝑗. Let 𝑌 = (𝑌1 + ⋯ + 𝑌𝑜)/𝑜. Then:

ℙ 𝑌 − 𝔽𝑌 ≥ 𝜇 ≤ 2 𝑓

− 2𝜇2𝑜2 σ𝑗=1

𝑜

𝑏𝑗−𝑐𝑗 2

Sums of independent random variables

Hoeffding’s inequality: Let 𝑌1, … , 𝑌𝑜 be a sequence of independent random variables where, for each 1 ≤ 𝑗 ≤ 𝑜, we have 𝑏𝑗 ≤ 𝑌𝑗 ≤ 𝑐𝑗. Let 𝑌 = (𝑌1 + ⋯ + 𝑌𝑜)/𝑜. Then:

ℙ 𝑌 − 𝔽𝑌 ≥ 𝜇 ≤ 2 𝑓

− 2𝜇2𝑜2 σ𝑗=1

𝑜

𝑏𝑗−𝑐𝑗 2

Suppose we wanted our poll from the previous slide to be correct with probability at least 1 − 𝜀. Chebyshev’s inequality would tell us we need at most 𝑃

1 𝜗2 𝜀 samples.

Sums of independent random variables

Hoeffding’s inequality: Let 𝑌1, … , 𝑌𝑜 be a sequence of independent random variables where, for each 1 ≤ 𝑗 ≤ 𝑜, we have 𝑏𝑗 ≤ 𝑌𝑗 ≤ 𝑐𝑗. Let 𝑌 = (𝑌1 + ⋯ + 𝑌𝑜)/𝑜. Then:

ℙ 𝑌 − 𝔽𝑌 ≥ 𝜇 ≤ 2 𝑓

− 2𝜇2𝑜2 σ𝑗=1

𝑜

𝑏𝑗−𝑐𝑗 2

Suppose we wanted our poll from the previous slide to be correct with probability at least 1 − 𝜀. Chebyshev’s inequality would tell us we need at most 𝑃

1 𝜗2 𝜀 samples.

Setting 𝑏𝑗 = 0, 𝑐𝑗 = 1, and 𝜇 = 𝜗 in Hoeffding’s inequality gives ℙ Ƹ 𝑞 − 𝑞 ≥ 𝜗 ≤ 2𝑓−2𝜗2𝑜 so we only need 𝑜 ≤ 𝑃

log 1

𝜀

𝜗2

samples.

Sums of independent random variables

Chernoff bound (multiplicative): Let 𝑌1, … , 𝑌𝑜 be a sequence of independent 0,1 -valued random variables. Let 𝑞𝑗 = 𝔽[𝑌𝑗], 𝑌 = 𝑌1 + 𝑌2 + ⋯ + 𝑌𝑜, 𝜈 = 𝔽[𝑌]. Then for every 𝛾 ≥ 1:

ℙ 𝑌 ≥ 𝛾𝜈 ≤ 𝑓𝛾−1 𝛾𝛾

𝜈

ℙ 𝑌 ≤ 𝜈/𝛾 ≤ 𝑓

1 𝛾−1

𝛾𝛾

𝜈

Sums of independent random variables

Chernoff bound (multiplicative): Let 𝑌1, … , 𝑌𝑜 be a sequence of independent 0,1 -valued random variables. Let 𝑞𝑗 = 𝔽[𝑌𝑗], 𝑌 = 𝑌1 + 𝑌2 + ⋯ + 𝑌𝑜, 𝜈 = 𝔽[𝑌]. Then for every 𝛾 ≥ 1: 𝑂 balls thrown randomly into 𝑂 bins. 𝑌𝑗 = 1 if ith ball ends up in first bin and 𝑌𝑗 = 0 otherwise. Then 𝑌 = # of balls in first bin. As we calculated earlier, 𝔽 𝑌 = 1 For 𝛾 ≈

log 𝑂 log log 𝑂, the Chernoff bound gives ℙ 𝑌 ≥ 𝛾 ≤ 1/𝑂2

Reproduce balls in bins:

ℙ 𝑌 ≥ 𝛾𝜈 ≤ 𝑓𝛾−1 𝛾𝛾

𝜈

ℙ 𝑌 ≤ 𝜈/𝛾 ≤ 𝑓

1 𝛾−1

𝛾𝛾

𝜈

Sums of independent random variables

Chernoff bound (multiplicative): Let 𝑌1, … , 𝑌𝑜 be a sequence of independent 0,1 -valued random variables. Let 𝑞𝑗 = 𝔽[𝑌𝑗], 𝑌 = 𝑌1 + 𝑌2 + ⋯ + 𝑌𝑜, 𝜈 = 𝔽[𝑌]. Then for every 𝛾 ≥ 1:

ℙ 𝑌 ≥ 𝛾𝜈 ≤ 𝑓𝛾−1 𝛾𝛾

𝜈

ℙ 𝑌 ≤ 𝜈/𝛾 ≤ 𝑓

1 𝛾−1

𝛾𝛾

𝜈

𝑂 balls thrown randomly into 𝑂 bins. 𝑌𝑗 = 1 if ith ball ends up in first bin and 𝑌𝑗 = 0 otherwise. Then 𝑌 = # of balls in first bin. As we calculated earlier, 𝔽 𝑌 = 1 For 𝛾 ≈

log 𝑂 log log 𝑂, the Chernoff bound gives ℙ 𝑌 ≥ 𝛾 ≤ 1/𝑂2

Reproduce balls in bins: This type of analysis works for much more complicated kinds of events (see homework #2)

(return to) Bloom filters

Heuristic analysis: If each location in 𝐵 is 0 with probability 𝑞(𝑙, 𝑂), then a false positive for 𝑦 ∉ 𝑇 should happen with probability at most 1 − 𝑞 𝑙, 𝑂

𝑙 ≈ 1 − 𝑓−𝑙𝑂 𝑁 𝑙

But the actual fraction of 0′𝑡 in the hash table is a random variable 𝑌𝑙,𝑂 with expectation 𝔽 𝑌𝑙,𝑂 = 𝑞 𝑙, 𝑂 To get the analysis right, we need a concentration bound: Want to say that 𝑌𝑙,𝑂 is close to its expected value with high probability. Let’s analyze!

(return to) Bloom filters

We have an array with 𝑁 bits and to hash an element 𝑦 ∈ 𝒱, we set the bits in positions ℎ1 𝑦 , ℎ2 𝑦 , … , ℎ𝑙 𝑦 to 1.

(return to) Bloom filters

Let 𝑌 be the # of 0’s in the hash table after 𝑂 elements are hashed. We have an array with 𝑁 bits and to hash an element 𝑦 ∈ 𝒱, we set the bits in positions ℎ1 𝑦 , ℎ2 𝑦 , … , ℎ𝑙 𝑦 to 1.

(return to) Bloom filters

Let 𝑌 be the # of 0’s in the hash table after 𝑂 elements are hashed. Consider the 𝑂 elements to hash: 𝑦1, 𝑦2, … , 𝑦𝑂 We have an array with 𝑁 bits and to hash an element 𝑦 ∈ 𝒱, we set the bits in positions ℎ1 𝑦 , ℎ2 𝑦 , … , ℎ𝑙 𝑦 to 1.

(return to) Bloom filters

Let 𝑌 be the # of 0’s in the hash table after 𝑂 elements are hashed. Consider the 𝑂 elements to hash: 𝑦1, 𝑦2, … , 𝑦𝑂 Let 𝐼 𝑦𝑗 = ℎ1 𝑦𝑗 , … , ℎ𝑙 𝑦𝑗 We have an array with 𝑁 bits and to hash an element 𝑦 ∈ 𝒱, we set the bits in positions ℎ1 𝑦 , ℎ2 𝑦 , … , ℎ𝑙 𝑦 to 1.

(return to) Bloom filters

Let 𝑌 be the # of 0’s in the hash table after 𝑂 elements are hashed. Consider the 𝑂 elements to hash: 𝑦1, 𝑦2, … , 𝑦𝑂 Let 𝐼 𝑦𝑗 = ℎ1 𝑦𝑗 , … , ℎ𝑙 𝑦𝑗 Define 𝑌

𝑘 = 𝔽 𝑌

𝐼 𝑦1 , 𝐼 𝑦2 , … , 𝐼 𝑦𝑘 to be the expected # of 0’s in the hash table after hashing the first 𝑘 elements. We have an array with 𝑁 bits and to hash an element 𝑦 ∈ 𝒱, we set the bits in positions ℎ1 𝑦 , ℎ2 𝑦 , … , ℎ𝑙 𝑦 to 1.

(return to) Bloom filters

Let 𝑌 be the # of 0’s in the hash table after 𝑂 elements are hashed. Consider the 𝑂 elements to hash: 𝑦1, 𝑦2, … , 𝑦𝑂 Let 𝐼 𝑦𝑗 = ℎ1 𝑦𝑗 , … , ℎ𝑙 𝑦𝑗 Define 𝑌

𝑘 = 𝔽 𝑌

𝐼 𝑦1 , 𝐼 𝑦2 , … , 𝐼 𝑦𝑘 to be the expected # of 0’s in the hash table after hashing the first 𝑘 elements. Note that 𝑦1, … , 𝑦𝑂 are any set of keys. The randomness here is all in the choice of the hash functions ℎ1, … , ℎ𝑙. We have an array with 𝑁 bits and to hash an element 𝑦 ∈ 𝒱, we set the bits in positions ℎ1 𝑦 , ℎ2 𝑦 , … , ℎ𝑙 𝑦 to 1.

(return to) Bloom filters

Let 𝑌 be the # of 0’s in the hash table after 𝑂 elements are hashed. Consider the 𝑂 elements to hash: 𝑦1, 𝑦2, … , 𝑦𝑂 𝐼 𝑦𝑗 = ℎ1 𝑦𝑗 , … , ℎ𝑙 𝑦𝑗 𝑌

𝑘 = 𝔽 𝑌

𝐼 𝑦1 , 𝐼 𝑦2 , … , 𝐼 𝑦𝑘 We calculated before that 𝑌0 = 𝔽 𝑌 = 𝑛 1 − 1

𝑛 𝑙𝑂

[why?]

(return to) Bloom filters

Let 𝑌 be the # of 0’s in the hash table after 𝑂 elements are hashed. Consider the 𝑂 elements to hash: 𝑦1, 𝑦2, … , 𝑦𝑂 𝐼 𝑦𝑗 = ℎ1 𝑦𝑗 , … , ℎ𝑙 𝑦𝑗 𝑌

𝑘 = 𝔽 𝑌

𝐼 𝑦1 , 𝐼 𝑦2 , … , 𝐼 𝑦𝑘 We calculated before that 𝑌0 = 𝔽 𝑌 = 𝑛 1 − 1

𝑛 𝑙𝑂

[why?] Now we want to know the probability that 𝑌 is much different from its expectation 𝑌0.

(return to) Bloom filters

Let 𝑌 be the # of 0’s in the hash table after 𝑂 elements are hashed. Consider the 𝑂 elements to hash: 𝑦1, 𝑦2, … , 𝑦𝑂 𝐼 𝑦𝑗 = ℎ1 𝑦𝑗 , … , ℎ𝑙 𝑦𝑗 𝑌

𝑘 = 𝔽 𝑌

𝐼 𝑦1 , 𝐼 𝑦2 , … , 𝐼 𝑦𝑘 We calculated before that 𝑌0 = 𝔽 𝑌 = 𝑛 1 − 1

𝑛 𝑙𝑂

[why?] Now we want to know the probability that 𝑌 is much different from its expectation 𝑌0. Claim #1: 𝑌

𝑘+1 − 𝑌 𝑘 ≤ 𝑙

for all 𝑘 = 1,2, … , 𝑂 − 1

(return to) Bloom filters

Let 𝑌 be the # of 0’s in the hash table after 𝑂 elements are hashed. Consider the 𝑂 elements to hash: 𝑦1, 𝑦2, … , 𝑦𝑂 𝐼 𝑦𝑗 = ℎ1 𝑦𝑗 , … , ℎ𝑙 𝑦𝑗 𝑌

𝑘 = 𝔽 𝑌

𝐼 𝑦1 , 𝐼 𝑦2 , … , 𝐼 𝑦𝑘 We calculated before that 𝑌0 = 𝔽 𝑌 = 𝑛 1 − 1

𝑛 𝑙𝑂

[why?] Now we want to know the probability that 𝑌 is much different from its expectation 𝑌0. Claim #1: 𝑌

𝑘+1 − 𝑌 𝑘 ≤ 𝑙

for all 𝑘 = 1,2, … , 𝑂 − 1 Claim #2: 𝔽 𝑌

𝑘+1

𝐼 𝑦1 , … , 𝐼 𝑦𝑘 = 𝑌

𝑘

for all 𝑘 = 1,2, … , 𝑂 − 1

(return to) Bloom filters

Let 𝑌 be the # of 0’s in the hash table after 𝑂 elements are hashed. Consider the 𝑂 elements to hash: 𝑦1, 𝑦2, … , 𝑦𝑂 𝐼 𝑦𝑗 = ℎ1 𝑦𝑗 , … , ℎ𝑙 𝑦𝑗 𝑌

𝑘 = 𝔽 𝑌

𝐼 𝑦1 , 𝐼 𝑦2 , … , 𝐼 𝑦𝑘 We calculated before that 𝑌0 = 𝔽 𝑌 = 𝑛 1 − 1

𝑛 𝑙𝑂

[why?] Now we want to know the probability that 𝑌 is much different from its expectation 𝑌0. Claim #1: 𝑌

𝑘+1 − 𝑌 𝑘 ≤ 𝑙

for all 𝑘 = 1,2, … , 𝑂 − 1 Claim #2: 𝔽 𝑌

𝑘+1

𝐼 𝑦1 , … , 𝐼 𝑦𝑘 = 𝑌

𝑘

for all 𝑘 = 1,2, … , 𝑂 − 1 Such a sequence of random variables is called a martingale

Azuma’s inequality

Suppose that {𝑌0, 𝑌1, … , 𝑌𝑂} is a martingale such that for some constants {𝑑

𝑘},

𝑌

𝑘+1 − 𝑌 𝑘 ≤ 𝑑 𝑘 for all 𝑘 = 0,1, … , 𝑂 − 1. Then for any 𝜇 > 0,

ℙ 𝑌𝑂 − 𝑌0 ≥ 𝜇 ≤ 2 exp − 𝜇2 2 𝑑1

2 + ⋯ + 𝑑𝑂 2

Azuma’s inequality

Suppose that {𝑌0, 𝑌1, … , 𝑌𝑂} is a martingale such that for some constants {𝑑

𝑘},

𝑌

𝑘+1 − 𝑌 𝑘 ≤ 𝑑 𝑘 for all 𝑘 = 0,1, … , 𝑂 − 1. Then for any 𝜇 > 0,

ℙ 𝑌𝑂 − 𝑌0 ≥ 𝜇 ≤ 2 exp − 𝜇2 2 𝑑1

2 + ⋯ + 𝑑𝑂 2

For our problem: 𝑑1 = 𝑑2 = ⋯ = 𝑑𝑂 = 𝑙 So the probability that the # of 0’s differs from its expectation by more than 𝜇 is at most 2 exp(−𝜇2/2𝑙2𝑂) So the deviation is ≈ 𝑙 𝑂 and is tightly concentrated in this window.

(take home) EXERCISE

Suppose that {𝑌0, 𝑌1, … , 𝑌𝑂} is a martingale such that for some constants {𝑑

𝑘},

𝑌

𝑘+1 − 𝑌 𝑘 ≤ 𝑑 𝑘 for all 𝑘 = 0,1, … , 𝑂 − 1. Then for any 𝜇 > 0,

ℙ 𝑌𝑂 − 𝑌0 ≥ 𝜇 ≤ 2 exp − 𝜇2 2 𝑑1

2 + ⋯ + 𝑑𝑂 2

For our problem: 𝑑1 = 𝑑2 = ⋯ = 𝑑𝑂 = 𝑙 So the probability that the # of 0’s differs from its expectation by more than 𝜇 is at most 2 exp(−𝜇2/2𝑙2𝑂) So the deviation is ≈ 𝑙 𝑂 and is tightly concentrated in this window. Improve the error probability to 2 exp(−𝜇2/2𝑙𝑂) using a different martingale.