Balls-into-Bins Model and Chernoff Bounds Advanced Algorithms - - PowerPoint PPT Presentation

Balls-into-Bins Model and Chernoff Bounds

Advanced Algorithms Nanjing University, Fall 2018

Balls-into-Bins Model

m balls

Balls-into-Bins Model

m balls n bins

Balls-into-Bins Model

m balls n bins uniformly and independently thrown into

Balls-into-Bins Model

m balls n bins uniformly and independently thrown into uniformly at random choose h: [m]→[n]

Balls-into-Bins Model

m balls n bins uniformly and independently thrown into uniformly at random choose h: [m]→[n] Question: probability that each ball lands in its own bin (h is 1-1)?

Balls-into-Bins Model

m balls n bins uniformly and independently thrown into uniformly at random choose h: [m]→[n] Question: probability that each ball lands in its own bin (h is 1-1)? Question: probability that every bin is not empty (h is onto)?

Balls-into-Bins Model

m balls n bins uniformly and independently thrown into uniformly at random choose h: [m]→[n] Question: probability that each ball lands in its own bin (h is 1-1)? Question: probability that every bin is not empty (h is onto)? Question: maximum number of balls in a bin (max{|h-1(i)|})?

Question: probability that each ball lands in its own bin (h is 1-1)?

Birthday Problem

Question: probability that each ball lands in its own bin (h is 1-1)?

Birthday Problem

Question: probability that each ball lands in its own bin (h is 1-1)?

Jan 1st Jan 2nd Jan 3rd Jan 31st Feb 1st

… …

Dec 31st

Birthday Problem

Question: probability that each ball lands in its own bin (h is 1-1)?

Jan 1st Jan 2nd Jan 3rd Jan 31st Feb 1st

… …

Dec 31st

Birthday Problem

Question: probability that each ball lands in its own bin (h is 1-1)?

Jan 1st Jan 2nd Jan 3rd Jan 31st Feb 1st

… …

Dec 31st

This probability is some constant when

Question: probability that every bin is not empty (h is onto)?

Question: probability that every bin is not empty (h is onto)?

Question: probability that every bin is not empty (h is onto)? Question: how many balls we need to throw to leave no empty bins?

Coupon Collector

Question: probability that every bin is not empty (h is onto)? Question: how many balls we need to throw to leave no empty bins?

Coupon Collector

Question: probability that every bin is not empty (h is onto)? Question: how many balls we need to throw to leave no empty bins?

…

Coupon Collector

Question: probability that every bin is not empty (h is onto)? Question: how many balls we need to throw to leave no empty bins?

…

Let Xi be the number of balls thrown until i bins are non-empty, given i-1 bins are already non-empty.

Coupon Collector

Question: probability that every bin is not empty (h is onto)? Question: how many balls we need to throw to leave no empty bins?

…

Let Xi be the number of balls thrown until i bins are non-empty, given i-1 bins are already non-empty. Xi is a geometric r.v. with parameter (n-i+1)/n.

Coupon Collector

Question: probability that every bin is not empty (h is onto)? Question: how many balls we need to throw to leave no empty bins?

…

Let Xi be the number of balls thrown until i bins are non-empty, given i-1 bins are already non-empty. Xi is a geometric r.v. with parameter (n-i+1)/n.

Load Balancing

Question: maximum number of balls in a bin (max{|h-1(i)|})?

Load Balancing

Question: maximum number of balls in a bin (max{|h-1(i)|})? Let Xi be the number of balls in bin i. That is, Xi = |h-1(i)|.

Load Balancing

Question: maximum number of balls in a bin (max{|h-1(i)|})? Let Xi be the number of balls in bin i. That is, Xi = |h-1(i)|.

Load Balancing

Question: maximum number of balls in a bin (max{|h-1(i)|})? Let Xi be the number of balls in bin i. That is, Xi = |h-1(i)|. Let Yij be i.r.v. taking value 1 iff ball j lands in bin i.

Load Balancing

Question: maximum number of balls in a bin (max{|h-1(i)|})? Let Xi be the number of balls in bin i. That is, Xi = |h-1(i)|. Let Yij be i.r.v. taking value 1 iff ball j lands in bin i.

Load Balancing

Question: maximum number of balls in a bin (max{|h-1(i)|})? Let Xi be the number of balls in bin i. That is, Xi = |h-1(i)|. Let Yij be i.r.v. taking value 1 iff ball j lands in bin i. Is max{𝔽(Xi)} what we want?

Load Balancing

Question: maximum number of balls in a bin (max{|h-1(i)|})? Let Xi be the number of balls in bin i. That is, Xi = |h-1(i)|. Let Yij be i.r.v. taking value 1 iff ball j lands in bin i. Is max{𝔽(Xi)} what we want? For every i, 𝔽(Xi) is m/n, so max{𝔽(Xi)} is simply m/n.

Load Balancing

Question: maximum number of balls in a bin?

Load Balancing

Question: maximum number of balls in a bin?

Load Balancing

Question: maximum number of balls in a bin? Something is not right…

Load Balancing

Question: maximum number of balls in a bin?

Load Balancing

with high probability (w.h.p.): We say an event happens with high probability (with respect to n) if it happens with probability at least 1-1/n.

Question: maximum number of balls in a bin?

Load Balancing

Load Balancing

Load Balancing

Load Balancing

Load Balancing

Load Balancing

let m=n

Load Balancing

let t = 3ln(n)/lnln(n)

Load Balancing

for sufficiently large n

Load Balancing

Load Balancing

Load Balancing

Load Balancing

Load Balancing

let m=nlg(n)

Load Balancing

let t = 4m/n = 4lg(n)

Load Balancing

let t = 4m/n = 4lg(n)

Load Balancing

Question: maximum number of balls in a bin (max{|h-1(i)|})? “m balls are thrown into n bins uniformly and independently at random” “uniformly at random choose h: [m]→[n]”

Concentration

balls into bins coin flipping

Concentration

balls into bins coin flipping Question: probability that X deviates more than 𝜀 from expectation?

Chernoff Bounds

Herman Chernoff

Chernoff Bounds

Herman Chernoff

Chernoff Bounds

(Convenient) Chernoff Bounds

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 p

Power of the Chernoff Bounds

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 p

Power of the Chernoff Bounds

Chernoff Bounds in Action:

Load Balancing

“m balls are thrown into n bins uniformly and independently at random” Question: maximum number of balls in a bin?

Chernoff Bounds in Action:

Load Balancing

“m balls are thrown into n bins uniformly and independently at random” Question: maximum number of balls in a bin?

Xi: load of bin i Yij: i.r.v. taking value 1 iff ball j lands in bin i

Chernoff Bounds in Action:

Load Balancing

“m balls are thrown into n bins uniformly and independently at random” Question: maximum number of balls in a bin?

Xi: load of bin i Yij: i.r.v. taking value 1 iff ball j lands in bin i

Chernoff Bounds in Action:

Load Balancing

“m balls are thrown into n bins uniformly and independently at random” Question: maximum number of balls in a bin?

Xi: load of bin i Yij: i.r.v. taking value 1 iff ball j lands in bin i For 𝑛 = 𝑜, 𝜈 = 1 implies

Chernoff Bounds in Action:

Load Balancing

“m balls are thrown into n bins uniformly and independently at random” Question: maximum number of balls in a bin?

Xi: load of bin i Yij: i.r.v. taking value 1 iff ball j lands in bin i For 𝑛 = 𝑜, 𝜈 = 1 implies when 𝑢 ≥

𝑓 ln 𝑜 ln ln 𝑜 and 𝑜 sufficiently large

Chernoff Bounds in Action:

Load Balancing

“m balls are thrown into n bins uniformly and independently at random” Question: maximum number of balls in a bin?

Xi: load of bin i Yij: i.r.v. taking value 1 iff ball j lands in bin i For 𝑛 = 𝑜, 𝜈 = 1 implies when 𝑢 ≥

𝑓 ln 𝑜 ln ln 𝑜 and 𝑜 sufficiently large

Chernoff Bounds in Action:

Load Balancing

“m balls are thrown into n bins uniformly and independently at random” Question: maximum number of balls in a bin?

Xi: load of bin i Yij: i.r.v. taking value 1 iff ball j lands in bin i For 𝑛 = 𝑜, 𝜈 = 1 implies when 𝑢 ≥

𝑓 ln 𝑜 ln ln 𝑜 and 𝑜 sufficiently large

when 𝑛 = Θ(𝑜) max load is 𝑃

log 𝑜 log log 𝑜 w.h.p.

Chernoff Bounds in Action:

Load Balancing

For 𝑛 ≥ 𝑜 ln 𝑜, 𝜈 ≥ ln 𝑜

“m balls are thrown into n bins uniformly and independently at random” Question: maximum number of balls in a bin?

Chernoff Bounds in Action:

Load Balancing

For 𝑛 ≥ 𝑜 ln 𝑜, 𝜈 ≥ ln 𝑜

“m balls are thrown into n bins uniformly and independently at random” Question: maximum number of balls in a bin?

Chernoff Bounds in Action:

Load Balancing

For 𝑛 ≥ 𝑜 ln 𝑜, 𝜈 ≥ ln 𝑜

“m balls are thrown into n bins uniformly and independently at random” Question: maximum number of balls in a bin?

Chernoff Bounds in Action:

Load Balancing

For 𝑛 ≥ 𝑜 ln 𝑜, 𝜈 ≥ ln 𝑜

“m balls are thrown into n bins uniformly and independently at random” Question: maximum number of balls in a bin?

Chernoff Bounds in Action:

Load Balancing

For 𝑛 ≥ 𝑜 ln 𝑜, 𝜈 ≥ ln 𝑜 when 𝑛 = Ω(𝑜 log 𝑜) max load is 𝑃

𝑛 𝑜

w.h.p.

“m balls are thrown into n bins uniformly and independently at random” Question: maximum number of balls in a bin?

Chernoff Bounds in Action:

Load Balancing

For 𝑛 ≥ 𝑜 ln 𝑜, 𝜈 ≥ ln 𝑜 when 𝑛 = Ω(𝑜 log 𝑜) max load is 𝑃

𝑛 𝑜

w.h.p.

“m balls are thrown into n bins uniformly and independently at random” Question: maximum number of balls in a bin?

Chernoff Bounds in Action:

Load Balancing

For 𝑛 ≥ 𝑜 ln 𝑜, 𝜈 ≥ ln 𝑜 when 𝑛 = Ω(𝑜 log 𝑜) max load is 𝑃

𝑛 𝑜

w.h.p. when 𝑛 = Ω(𝑜 log 𝑜) min load is Ω

𝑛 𝑜

w.h.p.

“m balls are thrown into n bins uniformly and independently at random” Question: maximum number of balls in a bin?

Load Balancing

“m balls are thrown into n bins uniformly and independently at random” Question: maximum number of balls in a bin?

Load Balancing

“m balls are thrown into n bins uniformly and independently at random” Question: maximum number of balls in a bin?

Chernoff Bounds

Generalization of Markov’s Inequality

Moment Generating Functions

Moment Generating Functions

by Taylor’s expansion:

𝜇 > 0, and generalized Markov’s inequality

𝜇 > 0, and generalized Markov’s inequality

𝜇 > 0, and generalized Markov’s inequality independence of Xi not linearity of expectation

𝜇 > 0, and generalized Markov’s inequality independence of Xi not linearity of expectation

𝜇 > 0, and generalized Markov’s inequality independence of Xi not linearity of expectation

𝜇 > 0, and generalized Markov’s inequality independence of Xi not linearity of expectation

𝜇 > 0, and generalized Markov’s inequality independence of Xi not linearity of expectation 1 + 𝑧 ≤ 𝑓𝑧

𝜇 > 0, and generalized Markov’s inequality independence of Xi not linearity of expectation 1 + 𝑧 ≤ 𝑓𝑧

𝜇 > 0, and generalized Markov’s inequality independence of Xi not linearity of expectation 1 + 𝑧 ≤ 𝑓𝑧

𝜇 > 0, and generalized Markov’s inequality independence of Xi not linearity of expectation 1 + 𝑧 ≤ 𝑓𝑧 minimized when 𝜇 = ln(1 + 𝜀)

𝜇 > 0, and generalized Markov’s inequality independence of Xi not linearity of expectation 1 + 𝑧 ≤ 𝑓𝑧 minimized when 𝜇 = ln(1 + 𝜀)

(a) apply Markov’s inequality to moment generating function

𝜇 > 0, and generalized Markov’s inequality independence of Xi not linearity of expectation 1 + 𝑧 ≤ 𝑓𝑧 minimized when 𝜇 = ln(1 + 𝜀)

(a) apply Markov’s inequality to moment generating function (b) bound the value of the moment generating function

𝜇 > 0, and generalized Markov’s inequality independence of Xi not linearity of expectation 1 + 𝑧 ≤ 𝑓𝑧 minimized when 𝜇 = ln(1 + 𝜀)

(a) apply Markov’s inequality to moment generating function (b) bound the value of the moment generating function (c) optimize the bound of the moment generating function

Chernoff Bounds

???

Chernoff Bounds

???

Hoeffding’s Inequality

(Convenient) Hoeffding’s Inequality

Hoeffding’s Lemma

𝜇 > 0, and generalized Markov’s inequality

𝜇 > 0, and generalized Markov’s inequality independence of Yi

𝜇 > 0, and generalized Markov’s inequality independence of Yi Hoeffding’s lemma

𝜇 > 0, and generalized Markov’s inequality independence of Yi Hoeffding’s lemma

𝜇 > 0, and generalized Markov’s inequality independence of Yi Hoeffding’s lemma minimized when 𝜇 =

4𝑢 σ𝑗=1

𝑜

𝑐𝑗−𝑏𝑗 2

Hoeffding’s Inequality

???

Hoeffding’s Inequality

???

Hoeffding’s Inequality in Action:

Randomized Quicksort

1 7 2 3 B 5 9 C 4 8 A 6 B 9 C 8 A 1 2 3 4 5 6 1 2 4 5 6 9 8 B C 4 6 1 9 C

sort n distinct elements using QuickSort choose pivot uniformly at random in each recursive call of QuickSort

Hoeffding’s Inequality in Action:

Randomized Quicksort

1 7 2 3 B 5 9 C 4 8 A 6 B 9 C 8 A 1 2 3 4 5 6 1 2 4 5 6 9 8 B C 4 6 1 9 C

sort n distinct elements using QuickSort choose pivot uniformly at random in each recursive call of QuickSort expected cost under adversarial input: Θ(𝑜 log 𝑜)

Hoeffding’s Inequality in Action:

Randomized Quicksort

1 7 2 3 B 5 9 C 4 8 A 6 B 9 C 8 A 1 2 3 4 5 6 1 2 4 5 6 9 8 B C 4 6 1 9 C

sort n distinct elements using QuickSort choose pivot uniformly at random in each recursive call of QuickSort expected cost under adversarial input: Θ(𝑜 log 𝑜) worst case cost under any input: Θ(𝑜2)

Hoeffding’s Inequality in Action:

Randomized Quicksort

1 7 2 3 B 5 9 C 4 8 A 6 B 9 C 8 A 1 2 3 4 5 6 1 2 4 5 6 9 8 B C 4 6 1 9 C

sort n distinct elements using QuickSort choose pivot uniformly at random in each recursive call of QuickSort expected cost under adversarial input: Θ(𝑜 log 𝑜) worst case cost under any input: Θ(𝑜2)

Question: probability that cost is 𝜕(𝑜 log 𝑜)?

Hoeffding’s Inequality in Action:

Randomized Quicksort

1 7 2 3 B 5 9 C 4 8 A 6 B 9 C 8 A 1 2 3 4 5 6 1 2 4 5 6 9 8 B C 4 6 1 9 C

Question: probability that cost is 𝜕(𝑜 log 𝑜)? The cost will be at most 30𝑜 lg 𝑜 with high probability.

Hoeffding’s Inequality in Action:

Randomized Quicksort

1 7 2 3 B 5 9 C 4 8 A 6 B 9 C 8 A 1 2 3 4 5 6 1 2 4 5 6 9 8 B C 4 6 1 9 C

Question: probability that cost is 𝜕(𝑜 log 𝑜)? The cost will be at most 30𝑜 lg 𝑜 with high probability.

Hoeffding’s Inequality in Action:

Randomized Quicksort

1 7 2 3 B 5 9 C 4 8 A 6 B 9 C 8 A 1 2 3 4 5 6 1 2 4 5 6 9 8 B C 4 6 1 9 C

Question: probability that cost is 𝜕(𝑜 log 𝑜)? The cost will be at most 30𝑜 lg 𝑜 with high probability.

Hoeffding’s Inequality in Action:

Randomized Quicksort

1 7 2 3 B 5 9 C 4 8 A 6 B 9 C 8 A 1 2 3 4 5 6 1 2 4 5 6 9 8 B C 4 6 1 9 C

Question: probability that cost is 𝜕(𝑜 log 𝑜)? The cost will be at most 30𝑜 lg 𝑜 with high probability.

Hoeffding’s Inequality in Action:

Randomized Quicksort

1 7 2 3 B 5 9 C 4 8 A 6 B 9 C 8 A 1 2 3 4 5 6 1 2 4 5 6 9 8 B C 4 6 1 9 C

Question: probability that cost is 𝜕(𝑜 log 𝑜)? The cost will be at most 30𝑜 lg 𝑜 with high probability.

Hoeffding’s Inequality in Action:

Randomized Quicksort

1 7 2 3 B 5 9 C 4 8 A 6 B 9 C 8 A 1 2 3 4 5 6 1 2 4 5 6 9 8 B C 4 6 1 9 C

Question: probability that cost is 𝜕(𝑜 log 𝑜)? The cost will be at most 30𝑜 lg 𝑜 with high probability.

Hoeffding’s Inequality in Action:

Randomized Quicksort

1 7 2 3 B 5 9 C 4 8 A 6 B 9 C 8 A 1 2 3 4 5 6 1 2 4 5 6 9 8 B C 4 6 1 9 C

Question: probability that cost is 𝜕(𝑜 log 𝑜)? The cost will be at most 30𝑜 lg 𝑜 with high probability.

Hoeffding’s Inequality in Action:

Randomized Quicksort

1 7 2 3 B 5 9 C 4 8 A 6 B 9 C 8 A 1 2 3 4 5 6 1 2 4 5 6 9 8 B C 4 6 1 9 C

Question: probability that cost is 𝜕(𝑜 log 𝑜)? The cost will be at most 30𝑜 lg 𝑜 with high probability.

???

Hoeffding’s Inequality in Action:

Randomized Quicksort

1 7 2 3 B 5 9 C 4 8 A 6 B 9 C 8 A 1 2 3 4 5 6 1 2 4 5 6 9 8 B C 4 6 1 9 C

Question: probability that cost is 𝜕(𝑜 log 𝑜)? The cost will be at most 30𝑜 lg 𝑜 with high probability.

Hoeffding’s Inequality in Action:

Randomized Quicksort

1 7 2 3 B 5 9 C 4 8 A 6 B 9 C 8 A 1 2 3 4 5 6 1 2 4 5 6 9 8 B C 4 6 1 9 C

Question: probability that cost is 𝜕(𝑜 log 𝑜)? The cost will be at most 30𝑜 lg 𝑜 with high probability.

Hoeffding’s Inequality in Action:

Randomized Quicksort

1 7 2 3 B 5 9 C 4 8 A 6 B 9 C 8 A 1 2 3 4 5 6 1 2 4 5 6 9 8 B C 4 6 1 9 C

Question: probability that cost is 𝜕(𝑜 log 𝑜)? The cost will be at most 30𝑜 lg 𝑜 with high probability.

Hoeffding’s Inequality in Action:

Randomized Quicksort

1 7 2 3 B 5 9 C 4 8 A 6 B 9 C 8 A 1 2 3 4 5 6 1 2 4 5 6 9 8 B C 4 6 1 9 C

Question: probability that cost is 𝜕(𝑜 log 𝑜)? The cost will be at most 30𝑜 lg 𝑜 with high probability.

(Some) Concentration Inequalities

Question: probability that X deviates more than 𝜀 from expectation?

(Some) Concentration Inequalities

Question: probability that X deviates more than 𝜀 from expectation?

(More) Load Balancing

Question: maximum number of balls in a bin? “m balls are thrown into n bins uniformly and independently at random”

200 Balls into 200 Bins

previous strategy new strategy

Power of Two Choices

“m balls are thrown into n bins in the following manner: for each ball, choose two bins uniformly and independently at random, then place the ball in the less loaded bin”

Power of Two Choices

Question: maximum number of balls in a bin? “m balls are thrown into n bins in the following manner: for each ball, choose two bins uniformly and independently at random, then place the ball in the less loaded bin”

Power of Two Choices

Question: maximum number of balls in a bin? “m balls are thrown into n bins in the following manner: for each ball, choose two bins uniformly and independently at random, then place the ball in the less loaded bin” Ο

log 𝑜 log log 𝑜 versus Ο log log 𝑜 , exponential gap

Power of Two Choices

“m balls are thrown into n bins in the following manner: for each ball, choose two bins uniformly and independently at random, then place the ball in the less loaded bin” the max loaded bin has Ο(log log 𝑜) balls, w.h.p.

Power of Two Choices

“m balls are thrown into n bins in the following manner: for each ball, choose two bins uniformly and independently at random, then place the ball in the less loaded bin” the max loaded bin has Ο(log log 𝑜) balls, w.h.p.

Power of Two Choices

“m balls are thrown into n bins in the following manner: for each ball, choose two bins uniformly and independently at random, then place the ball in the less loaded bin” the max loaded bin has Ο(log log 𝑜) balls, w.h.p.

Power of Two Choices

“m balls are thrown into n bins in the following manner: for each ball, choose two bins uniformly and independently at random, then place the ball in the less loaded bin” the max loaded bin has Ο(log log 𝑜) balls, w.h.p.

Power of Two Choices

“m balls are thrown into n bins in the following manner: for each ball, choose two bins uniformly and independently at random, then place the ball in the less loaded bin” the max loaded bin has Ο(log log 𝑜) balls, w.h.p.

Power of Two Choices

“m balls are thrown into n bins in the following manner: for each ball, choose two bins uniformly and independently at random, then place the ball in the less loaded bin” the max loaded bin has Ο(log log 𝑜) balls, w.h.p.

Power of Two Choices

“m balls are thrown into n bins in the following manner: for each ball, choose two bins uniformly and independently at random, then place the ball in the less loaded bin” the max loaded bin has Ο(log log 𝑜) balls, w.h.p.

Power of Two Choices

“m balls are thrown into n bins in the following manner: for each ball, choose two bins uniformly and independently at random, then place the ball in the less loaded bin” the max loaded bin has Ο(log log 𝑜) balls, w.h.p.

Power of Two Choices

“m balls are thrown into n bins in the following manner: for each ball, choose two bins uniformly and independently at random, then place the ball in the less loaded bin” the max loaded bin has Ο(log log 𝑜) balls, w.h.p.

Power of Two Choices

“m balls are thrown into n bins in the following manner: for each ball, choose two bins uniformly and independently at random, then place the ball in the less loaded bin” the max loaded bin has Ο(log log 𝑜) balls, w.h.p.

Power of Two Choices

“m balls are thrown into n bins in the following manner: for each ball, choose two bins uniformly and independently at random, then place the ball in the less loaded bin” the max loaded bin has Ο(log log 𝑜) balls, w.h.p.

Power of 𝑒 Choices

“m balls are thrown into n bins in the following manner: for each ball, choose 𝑒 ≥ 2 bins uniformly and independently at random, then place the ball in the least loaded bin”

Power of 𝑒 Choices

“m balls are thrown into n bins in the following manner: for each ball, choose 𝑒 ≥ 2 bins uniformly and independently at random, then place the ball in the least loaded bin” the max loaded bin has Ο(log(𝑒) 𝑜) balls, w.h.p.

Power of 𝑒 Choices

“m balls are thrown into n bins in the following manner: for each ball, choose 𝑒 ≥ 2 bins uniformly and independently at random, then place the ball in the least loaded bin” the max loaded bin has Ο(log(𝑒) 𝑜) balls, w.h.p.

Power of 𝑒 Choices

“m balls are thrown into n bins in the following manner: for each ball, choose 𝑒 ≥ 2 bins uniformly and independently at random, then place the ball in the least loaded bin” the max loaded bin has Ο(log(𝑒) 𝑜) balls, w.h.p. the max loaded bin has Ο

log log 𝑜 log 𝑒

balls, w.h.p.