Randomness in Computing L ECTURE 10 Last time Chernoff Bounds - - PowerPoint PPT Presentation

2/25/2020

Randomness in Computing

LECTURE 10

Last time

• Chernoff Bounds

Today

• Hoeffding Bounds
• Applications of Chernoff-

Hoeffding Bounds

• Estimating a Parameter
• Set Balancing

Sofya Raskhodnikova;Randomness in Computing

Sums of independent RVs

Chernoff Bound (Upper Tail). Let X1, … , 𝑌𝑜 be independent Bernoulli RVs. Let 𝑌 = 𝑌1 + ⋯ + 𝑌𝑜 and 𝜈 = 𝔽 𝑌 . Then

• (stronger) for any 𝜀 > 0,

Pr 𝑌 ≥ 1 + 𝜀 𝜈 ≤ 𝑓𝜀 1 + 𝜀 1+𝜀

𝜈

.

• (easier to use) for any 𝜀 ∈ (0,1],

Pr 𝑌 ≥ 1 + 𝜀 𝜈 ≤ 𝑓−𝜈𝜀2/3.

2/25/2020

Sofya Raskhodnikova; Randomness in Computing

Sums of independent RVs

Chernoff Bound (Lower Tail). Let X1, … , 𝑌𝑜 be independent Bernoulli RVs. Let 𝑌 = 𝑌1 + ⋯ + 𝑌𝑜 and 𝜈 = 𝔽 𝑌 . Then

• (stronger) for any 𝜀 ∈ (0,1),

Pr 𝑌 ≤ 1 − 𝜀 𝜈 ≤ 𝑓−𝜀 1 − 𝜀 1−𝜀

𝜈

.

• (easier to use) for any 𝜀 ∈ (0,1),

Pr 𝑌 ≤ 1 − 𝜀 𝜈 ≤ 𝑓−𝜈𝜀2/2.

2/25/2020

Sofya Raskhodnikova; Randomness in Computing

Sums of independent RVs

Chernoff Bound (Both Tails). Let X1, … , 𝑌𝑜 be independent Bernoulli RVs. Let 𝑌 = 𝑌1 + ⋯ + 𝑌𝑜 and 𝜈 = 𝔽 𝑌 . Then

• for any 𝜀 ∈ 0,1 ,

Pr |𝑌 − 𝜈| ≥ 𝜀𝜈 ≤ 2𝑓−𝜈𝜀2/3.

2/25/2020

Sofya Raskhodnikova; Randomness in Computing

Exercise 2

• The Halting Problem Team wins each hockey game they play

with probability 1/3. Assuming outcomes of the games are independent, derive an upper bound on the probability that they have a winning season in 𝑜 games.

• The Halting Problem Team hires a new coach, and critics

revise their probability of winning each game to 3/4. Derive an upper bound on the probability they suffer a losing season.

2/25/2020

Exercise 3

We throw 𝑜 balls uniformly and independently into 𝑜 bins. Let 𝑍

1 be the number of balls that fell into bin 1.

Determine 𝑛 such that Pr 𝑍

1 > 𝑛 ≤ 1 𝑜2 .

2/25/2020

Sofya Raskhodnikova; Randomness in Computing

Sums of independent RVs

Hoeffding Bound. Let X1, … , 𝑌𝑜 be independent RVs with 𝔽 𝑌𝑗 = 𝜈0 and Pr 𝑏 ≤ 𝑌𝑗 ≤ 𝑐 = 1. Let 𝑌 = 𝑌1 + ⋯ + 𝑌𝑜. Then

• (upper tail) Pr 𝑌 ≥ 𝜈0𝑜 + 𝜗𝑜 ≤ 𝑓−2𝑜𝜗2/ 𝑐−𝑏 2.
• (lower tail) Pr 𝑌 ≤ 𝜈0𝑜 − 𝜗𝑜 ≤ 𝑓−2𝑜𝜗2/ 𝑐−𝑏 2

2/25/2020

Sofya Raskhodnikova; Randomness in Computing

Application: Estimating a parameter

• Unknown: probability 𝑞 that a feature occurs in the population.
• Obtain an estimate by taking 𝑜 samples
• 𝑌 ∼ Bin(𝑜, 𝑞)
• Suppose 𝑌 =

𝑞𝑜.

• A 1 − 𝛿 confidence interval for parameter 𝑞 is [

𝑞 − 𝜀, 𝑞 + 𝜀] such that Pr[𝑞 ∈ [ 𝑞 − 𝜀, 𝑞 + 𝜀]] ≥ 1 − 𝛿.

• Find a tradeoff between 𝛿, 𝜀 and 𝑜.

2/25/2020

Sofya Raskhodnikova; Randomness in Computing

Application: Estimating a parameter

• A 1 − 𝛿 confidence interval for parameter 𝑞 is [

𝑞 − 𝜀, 𝑞 + 𝜀] such that Pr[𝑞 ∈ [ 𝑞 − 𝜀, 𝑞 + 𝜀]] ≥ 1 − 𝛿.

• Find a tradeoff between 𝛿, 𝜀 and 𝑜.

Solution: 𝔽 𝑌 = 𝑜𝑞

• Suppose 𝑞 ∉ [

𝑞 − 𝜀, 𝑞 + 𝜀]

• Case 1: 𝑞 <

𝑞 − 𝜀. Then 𝑞 > 𝑞 + 𝜀

• Case 2: 𝑞 >

𝑞 + 𝜀. Then 𝑞 < 𝑞 − 𝜀

• 𝛿 = 2 ⋅ 𝑓−2𝜀2𝑜

2/25/2020

Sofya Raskhodnikova; Randomness in Computing

Application: Set Balancing

• Given: an 𝑜 × 𝑛 matrix 𝐵 with 0-1 entries
• Definition: | (𝑦1, … , 𝑦𝑜) |∞ = max

𝑗∈[𝑜] |𝑦𝑗|

• Find: 𝑐 ∈ −1,1 𝑛 minimizing 𝐵𝑐

∞

𝑏11 𝑏12 … 𝑏1𝑛 𝑏21 𝑏22 … 𝑏2𝑛 … 𝑏𝑜1 𝑏𝑜2 … 𝑏𝑜𝑛 Partition subjects into two groups, so that each feature is balanced.

2/25/2020

Sofya Raskhodnikova; Randomness in Computing

𝒏 subjects 𝒐 features

Application: Set Balancing

• Algorithm: Choose each 𝑐𝑗 u.a.r. from −1,1 .
• Theorem. Pr

𝐵𝑐

∞ ≥

4𝑛 ln 𝑜 ≤ 2/𝑜.

2/25/2020

Sofya Raskhodnikova; Randomness in Computing