CS70: Lecture 33. Inequalities: An Overview Chebyshev Distribution - PowerPoint PPT Presentation

CS70: Lecture 33. Inequalities: An Overview Chebyshev Distribution Markov WLLN, Confidence Intervals (CI): Chebyshev vs. CLT p n p n p n 1. Review: Inequalities: Markov, Chebyshev 2. Law of Large Numbers p n � � n n n 3. Review: CLT µ µ a 4. Confidence Intervals: Chebyshev vs. CLT Pr [ X > a ] Pr [ | X − µ | > � ] Fraction of H ’s Fraction of H ’s Here is a classical application of Chebyshev’s inequality. How likely is it that the fraction of H ’s differs from 50 % ? M n = X 1 + ··· + X n , for n ≥ 1 . Let X m = 1 if the m -th flip of a fair coin is H and X m = 0 n otherwise. Pr [ | M n − 0 . 5 | ≥ 0 . 1 ] ≤ 25 n . Define M n = X 1 + ··· + X n , for n ≥ 1 . For n = 1 , 000, we find that this probability is less than 2 . 5 % . n As n → ∞ , this probability goes to zero. We want to estimate In fact, for any ε > 0, as n → ∞ , the probability that the fraction Pr [ | M n − 0 . 5 | ≥ 0 . 1 ] = Pr [ M n ≤ 0 . 4 or M n ≥ 0 . 6 ] . of H s is within ε > 0 of 50 % approaches 1: By Chebyshev, Pr [ | M n − 0 . 5 | ≤ ε ] → 1 . Pr [ | M n − 0 . 5 | ≥ 0 . 1 ] ≤ var [ M n ] ( 0 . 1 ) 2 = 100 var [ M n ] . This is an example of the (Weak) Law of Large Numbers. Now, We look at a general case next. var [ M n ] = 1 n 2 ( var [ X 1 ]+ ··· + var [ X n ]) = 1 n var [ X 1 ] ≤ 1 4 n . Var ( X i ) = p ( 1 − p ) ≤ ( . 5 )( . 5 ) = 1 4

Weak Law of Large Numbers Theorem Weak Law of Large Numbers Let X 1 , X 2 ,... be pairwise independent with the same distribution and mean µ . Then, for all ε > 0, Pr [ | X 1 + ··· + X n − µ | ≥ ε ] → 0 , as n → ∞ . n Proof: Let M n = X 1 + ··· + X n . Then n var [ M n ] = var [ X 1 + ··· + X n ] Pr [ | M n − µ | ≥ ε ] ≤ ε 2 n 2 ε 2 nvar [ X 1 ] = var [ X 1 ] = → 0 , as n → ∞ . n 2 ε 2 n ε 2 Recap: Normal (Gaussian) Distribution. Recap: Central Limit Theorem Confidence Interval (CI) for Mean: CLT Let X 1 , X 2 ,... be i.i.d. with mean µ and variance σ 2 . Let For any µ and σ , a normal (aka Gaussian ) random variable Y , Central Limit Theorem which we write as Y = N ( µ , σ 2 ) , has pdf Let X 1 , X 2 ,... be i.i.d. with E [ X 1 ] = µ and var ( X 1 ) = σ 2 . Define A n = X 1 + ··· + X n . n 1 S n := A n − µ σ / √ n = X 1 + ··· + X n − n µ 2 πσ 2 e − ( y − µ ) 2 / 2 σ 2 . f Y ( y ) = √ σ √ n . The CLT states that A n − µ σ / √ n = X 1 + ··· + X n − n µ σ √ n → N ( 0 , 1 ) as n → ∞ . Standard normal has µ = 0 and σ = 1 . Then, S n → N ( 0 , 1 ) , as n → ∞ . Thus, for n ≫ 1, one has That is, Pr [ − 2 ≤ ( A n − µ σ / √ n ) ≤ 2 ] ≈ 95 % . � α 1 − ∞ e − x 2 / 2 dx . Pr [ S n ≤ α ] → √ 2 π Equivalently, Pr [ µ ∈ [ A n − 2 σ √ n , A n + 2 σ 1 √ n ]] ≈ 95 % . E ( S n ) = σ / √ n ( E ( A n ) − µ ) = 0 That is, 1 Var ( S n ) = σ 2 / nVar ( A n ) = 1 . [ A n − 2 σ √ n , A n + 2 σ √ n ] is a 95 % − CI for µ . Note: Pr [ | Y − µ | > 1 . 65 σ ] = 10 %; Pr [ | Y − µ | > 2 σ ] = 5 % .

CI for Mean: CLT vs. Chebyshev Coins and CLT. Coins and CLT. Let X 1 , X 2 ,... be i.i.d. B ( p ) . Thus, X 1 + ··· + X n = B ( n , p ) . Let X 1 , X 2 ,... be i.i.d. B ( p ) . Thus, X 1 + ··· + X n = B ( n , p ) . Let X 1 , X 2 ,... be i.i.d. with mean µ and variance σ 2 . Let � Here, µ = p and σ = p ( 1 − p ) . CLT states that � Here, µ = p and σ = p ( 1 − p ) . CLT states that X 1 + ··· + X n − np A n = X 1 + ··· + X n → N ( 0 , 1 ) . X 1 + ··· + X n − np � n → N ( 0 , 1 ) . p ( 1 − p ) n � p ( 1 − p ) n The CLT states that and [ A n − 2 σ √ n , A n + 2 σ √ n ] is a 95 % − CI for µ X 1 + ··· + X n − n µ σ √ n → N ( 0 , 1 ) as n → ∞ . with A n = ( X 1 + ··· + X n ) / n . Hence, Also, [ A n − 2 σ √ n , A n + 2 σ [ A n − 2 σ √ n , A n + 2 σ √ n ] is a 95 % − CI for µ . √ n ] is a 95 % − CI for p . What would Chebyshev’s bound give us? Since σ ≤ 0 . 5 , [ A n − 4 . 5 σ √ n , A n + 4 . 5 σ [ A n − 20 . 5 √ n , A n + 20 . 5 √ n ] is a 95 % − CI for µ . ( Why ?) √ n ] is a 95 % − CI for p . Thus, Thus, the CLT provides a smaller confidence interval. [ A n − 1 √ n , A n + 1 √ n ] is a 95 % − CI for p . Summary Inequalities and Confidence Interals 1. Inequalities: Markov and Chebyshev Tail Bounds 2. Weak Law of Large Numbers 3. Confidence Intervals: Chebyshev Bounds vs. CLT Approx. ⇒ A n − µ 4. CLT: X n i.i.d. = σ / √ n → N ( 0 , 1 ) 5. CI: [ A n − 2 σ √ n , A n + 2 σ √ n ] = 95 % -CI for µ .