Chapter 10 Randomized Algorithms II High Probability CS 573: - PDF document

Chapter 10 Randomized Algorithms II – High Probability CS 573: Algorithms, Fall 2014 September 25, 2014 10.1 Movie... 10.2 Understanding the binomial distribution 10.2.0.1 Binomial distribution X n = numbers of heads when flipping a coin n times. Claim = ( n i ) � � Pr X n = i 2 n . � n � n ! Where: = ( n − k )! k ! . k � n � Indeed, is the number of ways to choose i elements out of n elements (i.e., pick which i coin flip i come up heads). Each specific such possibility (say 0100010 ... ) had probability 1 / 2 n . 10.2.0.2 Massive randomness.. Is not that random. Consider flipping a fair coin n times independently, head given 1, tail gives zero. How many heads? ...we get a binomial distribution. 1

2

3

4

10.2.0.3 Massive randomness.. Is not that random. This is known as concentration of mass . This is a very special case of the law of large numbers . 10.2.1 Side note... 10.2.1.1 Law of large numbers (weakest form)... Informal statement of law of large numbers For n large enough, the middle portion of the binomial distribution looks like (converges to) the nor- mal/Gaussian distribution. 10.2.1.2 Massive randomness.. Is not that random. Intuitive conclusion Randomized algorithm are unpredictable in the tactical level, but very predictable in the strategic level. 5

10.2.1.3 What is really hiding below the Normal distribution? Taken from ? . 10.3 QuickSort with high probability 10.4 AlgorithmQuickSort and Treaps with High Probability 10.4.0.4 Show that QuickSort running time is O ( n log n ) (A) QuickSort picks a pivot, splits into two subproblems, and continues recursively. (B) Track single element in input. (C) Game ends, when this element is alone in subproblem. (D) Show every element in input, participates ≤ 32 ln n rounds (with high enough probability). (E) E i : event i th element participates > 32 ln n rounds. (F) C QS : number of comparisons performed by QuickSort . (G) Running time O ( C QS ). � � � � i E i ] ≤ � n (H) Probability of failure is α = Pr C QS ≥ 32 n ln n ≤ Pr [ � i =1 Pr E i . ... by the union bound. 10.4.0.5 Show that QuickSort running time is O ( n log n ) � � � � i E i ] ≤ � n (A) Probability of failure is α = Pr C QS ≥ 32 n ln n ≤ Pr [ � i =1 Pr . E i (B) Union bound : for any two events A and B : Pr [ A ∪ B ] ≤ Pr [ A ] + Pr [ B ]. (C) Assume: Pr [ E i ] ≤ 1 /n 3 . � � � n � n 1 1 (D) Bad probability... α ≤ i =1 Pr E i ≤ n 3 = n 2 . i =1 (E) = ⇒ QuickSort performs ≤ 32 n ln n comparisons, w.h.p. (F) = ⇒ QuickSort runs in O ( n log n ) time, with high probability. 6

10.4.1 Proving that an element participates in small number of rounds 10.4.2 Proving that an element... 10.4.2.1 ... participates in small number of rounds. (A) n : number of elements in input for QuickSort . (B) x : Arbitrary element x in input. (C) S 1 : Input. (D) S i : input to i th level recursive call that include x . (E) x lucky in j th iteration, if balanced split... | S j +1 | ≤ (3 / 4) | S j | and | S j \ S j +1 | ≤ (3 / 4) | S j | (F) Y j = 1 ⇐ ⇒ x lucky in j th iteration. � � = 1 (G) Pr Y j 2 . (H) Observation : Y 1 , Y 2 , . . . , Y m are independent variables. (I) x can participate ≤ ρ = log 4 / 3 n ≤ 3 . 5 ln n rounds. (J) ...since | S j | ≤ n (3 / 4) # of lucky iteration in1 ...j . ⇒ | S k | ≤ (3 / 4) ρ n ≤ 1. (K) If ρ lucky rounds in first k rounds = 10.4.3 Proving that an element... 10.4.3.1 ... participates in small number of rounds. (A) Brain reset! (B) Q: How many rounds x participates in = how many coin flips till one gets ρ heads? (C) A: In expectation, 2 ρ times. 10.4.4 Proving that an element... 10.4.4.1 ... participates in small number of rounds. (A) Assume the following: Lemma 10.4.1. In M coin flips: Pr [# heads ≤ M/ 4] ≤ exp( − M/ 8) . (B) Set M = 32 ln n ≥ 8 ρ . (C) Pr [ Y j = 0] = Pr [ Y j = 1] = 1 / 2. (D) Y 1 , Y 2 , . . . , Y M are independent. (E) = ⇒ probability ≤ ρ ≤ M/ 4 ones in Y 1 , . . . , Y M is − M ≤ exp( − ρ ) ≤ 1 � � ≤ exp n 3 . 8 ⇒ probability x participates in M recursive calls of QuickSort ≤ 1 /n 3 . (F) = 10.4.5 Proving that an element... 10.4.5.1 ... participates in small number of rounds. (A) n input elements. Probability depth of recursion in QuickSort > 32 ln n is ≤ (1 /n 3 ) ∗ n = 1 /n 2 . (B) Result: 7

Theorem 10.4.2. With high probability (i.e., 1 − 1 /n 2 ) the depth of the recursion of QuickSort is ≤ 32 ln n . Thus, with high probability, the running time of QuickSort is O ( n log n ) . (C) Same result holds for MatchNutsAndBolts . 10.4.5.2 Alternative proof of high probability of QuickSort (A) T : n items to be sorted. (B) t ∈ T : element. (C) X i : the size of subproblem in i th level of recursion containing t . � � � ≤ 1 3 4 X i − 1 + 1 2 X i − 1 ≤ 7 (D) X 0 = n , and E X i � X i − 1 8 X i − 1 . � 2 � � � � � � � (E) ∀ random variables E X = E y X � � Y = y . E � � � � � i n. � i � � � � � � � � � � 7 = 7 7 7 (F) E X i = E y X i � X i − 1 = y ≤ E X i − 1 = y 8 y X i − 1 ≤ E [ X 0 ] = E 8 E � 8 8 10.4.5.3 Alternative proof of high probability of QuickSort � M n ≤ � � � 7 1 1 (A) M = 8 log 8 / 7 n : µ = E X M ≤ n 8 n = n 7 . 8 (B) Markov’s Inequality : For a non-negative variable X , and t > 0, we have: � � E [ X ] Pr X ≥ t ≤ . t (C) By Markov’s inequality: � � t participates ≤ E [ X M ] ≤ 1 � � Pr ≤ Pr X M ≥ 1 n 7 . > M recursive calls 1 (D) Probability any element of input participates > M recursive calls ≤ n (1 /n 7 ) ≤ 1 /n 6 . 10.5 Chernoff inequality 10.5.0.4 Preliminaries (A) X, Y : Random variables are independent if ∀ x, y : � � � � � � Pr ( X = x ) ∩ ( Y = y ) = Pr X = x · Pr Y = y . (B) The following is easy to prove: Claim 10.5.1. If X and Y are independent = ⇒ E [ XY ] = E [ X ] E [ Y ] . ⇒ Z = e X and W = e Y are independent. = 10.5.0.5 Chernoff inequality Theorem 10.5.2 (Chernoff inequality). X 1 , . . . , X n : n independent random variables, such that 1 Let Y = � n Pr [ X i = 1] = Pr [ X i = − 1] = 2 , for i = 1 , . . . , n . i =1 X i . Then, for any ∆ > 0 , we have � � � � − ∆ 2 / 2 n Pr Y ≥ ∆ ≤ exp . 8

10.5.0.6 Proof of Chernoff inequality Fix arbitrary t > 0: � � � � � � Pr Y ≥ ∆ = Pr tY ≥ t ∆ = Pr exp( tY ) ≥ exp( t ∆) � � exp( tY ) E ≤ , exp( t ∆) 10.5.1 Proof of Chernoff inequality 10.5.1.1 Continued... 2 e − t = e t + e − t = 1 2 e t + 1 � � exp( tX i ) E 2 � 1! + t 2 2! + t 3 � 1 1 + t = 3! + · · · 2 � 1! + t 2 2! − t 3 � + 1 1 − t 3! + · · · 2 = 1 + t 2 2! + + · · · + t 2 k (2 k )! + · · · . However: (2 k )! = k !( k + 1)( k + 2) · · · 2 k ≥ k !2 k . � i ∞ t 2 i ∞ t 2 i ∞ � t 2 � t 2 � 1 � � � � � exp( tX i ) = (2 i )! ≤ 2 i ( i !) = ≤ = ≤ exp . E i ! 2 2 i =0 i =0 i =0 n n � t 2 � nt 2 � �� �� �� � � � � � � � � � exp( tY ) = E exp tX i = E exp( tX i ) = exp( tX i ) ≤ exp = ≤ exp . E E 2 2 i i i =1 i =1 � � exp( tY ) � nt 2 � E exp � nt 2 � � � 2 Pr Y ≥ ∆ ≤ ≤ exp( t ∆) = exp 2 − t ∆ . exp( t ∆) Set t = ∆ /n :  � 2  − ∆ 2 � ∆ � �  n − ∆ � �  = exp Pr Y ≥ ∆ ≤ exp n ∆ . 2 n 2 n 10.5.2 Chernoff inequality... 10.5.2.1 ...what it really says By theorem: � n � n n � − ∆ 2 � � � � � i � � Pr Y ≥ ∆ = Pr Y = i = 2 n ≤ exp , 2 n i =∆ i = n/ 2+∆ / 2 9

10.5.3 Chernoff inequality... 10.5.3.1 symmetry Corollary 10.5.3. Let X 1 , . . . , X n be n independent random variables, such that Pr [ X i = 1] = Pr [ X i = − 1] = 1 2 , for i = 1 , . . . , n . Let Y = � n i =1 X i . Then, for any ∆ > 0 , we have − ∆ 2 � � � � Pr | Y | ≥ ∆ ≤ 2 exp . 2 n 10.5.3.2 Chernoff inequality for coin flips X 1 , . . . , X n be n independent coin flips, such that Pr [ X i = 1] = Pr [ X i = 0] = 1 2 , for i = 1 , . . . , n . Let Y = � n i =1 X i . Then, for any ∆ > 0, we have � n � − 2∆ 2 � � − 2∆ 2 � Y − n � � � Pr 2 − Y ≥ ∆ ≤ exp and Pr 2 ≥ ∆ ≤ exp . n n � − 2∆ 2 � � Y − n � � � � � � In particular, we have Pr � ≥ ∆ ≤ 2 exp . � � 2 n 10.5.3.3 The special case we needed Lemma 10.5.4. In a sequence of M coin flips, the probability that the number of ones is smaller than L ≤ M/ 4 is at most exp( − M/ 8) . Proof: Let Y = � m i =1 X i the sum of the M coin flips. By the above corollary, we have: � M 2 − Y ≥ M � M � � � � Pr Y ≤ L = Pr 2 − L = Pr 2 − Y ≥ ∆ , � � � − 2∆ 2 � where ∆ = M/ 2 − L ≥ M/ 4. Using the above Chernoff inequality, we get Pr Y ≤ L ≤ exp ≤ M exp( − M/ 8) . 10.6 The Chernoff Bound — General Case 10.6.1 The Chernoff Bound 10.6.1.1 The general problem Problem 10.6.1. Let X 1 , . . . X n be n independent Bernoulli trials, where � � � � Pr X i = 1 = p i and Pr X i = 0 = 1 − p i , and let denote � Y = X i µ = E [ Y ] . i Question: what is the probability that Y ≥ (1 + δ ) µ. 10