Chapter 12 Randomized Algorithms II High Probability NEW CS 473: - PDF document

Chapter 12 Randomized Algorithms II – High Probability NEW CS 473: Theory II, Fall 2015 October 6, 2015 12.1 Understanding the binomial distribution 12.1.0.1 Binomial distribution X n = numbers of heads when flipping a coin n times. Claim � n � � � Pr X n = i = 2 n . i � n n ! � Where: = ( n − k )! k ! . k � n � Indeed, is the number of ways to choose i elements out of n elements (i.e., pick which i coin flip i come up heads). Each specific such possibility (say 0100010 ... ) had probability 1 / 2 n . 12.1.0.2 Massive randomness.. Is not that random. Consider flipping a fair coin n times independently, head given 1, tail gives zero. How many heads? ...we get a binomial distribution. 1

2

3

4

12.1.0.3 Massive randomness.. Is not that random. This is known as concentration of mass . This is a very special case of the law of large numbers . 12.1.1 Side note... 12.1.1.1 Law of large numbers (weakest form)... Informal statement of law of large numbers For n large enough, the middle portion of the binomial distribution looks like (converges to) the nor- mal/Gaussian distribution. 12.1.1.2 Massive randomness.. Is not that random. Intuitive conclusion Randomized algorithm are unpredictable in the tactical level, but very predictable in the strategic level. 5

12.1.1.3 What is really hiding below the Normal distribution? Taken from Matouˇ sek and Neˇ setˇ ril [1998] . 12.2 QuickSort and Treaps with High Probability 12.2.0.1 Proof of high probability of QuickSort (A) T : n items to be sorted. (B) t ∈ T : element. (C) X i : the size of subproblem S i in i th level of recursion containing t . (D) X 0 = n , and ≤ Pr [lucky] 3 � � � X i � X i − 1 4 X i − 1 + Pr [unlucky] X i − 1 E � � 1 � 4 | S i | , 3 (E) Lucky = pivot used in S i is in rank 4 | S i | (F) Pr [lucky] = 1 / 2. (G) Pr [lucky] = 1 / 2. As such... ≤ 1 3 4 X i − 1 + 1 2 X i − 1 = 7 � � � X i � X i − 1 8 X i − 1 . E � 2 12.2.0.2 Proof of high probability of QuickSort (A) T : n items to be sorted. (B) t ∈ T : element. (C) X i : the size of subproblem in i th level of recursion containing t . � � � ≤ 1 4 X i − 1 + 1 3 2 X i − 1 ≤ 7 (D) X 0 = n , and E X i � X i − 1 8 X i − 1 . � 2 � � � � � � � (E) ∀ random variables E X = E y X � Y = y . E � 6

� � � � i E [ X 0 ] = � 7 � 7 � 7 � � � � � � � i n. � = 7 (F) E X i = E y X i � X i − 1 = y ≤ E X i − 1 = y 8 y X i − 1 ≤ E � 8 E 8 8 12.2.0.3 Proof of high probability of QuickSort � 7 � � � M n ≤ 1 1 (A) M = 8 log 8 / 7 n : µ = E X M ≤ n 8 n = n 7 . 8 � � (B) Markov’s Inequality : For a non-negative variable X , and t > 0, we have: Pr X ≥ t ≤ E [ X ] . t (C) By Markov’s inequality:   t participates ≤ E [ X M ] ≤ 1 � �  ≤ Pr Pr > M recursive X M ≥ 1 n 7 .  1 calls (D) Probability any element of input participates > M recursive calls ≤ n (1 /n 7 ) ≤ 1 /n 6 . 12.2.1 High probability via Chernoff inequality 12.2.1.1 Show that QuickSort running time is O ( n log n ) (A) QuickSort picks a pivot, splits into two subproblems, and continues recursively. (B) Track single element in input. (C) Game ends, when this element is alone in subproblem. (D) Show every element in input, participates ≤ 32 ln n rounds (with high enough probability). (E) E i : event i th element participates > 32 ln n rounds. (F) C QS : number of comparisons performed by QuickSort . (G) Running time O ( C QS ). � � � � i E i ] ≤ � n ≤ Pr [ � (H) Probability of failure is α = Pr C QS ≥ 32 n ln n i =1 Pr E i . ... by the union bound. 12.2.1.2 Show that QuickSort running time is O ( n log n ) � � � � i E i ] ≤ � n ≤ Pr [ � (A) Probability of failure is α = Pr C QS ≥ 32 n ln n i =1 Pr E i . (B) Union bound : for any two events A and B : Pr [ A ∪ B ] ≤ Pr [ A ] + Pr [ B ]. (C) Assume: Pr [ E i ] ≤ 1 /n 3 . � � (D) Bad probability... α ≤ � n ≤ � n 1 1 i =1 Pr n 3 = n 2 . E i i =1 (E) = ⇒ QuickSort performs ≤ 32 n ln n comparisons, w.h.p. (F) = ⇒ QuickSort runs in O ( n log n ) time, with high probability. 12.2.2 Proving that an element participates in small number of rounds 12.2.3 Proving that an element... 12.2.3.1 ... participates in small number of rounds. (A) n : number of elements in input for QuickSort . (B) x : Arbitrary element x in input. (C) S 1 : Input. 7

(D) S i : input to i th level recursive call that include x . (E) x lucky in j th iteration, if balanced split... | S j +1 | ≤ (3 / 4) | S j | and | S j \ S j +1 | ≤ (3 / 4) | S j | (F) Y j = 1 ⇐ ⇒ x lucky in j th iteration. � � = 1 (G) Pr Y j 2 . (H) Observation : Y 1 , Y 2 , . . . , Y m are independent variables. (I) x can participate ≤ ρ = log 4 / 3 n ≤ 3 . 5 ln n rounds. (J) ...since | S j | ≤ n (3 / 4) # of lucky iteration in1 ...j . ⇒ | S k | ≤ (3 / 4) ρ n ≤ 1. (K) If ρ lucky rounds in first k rounds = 12.2.4 Proving that an element... 12.2.4.1 ... participates in small number of rounds. (A) Brain reset! (B) Q: How many rounds x participates in = how many coin flips till one gets ρ heads? (C) A: In expectation, 2 ρ times. 12.2.5 Proving that an element... 12.2.5.1 ... participates in small number of rounds. (A) Assume the following: Lemma 12.2.1. In M coin flips: Pr [# heads ≤ M/ 4] ≤ exp( − M/ 8) . (B) Set M = 32 ln n ≥ 8 ρ . (C) Pr [ Y j = 0] = Pr [ Y j = 1] = 1 / 2. (D) Y 1 , Y 2 , . . . , Y M are independent. (E) = ⇒ probability ≤ ρ ≤ M/ 4 ones in Y 1 , . . . , Y M is � − M � ≤ exp( − ρ ) ≤ 1 ≤ exp n 3 . 8 ⇒ probability x participates in M recursive calls of QuickSort ≤ 1 /n 3 . (F) = 12.2.6 Proving that an element... 12.2.6.1 ... participates in small number of rounds. (A) n input elements. Probability depth of recursion in QuickSort > 32 ln n is ≤ (1 /n 3 ) ∗ n = 1 /n 2 . (B) Result: Theorem 12.2.2. With high probability (i.e., 1 − 1 /n 2 ) the depth of the recursion of QuickSort is ≤ 32 ln n . Thus, with high probability, the running time of QuickSort is O ( n log n ) . (C) Same result holds for MatchNutsAndBolts . 8

12.3 Chernoff inequality 12.3.0.1 Preliminaries (A) X, Y : Random variables are independent if ∀ x, y : � � � � � � Pr ( X = x ) ∩ ( Y = y ) = Pr X = x · Pr Y = y . (B) The following is easy to prove: Claim 12.3.1. If X and Y are independent = ⇒ E [ XY ] = E [ X ] E [ Y ] . ⇒ Z = e X and W = e Y are independent. = 12.3.0.2 Chernoff inequality Theorem 12.3.2 (Chernoff inequality). X 1 , . . . , X n : n independent random variables, such that Let Y = � n 1 Pr [ X i = 1] = Pr [ X i = − 1] = 2 , for i = 1 , . . . , n . i =1 X i . Then, for any ∆ > 0 , we have � � − ∆ 2 / 2 n � � Pr Y ≥ ∆ ≤ exp . 12.3.0.3 Proof of Chernoff inequality Fix arbitrary t > 0: � � � � � � Pr Y ≥ ∆ = Pr tY ≥ t ∆ = Pr exp( tY ) ≥ exp( t ∆) � � exp( tY ) E ≤ , exp( t ∆) 12.3.1 Proof of Chernoff inequality 12.3.1.1 Continued... 2 e − t = e t + e − t = 1 2 e t + 1 � � exp( tX i ) E 2 1! + t 2 2! + t 3 1 � 1 + t � = 3! + · · · 2 1! + t 2 2! − t 3 + 1 � 1 − t � 3! + · · · 2 = 1 + t 2 2! + + · · · + t 2 k (2 k )! + · · · . However: (2 k )! = k !( k + 1)( k + 2) · · · 2 k ≥ k !2 k . ∞ ∞ ∞ � i t 2 i t 2 i � t 2 � t 2 1 � � � � � � exp( tX i ) = (2 i )! ≤ 2 i ( i !) = ≤ = ≤ exp . E i ! 2 2 i =0 i =0 i =0 9

� �� �� �� � n n � t 2 � nt 2 � � � � � � � � exp( tY ) = E exp tX i = E exp( tX i ) = exp( tX i ) ≤ exp = ≤ exp . E E 2 2 i i i =1 i =1 � � � � nt 2 exp( tY ) exp E � nt 2 � � � 2 Pr Y ≥ ∆ ≤ ≤ exp( t ∆) = exp 2 − t ∆ . exp( t ∆) Set t = ∆ /n : � � � 2 − ∆ 2 n � ∆ − ∆ � � � � Pr Y ≥ ∆ ≤ exp n ∆ = exp . 2 n 2 n 12.3.2 Chernoff inequality... 12.3.2.1 ...what it really says By theorem: n n � n � − ∆ 2 � � � � � � � � i Pr Y ≥ ∆ = Pr Y = i = 2 n ≤ exp , 2 n i =∆ i = n/ 2+∆ / 2 12.3.3 Chernoff inequality... 12.3.3.1 symmetry Corollary 12.3.3. Let X 1 , . . . , X n be n independent random variables, such that Pr [ X i = 1] = Pr [ X i = − 1] = 2 , for i = 1 , . . . , n . Let Y = � n 1 i =1 X i . Then, for any ∆ > 0 , we have − ∆ 2 � � � � Pr | Y | ≥ ∆ ≤ 2 exp . 2 n 12.3.3.2 Chernoff inequality for coin flips X 1 , . . . , X n be n independent coin flips, such that Pr [ X i = 1] = Pr [ X i = 0] = 1 2 , for i = 1 , . . . , n . Let Y = � n i =1 X i . Then, for any ∆ > 0, we have � n − 2∆ 2 � � � Pr 2 − Y ≥ ∆ ≤ exp n − 2∆ 2 � � Y − n � � and Pr 2 ≥ ∆ ≤ exp . n − 2∆ 2 � Y − n � � � � � � In particular, we have Pr � ≥ ∆ ≤ 2 exp . � � 2 n Note: Variables X i ∈ { 0 , 1 } . Previous slide X i ∈ {− 1 , 1 } (different result!). 10