CS4102 Algorithms Summer 2020 Warm up Show log ! = ( log ) Hint: - PowerPoint PPT Presentation

CS4102 Algorithms Summer 2020 Warm up Show log 𝑜! = Θ(𝑜 log 𝑜) Hint: show 𝑜! ≤ 𝑜 𝑜 𝑜 𝑜 2 Hint 2: show 𝑜! ≥ 2 1

log 𝑜! = 𝑃 𝑜 log 𝑜 𝑜! = 𝑜 ⋅ 𝑜 − 1 ⋅ 𝑜 − 2 ⋅ … ⋅ 2 ⋅ 1 = < < < < 𝑜 𝑜 = 𝑜 ⋅ 𝑜 ⋅ 𝑜 ⋅ … ⋅ 𝑜 ⋅ 𝑜 𝑜! ≤ 𝑜 𝑜 ⇒ log 𝑜! ≤ log 𝑜 𝑜 ⇒ log 𝑜! ≤ 𝑜 log 𝑜 ⇒ log 𝑜! = 𝑃(𝑜 log 𝑜) 2

log 𝑜! = Ω 𝑜 log 𝑜 𝑜! = 𝑜 ⋅ 𝑜 − 1 ⋅ 𝑜 − 2 ⋅ … ⋅ 𝑜 2 ⋅ 𝑜 2 − 1 ⋅ … ⋅ 2 ⋅ 1 > > > = > > = 𝑜 𝑜 2 = 𝑜 2 ⋅ 𝑜 2 ⋅ 𝑜 2 ⋅ … ⋅ 𝑜 2 ⋅ 1 ⋅ … ⋅ 1 ⋅ 1 2 𝑜 𝑜! ≥ 𝑜 2 2 𝑜 𝑜 2 ⇒ log 𝑜! ≥ log 2 ⇒ log 𝑜! ≥ 𝑜 2 log 𝑜 2 ⇒ log 𝑜! = Ω(𝑜 log 𝑜) 3

Median of Medians, Run Time Θ(𝑜) 1. Break list into chunks of 5 Θ(𝑜) 2. Find the median of each chunk 3. Return median of medians (using Quickselect) 𝑇 𝑜 5 𝑁 𝑜 = 𝑇 𝑜 5 + Θ(𝑜) 4

Quickselect 𝑁 𝑜 = 𝑇 𝑜 𝑇 𝑜 ≤ 𝑇 7𝑜 5 + Θ(𝑜) 10 + 𝑁 𝑜 + Θ(𝑜) = 𝑇 7𝑜 10 + 𝑇 𝑜 5 + Θ(𝑜) = 𝑇 7𝑜 10 + 𝑇 2𝑜 + Θ(𝑜) 10 ≤ 𝑇 9𝑜 + Θ(𝑜) Because 𝑇 𝑜 = Ω(𝑜) 10 Master theorem Case 3! 𝑇 𝑜 = O(𝑜) 𝑇 𝑜 = Θ(𝑜) 5

Phew! Back to Quicksort Using Quickselect, with a median-of-medians partition: 2 5 1 3 6 4 7 8 10 9 11 12 2 1 3 5 6 4 7 8 9 10 11 12 Then we divide in half each time 𝑈 𝑜 = 2𝑈 𝑜 2 + Θ(𝑜) 𝑈 𝑜 = Θ(𝑜 log 𝑜) 6

Random Pivot • Using Quickselect to pick median guarantees Θ(𝑜 log 𝑜) run time – Approach has very large constants – If you really want Θ(𝑜 log 𝑜) , better off using MergeSort • Better approach: Random pivot – Very small constant (very fast algorithm) – Expected to run in Θ(𝑜 log 𝑜) time • Why? Unbalanced partitions are very unlikely – Other options: Median of 5 7

Quicksort Run Time If the pivot is always 𝑜 th order statistic: 10 10 + 𝑈 9𝑜 𝑜 𝑈 𝑜 = 𝑈 + 𝑜 10 8

10 + 𝑈 9𝑜 𝑜 𝑈 𝑜 = 𝑈 10 + 𝑜 𝑜 𝑜 𝑜 𝑜/10 9𝑜/10 𝑜 𝑜 10 9𝑜 10 + 9𝑜/100 81𝑜/100 9𝑜/ 100 𝑜/100 𝑜 100 9𝑜 100 9𝑜 100 81𝑜 100 𝑜 + + + log 10 𝑜 … … … … 9 1 1 + 1 1 1 + 1 1 + 1

Quicksort Run Time If the pivot is always 𝑜 th order statistic: 10 10 + 𝑈 9𝑜 𝑜 𝑈 𝑜 = 𝑈 + 𝑜 10 𝑈 𝑜 = Θ(𝑜 log 𝑜) 10

Quicksort Run Time If the pivot is always 𝑒 th order statistic: 1 5 2 3 6 4 7 8 10 9 11 12 1 2 3 5 6 4 7 8 10 9 11 12 Then we shorten by 𝑒 each time 𝑈 𝑜 = 𝑈 𝑜 − 𝑒 + 𝑜 𝑈 𝑜 = 𝑃(𝑜 2 ) What’s the probability of this occurring? 11

Probability of 𝑜 2 run time We must consistently select pivot from within the first 𝑒 terms 𝑒 Probability first pivot is among 𝑒 smallest: 𝑜 𝑒 Probability second pivot is among 𝑒 smallest: 𝑜−𝑒 Probability all pivots are among 𝑒 smallest: 𝑒 𝑒 𝑜 − 2𝑒 ⋅ … ⋅ 𝑒 𝑒 1 𝑜 ⋅ 𝑜 − 𝑒 ⋅ 2𝑒 ⋅ 1 = 𝑜 𝑒 ! 12

Formal Argument for 𝑜 log 𝑜 Average • Remember, run time counts comparisons! • Quicksort only compares against a pivot – Element 𝑗 only compared to element 𝑘 if one of them was the pivot 13

Formal Argument for 𝑜 log 𝑜 Average • What is the probability of comparing two given elements? 1 2 3 4 5 6 7 8 9 10 11 12 • (Probability of comparing 3 and 4) = 1 – Why? Otherwise I wouldn’t know which came first – ANY sorting algorithm must compare adjacent elements 14

Formal Argument for 𝑜 log 𝑜 Average • What is the probability of comparing two given elements? 1 2 3 4 5 6 7 8 9 10 11 12 • (Probability of comparing 1 and 12) = 2 12 – Why? • I only compare 1 with 12 if either was chosen as the first pivot • Otherwise they would be divided into opposite sublists 15

Formal Argument for 𝑜 log 𝑜 Average • Probability of comparing 𝑗 with 𝑘 ( 𝑘 > 𝑗 ): – dependent on the number of elements between 𝑗 and 𝑘 1 – 𝑘−𝑗+1 • Expected number of comparisons: 1 – 𝑗<𝑘 𝑘−𝑗+1 16

Expected number of Comparisons 1 Consider when 𝑗 = 1 𝑘 − 𝑗 + 1 𝑗<𝑘 1 2 3 4 5 6 7 8 9 10 11 12 Compared if 1 or 2 are chosen as pivot (these will always be compared) 2 Sum so far: 2 17

Expected number of Comparisons 1 Consider when 𝑗 = 1 𝑘 − 𝑗 + 1 𝑗<𝑘 1 2 3 4 5 6 7 8 9 10 11 12 Compared if 1 or 3 are chosen as pivot (but never if 2 is ever chosen) 2 2 2 + Sum so far: 3 18

Expected number of Comparisons 1 Consider when 𝑗 = 1 𝑘 − 𝑗 + 1 𝑗<𝑘 1 2 3 4 5 6 7 8 9 10 11 12 Compared if 1 or 4 are chosen as pivot (but never if 2 or 3 are chosen) 2 2 2 2 + 3 + Sum so far: 4 19

Expected number of Comparisons 1 Consider when 𝑗 = 1 𝑘 − 𝑗 + 1 𝑗<𝑘 1 2 3 4 5 6 7 8 9 10 11 12 Compared if 1 or 12 are chosen as pivot (but never if 2 -> 11 are chosen) 2 2 2 2 2 2 + 3 + 4 + 5 + ⋯ + Overall sum: 𝑜 20

Expected number of Comparisons 1 𝑘 − 𝑗 + 1 𝑗<𝑘 1 1 1 1 When 𝑗 = 1 : 2 2 + 3 + 4 + ⋯ + 𝑜 𝑜 terms overall 1 ≤ 2𝑜 1 2 + 1 3 + ⋯ + 1 Θ(log 𝑜) 𝑜 𝑘 − 𝑗 + 1 𝑗<𝑘 Quicksort overall: expected Θ 𝑜 log 𝑜 21

Sorting, so far • Sorting algorithms we have discussed: 𝑃(𝑜 log 𝑜) – Mergesort 𝑃(𝑜 log 𝑜) – Quicksort • Other sorting algorithms (will discuss): 𝑃(𝑜 2 ) – Bubblesort 𝑃(𝑜 2 ) – Insertionsort 𝑃(𝑜 log 𝑜) – Heapsort Can we do better than 𝑃(𝑜 log 𝑜) ? 22

Worst Case Lower Bounds • Prove that there is no algorithm which can sort faster than 𝑃(𝑜 log 𝑜) – Every algorithm, in the worst case, must have a certain lower bound • Non-existence proof! – Very hard to do 23

Strategy: Decision Tree • Sorting algorithms use comparisons to figure out the order of input elements • Draw tree to illustrate all possible execution paths One comparison Result of Possible comparison execution path < >or<? > >or<? >or<? < < > > >or<? >or<? >or<? >or<? < < > < > < > > >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? … … … … Permutation … … [1,2,3,4,5] [2,1,3,4,5] [5,2,4,1,3] [5,4,3,2,1] of sorted list 24

Strategy: Decision Tree • Worst case run time is the longest execution path • i.e., “height” of the decision tree One comparison Result of Possible comparison execution path < >or<? > >or<? >or<? < < > > >or<? >or<? >or<? >or<? < < > < > < > > log 𝑜! >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? Θ(𝑜 log 𝑜) … … … … Permutation … … [1,2,3,4,5] [2,1,3,4,5] [5,2,4,1,3] [5,4,3,2,1] of sorted list 𝑜! Possible permutations 25

Strategy: Decision Tree • Conclusion: Worst Case Optimal run time of sorting is Θ(𝑜 log 𝑜) – There is no (comparison-based) sorting algorithm with run time 𝑝(𝑜 log 𝑜) One comparison Result of Possible comparison execution path < >or<? > >or<? >or<? < < > > >or<? >or<? >or<? >or<? < < > < > < > > log 𝑜! >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? Θ(𝑜 log 𝑜) … … … … Permutation … … [1,2,3,4,5] [2,1,3,4,5] [5,2,4,1,3] [5,4,3,2,1] of sorted list 𝑜! Possible permutations 26

Sorting, so far • Sorting algorithms we have discussed: 𝑃(𝑜 log 𝑜) – Mergesort Optimal! 𝑃(𝑜 log 𝑜) – Quicksort Optimal! • Other sorting algorithms (will discuss): 𝑃(𝑜 2 ) – Bubblesort 𝑃(𝑜 2 ) – Insertionsort 𝑃(𝑜 log 𝑜) – Heapsort Optimal! 27

Speed Isn’t Everything • Important properties of sorting algorithms: • Run Time – Asymptotic Complexity – Constants • In Place (or In-Situ) – Done with only constant additional space • Adaptive – Faster if list is nearly sorted • Stable – Equal elements remain in original order • Parallelizable – Runs faster with multiple computers 28

Mergesort • Run Time? Divide: – Break 𝑜 -element list into two lists of 𝑜 2 elements Θ(𝑜 log 𝑜) • Conquer: – If 𝑜 > 1 : Sort each sublist recursively Optimal! – If 𝑜 = 1 : List is already sorted (base case) • Combine: – Merge together sorted sublists into one sorted list In Place? Adaptive? Stable? No No Yes! (usually)