CS4102 Algorithms Summer 2020 Warm up Show log ! = ( log ) Hint: - - PowerPoint PPT Presentation

Warm up Show log 𝑜! = Θ(𝑜 log 𝑜) Hint: show 𝑜! ≤ 𝑜𝑜 Hint 2: show 𝑜! ≥

𝑜 2

𝑜 2

1

CS4102 Algorithms

Summer 2020

2

log 𝑜! = 𝑃 𝑜 log 𝑜

𝑜! = 𝑜 ⋅ 𝑜 − 1 ⋅ 𝑜 − 2 ⋅ … ⋅ 2 ⋅ 1 𝑜𝑜 = 𝑜 ⋅ 𝑜 ⋅ 𝑜 ⋅ … ⋅ 𝑜 ⋅ 𝑜 = < < < < 𝑜! ≤ 𝑜𝑜 ⇒ log 𝑜! ≤ log 𝑜𝑜 ⇒ log 𝑜! ≤ 𝑜 log 𝑜 ⇒ log 𝑜! = 𝑃(𝑜 log 𝑜)

3

log 𝑜! = Ω 𝑜 log 𝑜

𝑜! = 𝑜 ⋅ 𝑜 − 1 ⋅ 𝑜 − 2 ⋅ … ⋅ 𝑜 2 ⋅ 𝑜 2 − 1 ⋅ … ⋅ 2 ⋅ 1 𝑜 2

𝑜 2 = 𝑜

2 ⋅ 𝑜 2 ⋅ 𝑜 2 ⋅ … ⋅ 𝑜 2 ⋅ 1 ⋅ … ⋅ 1 ⋅ 1 > > > > = 𝑜! ≥ 𝑜 2

𝑜 2

⇒ log 𝑜! ≥ log 𝑜 2

𝑜 2

⇒ log 𝑜! ≥ 𝑜 2 log 𝑜 2 ⇒ log 𝑜! = Ω(𝑜 log 𝑜) > =

Median of Medians, Run Time

4

• 1. Break list into chunks of 5
• 2. Find the median of each chunk
• 3. Return median of medians (using Quickselect)

Θ(𝑜) Θ(𝑜) 𝑇 𝑜 5 𝑁 𝑜 = 𝑇 𝑜 5 + Θ(𝑜)

Quickselect

5

𝑁 𝑜 = 𝑇 𝑜 5 + Θ(𝑜) 𝑇 𝑜 ≤ 𝑇 7𝑜 10 + 𝑁 𝑜 + Θ(𝑜) = 𝑇 7𝑜 10 + 𝑇 𝑜 5 + Θ(𝑜) 𝑇 𝑜 = O(𝑜) = 𝑇 7𝑜 10 + 𝑇 2𝑜 10 + Θ(𝑜) ≤ 𝑇 9𝑜 10 + Θ(𝑜)

Master theorem Case 3! Because 𝑇 𝑜 = Ω(𝑜)

𝑇 𝑜 = Θ(𝑜)

Phew! Back to Quicksort

Then we divide in half each time

6

2 5 1 3 6 4 7 8 10 9 11 12 2 1 3 5 6 4 7 8 9 10 11 12

𝑈 𝑜 = 2𝑈 𝑜 2 + Θ(𝑜)

Using Quickselect, with a median-of-medians partition:

𝑈 𝑜 = Θ(𝑜 log 𝑜)

Random Pivot

• Using Quickselect to pick median guarantees Θ(𝑜 log 𝑜) run

time

– Approach has very large constants – If you really want Θ(𝑜 log 𝑜), better off using MergeSort

• Better approach: Random pivot

– Very small constant (very fast algorithm) – Expected to run in Θ(𝑜 log 𝑜) time

• Why? Unbalanced partitions are very unlikely

– Other options: Median of 5

7

Quicksort Run Time

8

𝑈 𝑜 = 𝑈 𝑜 10 + 𝑈 9𝑜 10 + 𝑜

If the pivot is always 𝑜

10

th order statistic:

𝑜

𝑈 𝑜 = 𝑈 𝑜 10 + 𝑈 9𝑜 10 + 𝑜

𝑜 10 9𝑜 10 𝑜 100 9𝑜 100 9𝑜 100 81𝑜 100

… … … …

1 1 1 1

𝑜 𝑜/10 9𝑜/10 𝑜/100 9𝑜/100 9𝑜/100 81𝑜/100 1 1 1 1

𝑜 𝑜 𝑜 + + + + + + +

log 10

9

𝑜

Quicksort Run Time

10

𝑈 𝑜 = 𝑈 𝑜 10 + 𝑈 9𝑜 10 + 𝑜

If the pivot is always 𝑜

10

th order statistic:

𝑈 𝑜 = Θ(𝑜 log 𝑜)

Quicksort Run Time

Then we shorten by 𝑒 each time

11

1 5 2 3 6 4 7 8 10 9 11 12 1 2 3 5 6 4 7 8 10 9 11 12

𝑈 𝑜 = 𝑈 𝑜 − 𝑒 + 𝑜

If the pivot is always 𝑒th order statistic:

𝑈 𝑜 = 𝑃(𝑜2) What’s the probability of this occurring?

Probability of 𝑜2 run time

We must consistently select pivot from within the first 𝑒 terms

12

Probability first pivot is among 𝑒 smallest:

𝑒 𝑜

Probability second pivot is among 𝑒 smallest:

𝑒 𝑜−𝑒

Probability all pivots are among 𝑒 smallest: 𝑒 𝑜 ⋅ 𝑒 𝑜 − 𝑒 ⋅ 𝑒 𝑜 − 2𝑒 ⋅ … ⋅ 𝑒 2𝑒 ⋅ 1 = 1 𝑜 𝑒 !

Formal Argument for 𝑜 log 𝑜 Average

• Remember, run time counts comparisons!
• Quicksort only compares against a pivot

– Element 𝑗 only compared to element 𝑘 if one of them was the pivot

13

Formal Argument for 𝑜 log 𝑜 Average

• What is the probability of comparing two

given elements?

14

1 2 3 4 5 6 7 8 9 10 11 12

• (Probability of comparing 3 and 4) = 1

– Why? Otherwise I wouldn’t know which came first – ANY sorting algorithm must compare adjacent elements

Formal Argument for 𝑜 log 𝑜 Average

• What is the probability of comparing two

given elements?

15

1 2 3 4 5 6 7 8 9 10 11 12

• (Probability of comparing 1 and 12) = 2

12

– Why?

• I only compare 1 with 12 if either was chosen as the

first pivot

• Otherwise they would be divided into opposite sublists

Formal Argument for 𝑜 log 𝑜 Average

• Probability of comparing 𝑗 with 𝑘 (𝑘 > 𝑗):

– dependent on the number of elements between 𝑗 and 𝑘 –

1 𝑘−𝑗+1

• Expected number of comparisons:

–

1 𝑘−𝑗+1 𝑗<𝑘

16

Expected number of Comparisons

1 𝑘 − 𝑗 + 1

𝑗<𝑘

17

Consider when 𝑗 = 1 Sum so far:

2 2

1 2 3 4 5 6 7 8 9 10 11 12

Compared if 1 or 2 are chosen as pivot (these will always be compared)

Expected number of Comparisons

1 𝑘 − 𝑗 + 1

𝑗<𝑘

18

Consider when 𝑗 = 1 Sum so far:

2 2 + 2 3

1 2 3 4 5 6 7 8 9 10 11 12

Compared if 1 or 3 are chosen as pivot (but never if 2 is ever chosen)

Expected number of Comparisons

1 𝑘 − 𝑗 + 1

𝑗<𝑘

19

Consider when 𝑗 = 1 Sum so far:

2 2 + 2 3 + 2 4

1 2 3 4 5 6 7 8 9 10 11 12

Compared if 1 or 4 are chosen as pivot (but never if 2 or 3 are chosen)

Expected number of Comparisons

1 𝑘 − 𝑗 + 1

𝑗<𝑘

20

Consider when 𝑗 = 1 Overall sum:

2 2 + 2 3 + 2 4 + 2 5 + ⋯ + 2 𝑜

1 2 3 4 5 6 7 8 9 10 11 12

Compared if 1 or 12 are chosen as pivot (but never if 2 -> 11 are chosen)

Expected number of Comparisons

1 𝑘 − 𝑗 + 1

𝑗<𝑘

21

When 𝑗 = 1: 2

1 2 + 1 3 + 1 4 + ⋯ + 1 𝑜

𝑜 terms overall 1 𝑘 − 𝑗 + 1

𝑗<𝑘

≤ 2𝑜 1 2 + 1 3 + ⋯ + 1 𝑜

Θ(log 𝑜) Quicksort overall: expected Θ 𝑜 log 𝑜

Sorting, so far

• Sorting algorithms we have discussed:

– Mergesort – Quicksort

• Other sorting algorithms (will discuss):

– Bubblesort – Insertionsort – Heapsort

22

𝑃(𝑜 log 𝑜) 𝑃(𝑜 log 𝑜) 𝑃(𝑜 log 𝑜) 𝑃(𝑜2) 𝑃(𝑜2) Can we do better than 𝑃(𝑜 log 𝑜)?

Worst Case Lower Bounds

• Prove that there is no algorithm which can sort faster than

𝑃(𝑜 log 𝑜)

– Every algorithm, in the worst case, must have a certain lower bound

• Non-existence proof!

– Very hard to do

23

Strategy: Decision Tree

• Sorting algorithms use comparisons to figure out the order of input

elements

• Draw tree to illustrate all possible execution paths

24

>or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? [1,2,3,4,5] [2,1,3,4,5] [5,2,4,1,3] [5,4,3,2,1]

… … … … … … > < < > < > > > > < < < < > One comparison Result of comparison Permutation

• f sorted list

Possible execution path

Strategy: Decision Tree

• Worst case run time is the longest execution path
• i.e., “height” of the decision tree

25

>or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? [1,2,3,4,5] [2,1,3,4,5] [5,2,4,1,3] [5,4,3,2,1]

… … … … … … > < < > < > > > > < < < < > One comparison Result of comparison Permutation

• f sorted list

Possible execution path 𝑜! Possible permutations

log 𝑜! Θ(𝑜 log 𝑜)

Strategy: Decision Tree

• Conclusion: Worst Case Optimal run time of sorting is Θ(𝑜 log 𝑜)

– There is no (comparison-based) sorting algorithm with run time 𝑝(𝑜 log 𝑜)

26

>or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? >or<? [1,2,3,4,5] [2,1,3,4,5] [5,2,4,1,3] [5,4,3,2,1]

… … … … … … > < < > < > > > > < < < < > One comparison Result of comparison Permutation

• f sorted list

Possible execution path 𝑜! Possible permutations

log 𝑜! Θ(𝑜 log 𝑜)

Sorting, so far

• Sorting algorithms we have discussed:

– Mergesort – Quicksort

• Other sorting algorithms (will discuss):

– Bubblesort – Insertionsort – Heapsort

27

𝑃(𝑜 log 𝑜) 𝑃(𝑜 log 𝑜) 𝑃(𝑜 log 𝑜) 𝑃(𝑜2) 𝑃(𝑜2) Optimal! Optimal! Optimal!

Speed Isn’t Everything

• Important properties of sorting algorithms:
• Run Time

– Asymptotic Complexity – Constants

• In Place (or In-Situ)

– Done with only constant additional space

• Adaptive

– Faster if list is nearly sorted

• Stable

– Equal elements remain in original order

• Parallelizable

– Runs faster with multiple computers

28

Mergesort

• Divide:

– Break 𝑜-element list into two lists of 𝑜 2 elements

• Conquer:

– If 𝑜 > 1: Sort each sublist recursively – If 𝑜 = 1: List is already sorted (base case)

• Combine:

– Merge together sorted sublists into one sorted list

Run Time? Θ(𝑜 log 𝑜) Optimal! In Place? Adaptive? Stable? No No Yes! (usually)

Merge

• Combine: Merge sorted sublists into one sorted list
• We have:

– 2 sorted lists (𝑀1, 𝑀2) – 1 output list (𝑀𝑝𝑣𝑢)

While (𝑀1 and 𝑀2 not empty): If 𝑀1 0 ≤ 𝑀2[0]: 𝑀𝑝𝑣𝑢.append(𝑀1.pop()) Else: 𝑀𝑝𝑣𝑢.append(𝑀2.pop()) 𝑀𝑝𝑣𝑢.append(𝑀1) 𝑀𝑝𝑣𝑢.append(𝑀2)

Stable: If elements are equal, leftmost comes first

30

Mergesort

• Divide:

– Break 𝑜-element list into two lists of 𝑜 2 elements

• Conquer:

– If 𝑜 > 1: Sort each sublist recursively – If 𝑜 = 1: List is already sorted (base case)

• Combine:

– Merge together sorted sublists into one sorted list

Run Time? Θ(𝑜 log 𝑜) Optimal! In Place? Adaptive? Stable? Parallelizable? No No Yes! (usually) Yes!

31

Mergesort

• Divide:

– Break 𝑜-element list into two lists of 𝑜 2 elements

• Conquer:

– If 𝑜 > 1:

• Sort each sublist recursively

– If 𝑜 = 1:

• List is already sorted (base case)
• Combine:

– Merge together sorted sublists into one sorted list

32

Parallelizable: Allow different machines to work

• n each sublist

Mergesort (Sequential)

𝑜 total / level log2 𝑜 levels

• f recursion

𝑜

𝑈 𝑜 = 2𝑈(𝑜 2 ) + 𝑜

𝑜 2 𝑜 2 𝑜 4 𝑜 4 𝑜 4 𝑜 4

… … … …

1 1 1

…

1 1 1

𝑜 𝑜 2 𝑜 2 𝑜 4 𝑜 4 𝑜 4 𝑜 4 1 1 1 1 1 1

Run Time: Θ(𝑜 log 𝑜)

Mergesort (Parallel)

𝑜

𝑈 𝑜 = 𝑈(𝑜 2 ) + 𝑜

𝑜 2 𝑜 2 𝑜 4 𝑜 4 𝑜 4 𝑜 4

… … … …

1 1 1

…

1 1 1

𝑜 𝑜 2 𝑜 2 𝑜 4 𝑜 4 𝑜 4 𝑜 4 1 1 1 1 1 1

Run Time: Θ(𝑜) Done in Parallel

𝑜 2 𝑜 4 1

Quicksort

Run Time? Θ(𝑜 log 𝑜)

(almost always) Better constants than Mergesort

In Place? Adaptive? Stable? kinda No! No Parallelizable? Yes!

• Idea: pick a partition element, recursively sort

two sublists around that element

• Divide: select an element 𝑞, Partition(𝑞)
• Conquer: recursively sort left and right sublists
• Combine: Nothing!

Uses stack for recursive calls

Bubble Sort

• Idea: March through list, swapping adjacent

elements if out of order, repeat until sorted

36

8 5 7 9 12 10 1 2 4 3 6 11 5 8 7 9 12 10 1 2 4 3 6 11 5 7 8 9 12 10 1 2 4 3 6 11 5 7 8 9 12 10 1 2 4 3 6 11

Bubble Sort

Run Time? Θ(𝑜2)

Constants worse than Insertion Sort

In Place? Adaptive? Yes Kinda

• Idea: March through list, swapping adjacent

elements if out of order, repeat until sorted

“Compared to straight insertion […], bubble sorting requires a more complicated program and takes about twice as long!” –Donald Knuth

Bubble Sort is “almost” Adaptive

• Idea: March through list, swapping adjacent

elements if out of order

38

1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12

Only makes one “pass”

2 3 4 5 6 7 8 9 10 11 12 1

After one “pass”

2 3 4 5 6 7 8 9 10 11 1 12

Requires 𝑜 passes, thus is 𝑃(𝑜2)

Bubble Sort

Run Time? Θ(𝑜2)

Constants worse than Insertion Sort

In Place? Adaptive? Stable? Yes! Kinda Not really Yes Parallelizable? No

• Idea: March through list, swapping adjacent

elements if out of order, repeat until sorted

"the bubble sort seems to have nothing to recommend it, except a catchy name and the fact that it leads to some interesting theoretical problems” –Donald Knuth, The Art of Computer Programming

Insertion Sort

• Idea: Maintain a sorted list prefix, extend that

prefix by “inserting” the next element

40

3 5 7 8 10 12 9 2 4 6 1 11 Sorted Prefix 3 5 7 8 10 9 12 2 4 6 1 11 3 5 7 8 9 10 12 2 4 6 1 11 3 5 7 8 9 10 12 2 4 6 1 11 Sorted Prefix

Insertion Sort

Run Time? Θ(𝑜2)

(but with very small constants) Great for short lists!

In Place? Adaptive? Yes! Yes

• Idea: Maintain a sorted list prefix, extend that

prefix by “inserting” the next element

Insertion Sort is Adaptive

• Idea: Maintain a sorted list prefix, extend that

prefix by “inserting” the next element

42

1 2 3 4 5 6 7 8 9 10 11 12 Sorted Prefix 1 2 3 4 5 6 7 8 9 10 11 12 Sorted Prefix

Only one comparison needed per element! Runtime: 𝑃(𝑜)

Insertion Sort

Run Time? In Place? Adaptive? Stable? Yes! Yes Yes

• Idea: Maintain a sorted list prefix, extend that

prefix by “inserting” the next element

Θ(𝑜2)

(but with very small constants) Great for short lists!

Insertion Sort is Stable

• Idea: Maintain a sorted list prefix, extend that

prefix by “inserting” the next element

44

3 5 7 8 10 12 10’ 2 4 6 1 11 Sorted Prefix 3 5 7 8 10 10’ 12 2 4 6 1 11 3 5 7 8 10 10’ 12 2 4 6 1 11 Sorted Prefix

The “second” 10 will stay to the right

Insertion Sort

Run Time? In Place? Adaptive? Stable? Yes! Yes Yes Parallelizable? No

• Idea: Maintain a sorted list prefix, extend that

prefix by “inserting” the next element

Online? Yes

Can sort a list as it is received, i.e., don’t need the entire list to begin sorting

“All things considered, it’s actually a pretty good sorting algorithm!” –Nate Brunelle

Θ(𝑜2)

(but with very small constants) Great for short lists!

Heap Sort

• Idea: Build a Heap, repeatedly extract max element

from the heap to build sorted list Right-to-Left

46

10 9 6 8 7 5 2 4 1 3 10 9 6 8 7 5 2 4 1 3

Max Heap Property: Each node is larger than its children

1 2 3 4 5 6 7 8 9 10 1 2 3 4 6 5 7 8 9 10

Heap Sort

• Remove the Max element (i.e. the root) from the

Heap: replace with last element, call Heapify(root)

47

3 9 6 8 7 5 2 4 1 3 9 6 8 7 5 2 4 1

Max Heap Property: Each node is larger than its children

1 2 3 4 5 6 7 8 9 10 1 2 3 4 6 5 7 8 9

Heapify(node): if node satisfies heap property, done. Else swap with largest child and recurse on that subtree

Heap Sort

• Remove the Max element (i.e. the root) from the

Heap: replace with last element, call Heapify(root)

48

9 3 6 8 7 5 2 4 1 9 3 6 8 7 5 2 4 1

Max Heap Property: Each node is larger than its children

1 2 3 4 5 6 7 8 9 10 1 2 3 4 6 5 7 8 9

Heapify(node): if node satisfies heap property, done. Else swap with largest child and recurse on that subtree

Heap Sort

• Remove the Max element (i.e. the root) from the

Heap: replace with last element, call Heapify(root)

49

9 8 6 3 7 5 2 4 1 9 8 6 3 7 5 2 4 1

Max Heap Property: Each node is larger than its children

1 2 3 4 5 6 7 8 9 10 1 2 3 4 6 5 7 8 9

Heapify(node): if node satisfies heap property, done. Else swap with largest child and recurse on that subtree

Heap Sort

• Remove the Max element (i.e. the root) from the

Heap: replace with last element, call Heapify(root)

50

9 8 6 4 7 5 2 3 1 9 8 6 4 7 5 2 3 1

Max Heap Property: Each node is larger than its children

1 2 3 4 5 6 7 8 9 10 1 2 3 4 6 5 7 8 9

Heapify(node): if node satisfies heap property, done. Else swap with largest child and recurse on that subtree

Heap Sort

Run Time? Θ(𝑜 log 𝑜)

Constants worse than Quick Sort

In Place? Yes!

• Idea: Build a Heap, repeatedly extract max

element from the heap to build sorted list Right- to-Left

When removing an element from the heap, move it to the (now unoccupied) end of the list

In Place Heap Sort

• Idea: When removing an element from the heap,

move it to the (now unoccupied) end of the list

52

10 9 6 8 7 5 2 4 1 3 10 9 6 8 7 5 2 4 1 3

Max Heap Property: Each node is larger than its children

1 2 3 4 5 6 7 8 9 10 1 2 3 4 6 5 7 8 9 10

In Place Heap Sort

• Idea: When removing an element from the heap,

move it to the (now unoccupied) end of the list

53

3 9 6 8 7 5 2 4 1 3 9 6 8 7 5 2 4 1 10

Max Heap Property: Each node is larger than its children

1 2 3 4 5 6 7 8 9 10 1 2 3 4 6 5 7 8 9

In Place Heap Sort

• Idea: When removing an element from the heap,

move it to the (now unoccupied) end of the list

54

9 8 6 4 7 5 2 3 1 9 8 6 4 7 5 2 3 1 10

Max Heap Property: Each node is larger than its children

1 2 3 4 5 6 7 8 9 10 1 2 3 4 6 5 7 8 9

In Place Heap Sort

• Idea: When removing an element from the heap,

move it to the (now unoccupied) end of the list

55

8 7 6 4 1 5 2 3 8 7 6 4 1 5 2 3 9 10

Max Heap Property: Each node is larger than its children

1 2 3 4 5 6 7 8 9 10 1 2 3 4 6 5 7 8

In Place Heap Sort

• Idea: When removing an element from the heap,

move it to the (now unoccupied) end of the list

56

7 4 6 3 1 5 2 7 4 6 3 1 5 2 8 9 10

Max Heap Property: Each node is larger than its children

1 2 3 4 5 6 7 8 9 10 1 2 3 4 6 5 7

Heap Sort

Run Time? Θ(𝑜 log 𝑜)

Constants worse than Quick Sort

In Place? Adaptive? Stable? Yes! No No Parallelizable? No

• Idea: Build a Heap, repeatedly extract max

element from the heap to build sorted list Right- to-Left

Sorting in Linear Time

• Cannot be comparison-based
• Need to make some sort of assumption about the contents of

the list

– Small number of unique values – Small range of values – Etc.

58

Counting Sort

59

• Idea: Count how many things are less than each element

Range is [1, 𝑙] (here [1,6]) make an array 𝐷 of size 𝑙 populate with counts of each value

3 6 6 1 3 4 1 6 1 2 3 4 5 6 7 8 2 2 1 3 1 2 3 4 5 6

𝐷 = For 𝑗 in 𝑀: + +C 𝑀 𝑗 1. 𝑀 = Take “running sum” of 𝐷 to count things less than each value

2 2 4 5 5 8 1 2 3 4 5 6

𝐷 = For 𝑗 = 1 to len(𝐷): 𝐷 𝑗 = 𝐷 𝑗 − 1 + 𝐷[𝑗] 2.

running sum To sort: last item of value 3 goes at index 4

Counting Sort

60

• Idea: Count how many things are less than each element

3 6 6 1 3 4 1 6 1 2 3 4 5 6 7 8

𝑀 = For each element of 𝑀 (last to first): Use 𝐷 to find its proper place in 𝐶 Decrement that position of C

2 2 4 5 5 8 1 2 3 4 5 6

𝐷 =

Last item of value 6 goes at index 8

1 2 3 4 5 6 7 8

𝐶 = For 𝑗 = len(𝑀) downto 1: 𝐶 𝐷 𝑀 𝑗 = 𝑀 𝑗 𝐷 𝑀 𝑗 = 𝐷 𝑀 𝑗 − 1

7 6

Counting Sort

61

• Idea: Count how many things are less than each element

3 6 6 1 3 4 1 6 1 2 3 4 5 6 7 8

𝑀 = For each element of 𝑀 (last to first): Use 𝐷 to find its proper place in 𝐶 Decrement that position of C

2 2 4 5 5 7 1 2 3 4 5 6

𝐷 =

Last item of value 1 goes at index 2

6 1 2 3 4 5 6 7 8

𝐶 = For 𝑗 = len(𝑀) downto 1: 𝐶 𝐷 𝑀 𝑗 = 𝑀 𝑗 𝐷 𝑀 𝑗 = 𝐷 𝑀 𝑗 − 1

1 1

Run Time: 𝑃 𝑜 + 𝑙 Memory: 𝑃 𝑜 + 𝑙

Counting Sort

• Why not always use counting sort?
• For 64-bit numbers, requires an array of length 264 > 1019

– 5 GHz CPU will require > 116 years to initialize the array – 18 Exabytes of data

• Total amount of data that Google has (?)

62

One Exabyte = 1018 bytes 1 million terabytes (TB) 1 billion gigabytes (GB) 100,000 x Library of Congress (print)

12 Exabytes

63 https://en.wikipedia.org/wiki/Utah_Data_Center

Radix Sort

• Idea: Stable sort on each digit, from least

significant to most significant

64

103 801 401 323 255 823 999 101 1 2 3 4 5 6 7

Place each element into a “bucket” according to its 1’s place

999 018 255 555 245 103 323 823 113 512 113 901 555 512 245 800 018 121 8 9 10 11 12 13 14 15 1 2 3 4 5 6 7 801 401 101 901 121 800 8 9

Radix Sort

• Idea: Stable sort on each digit, from least

significant to most significant

65

Place each element into a “bucket” according to its 10’s place

999 018 255 555 245 103 323 823 113 512 1 2 3 4 5 6 7 801 401 101 901 121 800 8 9 999 255 555 245 121 323 823 1 2 3 4 5 6 7 512 113 018 800 801 401 101 901 103 8 9

Radix Sort

• Idea: Stable sort on each digit, from least

significant to most significant

66

Place each element into a “bucket” according to its 100’s place

999 255 555 245 121 323 823 1 2 3 4 5 6 7 512 113 018 800 801 401 101 901 103 8 9 901 999 800 801 823 512 555 401 323 245 255 1 2 3 4 5 6 7 101 103 113 121 018 8 9

Run Time: 𝑃 𝑒 𝑜 + 𝑐 𝑒 = digits in largest value 𝑐 = base of representation