[PPT] - 7. Sorting I Selection Sort, Insertion Sort, Bubblesort PowerPoint Presentation

SLIDE 1

7. Sorting I

Simple Sorting

196

7.1 Simple Sorting

Selection Sort, Insertion Sort, Bubblesort [Ottman/Widmayer, Kap. 2.1, Cormen et al, Kap. 2.1, 2.2, Exercise 2.2-2, Problem 2-2

197

Problem

Input: An array A = (A[1], ..., A[n]) with length n. Output: a permutation A′ of A, that is sorted: A′[i] ≤ A′[j] for all

1 ≤ i ≤ j ≤ n.

198

Algorithm: IsSorted(A)

Input : Array A = (A[1], ..., A[n]) with length n. Output : Boolean decision “sorted” or “not sorted” for i ← 1 to n − 1 do if A[i] > A[i + 1] then return “not sorted”; return “sorted”;

199

SLIDE 2

Observation

IsSorted(A):“not sorted”, if A[i] > A[i + 1] for an i.

⇒ idea:

for j ← 1 to n − 1 do if A[j] > A[j + 1] then swap(A[j], A[j + 1]);

200

Give it a try

5 6 2 8 4 1

(j = 1)

5 6 2 8 4 1

(j = 2)

5 2 6 8 4 1

(j = 3)

5 2 6 8 4 1

(j = 4)

5 2 6 4 8 1

(j = 5)

5 2 6 4 1 8 Not sorted! . But the greatest element moves to the right

⇒ new idea!

201

Try it out

5 6 2 8 4 1 (j = 1, i = 1) 5 6 2 8 4 1 (j = 2) 5 2 6 8 4 1 (j = 3) 5 2 6 8 4 1 (j = 4) 5 2 6 4 8 1 (j = 5) 5 2 6 4 1 8 (j = 1, i = 2) 2 5 6 4 1 8 (j = 2) 2 5 6 4 1 8 (j = 3) 2 5 4 6 1 8 (j = 4) 2 5 4 1 6 8 (j = 1, i = 3) 2 5 4 1 6 8 (j = 2) 2 4 5 1 6 8 (j = 3) 2 4 1 5 6 8 (j = 1, i = 4) 2 4 1 5 6 8 (j = 2) 2 1 4 5 6 8 (i = 1, j = 5) 1 2 4 5 6 8

Apply the procedure iteratively. For A[1, . . . , n], then A[1, . . . , n − 1], then A[1, . . . , n − 2], etc.

202

Algorithm: Bubblesort

Input : Array A = (A[1], . . . , A[n]), n ≥ 0. Output : Sorted Array A for i ← 1 to n − 1 do for j ← 1 to n − i do if A[j] > A[j + 1] then swap(A[j], A[j + 1]);

203

SLIDE 3

Analysis

Number key comparisons n−1

i=1 (n − i) = n(n−1) 2

= Θ(n2).

Number swaps in the worst case: Θ(n2) ? What is the worst case? ! If A is sorted in decreasing order. ? Algorithm can be adapted such that it terminates when the array is sorted.

Key comparisons and swaps of the modified algorithm in the best case?

! Key comparisons = n − 1. Swaps = 0.

204

Selection Sort

5 6 2 8 4 1

(i = 1)

1 6 2 8 4 5

(i = 2)

1 2 6 8 4 5

(i = 3)

1 2 4 8 6 5

(i = 4)

1 2 4 5 6 8

(i = 5)

1 2 4 5 6 8

(i = 6)

1 2 4 5 6 8 Iterative procedure as for Bubblesort. Selection of the smallest (or largest) element by immediate search.

205

Algorithm: Selection Sort

Input : Array A = (A[1], . . . , A[n]), n ≥ 0. Output : Sorted Array A for i ← 1 to n − 1 do p ← i for j ← i + 1 to n do if A[j] < A[p] then p ← j; swap(A[i], A[p])

206

Analysis

Number comparisons in worst case: Θ(n2). Number swaps in the worst case: n − 1 = Θ(n) Best case number comparisons: Θ(n2).

207

SLIDE 4

Insertion Sort

5 6 2 8 4 1

(i = 1)

5 6 2 8 4 1

(i = 2)

5 6 2 8 4 1

(i = 3)

2 5 6 8 4 1

(i = 4)

2 5 6 8 4 1

(i = 5)

2 4 5 6 8 1

(i = 6)

1 2 4 5 6 8 Iterative procedure:

i = 1...n

Determine insertion position for element i. Insert element i array block movement potentially required

208

Insertion Sort

? What is the disadvantage of this algorithm compared to sorting by selection? ! Many element movements in the worst case. ? What is the advantage of this algorithm compared to selection sort? ! The search domain (insertion interval) is already sorted. Consequently: binary search possible.

209

Algorithm: Insertion Sort

Input : Array A = (A[1], . . . , A[n]), n ≥ 0. Output : Sorted Array A for i ← 2 to n do x ← A[i] p ← BinarySearch(A[1...i − 1], x); // Smallest p ∈ [1, i] with A[p] ≥ x for j ← i − 1 downto p do A[j + 1] ← A[j] A[p] ← x

210

Analysis

Number comparisons in the worst case:

n−1

k=1 a · log k = a log((n − 1)!) ∈ O(n log n).

Number comparisons in the best case Θ(n log n).4 Number swaps in the worst case n

k=2(k − 1) ∈ Θ(n2)

4With slight modification of the function BinarySearch for the minimum / maximum: Θ(n) 211

SLIDE 5

Different point of view

Sorting node:

≷

8 4

212

Different point of view

5 6

≷

2 ≷ ≷

8 ≷ ≷ ≷

4 ≷ ≷ ≷ ≷

1 ≷ ≷ ≷ ≷ ≷

1 2 4 5 6 8

5 6 2 5 6 8 2 5 4 2 4 8 1 2 1 2 5 5 4 2 6 8 5 4 6 5 4 6 5 8 6 5 8 6 6 8

Like selection sort [and like Bubblesort]

213

Different point of view

5 6

≷

2 ≷ ≷

8 ≷ ≷ ≷

4 ≷ ≷ ≷ ≷

1 ≷ ≷ ≷ ≷ ≷

1 2 4 5 6 8

5 6 5 6 2 5 2 5 6 8 8 8 2 5 6 8 4 4 5 6 2 4 5 6 8 1 2 4 5 6 1 2 4 5 6 8

Like insertion sort

214

Conclusion

In a certain sense, Selection Sort, Bubble Sort and Insertion Sort provide the same kind of sort strategy. Will be made more precise. 5

5In the part about parallel sorting networks. For the sequential code of course the observations as described above still

hold.

215

SLIDE 6

Shellsort

Insertion sort on subsequences of the form (Ak·i) (i ∈ ◆) with decreasing distances k. Last considered distance must be k = 1. Good sequences: for example sequences with distances

k ∈ {2i3j|0 ≤ i, j}.

216

Shellsort

9 8 7 6 5 4 3 2 1 1 8 7 6 5 4 3 2 9 insertion sort, k = 4 1 7 6 5 4 3 2 9 8 1 3 6 5 4 7 2 9 8 1 3 2 5 4 7 6 9 8 1 3 2 5 4 7 6 9 8 insertion sort, k = 2 1 3 2 5 4 7 6 9 8 1 2 3 4 5 6 7 8 9 insertion sort, k = 1

217

8. Sorting II

Heapsort, Quicksort, Mergesort

218

8.1 Heapsort

[Ottman/Widmayer, Kap. 2.3, Cormen et al, Kap. 6]

219

SLIDE 7

Heapsort

Inspiration from selectsort: fast insertion Inspiration from insertion sort: fast determination of position ? Can we have the best of both worlds? ! Yes, but it requires some more thinking...

220

[Max-]Heap6

Binary tree with the following prop- erties

1 complete up to the lowest

level

2 Gaps (if any) of the tree in

the last level to the right

3 Heap-Condition:

Max-(Min-)Heap: key of a child smaller (greater) thant that of the parent node

Wurzel

22 20 16 3 2 12 8 11 18 15 14 17

parent child leaves

6Heap(data structure), not: as in “heap and stack” (memory allocation) 221

Heap and Array

Tree → Array: children(i) = {2i, 2i + 1} parent(i) = ⌊i/2⌋

22 1 20 2 18 3 16 4 12 5 15 6 17 7 3 8 2 9 8 10 11 11 14 12

Vater Kinder

22 20 16 3 2 12 8 11 18 15 14 17 [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]

Depends on the starting index7

7For array that start at 0: {2i, 2i + 1} → {2i + 1, 2i + 2}, ⌊i/2⌋ → ⌊(i − 1)/2⌋ 222

Recursive heap structure

A heap consists of two heaps:

22 20 16 3 2 12 8 11 18 15 14 17

223

SLIDE 8

Insert

Insert new element at the first free

position. Potentially violates the heap

property. Reestablish heap property: climb successively Worst case number of operations:

O(log n)

22 20 16 3 2 12 8 11 18 15 14 17 22 20 16 3 2 12 8 11 21 18 14 15 17

224

Remove the maximum

Replace the maximum by the lower right element Reestablish heap property: sink successively (in the direction of the greater child) Worst case number of operations:

O(log n)

21 20 16 3 2 12 8 11 18 15 14 17 20 16 14 3 2 12 8 11 18 15 17

225

Algorithm Sink(A, i, m)

Input : Array A with heap structure for the children of i. Last element m. Output : Array A with heap structure for i with last element m. while 2i ≤ m do j ← 2i; // j left child if j < m and A[j] < A[j + 1] then j ← j + 1; // j right child with greater key if A[i] < A[j] then swap(A[i], A[j]) i ← j; // keep sinking else i ← m; // sinking finished

226

Sort heap

A[1, ..., n] is a Heap.

227

SLIDE 9

Heap creation

Observation: Every leaf of a heap is trivially a correct heap. Consequence: Induction from below!

228

Algorithm HeapSort(A, n)

Input : Array A with length n. Output : A sorted. // Build the heap. for i ← n/2 downto 1 do Sink(A, i, n); // Now A is a heap. for i ← n downto 2 do swap(A[1], A[i]) Sink(A, 1, i − 1) // Now A is sorted.

229

Analysis: sorting a heap

Sink traverses at most log n nodes. For each node 2 key

comparisons. ⇒ sorting a heap costs in the worst case 2 log n

comparisons. Number of memory movements of sorting a heap also O(n log n).

230

Analysis: creating a heap

Calls to sink: n/2. Thus number of comparisons and movements:

v(n) ∈ O(n log n).

But mean length of sinking paths is much smaller:

v(n) =

⌊log n⌋

h=0

n 2h+1

· c · h ∈ O(n

⌊log n⌋

h=0

h 2h)

with s(x) := ∞

k=0 kxk = x (1−x)2

(0 < x < 1) 8 and s(1

2) = 2:

v(n) ∈ O(n).

8f(x) = 1 1−x = 1 + x + x2... ⇒ f′(x) = 1 (1−x)2 = 1 + 2x + ... 231

SLIDE 10

8.2 Mergesort

[Ottman/Widmayer, Kap. 2.4, Cormen et al, Kap. 2.3],

232

Intermediate result

Heapsort: O(n log n) Comparisons and movements. ? Disadvantages of heapsort? ! Missing locality: heapsort jumps around in the sorted array (negative cache effect). ! Two comparisons required before each necessary memory movement.

233

Mergesort

Divide and Conquer! Assumption: two halves of the array A are already sorted. Minimum of A can be evaluated with two comparisons. Iteratively: sort the pre-sorted array A in O(n).

234

Merge

1 4 7 9 16 2 3 10 11 12 1 2 3 4 7 9 10 11 12 16

235

SLIDE 11

Algorithm Merge(A, l, m, r)

Input : Array A with length n, indexes 1 ≤ l ≤ m ≤ r ≤ n. A[l, . . . , m], A[m + 1, . . . , r] sorted Output : A[l, . . . , r] sorted

1 B ← new Array(r − l + 1) 2 i ← l; j ← m + 1; k ← 1 3 while i ≤ m and j ≤ r do 4

if A[i] ≤ A[j] then B[k] ← A[i]; i ← i + 1

5

else B[k] ← A[j]; j ← j + 1

6

k ← k + 1;

7 while i ≤ m do B[k] ← A[i]; i ← i + 1; k ← k + 1 8 while j ≤ r do B[k] ← A[j]; j ← j + 1; k ← k + 1 9 for k ← l to r do A[k] ← B[k − l + 1]

236

Correctness

Hypothesis: after k iterations of the loop in line 3 B[1, . . . , k] is sorted and B[k] ≤ A[i], if i ≤ m and B[k] ≤ A[j] if j ≤ r. Proof by induction:

Base case: the empty array B[1, . . . , 0] is trivially sorted. Induction step (k → k + 1): wlog A[i] ≤ A[j], i ≤ m, j ≤ r.

B[1, . . . , k] is sorted by hypothesis and B[k] ≤ A[i].

After B[k + 1] ← A[i] B[1, . . . , k + 1] is sorted.

B[k + 1] = A[i] ≤ A[i + 1] (if i + 1 ≤ m) and B[k + 1] ≤ A[j] if j ≤ r. k ← k + 1, i ← i + 1: Statement holds again.

237

Analysis (Merge)

Lemma If: array A with length n, indexes 1 ≤ l < r ≤ n. m = ⌊(l + r)/2⌋ and A[l, . . . , m], A[m + 1, . . . , r] sorted. Then: in the call of Merge(A, l, m, r) a number of Θ(r − l) key movements and comparisons are executed. Proof: straightforward(Inspect the algorithm and count the

perations.)

SLIDE 12

Algorithm recursive 2-way Mergesort(A, l, r)

Input : Array A with length n. 1 ≤ l ≤ r ≤ n Output : Array A[l, . . . , r] sorted. if l < r then m ← ⌊(l + r)/2⌋ // middle position Mergesort(A, l, m) // sort lower half Mergesort(A, m + 1, r) // sort higher half Merge(A, l, m, r) // Merge subsequences

240

Analysis

Recursion equation for the number of comparisons and key movements:

C(n) = C( n 2

) + C(

n 2

) + Θ(n) ∈ Θ(n log n)

241

Algorithm StraightMergesort(A)

Avoid recursion: merge sequences of length 1, 2, 4, ... directly

Input : Array A with length n Output : Array A sorted length ← 1 while length < n do // Iterate over lengths n r ← 0 while r + length < n do // Iterate over subsequences l ← r + 1 m ← l + length − 1 r ← min(m + length, n) Merge(A, l, m, r) length ← length · 2

242

Analysis

Like the recursive variant, the straight 2-way mergesort always executes a number of Θ(n log n) key comparisons and key movements.

243

SLIDE 13

Natural 2-way mergesort

Observation: the variants above do not make use of any presorting and always execute Θ(n log n) memory movements. ? How can partially presorted arrays be sorted better? ! Recursive merging of previously sorted parts (runs) of A.

244

Natural 2-way mergesort

5 6 2 4 8 3 9 7 1 2 4 5 6 8 3 7 9 1 2 3 4 5 6 7 8 9 1 1 2 3 4 5 6 7 8 9

245

Algorithm NaturalMergesort(A)

Input : Array A with length n > 0 Output : Array A sorted repeat r ← 0 while r < n do l ← r + 1 m ← l; while m < n and A[m + 1] ≥ A[m] do m ← m + 1 if m < n then r ← m + 1; while r < n and A[r + 1] ≥ A[r] do r ← r + 1 Merge(A, l, m, r); elser ← n until l = 1

246

Analysis

In the best case, natural merge sort requires only n − 1 comparisons. ? Is it also asymptotically better than StraightMergesort on average?

! No. Given the assumption of pairwise distinct keys, on average there are n/2 positions i with ki > ki+1, i.e. n/2 runs. Only one iteration is saved on average.

Natural mergesort executes in the worst case and on average a number of Θ(n log n) comparisons and memory movements.

247

SLIDE 14

8.3 Quicksort

[Ottman/Widmayer, Kap. 2.2, Cormen et al, Kap. 7]

248

Quicksort

? What is the disadvantage of Mergesort? ! Requires Θ(n) storage for merging. ? How could we reduce the merge costs? ! Make sure that the left part contains only smaller elements than the right part. ? How? ! Pivot and Partition!

249

Quicksort (arbitrary pivot)

2 4 5 6 8 3 7 9 1 2 1 3 6 8 5 7 9 4 1 2 3 4 5 8 7 9 6 1 2 3 4 5 6 7 9 8 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9

250

Algorithm Quicksort(A[l, . . . , r]

Input : Array A with length n. 1 ≤ l ≤ r ≤ n. Output : Array A, sorted between l and r. if l < r then Choose pivot p ∈ A[l, . . . , r] k ← Partition(A[l, . . . , r], p) Quicksort(A[l, . . . , k − 1]) Quicksort(A[k + 1, . . . , r])

251

SLIDE 15

Reminder: algorithm Partition(A[l, . . . , r], p)

Input : Array A, that contains the pivot p in [l, r] at least once. Output : Array A partitioned around p. Returns the position of p. while l ≤ r do while A[l] < p do l ← l + 1 while A[r] > p do r ← r − 1 swap(A[l], A[r]) if A[l] = A[r] then // Only for keys that are not pairwise different l ← l + 1 return l-1

252

Analysis: number comparisons

Best case. Pivot = median; number comparisons:

T(n) = 2T(n/2) + c · n, T(1) = 0 ⇒ T(n) ∈ O(n log n)

Worst case. Pivot = min or max; number comparisons:

T(n) = T(n − 1) + c · n, T(1) = 0 ⇒ T(n) ∈ Θ(n2)

253

Analysis: number swaps

Result of a call to partition (pivot 3):

2 1 3 6 8 5 7 9 4

? How many swaps have taken place? ! 2. The maximum number of swaps is given by the number of keys in the smaller part.

254

Analysis: number swaps

Intellectual game Each key from the smaller part pay a coin when swapped. When a key has paid a coin then the domain containing the key is less than or equal to half the previous size. Every key needs to pay at most log n coins. But there are only n keys. Consequence: there are O(n log n) key swaps in the worst case.

255

SLIDE 16

Randomized Quicksort

Despite the worst case running time of Θ(n2), quicksort is used practically very often. Reason: quadratic running time unlikely provided that the choice of the pivot and the pre-sorting are not very disadvantageous. Avoidance: randomly choose pivot. Draw uniformly from [l, r].

256

Analysis (randomized quicksort)

Expected number of compared keys with input length n:

T(n) = (n − 1) + 1 n

n

k=1

(T(k − 1) + T(n − k)) , T(0) = T(1) = 0

Claim T(n) ≤ 4n log n. Proof by induction: Base case straightforward for n = 0 (with 0 log 0 := 0) and for n = 1. Hypothesis: T(n) ≤ 4n log n for some n. Induction step: (n − 1 → n)

257

Analysis (randomized quicksort)

T(n) = n − 1 + 2 n

n−1

k=0

T(k)

H

≤ n − 1 + 2 n

n−1

k=0

4k log k = n − 1 +

n/2

k=1

4k log k

≤log n−1

+

n−1

k=n/2+1

4k log k

≤log n

≤ n − 1 + 8 n  (log n − 1)

n/2

k=1

k + log n

n−1

k=n/2+1

k   = n − 1 + 8 n

(log n) · n(n − 1)

2 − n 4 n 2 + 1

= 4n log n − 4 log n − 3 ≤ 4n log n

258

Analysis (randomized quicksort)

Theorem On average randomized quicksort requires O(n · log n) comparisons.

259

SLIDE 17

Practical considerations

Worst case recursion depth n − 19. Then also a memory consumption of O(n). Can be avoided: recursion only on the smaller part. Then guaranteed O(log n) worst case recursion depth and memory consumption.

9stack overflow possible! 260

Quicksort with logarithmic memory consumption

Input : Array A with length n. 1 ≤ l ≤ r ≤ n. Output : Array A, sorted between l and r. while l < r do Choose pivot p ∈ A[l, . . . , r] k ← Partition(A[l, . . . , r], p) if k − l < r − k then Quicksort(A[l, . . . , k − 1]) l ← k + 1 else Quicksort(A[k + 1, . . . , r]) r ← k − 1

The call of Quicksort(A[l, . . . , r]) in the original algorithm has moved to iteration (tail recursion!): the if-statement became a while-statement.

261

Practical considerations.

Practically the pivot is often the median of three elements. For example: Median3(A[l], A[r], A[⌊l + r/2⌋]). There is a variant of quicksort that requires only constant storage. Idea: store the old pivot at the position of the new pivot.

262