Average-Case and Distributional Analysis of Java 7s Dual Pivot - - PowerPoint PPT Presentation

Average-Case and Distributional Analysis of Java 7’s Dual Pivot Quicksort

Markus E. Nebel

based on joint work with Ralph Neininger and Sebastian Wild

AofA 2013 Menorca, Spain

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 1 / 22

Sorting Algorithms in Practice

Many inventions by algorithms comunity vs. Few methods successful in practice C C+ + Java 6                Quicksort

+Mergesort variant as stable sort

.NET Haskell Python Timsort

Sorting methods listed on Wikipedia Sorting methods of standard libraries for random access data Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 2 / 22

Sorting Algorithms in Practice

Many inventions by algorithms comunity vs. Few methods successful in practice C C+ + Java 6                Quicksort

+Mergesort variant as stable sort

.NET Haskell Python Timsort

Sorting methods listed on Wikipedia Sorting methods of standard libraries for random access data Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 2 / 22

History of Quicksort in Practice

1961,62 Hoare: first publication, average case analysis 1969 Singleton: median-of-three & Insertionsort on small subarrays 1975-78 Sedgewick: detailled analysis of many optimizations 1993 Bentley, McIlroy: Engineering a Sort Function 1997 Musser: O(n log n) worst case by bounded recursion depth Basic algorithm settled since 1961; latest tweaks from 1990’s. Since then: Almost identical in all programming libraries!

1961 1969 1975 ’78 1993 1997 ’62 ’77 today Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 3 / 22

History of Quicksort in Practice

1961,62 Hoare: first publication, average case analysis 1969 Singleton: median-of-three & Insertionsort on small subarrays 1975-78 Sedgewick: detailled analysis of many optimizations 1993 Bentley, McIlroy: Engineering a Sort Function 1997 Musser: O(n log n) worst case by bounded recursion depth Basic algorithm settled since 1961; latest tweaks from 1990’s. Since then: Almost identical in all programming libraries!

1961 1969 1975 ’78 1993 1997 ’62 ’77 today Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 3 / 22

History of Quicksort in Practice

1961,62 Hoare: first publication, average case analysis 1969 Singleton: median-of-three & Insertionsort on small subarrays 1975-78 Sedgewick: detailled analysis of many optimizations 1993 Bentley, McIlroy: Engineering a Sort Function 1997 Musser: O(n log n) worst case by bounded recursion depth Basic algorithm settled since 1961; latest tweaks from 1990’s. Since then: Almost identical in all programming libraries! Until 2009: Java 7 switches to a new dual pivot Quicksort!

• Sept. 2009 Vladimir Yaroslavskiy announced algorithm on Java core library mailing

list July 2011 public release of Java 7 with Yaroslavskiy’s Quicksort.

1961 1969 1975 ’78 1993 1997 ’62 ’77 today 2009 Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 3 / 22

Running Time Experiments

Why switch to new, unknown algorithm?

0.5 1 1.5 2 ·106 7 8 9 n time 10−6 · n ln n Java 6 Library Normalized Java runtimes (in ms). Average and standard deviation of 1000 random permutations per size. Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 4 / 22

Running Time Experiments

Why switch to new, unknown algorithm? Because it is faster!

0.5 1 1.5 2 ·106 7 8 9 n time 10−6 · n ln n Java 6 Library Java 7 Library Normalized Java runtimes (in ms). Average and standard deviation of 1000 random permutations per size. Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 4 / 22

Running Time Experiments

Why switch to new, unknown algorithm? Because it is faster!

0.5 1 1.5 2 ·106 7 8 9 n time 10−6 · n ln n Java 6 Library Java 7 Library Classic Quicksort Yaroslavskiy Normalized Java runtimes (in ms). Average and standard deviation of 1000 random permutations per size.

remains true for basic variants of algorithms: vs. !

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 4 / 22

Dual Pivot Quicksort

High Level Algorithm:

1

Partition array arround two pivots p q.

2

Sort 3 subarrays recursively.

How to do partitioning?

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 5 / 22

Dual Pivot Quicksort

High Level Algorithm:

1

Partition array arround two pivots p q.

2

Sort 3 subarrays recursively.

How to do partitioning?

1

For each element x, determine its class

small for x < p medium for p < x < q large for q < x

by comparing x to p and/or q

2

Arrange elements according to classes

p q Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 5 / 22

Dual Pivot Quicksort – Previous Work

Robert Sedgewick, 1975

in-place dual pivot Quicksort implementation more comparisons and swaps than classic Quicksort

Pascal Hennequin, 1991

comparisons for list-based Quicksort with r pivots r = 2 same #comparisons as classic Quicksort in one partitioning step: 5

3 comparisons per element

r > 2 very small savings, but complicated partitioning

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 6 / 22

Dual Pivot Quicksort – Previous Work

Robert Sedgewick, 1975

in-place dual pivot Quicksort implementation more comparisons and swaps than classic Quicksort

Pascal Hennequin, 1991

comparisons for list-based Quicksort with r pivots r = 2 same #comparisons as classic Quicksort in one partitioning step: 5

3 comparisons per element

r > 2 very small savings, but complicated partitioning

Using two pivots does not pay, and ... ... no theoretical explanation for impressive speedup.

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 6 / 22

Dual Pivot Quicksort – Comparisons

How many comparisons to determine classes ( small , medium or large ) ? Assume, we first compare with p. small elements need 1, others 2 comparisons

• n average: 1

3 of all elements are small

1

3 · 1 + 2 3 · 2 = 5 3 comparisons per element

if inputs are uniform random permutations, classes of x and y are independent Any partitioning method needs at least

5 3(n − 2) ∼ 20 12n comparisons on average?

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 7 / 22

Dual Pivot Quicksort – Comparisons

How many comparisons to determine classes ( small , medium or large ) ? Assume, we first compare with p. small elements need 1, others 2 comparisons

• n average: 1

3 of all elements are small

1

3 · 1 + 2 3 · 2 = 5 3 comparisons per element

if inputs are uniform random permutations, classes of x and y are independent Any partitioning method needs at least

5 3(n − 2) ∼ 20 12n comparisons on average?

No! (Stay tuned . . . )

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 7 / 22

Beating the “Lower Bound”

∼ 20

12n comparisons only needed,

if there is one comparison location (giving rise to fixed order like first compare with p), then checks for x and y independent But: Can have several comparison locations! Here: Assume two locations C1 and C2 s. t.

C1 first compares with p. C2 first compares with q. C1 executed often, iff p is large. C2 executed often, iff q is small.

• C1 executed often

iff many small elements iff good chance that C1 needs only one comparison

(C2 similar)

less comparisons than 5

3 per elements on average

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 8 / 22

Yaroslavskiy’s Quicksort

5

while k g

6

Ck if A[k] < p

7

Swap A[k] and A[ℓ] ; ℓ := ℓ + 1

8

else C′

k

if A[k] q

9

Cg while A[g] > q and k < g do g := g − 1 end while

10

Swap A[k] and A[g] ; g := g − 1

11

C′

g

if A[k] < p

12

Swap A[k] and A[ℓ] ; ℓ := ℓ + 1

13

end if

14

end if

15

k := k + 1

16

end while

2 comparison locations Ck handles pointer k Cg handles pointer g Ck first checks < p C′

k if needed q

Cg first checks > q C′

g if needed < p

Invariant: < p ℓ → > q g ← p ◦ q k → ?

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 9 / 22

Analysis of Yaroslavskiy’s Algorithm

In this talk:

leading term asymptotics of comparisons (we have results for swaps and Java bytecodes too) distribution and correlation of costs effect of pivot sampling

Cn expected #comparisons to sort random permutation of {1, . . . , n} Cn satisfies recurrence relation Cn = cn +

2 n(n−1)

• 1p<qn
• Cp−1 + Cq−p−1 + Cn−q
• ,

with cn expected #comparisons in first partitioning step recurrence solvable by standard methods

• linear cn ∼ a · n yields Cn ∼ 6

5a · n ln n.

need to compute cn

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 10 / 22

Analysis of Yaroslavskiy’s Algorithm

first comparison for all elements (at Ck or Cg ) ∼ n comparisons second comparison for some elements at C′

k resp. C′ g

. . . but how often are C′

k resp. C′ g reached?

C′

k : all non- small elements reached by pointer k.

C′

g : all non- large elements reached by pointer g.

second comparison for medium elements not avoidable ∼ 1

3n comparisons in expectation

it remains to count: large elements reached by k and small elements reached by g.

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 11 / 22

Analysis of Yaroslavskiy’s Algorithm

Second comparisons for small and large elements? Depends on location! C′

k l @ K: number of large elements at positions K.

C′

g s @ G: number of small elements at positions G.

1

Recall invariant: < p ℓ → > q g ← p ◦ q k → ? k and g cross at (rank of) q

p q positions K = {2, . . . , q − 1} G = {q, . . . , n − 1} l @ K = 3 s @ G = 2

|K| ∼ q, |G| ∼ n − q

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 12 / 22

Distribution of l @ K and s @ G

Assume p and q are fixed.

2

How many small and large elements?

#small =

• {1, . . . , p − 1}
• = p − 1

#large =

• {q + 1, . . . , n}
• = n − q

|K|, |G|, #small and #large are constant (for given p and q). But: l @ K and s @ G are random even for fixed p and q.

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 13 / 22

Distribution of l @ K and s @ G

Assume p and q are fixed.

2

How many small and large elements?

#small =

• {1, . . . , p − 1}
• = p − 1

#large =

• {q + 1, . . . , n}
• = n − q

|K|, |G|, #small and #large are constant (for given p and q). But: l @ K and s @ G are random even for fixed p and q.

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 13 / 22

Distribution of l @ K and s @ G

3

Conditional Distribution of l @ K

We draw positions of large elements at random. n − 2 positions (≡ urn with n − 2 balls) draw #large positions without replacement (≡ number of draws is #large) |K| positions contribute to l @ K (≡ maximum number of successes is |K|)

l @ K D = Hypergeometric (#large, |K|, n − 2) E [l @ K | p, q] = #large · |K| n − 2 ∼

?? , 2

(n − q) · q n − 2

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 14 / 22

Distribution of l @ K and s @ G

3

Conditional Distribution of l @ K

We draw positions of large elements at random. n − 2 positions (≡ urn with n − 2 balls) draw #large positions without replacement (≡ number of draws is #large) |K| positions contribute to l @ K (≡ maximum number of successes is |K|)

l @ K D = Hypergeometric (#large, |K|, n − 2) E [l @ K | p, q] = #large · |K| n − 2 ∼

?? , 2

(n − q) · q n − 2

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 14 / 22

Analysis of Yaroslavskiy’s Algorithm

law of total expectation: E [l @ K] =

• 1p<qn

Pr[pivots (p, q)] · (n − q) q−2

n−2 ∼ 1 6n

Similarly: E [s @ G] ∼

1 12n.

Summing up contributions: cn ∼ n

first comparisons

+ 1

3n

medium elements

+ 1

6n

large elements at C′

k

+ 1

12n

small elements at C′

g

=

19 12 n

Recall: “lower bound” was 20

12n.

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 15 / 22

Distribution of costs

The contraction method can be used to show Theorem

For the number Cn of key comparisons used by Yaroslavskiy’s Quicksort when operating

• n a uniformly at random distributed permutation we have

Cn − E[Cn] n → C ∗, (n → ∞), where the convergence is in distribution and with second moments. The distribution of C ∗ is determined as the unique fixed point, subject to E[X] = 0 and E[X 2] < ∞, of X

D

= 1 + (D1 + D2)(D2 + 2D3) +

3

• j=1
• DjX (j) + 19

10Dj ln Dj

• ,

where (D1, D2, D3), X (1), X (2) and X (3) are independent and X (j) has the same distribution as X for j ∈ {1, 2, 3}. Moreover, we have, as n → ∞, Var(Cn) ∼ σ2

Cn2

with σ2

C

=

2231 360 − 361 600π2 = 0.25901. . . Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 16 / 22

Distribution of costs

Exact distribution Cn n , and Cn − E[Cn] n , n = 5 . . . 25.

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 17 / 22

Covariance between comparisons and swaps

Theorem

For the number Cn of key comparisons and the number Sn of swaps used by Yaroslavskiy’s algorithm on a random permutation, we have for n → ∞ Cov(Cn, Sn) ∼ σC,S n2 with σC,S =

28 15 − 19 100π2

= −0.00855817. . . The correlation coefficient of Cn and Sn is consequently ρ = Cov(Cn, Sn)

• Var(Cn)
• Var(Sn)

≈ −0.0512112. . .

Remark: Note the changed behavior compared to classic quicksort where a strong negative correlation (−0.86404) is observed!

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 18 / 22

Pivot Sampling

0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 α1 α2

α1 (α2) the percentage of small (medium) elements.

Black (white) dot shows

• ptimal (symmetric) choice

for the pivots (exact order statistics k → ∞); dashed black (dotted white) lines represent “equi-cost-ant” to optimum (equidistant from symmetric) pivot choices; a smaller sample together with optimal choice can beat symmetric choice for larger sample (in number of Java bytecodes).

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 19 / 22

Pivot Sampling

0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 α1 α2

α1 (α2) the percentage of small (medium) elements.

Black (white) dot shows

• ptimal (symmetric) choice

for the pivots (exact order statistics k → ∞); dashed black (dotted white) lines represent “equi-cost-ant” to optimum (equidistant from symmetric) pivot choices; a smaller sample together with optimal choice can beat symmetric choice for larger sample (in number of Java bytecodes).

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 19 / 22

Discussion

Comparisons:

Yaroslavskiy needs ∼ 6

5 · 19 12 n ln n = 1.9 n ln n on average.

Classic Quicksort needs ∼ 2 n ln n comparisons!

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 20 / 22

Discussion – we did not really succeed

Comparisons:

Yaroslavskiy needs ∼ 6

5 · 19 12 n ln n = 1.9 n ln n on average.

Classic Quicksort needs ∼ 2 n ln n comparisons!

Swaps:

∼ 0.6 n ln n swaps for Yaroslavskiy’s algorithm vs. ∼ 0.3 n ln n swaps for classic Quicksort

Bytecodes:

∼ 21.7n ln(n) − 3.56319n Java bytecodes for Yaroslavskiy’s algorithm vs. ∼ 18n ln(n) + 6.21488n Java bytecodes for classic Quicksort

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 20 / 22

Conclusion

Yaroslavskiy’s quicksort is a perfect textbook example to demonstrate how well methods from AofA are developed; the depth of results obtainable (precise expectations, distributions, covariances, ...) by those methods; how AofA can guide engineering of an algorithm (pivot sampling, switch to insertionsort, ...). However, our sophisticated machinery fails to explain the practical efficiency of Yaroslavskiy’s algorithms; (presumably) it would be important to get access to branch mispredictions and/or cache misses.

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 21 / 22

Many thanks for your attention!

Markus E. Nebel Java 7’s Dual Pivot Quicksort 2013/18/5 22 / 22