Algorithm Engineering (aka. How to Write Fast Code) CS26 S260 - - PowerPoint PPT Presentation

Algorithm Engineering

(aka. How to Write Fast Code)

I/O Algorithms and Parallel Samplesort

CS26 S260 – Lecture cture 6 Yan n Gu

CS260: Algorithm Engineering Lecture 6

2

The I/O Model Sampling in Algorithm Design Parallel Samplesort

3

• The I/O

O model el has two speci cial al memor mory y transfer sfer instructions: ructions:

• Read transfe

nsfer: load a block from slow memory

• Write

te transf sfer: write a block to slow memory

• The co

comp mplexi lexity ty of an algor

• rithm

ithm on the I/O O model del (I/O O co complexi plexity) ty) is measur sured ed by: y: #( #(rea ead tran ansfe sfers) rs) + #( #(write e transfe ansfers) rs)

Last week - The I/O model

CPU

Fast Memory Slow Memory

1 1

𝑁/𝐶

𝐶

Cache-Oblivious Algorithms

• Alg

lgorit ithms hms not paramete meteriz ized ed by 𝐶 or 𝑁

• These algorithms are unaware of the parameters of the memory

hierarchy

• Analy

lyze ze in in the id ideal l cache model el — same e as the I/O m model l except pt optim imal al repla laceme ement nt is is assum sumed ed

CPU

Fast Memory

1 1

𝑁/𝐶

𝐶

Slow Memory

CS260: Algorithm Engineering Lecture 6

6

The I/O Model Sampling in Algorithm Design Parallel Samplesort

• Yan has an array {𝒃𝟏, 𝒃𝟐, … , 𝒃𝒐−𝟐} such that 𝒃𝒋 = 𝟏 or 𝟐, and

Yan wants to know how many 𝟏(s) in the array

• Scan, linear work, can be parallelized
• Sounds like a good idea?

Why Sampling?

• Yan has an array {𝒃𝟏, 𝒃𝟐, … , 𝒃𝒐−𝟐} and a function 𝒈(⋅) such that

𝒈(𝒃𝒋) = 𝟏 or 𝟐, and Yan wants to know how many 𝒈(𝒃𝒋) = 𝟏

Why Sampling?

• Yan has an array {𝒃𝟏, 𝒃𝟐, … , 𝒃𝒐−𝟐} and 𝒐 function 𝒈𝟐 ⋅ , … , 𝒈𝒐(⋅)

such that 𝒈𝒌(𝒃𝒋) = 𝟏 or 𝟐, and Yan wants to know how many 𝒈𝒌(𝒃𝒋) = 𝟏

• Takes quadratic work, does not work for reasonable input size
• Examples:
• Find the median 𝑛 of 𝑏𝑗, 𝑔

𝑛 𝑏𝑗 = "𝑏𝑗 < 𝑛", check if #(𝑔 𝑏𝑘 𝑏𝑗 = 0) is 𝑜/2

• Find a good pivot 𝑞 in quicksort (e.g.,

𝑜 4 ≤ #(𝑔 𝑞 𝑏𝑗 = 0) ≤ 3𝑜 4 )

• Guarantee all sorts of properties in graph, geometry and other algorithms

Why Sampling?

• Yan has an array {𝒃𝟏, 𝒃𝟐, … , 𝒃𝒐−𝟐} and 𝒐 function 𝒈 ⋅ such that

𝒈(𝒃𝒋) = 𝟏 or 𝟐, and Yan wants to know how many 𝒈(𝒃𝒋) = 𝟏

• Uniformly randomly pick 𝒍 elements, compute the 𝒈 𝒃𝒋 = 𝟏

case (denoted as 𝒍𝟏), and estimate by

𝒐⋅𝒍𝟏 𝒍

• As long as 𝑙 is sufficiently large, we are “confident” with our estimation
• On the other hand, when 𝑙 is small, the result can be random
• When is the estimation good?
• What is “good”?

Approximate Solution: Sampling

• What is “good”?
• With high probability (informal): happens with probability 1 − 𝑜−𝑑 for any

constant 𝑑 > 0

• This is large when 𝑜 is reasonably large, like > 106
• When is the estimation good?
• Claim: when 𝑙0 is Ω log 𝑜
• How can reality off from the estimate?

Approximate Solution: Sampling

• When is the estimation good?
• Claim: when 𝑙0 is Ω log 𝑜
• How can reality off from the estimate?
• Assume there are 𝑨 elements with 𝒈(𝒃𝒋) = 𝟏, and we have 𝑙 samples with

𝑙0 hits. The expected #hits E 𝑙0 = 𝑙𝑨/𝑜.

• The probability that this is off by 100% (i.e., 𝑙0 > 2𝑙𝑨/𝑜) is 𝑓−𝑙𝑨

3𝑜

Approximate Solution: Sampling

Chernoff bound: for 𝑜 independent random variables in {0, 1}, let 𝑌 be the sum, and 𝜈 = E 𝑌 , then for any 0 ≤ 𝜀 ≤ 1, Pr 𝑌 ≥ 1 + 𝜀 𝜈 ≤ 𝑓−𝜀2𝜈

3

• When is the estimation good?
• Claim: when 𝑙0 is Ω log 𝑜
• How can reality off from the estimate?
• Assume there are 𝑨 elements with 𝒈(𝒃𝒋) = 𝟏, and we have 𝑙 samples with

𝑙0 hits. The expected #hits E 𝑙0 = 𝑙𝑨/𝑜.

• The probability that this is off by 100% (i.e., 𝑙0 > 2𝑙𝑨/𝑜) is 𝑓−𝑙𝑨

3𝑜

• Since 𝑙0 ≈ 𝑙𝑨/𝑜, 𝑓−𝑙𝑨

3𝑜 is 𝑜−𝑑 when 𝑙0 = Ω log 𝑜 , because

𝑓−𝑙𝑨

3𝑜 ≈ 𝑓−𝑙0 3 < 𝑓−𝑑′ log2𝑜 = 𝑜−𝑑

Approximate Solution: Sampling

• When is the estimation good?
• Claim: when 𝑙0 is Ω log 𝑜
• How can reality off from the estimate?
• Assume there are 𝑨 elements with 𝒈(𝒃𝒋) = 𝟏, and we have 𝑙 samples with

𝑙0 hits. The expected #hits E 𝑙0 = 𝑙𝑨/𝑜.

• The probability that this is off by 1% (i.e., 𝑙0 > 1.01𝑙𝑨/𝑜) is 𝑓−𝜀2𝑙𝑨

3𝑜

• Since 𝑙0 ≈ 𝑙𝑨/𝑜, 𝑓−𝜀2𝑙𝑨

3𝑜 is 𝑜−𝑑 when 𝑙0 = Ω log 𝑜 , because

𝑓−𝜀2𝑙𝑨

3𝑜 ≈ 𝑓− 𝑙0 3⋅1002 < 𝑓−𝑑′ log2𝑜 = 𝑜−𝑑

Approximate Solution: Sampling

Chernoff bound: for 𝑜 independent random variables in {0, 1}, let 𝑌 be the sum, and 𝜈 = E 𝑌 , then for any 0 < 𝜀 < 1, Pr 𝑌 ≥ 1 + 𝜀 𝜈 ≤ 𝑓−𝜀2𝜈

3

• Example Applications:
• Find the median 𝑛 of 𝑏𝑗, 𝑔 𝑏𝑗 = "𝑏𝑗 < 𝑛", check if #(𝑔

𝑏𝑘 𝑏𝑗 = 0) is 𝑜/2

• Find a good pivot 𝑞 in quicksort (e.g.,

𝑜 4 ≤ #(𝑔 𝑞 𝑏𝑗 = 0) ≤ 3𝑜 4 )

• Guarantee all sorts of properties in graph, geometry and other algorithms
• Take some samples! Uniformly randomly pick 𝒍 elements,

compute the 𝒈 𝒃𝒋 = 𝟏 case (denoted as 𝒍𝟏), and estimate by

𝒐⋅𝒍𝟏 𝒍

• 4 sample hits gives you reasonable result
• 20 sample hits gives you confident
• 100 sample hits is sufficient!
• Remember: only hits count

Rule of Thumbs for Sampling

CS260: Algorithm Engineering Lecture 6

16

The I/O Model Sampling in Algorithm Design Parallel Samplesort

Parallel and I/O-efficient Sorting Algorithms

• Cla

lassi sic c sortin ing g alg lgorit ithm hms s are easy y to b be p parallel lleliz ized ed

• Quicksort: find a “good” pivot, apply partition (filter) to find

elements that are smaller and that are larger, and recurse

• Mergesort: apply parallel merge for log2 𝑜 rounds
• But not I/O efficient since we need log2 𝑜 rounds of global data

movement

• We now introduce samplesort, which is both highly in parallel and

I/O efficient

Sample-sort outline

Analo logou gous s to mult ltiw iway ay quic ickso ksort 1.

• 1. Sp

Spli lit in input ut array in into 𝑂 contiguo iguous us suba barra rrays ys of siz ize 𝑂. So Sort subar arrays rays recursi sivel vely

… 𝑂, sorted 𝑂

Sample-sort outline

𝑂, sorted …

Analo logou gous s to mult ltiw iway ay quic ickso ksort 1.

• 1. Sp

Spli lit in input ut array in into 𝑂 contiguo iguous us suba barra rrays ys of siz ize 𝑂. So Sort subar arrays rays recursi sivel vely y (sequ equent entia ially lly)

Sample-sort outline

2.

• 2. Choo
• ose

se 𝑂 − 1 “good” pivots 𝑞1 ≤ 𝑞2 ≤ ⋯ ≤ 𝑞 𝑂−1 3.

• 3. Dis

istribu ribute te su subar barrays rays in into

• buckets

ckets, , ac accordin

• rding

g to pivot vots

𝑂, sorted … Bucket 1 Bucket 2 Bucket 𝑂 ≤ 𝑞1 ≤ ≤ 𝑞2 ≤ ⋯ ≤ 𝑞 𝑂−1 ≤

Size ≈ 𝑂

4.

• 4. Recurs

cursively ively sort rt the buckets ckets 5.

• 5. Copy

py conca

• ncatenated

tenated buckets ckets bac ack k to input put ar arra ray

Sample-sort outline

Bucket 1 Bucket 2 Bucket 𝑂 ≤ 𝑞1 ≤ ≤ 𝑞2 ≤ ⋯ ≤ 𝑞 𝑂−1 ≤ sorted

Choosing good pivots based on sampling

2.

• 2. Cho

hoose

• se 𝑂 − 1 “good” pivots 𝑞1 ≤ 𝑞2 ≤ ⋯ ≤

𝑞 𝑂−1

Can an be ac achieved ieved by y ra randoml domly y pic ick k 𝑑 𝑂 log 𝑂 ra rando dom m sam amples les, , sort rt them m an and pick ck the eve very ry 𝑑 log 𝑂 -th th element ment This is step p is fa fast

Sequential local sorts (e.g., call stl::sort)

1.

• 1. Sp

Spli lit in input ut array in into 𝑂 contiguo iguous us subar array ays of siz ize 𝑂. So Sort rt suba barray rrays s re recu cursi rsivel vely y (sequen quentia ially) lly) 4. Recur ursi sively vely sort the buckets ets (sequ quenti ential al)

… 𝑂, sorted Bucket 1 Bucket 2 Bucket 𝑂 ≤ 𝑞1 ≤ ≤ 𝑞2 ≤ ⋯ ≤ 𝑞 𝑂−1 ≤

Key Part: the Distribution Phase

3. . Dis istribute ribute su subarr arrays ays in into to buck uckets ets, , ac according cording to pivot vots

𝑂, sorted … Bucket 1 Bucket 2 Bucket 𝑂 ≤ 𝑞1 ≤ ≤ 𝑞2 ≤ ⋯ ≤ 𝑞 𝑂−1 ≤

Size ≈ 𝑂

Key Part: the Distribution Phase

• For si

simpli plicity ity, , assum sume e 𝒐 = 𝟐𝟕, a and the in input ut is is [𝟐, 𝟑, 𝟒, 𝟓, 𝟐, 𝟐, 𝟒, 𝟒, 𝟐, 𝟑, 𝟑, 𝟓, 𝟐, 𝟑, 𝟓, 𝟓]

• Fir

irst, , ge get the count t for each subar array ray in in ea each bucket et [𝟐, 𝟐, 𝟐, 𝟐, 𝟑, 𝟏, 𝟑, 𝟏, 𝟐, 𝟑, 𝟏, 𝟐, 𝟐, 𝟐, 𝟏, 𝟑]

• Then,

, transpos spose e the array and d scan to co compute ute the offse fsets ts [𝟐, 𝟑, 𝟐, 𝟐, 𝟐, 𝟏, 𝟑, 𝟐, 𝟐, 𝟑, 𝟏, 𝟏, 𝟐, 𝟏, 𝟐, 𝟑] [𝟏, 𝟐, 𝟒, 𝟓, 𝟔, 𝟕, 𝟕, 𝟗, 𝟘, 𝟐𝟏, 𝟐𝟑, 𝟐𝟑, 𝟐𝟑, 𝟐𝟒, 𝟐𝟒, 𝟐𝟓]

• Lastly

ly, , move e each ele lement ent to th the correspo pond nding ing bucket et [∅, ∅, ∅, ∅, ∅, ∅, ∅, ∅, ∅, ∅, ∅, ∅, ∅, ∅, ∅, ∅]

25

[𝟐, ∅, ∅, ∅, ∅, 𝟑, ∅, ∅, ∅, 𝟒, ∅, ∅, 𝟓, ∅, ∅, ∅] [𝟐, 𝟐, 𝟐, ∅, ∅, 𝟑, ∅, ∅, ∅, 𝟒, 𝟒, 𝟒, 𝟓, ∅, ∅, ∅] [𝟐, 𝟐, 𝟐, 𝟐, 𝟐, 𝟑, 𝟑, 𝟑, 𝟑, 𝟒, 𝟒, 𝟒, 𝟓, 𝟓, 𝟓, 𝟓]

Additional Details Left for You

• How to

to d decid ide the count of each bucket et in in ea each suba barray ay

• Hint: use a (sequential) merge algorithm
• How to

to tr transpos spose the array for counts ts and writ ite the ele lements ents to b bucket ets s I/O effi ficie ientl ntly

• Hint: use divide-and-conquer
• Fin

ind d the best t #piv ivots ts and #subar arrays rays

• How does #pivots and #subarrays affect performance?

26

Samplesort is I/O-efficient

• Only

ly need d two rounds ds of gl global l data accesses sses

• For input size 𝑜 between 10 million and 100 billion
• In the mid

idterm erm project, t, you can choose e to im imple lemen ment this is alg lgorit ithm hm and engi ginee neer r the perfo forman rmance

• This is harder than matrix multiplication, but easier than semisort
• Expected score is 100%
• Dis

iscussio ussion: n: what t is is th the work for sampl plesor esort? And what t about t depth? th?

27

Next lecture: Semisort

• https:

ps://ww www. w.cs.uc cs.ucr.ed .edu/~ u/~yg ygu/ u/te teachi aching ng/alge algeng ng/al alge geng.htm ng.html

• https:

ps://il ilear earn.ucr n.ucr.ed .edu/ u/web webap apps/ ps/blac blackboa board/ d/exec execute/ ute/ann announc unc ement? nt?met method=se hod=sear arch&co ch&cont ntext=c ext=course& urse&cour

• urse_

se_id=_3 id=_307782_ 1

28