SLIDE 1
CS 573: Algorithms, Fall 2013
Randomized Algorithms
Lecture 9
September 24, 2013
Sariel (UIUC) CS573 1 Fall 2013 1 / 32
Part I
. .
Randomized Algorithms
Sariel (UIUC) CS573 2 Fall 2013 2 / 32
Randomized Algorithms Lecture 9 September 24, 2013 Sariel (UIUC) - - PowerPoint PPT Presentation
Randomized Algorithms Lecture 9 September 24, 2013 Sariel (UIUC) - - PowerPoint PPT Presentation
CS 573: Algorithms, Fall 2013 Randomized Algorithms Lecture 9 September 24, 2013 Sariel (UIUC) CS573 1 Fall 2013 1 / 32 Part I . Randomized Algorithms . Sariel (UIUC) CS573 2 Fall 2013 2 / 32 Probability - quick review Definitions
SLIDE 2
SLIDE 3
Probability - quick review
Definitions
.
Definition (Informal)
. . Random variable: a function from probability space to I R. Associates value ∀ atomic events in probability space. .
Definition
. . The conditional probability of X given Y is Pr
[
X = x
- Y = y
]
= Pr
[
(X = x) ∩ (Y = y)
]
Pr
[
Y = y
]
. Equivalent to Pr
[
(X = x) ∩ (Y = y)
]
= Pr
[
X = x
- Y = y
]
∗ Pr
[
Y = y
]
.
Sariel (UIUC) CS573 3 Fall 2013 3 / 32
SLIDE 4
Probability - quick review
Definitions
.
Definition (Informal)
. . Random variable: a function from probability space to I R. Associates value ∀ atomic events in probability space. .
Definition
. . The conditional probability of X given Y is Pr
[
X = x
- Y = y
]
= Pr
[
(X = x) ∩ (Y = y)
]
Pr
[
Y = y
]
. Equivalent to Pr
[
(X = x) ∩ (Y = y)
]
= Pr
[
X = x
- Y = y
]
∗ Pr
[
Y = y
]
.
Sariel (UIUC) CS573 3 Fall 2013 3 / 32
SLIDE 5
Probability - quick review
Definitions
.
Definition (Informal)
. . Random variable: a function from probability space to I R. Associates value ∀ atomic events in probability space. .
Definition
. . The conditional probability of X given Y is Pr
[
X = x
- Y = y
]
= Pr
[
(X = x) ∩ (Y = y)
]
Pr
[
Y = y
]
. Equivalent to Pr
[
(X = x) ∩ (Y = y)
]
= Pr
[
X = x
- Y = y
]
∗ Pr
[
Y = y
]
.
Sariel (UIUC) CS573 3 Fall 2013 3 / 32
SLIDE 6
Probability - quick review
Even more definitions
.
Definition
. . The events X = x and Y = y are independent, if Pr[X = x ∩ Y = y] = Pr[X = x] · Pr[Y = y] . ≡ .
Definition
. . The expectation of a random variable X its average value: E
[
X
]
=
∑
x
x · Pr[X = x] ,
Sariel (UIUC) CS573 4 Fall 2013 4 / 32
SLIDE 7
Probability - quick review
Even more definitions
.
Definition
. . The events X = x and Y = y are independent, if Pr[X = x ∩ Y = y] = Pr[X = x] · Pr[Y = y] . ≡ Pr
[
X = x
- Y = y
]
= Pr[X = x] . .
Definition
. . The expectation of a random variable X its average value: E
[
X
]
=
∑
x
x · Pr[X = x] ,
Sariel (UIUC) CS573 4 Fall 2013 4 / 32
SLIDE 8
Linearity of expectations
.
Lemma (Linearity of expectation.)
. . ∀ random variables X and Y: E
[
X + Y
]
= E
[
X
]
+ E
[
Y
]
. .
Proof.
. . Use definitions, do the math. See notes for details.
Sariel (UIUC) CS573 5 Fall 2013 5 / 32
SLIDE 9
Probability - quick review
Conditional Expectation
.
Definition
. . X, Y: random variables. The conditional expectation of X given Y (i.e., you know Y = y): E
[
X
- Y
]
= E
[
X
- Y = y
]
=
∑
x
x ∗ Pr
[
X = x
- Y = y
]
. E[X] is a number. f (y) = E
[
X
- Y = y
]
is a function.
Sariel (UIUC) CS573 6 Fall 2013 6 / 32
SLIDE 10
Conditional Expectation
.
Lemma
. . ∀ X, Y (not necessarily independent): E[X] = E
[
E
[
X
- Y
]]
. E
[
E
[
X
- Y
]]
= Ey
[
E
[
X
- Y = y
]]
.
Proof.
. . Use definitions, and do the math. See class notes.
Sariel (UIUC) CS573 7 Fall 2013 7 / 32
SLIDE 11
Conditional Expectation
.
Lemma
. . ∀ X, Y (not necessarily independent): E[X] = E
[
E
[
X
- Y
]]
. E
[
E
[
X
- Y
]]
= Ey
[
E
[
X
- Y = y
]]
.
Proof.
. . Use definitions, and do the math. See class notes.
Sariel (UIUC) CS573 7 Fall 2013 7 / 32
SLIDE 12
Conditional Expectation
.
Lemma
. . ∀ X, Y (not necessarily independent): E[X] = E
[
E
[
X
- Y
]]
. E
[
E
[
X
- Y
]]
= Ey
[
E
[
X
- Y = y
]]
.
Proof.
. . Use definitions, and do the math. See class notes.
Sariel (UIUC) CS573 7 Fall 2013 7 / 32
SLIDE 13
Sorting Nuts & Bolts
.
Problem (Sorting Nuts and Bolts)
. . .
1
Input: Set n nuts + n bolts. . .
2
Every nut have a matching bolt. . .
3
All diff sizes. . .
4
Task: Match nuts to bolts. (In sorted order). . .
5
Restriction: You can only compare a nut to a bolt. .
6
Q: How to match the n nuts to the n bolts quickly?
Sariel (UIUC) CS573 8 Fall 2013 8 / 32
SLIDE 14
Sorting Nuts & Bolts
.
Problem (Sorting Nuts and Bolts)
. . .
1
Input: Set n nuts + n bolts. . .
2
Every nut have a matching bolt. . .
3
All diff sizes. . .
4
Task: Match nuts to bolts. (In sorted order). .
5
Restriction: You can only compare a nut to a bolt. .
6
Q: How to match the n nuts to the n bolts quickly?
Sariel (UIUC) CS573 8 Fall 2013 8 / 32
SLIDE 15
Sorting Nuts & Bolts
.
Problem (Sorting Nuts and Bolts)
. . .
1
Input: Set n nuts + n bolts. . .
2
Every nut have a matching bolt. . .
3
All diff sizes. . .
4
Task: Match nuts to bolts. (In sorted order). . .
5
Restriction: You can only compare a nut to a bolt. . .
6
Q: How to match the n nuts to the n bolts quickly?
Sariel (UIUC) CS573 8 Fall 2013 8 / 32
SLIDE 16
Sorting Nuts & Bolts
.
Problem (Sorting Nuts and Bolts)
. . .
1
Input: Set n nuts + n bolts. . .
2
Every nut have a matching bolt. . .
3
All diff sizes. . .
4
Task: Match nuts to bolts. (In sorted order). . .
5
Restriction: You can only compare a nut to a bolt. .
6
Q: How to match the n nuts to the n bolts quickly?
Sariel (UIUC) CS573 8 Fall 2013 8 / 32
SLIDE 17
Sorting Nuts & Bolts
.
Problem (Sorting Nuts and Bolts)
. . .
1
Input: Set n nuts + n bolts. . .
2
Every nut have a matching bolt. . .
3
All diff sizes. . .
4
Task: Match nuts to bolts. (In sorted order). . .
5
Restriction: You can only compare a nut to a bolt. .
6
Q: How to match the n nuts to the n bolts quickly?
Sariel (UIUC) CS573 8 Fall 2013 8 / 32
SLIDE 18
Sorting nuts & bolts...
Algorithm
. .
1
Naive algorithm... . .
2
...better algorithm?
Sariel (UIUC) CS573 9 Fall 2013 9 / 32
SLIDE 19
Sorting nuts & bolts...
Algorithm
. .
1
Naive algorithm... . .
2
...better algorithm?
Sariel (UIUC) CS573 9 Fall 2013 9 / 32
SLIDE 20
Sorting nuts & bolts...
MatchNutsAndBolts(N: nuts, B: bolts) Pick a random nut npivot from N Find its matching bolt bpivot in B BL ← All bolts in B smaller than npivot NL ← All nuts in N smaller than bpivot BR ← All bolts in B larger than npivot NR ← All nuts in N larger than bpivot MatchNutsAndBolts(NR,BR) MatchNutsAndBolts(NL,BL)
QuickSort style...
Sariel (UIUC) CS573 10 Fall 2013 10 / 32
SLIDE 21
Sorting nuts & bolts...
MatchNutsAndBolts(N: nuts, B: bolts) Pick a random nut npivot from N Find its matching bolt bpivot in B BL ← All bolts in B smaller than npivot NL ← All nuts in N smaller than bpivot BR ← All bolts in B larger than npivot NR ← All nuts in N larger than bpivot MatchNutsAndBolts(NR,BR) MatchNutsAndBolts(NL,BL)
QuickSort style...
Sariel (UIUC) CS573 10 Fall 2013 10 / 32
SLIDE 22
What is running time for randomized algorithms?
Definitions
.
Definition
. . RT(U): random variable – running time of the algorithm on input U. .
Definition
. . Expected running time E[RT(U)] for input U. .
Definition
. . expected running-time of algorithm for input size n: T(n) = max
U is an input of size n E
[
RT(U)
]
.
Sariel (UIUC) CS573 11 Fall 2013 11 / 32
SLIDE 23
What is running time for randomized algorithms?
Definitions
.
Definition
. . RT(U): random variable – running time of the algorithm on input U. .
Definition
. . Expected running time E[RT(U)] for input U. .
Definition
. . expected running-time of algorithm for input size n: T(n) = max
U is an input of size n E
[
RT(U)
]
.
Sariel (UIUC) CS573 11 Fall 2013 11 / 32
SLIDE 24
What is running time for randomized algorithms?
Definitions
.
Definition
. . RT(U): random variable – running time of the algorithm on input U. .
Definition
. . Expected running time E[RT(U)] for input U. .
Definition
. . expected running-time of algorithm for input size n: T(n) = max
U is an input of size n E
[
RT(U)
]
.
Sariel (UIUC) CS573 11 Fall 2013 11 / 32
SLIDE 25
What is running time for randomized algorithms?
More definitions
.
Definition
. . rank(x): rank of element x ∈ S = number of elements in S smaller or equal to x.
Sariel (UIUC) CS573 12 Fall 2013 12 / 32
SLIDE 26
Nuts and bolts running time
.
Theorem
. . Expected running time MatchNutsAndBolts (QuickSort) is T(n) = O(n log n). Worst case is O(n2). .
Proof.
. . Pr[rank(npivot) = k] = 1
- n. Thus,
T(n) = E
k=rank(npivot)
[
O(n) + T(k − 1) + T(n − k)
]
Sariel (UIUC) CS573 13 Fall 2013 13 / 32
SLIDE 27
Nuts and bolts running time
.
Theorem
. . Expected running time MatchNutsAndBolts (QuickSort) is T(n) = O(n log n). Worst case is O(n2). .
Proof.
. . Pr[rank(npivot) = k] = 1
- n. Thus,
T(n) = E
k=rank(npivot)
[
O(n) + T(k − 1) + T(n − k)
]
= O(n) + E
k [T(k − 1) + T(n − k)]
Sariel (UIUC) CS573 13 Fall 2013 13 / 32
SLIDE 28
Nuts and bolts running time
.
Theorem
. . Expected running time MatchNutsAndBolts (QuickSort) is T(n) = O(n log n). Worst case is O(n2). .
Proof.
. . Pr[rank(npivot) = k] = 1
- n. Thus,
T(n) = O(n) + E
k [T(k − 1) + T(n − k)]
Sariel (UIUC) CS573 13 Fall 2013 13 / 32
SLIDE 29
Nuts and bolts running time
.
Theorem
. . Expected running time MatchNutsAndBolts (QuickSort) is T(n) = O(n log n). Worst case is O(n2). .
Proof.
. . Pr[rank(npivot) = k] = 1
- n. Thus,
T(n) = O(n) + E
k [T(k − 1) + T(n − k)]
= O(n) +
n
∑
k=1
Pr[Rank(Pivot) = k] ∗(T(k − 1) + T(n − k))
Sariel (UIUC) CS573 13 Fall 2013 13 / 32
SLIDE 30
Nuts and bolts running time
.
Theorem
. . Expected running time MatchNutsAndBolts (QuickSort) is T(n) = O(n log n). Worst case is O(n2). .
Proof.
. . Pr[rank(npivot) = k] = 1
- n. Thus,
T(n) = O(n) +
n
∑
k=1
Pr[Rank(Pivot) = k] ∗(T(k − 1) + T(n − k))
Sariel (UIUC) CS573 13 Fall 2013 13 / 32
SLIDE 31
Nuts and bolts running time
.
Theorem
. . Expected running time MatchNutsAndBolts (QuickSort) is T(n) = O(n log n). Worst case is O(n2). .
Proof.
. . Pr[rank(npivot) = k] = 1
- n. Thus,
T(n) = O(n) +
n
∑
k=1
Pr[Rank(Pivot) = k] ∗(T(k − 1) + T(n − k)) = O(n) +
n
∑
k=1
1 n ·(T(k − 1) + T(n − k)) ,
Sariel (UIUC) CS573 13 Fall 2013 13 / 32
SLIDE 32
Nuts and bolts running time
.
Theorem
. . Expected running time MatchNutsAndBolts (QuickSort) is T(n) = O(n log n). Worst case is O(n2). .
Proof.
. . Pr[rank(npivot) = k] = 1
- n. Thus,
T(n) = O(n) +
n
∑
k=1
1 n ·(T(k − 1) + T(n − k)) ,
Sariel (UIUC) CS573 13 Fall 2013 13 / 32
SLIDE 33
Nuts and bolts running time
.
Theorem
. . Expected running time MatchNutsAndBolts (QuickSort) is T(n) = O(n log n). Worst case is O(n2). .
Proof.
. . Pr[rank(npivot) = k] = 1
- n. Thus,
T(n) = O(n) +
n
∑
k=1
1 n ·(T(k − 1) + T(n − k)) , Solution is T(n) = O(n log n).
Sariel (UIUC) CS573 13 Fall 2013 13 / 32
SLIDE 34
Alternative intuitive analysis...
Which is not formally correct
. .
1
MatchNutsAndBolts is lucky if n
4 ≤ rank(npivot) ≤ 3 4n.
. .
2
Pr[“lucky”] = 1/2. . .
3
T(n) ≤ O(n) + Pr[“lucky”] ∗ (T(n/4) + T(3n/4)) + Pr[“unlucky”] ∗ T(n). . .
4
T(n) = O(n) + 1
2 ∗
(
T(n
4) + T(3 4n)
)
+ 1
2T(n).
. .
5
Rewriting: T(n) = O(n) + T(n/4) + T((3/4)n). . .
6
... solution is O(n log n).
Sariel (UIUC) CS573 14 Fall 2013 14 / 32
SLIDE 35
Alternative intuitive analysis...
Which is not formally correct
. .
1
MatchNutsAndBolts is lucky if n
4 ≤ rank(npivot) ≤ 3 4n.
. .
2
Pr[“lucky”] = 1/2. . .
3
T(n) ≤ O(n) + Pr[“lucky”] ∗ (T(n/4) + T(3n/4)) + Pr[“unlucky”] ∗ T(n). . .
4
T(n) = O(n) + 1
2 ∗
(
T(n
4) + T(3 4n)
)
+ 1
2T(n).
. .
5
Rewriting: T(n) = O(n) + T(n/4) + T((3/4)n). . .
6
... solution is O(n log n).
Sariel (UIUC) CS573 14 Fall 2013 14 / 32
SLIDE 36
Alternative intuitive analysis...
Which is not formally correct
. .
1
MatchNutsAndBolts is lucky if n
4 ≤ rank(npivot) ≤ 3 4n.
. .
2
Pr[“lucky”] = 1/2. . .
3
T(n) ≤ O(n) + Pr[“lucky”] ∗ (T(n/4) + T(3n/4)) + Pr[“unlucky”] ∗ T(n). . .
4
T(n) = O(n) + 1
2 ∗
(
T(n
4) + T(3 4n)
)
+ 1
2T(n).
. .
5
Rewriting: T(n) = O(n) + T(n/4) + T((3/4)n). . .
6
... solution is O(n log n).
Sariel (UIUC) CS573 14 Fall 2013 14 / 32
SLIDE 37
Alternative intuitive analysis...
Which is not formally correct
. .
1
MatchNutsAndBolts is lucky if n
4 ≤ rank(npivot) ≤ 3 4n.
. .
2
Pr[“lucky”] = 1/2. . .
3
T(n) ≤ O(n) + Pr[“lucky”] ∗ (T(n/4) + T(3n/4)) + Pr[“unlucky”] ∗ T(n). . .
4
T(n) = O(n) + 1
2 ∗
(
T(n
4) + T(3 4n)
)
+ 1
2T(n).
. .
5
Rewriting: T(n) = O(n) + T(n/4) + T((3/4)n). . .
6
... solution is O(n log n).
Sariel (UIUC) CS573 14 Fall 2013 14 / 32
SLIDE 38
Alternative intuitive analysis...
Which is not formally correct
. .
1
MatchNutsAndBolts is lucky if n
4 ≤ rank(npivot) ≤ 3 4n.
. .
2
Pr[“lucky”] = 1/2. . .
3
T(n) ≤ O(n) + Pr[“lucky”] ∗ (T(n/4) + T(3n/4)) + Pr[“unlucky”] ∗ T(n). . .
4
T(n) = O(n) + 1
2 ∗
(
T(n
4) + T(3 4n)
)
+ 1
2T(n).
. .
5
Rewriting: T(n) = O(n) + T(n/4) + T((3/4)n). . .
6
... solution is O(n log n).
Sariel (UIUC) CS573 14 Fall 2013 14 / 32
SLIDE 39
Alternative intuitive analysis...
Which is not formally correct
. .
1
MatchNutsAndBolts is lucky if n
4 ≤ rank(npivot) ≤ 3 4n.
. .
2
Pr[“lucky”] = 1/2. . .
3
T(n) ≤ O(n) + Pr[“lucky”] ∗ (T(n/4) + T(3n/4)) + Pr[“unlucky”] ∗ T(n). . .
4
T(n) = O(n) + 1
2 ∗
(
T(n
4) + T(3 4n)
)
+ 1
2T(n).
. .
5
Rewriting: T(n) = O(n) + T(n/4) + T((3/4)n). . .
6
... solution is O(n log n).
Sariel (UIUC) CS573 14 Fall 2013 14 / 32
SLIDE 40
Worst case vs. average case
Expected running time of a randomized algorithm is T(n) = max
U is an input of size n E
[
RT(U)
]
, Worst case running time of deterministic algorithm: T(n) = max
U is an input of size n RT(U),
Sariel (UIUC) CS573 15 Fall 2013 15 / 32
SLIDE 41
High Probability running time...
.
Definition
. . Running time Alg is O(f (n)) with high probability if Pr
[
RT(Alg(n)) ≥ c · f (n)
]
= o(1). = ⇒ Pr
[
RT(Alg) > c ∗ f (n)
]
→ 0 as n → ∞. Usually use weaker def: Pr
[
RT(Alg(n)) ≥ c · f (n)
]
≤ 1 nd , Technical reasons... also assume that E[RT(Alg(n))] = O(f (n)).
Sariel (UIUC) CS573 16 Fall 2013 16 / 32
SLIDE 42
High Probability running time...
.
Definition
. . Running time Alg is O(f (n)) with high probability if Pr
[
RT(Alg(n)) ≥ c · f (n)
]
= o(1). = ⇒ Pr
[
RT(Alg) > c ∗ f (n)
]
→ 0 as n → ∞. Usually use weaker def: Pr
[
RT(Alg(n)) ≥ c · f (n)
]
≤ 1 nd , Technical reasons... also assume that E[RT(Alg(n))] = O(f (n)).
Sariel (UIUC) CS573 16 Fall 2013 16 / 32
SLIDE 43
High Probability running time...
.
Definition
. . Running time Alg is O(f (n)) with high probability if Pr
[
RT(Alg(n)) ≥ c · f (n)
]
= o(1). = ⇒ Pr
[
RT(Alg) > c ∗ f (n)
]
→ 0 as n → ∞. Usually use weaker def: Pr
[
RT(Alg(n)) ≥ c · f (n)
]
≤ 1 nd , Technical reasons... also assume that E[RT(Alg(n))] = O(f (n)).
Sariel (UIUC) CS573 16 Fall 2013 16 / 32
SLIDE 44
High Probability running time...
.
Definition
. . Running time Alg is O(f (n)) with high probability if Pr
[
RT(Alg(n)) ≥ c · f (n)
]
= o(1). = ⇒ Pr
[
RT(Alg) > c ∗ f (n)
]
→ 0 as n → ∞. Usually use weaker def: Pr
[
RT(Alg(n)) ≥ c · f (n)
]
≤ 1 nd , Technical reasons... also assume that E[RT(Alg(n))] = O(f (n)).
Sariel (UIUC) CS573 16 Fall 2013 16 / 32
SLIDE 45
Part II
. .
Slick analysis of QuickSort
Sariel (UIUC) CS573 17 Fall 2013 17 / 32
SLIDE 46
A Slick Analysis of QuickSort
Let Q(A) be number of comparisons done on input array A: . .
1
For 1 ≤ i < j < n let Rij be the event that rank i element is compared with rank j element. . .
2
Xij: indicator random variable for Rij. Xij = 1 ⇐ ⇒ rank i element compared with rank j element,
- therwise 0.
Q(A) =
∑
1≤i<j≤n
Xij and hence by linearity of expectation, E
[
Q(A)
]
=
∑
1≤i<j≤n
E
[
Xij
]
=
∑
1≤i<j≤n
Pr
[
Rij
]
.
Sariel (UIUC) CS573 18 Fall 2013 18 / 32
SLIDE 47
A Slick Analysis of QuickSort
Let Q(A) be number of comparisons done on input array A: . .
1
For 1 ≤ i < j < n let Rij be the event that rank i element is compared with rank j element. . .
2
Xij: indicator random variable for Rij. Xij = 1 ⇐ ⇒ rank i element compared with rank j element,
- therwise 0.
Q(A) =
∑
1≤i<j≤n
Xij and hence by linearity of expectation, E
[
Q(A)
]
=
∑
1≤i<j≤n
E
[
Xij
]
=
∑
1≤i<j≤n
Pr
[
Rij
]
.
Sariel (UIUC) CS573 18 Fall 2013 18 / 32
SLIDE 48
A Slick Analysis of QuickSort
Rij = rank i element is compared with rank j element. Question: What is Pr[Rij]?
7 5 9 1 3 4 8 6
Sariel (UIUC) CS573 19 Fall 2013 19 / 32
SLIDE 49
A Slick Analysis of QuickSort
Rij = rank i element is compared with rank j element. Question: What is Pr[Rij]? With ranks:
7 5 9 1 3 4 8 6
1 2 3 4 5 6 7 8
Sariel (UIUC) CS573 19 Fall 2013 19 / 32
SLIDE 50
A Slick Analysis of QuickSort
Rij = rank i element is compared with rank j element. Question: What is Pr[Rij]? With ranks:
7 5 9 1 3 4 8 6
1 2 3 4 5 6 7 8 As such, probability of comparing 5 to 8 is Pr[R4,7].
Sariel (UIUC) CS573 19 Fall 2013 19 / 32
SLIDE 51
A Slick Analysis of QuickSort
Rij = rank i element is compared with rank j element. Question: What is Pr[Rij]? With ranks:
7 5 9 1 3 4 8 6
1 2 3 4 5 6 7 8 .
1
If pivot too small (say 3 [rank 2]). Partition and call recursively:
7 5 9 1 3 4 8 6
= ⇒
7 5 9 3 4 8 6 1
Decision if to compare 5 to 8 is moved to subproblem.
Sariel (UIUC) CS573 19 Fall 2013 19 / 32
SLIDE 52
A Slick Analysis of QuickSort
Rij = rank i element is compared with rank j element. Question: What is Pr[Rij]? With ranks:
7 5 9 1 3 4 8 6
1 2 3 4 5 6 7 8 .
1
If pivot too small (say 3 [rank 2]). Partition and call recursively:
7 5 9 1 3 4 8 6
= ⇒
7 5 9 3 4 8 6 1
Decision if to compare 5 to 8 is moved to subproblem. . .
2
If pivot too large (say 9 [rank 8]):
7 5 9 1 3 4 8 6 7 5 9 1 3 4 8 6
= ⇒
7 5 1 3 4 8 6 9
Decision if to compare 5 to 8 moved to subproblem.
Sariel (UIUC) CS573 19 Fall 2013 19 / 32
SLIDE 53
A Slick Analysis of QuickSort
Question: What is Pr[Ri,j]?
7 5 9 1 3 4 8 6
1 2 3 4 5 6 7 8 As such, probability of com- paring 5 to 8 is Pr[R4,7]. . .
1
If pivot is 5 (rank 4). Bingo!
7 5 9 1 3 4 8 6
= ⇒
1 3 4 5 7 9 8 6
Sariel (UIUC) CS573 20 Fall 2013 20 / 32
SLIDE 54
A Slick Analysis of QuickSort
Question: What is Pr[Ri,j]?
7 5 9 1 3 4 8 6
1 2 3 4 5 6 7 8 As such, probability of com- paring 5 to 8 is Pr[R4,7]. . .
1
If pivot is 5 (rank 4). Bingo!
7 5 9 1 3 4 8 6
= ⇒
1 3 4 5 7 9 8 6
.
2
If pivot is 8 (rank 7). Bingo!
7 5 9 1 3 4 8 6
= ⇒
7 5 9 1 3 4 6 8
Sariel (UIUC) CS573 20 Fall 2013 20 / 32
SLIDE 55
A Slick Analysis of QuickSort
Question: What is Pr[Ri,j]?
7 5 9 1 3 4 8 6
1 2 3 4 5 6 7 8 As such, probability of com- paring 5 to 8 is Pr[R4,7]. . .
1
If pivot is 5 (rank 4). Bingo!
7 5 9 1 3 4 8 6
= ⇒
1 3 4 5 7 9 8 6
.
2
If pivot is 8 (rank 7). Bingo!
7 5 9 1 3 4 8 6
= ⇒
7 5 9 1 3 4 6 8
.
3
If pivot in between the two numbers (say 6 [rank 5]):
7 5 9 1 3 4 8 6
= ⇒
7 5 9 1 3 4 6 8
5 and 8 will never be compared to each other.
Sariel (UIUC) CS573 20 Fall 2013 20 / 32
SLIDE 56
A Slick Analysis of QuickSort
Question: What is Pr[Ri,j]?
.
Conclusion:
. . Ri,j happens if and only if: ith or jth ranked element is the first pivot out of ith to jth ranked elements. .
How to analyze this?
. . Thinking acrobatics! .
1
Assign every element in the array a random priority (say in [0, 1]). .
2
Choose pivot to be the element with lowest priority in subproblem. . .
3
Equivalent to picking pivot uniformly at random (as QuickSort do).
Sariel (UIUC) CS573 21 Fall 2013 21 / 32
SLIDE 57
A Slick Analysis of QuickSort
Question: What is Pr[Ri,j]?
.
How to analyze this?
. . Thinking acrobatics! . .
1
Assign every element in the array a random priority (say in [0, 1]). . .
2
Choose pivot to be the element with lowest priority in subproblem. = ⇒ Ri,j happens if either i or j have lowest priority out of elements rank i to j, There are k = j − i + 1 relevant elements. Pr
[
Ri,j
]
= 2 k = 2 j − i + 1.
Sariel (UIUC) CS573 22 Fall 2013 22 / 32
SLIDE 58
A Slick Analysis of QuickSort
Question: What is Pr[Ri,j]?
.
How to analyze this?
. . Thinking acrobatics! . .
1
Assign every element in the array a random priority (say in [0, 1]). . .
2
Choose pivot to be the element with lowest priority in subproblem. = ⇒ Ri,j happens if either i or j have lowest priority out of elements rank i to j, There are k = j − i + 1 relevant elements. Pr
[
Ri,j
]
= 2 k = 2 j − i + 1.
Sariel (UIUC) CS573 22 Fall 2013 22 / 32
SLIDE 59
A Slick Analysis of QuickSort
Question: What is Pr[Rij]? .
Lemma
. . Pr
[
Rij
]
=
2 j−i+1.
.
Proof.
. . Let a1, . . . , ai, . . . , aj, . . . , an be elements of A in sorted order. Let S = {ai, ai+1, . . . , aj} Observation: If pivot is chosen outside S then all of S either in left array or right array. Observation: ai and aj separated when a pivot is chosen from S for the first time. Once separated no comparison. Observation: ai is compared with aj if and only if either ai or aj is chosen as a pivot from S at separation...
Sariel (UIUC) CS573 23 Fall 2013 23 / 32
SLIDE 60
A Slick Analysis of QuickSort
Question: What is Pr[Rij]? .
Lemma
. . Pr
[
Rij
]
=
2 j−i+1.
.
Proof.
. . Let a1, . . . , ai, . . . , aj, . . . , an be elements of A in sorted order. Let S = {ai, ai+1, . . . , aj} Observation: If pivot is chosen outside S then all of S either in left array or right array. Observation: ai and aj separated when a pivot is chosen from S for the first time. Once separated no comparison. Observation: ai is compared with aj if and only if either ai or aj is chosen as a pivot from S at separation...
Sariel (UIUC) CS573 23 Fall 2013 23 / 32
SLIDE 61
A Slick Analysis of QuickSort
Question: What is Pr[Rij]? .
Lemma
. . Pr
[
Rij
]
=
2 j−i+1.
.
Proof.
. . Let a1, . . . , ai, . . . , aj, . . . , an be elements of A in sorted order. Let S = {ai, ai+1, . . . , aj} Observation: If pivot is chosen outside S then all of S either in left array or right array. Observation: ai and aj separated when a pivot is chosen from S for the first time. Once separated no comparison. Observation: ai is compared with aj if and only if either ai or aj is chosen as a pivot from S at separation...
Sariel (UIUC) CS573 23 Fall 2013 23 / 32
SLIDE 62
A Slick Analysis of QuickSort
Continued...
.
Lemma
. . Pr
[
Rij
]
=
2 j−i+1.
.
Proof.
. . Let a1, . . . , ai, . . . , aj, . . . , an be sort of A. Let S = {ai, ai+1, . . . , aj} Observation: ai is compared with aj if and only if either ai or aj is chosen as a pivot from S at separation. Observation: Given that pivot is chosen from S the probability that it is ai or aj is exactly 2/|S| = 2/(j − i + 1) since the pivot is chosen uniformly at random from the array.
Sariel (UIUC) CS573 24 Fall 2013 24 / 32
SLIDE 63
A Slick Analysis of QuickSort
Continued...
E
[
Q(A)
]
=
∑
1≤i<j≤n
E[Xij] =
∑
1≤i<j≤n
Pr[Rij] . .
Lemma
. . Pr[Rij] =
2 j−i+1.
E
[
Q(A)
]
=
∑
1≤i<j≤n
2 j − i + 1
Sariel (UIUC) CS573 25 Fall 2013 25 / 32
SLIDE 64
A Slick Analysis of QuickSort
Continued...
.
Lemma
. . Pr[Rij] =
2 j−i+1.
E
[
Q(A)
]
=
∑
1≤i<j≤n
Pr
[
Rij
]
=
∑
1≤i<j≤n
2 j − i + 1
Sariel (UIUC) CS573 25 Fall 2013 25 / 32
SLIDE 65
A Slick Analysis of QuickSort
Continued...
.
Lemma
. . Pr[Rij] =
2 j−i+1.
E
[
Q(A)
]
=
∑
1≤i<j≤n
2 j − i + 1
Sariel (UIUC) CS573 25 Fall 2013 25 / 32
SLIDE 66
A Slick Analysis of QuickSort
Continued...
.
Lemma
. . Pr[Rij] =
2 j−i+1.
E
[
Q(A)
]
=
∑
1≤i<j≤n
2 j − i + 1 =
n−1
∑
i=1 n
∑
j=i+1
2 j − i + 1
Sariel (UIUC) CS573 25 Fall 2013 25 / 32
SLIDE 67
A Slick Analysis of QuickSort
Continued...
.
Lemma
. . Pr[Rij] =
2 j−i+1.
E
[
Q(A)
]
=
n−1
∑
i=1 n
∑
j=i+1
2 j − i + 1
Sariel (UIUC) CS573 25 Fall 2013 25 / 32
SLIDE 68
A Slick Analysis of QuickSort
Continued...
.
Lemma
. . Pr[Rij] =
2 j−i+1.
E
[
Q(A)
]
= 2
n−1
∑
i=1 n
∑
i<j
1 j − i + 1
Sariel (UIUC) CS573 25 Fall 2013 25 / 32
SLIDE 69
A Slick Analysis of QuickSort
Continued...
.
Lemma
. . Pr[Rij] =
2 j−i+1.
E
[
Q(A)
]
= 2
n−1
∑
i=1 n
∑
i<j
1 j − i + 1
Sariel (UIUC) CS573 25 Fall 2013 25 / 32
SLIDE 70
A Slick Analysis of QuickSort
Continued...
.
Lemma
. . Pr[Rij] =
2 j−i+1.
E
[
Q(A)
]
= 2
n−1
∑
i=1 n
∑
i<j
1 j − i + 1 ≤ 2
n−1
∑
i=1 n−i+1
∑
∆=2
1 ∆
Sariel (UIUC) CS573 25 Fall 2013 25 / 32
SLIDE 71
A Slick Analysis of QuickSort
Continued...
.
Lemma
. . Pr[Rij] =
2 j−i+1.
E
[
Q(A)
]
= 2
n−1
∑
i=1 n
∑
i<j
1 j − i + 1 ≤ 2
n−1
∑
i=1 n−i+1
∑
∆=2
1 ∆ ≤ 2
n−1
∑
i=1
(Hn−i+1 − 1) ≤ 2
∑
1≤i<n
Hn
Sariel (UIUC) CS573 25 Fall 2013 25 / 32
SLIDE 72
A Slick Analysis of QuickSort
Continued...
.
Lemma
. . Pr[Rij] =
2 j−i+1.
E
[
Q(A)
]
= 2
n−1
∑
i=1 n
∑
i<j
1 j − i + 1 ≤ 2
n−1
∑
i=1 n−i+1
∑
∆=2
1 ∆ ≤ 2
n−1
∑
i=1
(Hn−i+1 − 1) ≤ 2
∑
1≤i<n
Hn ≤ 2nHn = O(n log n)
Sariel (UIUC) CS573 25 Fall 2013 25 / 32
SLIDE 73
Part III
. .
Quick Select
Sariel (UIUC) CS573 26 Fall 2013 26 / 32
SLIDE 74
Randomized Quick Selection
Input Unsorted array A of n integers Goal Find the jth smallest number in A (rank j number) .
Randomized Quick Selection
. . .
1
Pick a pivot element uniformly at random from the array . .
2
Split array into 3 subarrays: those smaller than pivot, those larger than pivot, and the pivot itself. . .
3
Return pivot if rank of pivot is j. . .
4
Otherwise recurse on one of the arrays depending on j and their sizes.
Sariel (UIUC) CS573 27 Fall 2013 27 / 32
SLIDE 75
Algorithm for Randomized Selection
Assume for simplicity that A has distinct elements.
QuickSelect(A, j): Pick pivot x uniformly at random from A Partition A into Aless, x, and Agreater using x as pivot
if (|Aless| = j − 1) then return x if (|Aless| ≥ j) then return QuickSelect(Aless, j) else return QuickSelect(Agreater, j − |Aless| − 1)
Sariel (UIUC) CS573 28 Fall 2013 28 / 32
SLIDE 76
QuickSelect analysis
. .
1
S1, S2, . . . , Sk be the subproblems considered by the algorithm. Here |S1| = n. . .
2
Si would be successful if |Si| ≤ (3/4) |Si−1| . .
3
Y1 = number of recursive calls till first successful iteration. Clearly, total work till this happens is O(Y1n). . .
4
ni = size of the subproblem immediately after the (i − 1)th successful iteration. . .
5
Yi = number of recursive calls after the (i − 1)th successful call, till the ith successful iteration. . .
6
Running time is O(∑
i niYi).
Sariel (UIUC) CS573 29 Fall 2013 29 / 32
SLIDE 77
QuickSelect analysis
.
Example
. . Si = subarray used in ith recursive call |Si| = size of this subarray Red indicates successful iteration. Inst’ S1 S2 S3 S4 S5 S6 S7 S8 S9 |Si| 100 70 60 50 40 30 25 5 2 Succ’ Y1 = 2 Y2 = 4 Y3 = 2 Y4 = 1 ni = n1 = 100 n2 = 60 n3 = 25 n4 = 2 .
1
All the subproblems after (i − 1)th successful iteration till ith successful iteration have size ≤ ni. . .
2
Total work: O(∑
i niYi).
Sariel (UIUC) CS573 30 Fall 2013 30 / 32
SLIDE 78
QuickSelect analysis
.
Example
. . Si = subarray used in ith recursive call |Si| = size of this subarray Red indicates successful iteration. Inst’ S1 S2 S3 S4 S5 S6 S7 S8 S9 |Si| 100 70 60 50 40 30 25 5 2 Succ’ Y1 = 2 Y2 = 4 Y3 = 2 Y4 = 1 ni = n1 = 100 n2 = 60 n3 = 25 n4 = 2 .
1
All the subproblems after (i − 1)th successful iteration till ith successful iteration have size ≤ ni. . .
2
Total work: O(∑
i niYi).
Sariel (UIUC) CS573 30 Fall 2013 30 / 32
SLIDE 79
QuickSelect analysis
Total work: O(∑
i niYi).
We have: . .
1
ni ≤ (3/4)ni−1 ≤ (3/4)i−1n. . .
2
Yi is a random variable with geometric distribution Probability of Yi = k is 1/2i. . .
3
E[Yi] = 2. As such, expected work is proportional to E
[∑
i
niYi
]
=
∑
i
E
[
niYi
]
≤
∑
i
E
[
(3/4)i−1nYi
]
= n
∑
i
(3/4)i−1 E
[
Yi
]
= n
∑
i=1
(3/4)i−12 ≤ 8n.
Sariel (UIUC) CS573 31 Fall 2013 31 / 32
SLIDE 80
QuickSelect analysis
.
Theorem
. . The expected running time of QuickSelect is O(n).
Sariel (UIUC) CS573 32 Fall 2013 32 / 32
SLIDE 81
Notes
Sariel (UIUC) CS573 33 Fall 2013 33 / 32
SLIDE 82
Notes
Sariel (UIUC) CS573 34 Fall 2013 34 / 32
SLIDE 83
Notes
Sariel (UIUC) CS573 35 Fall 2013 35 / 32
SLIDE 84
Notes
Sariel (UIUC) CS573 36 Fall 2013 36 / 32