SLIDE 1
CS 573: Algorithms, Fall 2013
Entropy and Shannon’s Theorem
Lecture 28
December 10, 2013
Sariel (UIUC) CS573 1 Fall 2013 1 / 24
Part I
. .
Entropy
Sariel (UIUC) CS573 2 Fall 2013 2 / 24
Entropy and Shannons Theorem Lecture 28 December 10, 2013 Sariel - - PowerPoint PPT Presentation
Entropy and Shannons Theorem Lecture 28 December 10, 2013 Sariel - - PowerPoint PPT Presentation
CS 573: Algorithms, Fall 2013 Entropy and Shannons Theorem Lecture 28 December 10, 2013 Sariel (UIUC) CS573 1 Fall 2013 1 / 24 Part I . Entropy . Sariel (UIUC) CS573 2 Fall 2013 2 / 24 Part II . Extracting randomness .
SLIDE 2
SLIDE 3
Part II
. .
Extracting randomness
Sariel (UIUC) CS573 3 Fall 2013 3 / 24
SLIDE 4
Storing all strings of length n and j bits on
. .
1
Sn,j: set of all strings of length n with j ones in them. . .
2
Tn,j: prefix tree storing all Tn,j.
T0,0 T1,1 T1,0
Sariel (UIUC) CS573 4 Fall 2013 4 / 24
SLIDE 5
Prefix tree ∀ binary strings of length n with j ones
Tn,j
1
Tn−1,j−1 Tn−1,j
Tn,0
Tn−1,0
Tn,n
1
Tn−1,n−1
Sariel (UIUC) CS573 5 Fall 2013 5 / 24
SLIDE 6
Prefix tree ∀ binary strings of length n with j ones
Tn,j
1
Tn−1,j−1 Tn−1,j
# of leafs: |Tn,j| = |Tn−1,j| + |Tn−1,j−1| Tn,0
Tn−1,0
Tn,n
1
Tn−1,n−1
Sariel (UIUC) CS573 5 Fall 2013 5 / 24
SLIDE 7
Prefix tree ∀ binary strings of length n with j ones
Tn,j
1
Tn−1,j−1 Tn−1,j
# of leafs: |Tn,j| = |Tn−1,j| + |Tn−1,j−1|
(n
j
)
=
(n−1
j
)
+
(n−1
j−1
)
Tn,0
Tn−1,0
Tn,n
1
Tn−1,n−1
Sariel (UIUC) CS573 5 Fall 2013 5 / 24
SLIDE 8
Prefix tree ∀ binary strings of length n with j ones
Tn,j
1
Tn−1,j−1 Tn−1,j
# of leafs: |Tn,j| = |Tn−1,j| + |Tn−1,j−1|
(n
j
)
=
(n−1
j
)
+
(n−1
j−1
)
= ⇒ |Tn,j| =
(n
j
)
. Tn,0
Tn−1,0
Tn,n
1
Tn−1,n−1
Sariel (UIUC) CS573 5 Fall 2013 5 / 24
SLIDE 9
Encoding a string in Sn,j
. .
1
Tn,j leafs corresponds to strings of Sn,j. . .
2
Order all strings of Sn,j order in lexicographical ordering . .
3
≡ ordering leafs of Tn,j from left to right. . .
4
Input: s ∈ Sn,j: compute index of s in sorted set Sn,j. . .
5
EncodeBinomCoeff(s) denote this polytime procedure.
Sariel (UIUC) CS573 6 Fall 2013 6 / 24
SLIDE 10
Encoding a string in Sn,j
. .
1
Tn,j leafs corresponds to strings of Sn,j. . .
2
Order all strings of Sn,j order in lexicographical ordering . .
3
≡ ordering leafs of Tn,j from left to right. . .
4
Input: s ∈ Sn,j: compute index of s in sorted set Sn,j. . .
5
EncodeBinomCoeff(s) denote this polytime procedure.
Sariel (UIUC) CS573 6 Fall 2013 6 / 24
SLIDE 11
Encoding a string in Sn,j
. .
1
Tn,j leafs corresponds to strings of Sn,j. . .
2
Order all strings of Sn,j order in lexicographical ordering . .
3
≡ ordering leafs of Tn,j from left to right. Tn,j
1
Tn−1,j−1 Tn−1,j
.
4
Input: s ∈ Sn,j: compute index of s in sorted set Sn,j. . .
5
EncodeBinomCoeff(s) denote this polytime procedure.
Sariel (UIUC) CS573 6 Fall 2013 6 / 24
SLIDE 12
Encoding a string in Sn,j
. .
1
Tn,j leafs corresponds to strings of Sn,j. . .
2
Order all strings of Sn,j order in lexicographical ordering . .
3
≡ ordering leafs of Tn,j from left to right. Tn,j
1
Tn−1,j−1 Tn−1,j
.
4
Input: s ∈ Sn,j: compute index of s in sorted set Sn,j. . .
5
EncodeBinomCoeff(s) denote this polytime procedure.
Sariel (UIUC) CS573 6 Fall 2013 6 / 24
SLIDE 13
Encoding a string in Sn,j
. .
1
Tn,j leafs corresponds to strings of Sn,j. . .
2
Order all strings of Sn,j order in lexicographical ordering . .
3
≡ ordering leafs of Tn,j from left to right. Tn,j
1
Tn−1,j−1 Tn−1,j
.
4
Input: s ∈ Sn,j: compute index of s in sorted set Sn,j. . .
5
EncodeBinomCoeff(s) denote this polytime procedure.
Sariel (UIUC) CS573 6 Fall 2013 6 / 24
SLIDE 14
Decoding a string in Sn,j
. .
1
Tn,j leafs corresponds to strings of Sn,j. . .
2
Order all strings of Sn,j order in lexicographical ordering . .
3
≡ ordering leafs of Tn,j from left to right. . .
4
x ∈
{
1, . . . ,
(n
j
)}
: compute xth string in Sn,j in polytime. . .
5
DecodeBinomCoeff (x) denote this procedure.
Sariel (UIUC) CS573 7 Fall 2013 7 / 24
SLIDE 15
Decoding a string in Sn,j
. .
1
Tn,j leafs corresponds to strings of Sn,j. . .
2
Order all strings of Sn,j order in lexicographical ordering . .
3
≡ ordering leafs of Tn,j from left to right. . .
4
x ∈
{
1, . . . ,
(n
j
)}
: compute xth string in Sn,j in polytime. . .
5
DecodeBinomCoeff (x) denote this procedure.
Sariel (UIUC) CS573 7 Fall 2013 7 / 24
SLIDE 16
Decoding a string in Sn,j
. .
1
Tn,j leafs corresponds to strings of Sn,j. . .
2
Order all strings of Sn,j order in lexicographical ordering . .
3
≡ ordering leafs of Tn,j from left to right. Tn,j
1
Tn−1,j−1 Tn−1,j
.
4
x ∈
{
1, . . . ,
(n
j
)}
: compute xth string in Sn,j in polytime. . .
5
DecodeBinomCoeff (x) denote this procedure.
Sariel (UIUC) CS573 7 Fall 2013 7 / 24
SLIDE 17
Decoding a string in Sn,j
. .
1
Tn,j leafs corresponds to strings of Sn,j. . .
2
Order all strings of Sn,j order in lexicographical ordering . .
3
≡ ordering leafs of Tn,j from left to right. Tn,j
1
Tn−1,j−1 Tn−1,j
.
4
x ∈
{
1, . . . ,
(n
j
)}
: compute xth string in Sn,j in polytime. . .
5
DecodeBinomCoeff (x) denote this procedure.
Sariel (UIUC) CS573 7 Fall 2013 7 / 24
SLIDE 18
Decoding a string in Sn,j
. .
1
Tn,j leafs corresponds to strings of Sn,j. . .
2
Order all strings of Sn,j order in lexicographical ordering . .
3
≡ ordering leafs of Tn,j from left to right. Tn,j
1
Tn−1,j−1 Tn−1,j
.
4
x ∈
{
1, . . . ,
(n
j
)}
: compute xth string in Sn,j in polytime. . .
5
DecodeBinomCoeff (x) denote this procedure.
Sariel (UIUC) CS573 7 Fall 2013 7 / 24
SLIDE 19
Encoding/decoding strings of Sn,j
.
Lemma
. . Sn,j: Set of binary strings of length n with j ones, sorted lexicographically. . .
1
EncodeBinomCoeff(α): Input is string α ∈ Sn,j, compute index x of α in Sn,j in polynomial time in n. . .
2
DecodeBinomCoeff(x): Input index x ∈
{
1, . . . ,
(n
j
)}
. Output xth string α in Sn,j, in time O(polylog n + n).
Sariel (UIUC) CS573 8 Fall 2013 8 / 24
SLIDE 20
Extracting randomness
.
Theorem
. . Consider a coin that comes up heads with probability p > 1/2. For any constant δ > 0 and for n sufficiently large: (A) One can extract, from an input of a sequence of n flips, an
- utput sequence of (1 − δ)nH(p) (unbiased) independent
random bits. (B) One can not extract more than nH(p) bits from such a sequence.
Sariel (UIUC) CS573 9 Fall 2013 9 / 24
SLIDE 21
Proof...
. .
1
There are
(n
j
)
input strings with exactly j heads. . .
2
each has probability pj(1 − p)n−j. . .
3
map string s like that to index number in the set Sj =
{
1, . . . ,
(n
j
)}
. . .
4
Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j. . .
5
x ← EncodeBinomCoeff(s) . .
6
x uniform distributed in {1, . . . , N}, N =
(n
j
)
. . .
7
Seen in previous lecture... . .
8
... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N. . .
9
Extract bits using ExtractRandomness(x, N):.
Sariel (UIUC) CS573 10 Fall 2013 10 / 24
SLIDE 22
Proof...
. .
1
There are
(n
j
)
input strings with exactly j heads. . .
2
each has probability pj(1 − p)n−j. . .
3
map string s like that to index number in the set Sj =
{
1, . . . ,
(n
j
)}
. . .
4
Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j. . .
5
x ← EncodeBinomCoeff(s) . .
6
x uniform distributed in {1, . . . , N}, N =
(n
j
)
. . .
7
Seen in previous lecture... . .
8
... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N. . .
9
Extract bits using ExtractRandomness(x, N):.
Sariel (UIUC) CS573 10 Fall 2013 10 / 24
SLIDE 23
Proof...
. .
1
There are
(n
j
)
input strings with exactly j heads. . .
2
each has probability pj(1 − p)n−j. . .
3
map string s like that to index number in the set Sj =
{
1, . . . ,
(n
j
)}
. . .
4
Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j. . .
5
x ← EncodeBinomCoeff(s) . .
6
x uniform distributed in {1, . . . , N}, N =
(n
j
)
. . .
7
Seen in previous lecture... . .
8
... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N. . .
9
Extract bits using ExtractRandomness(x, N):.
Sariel (UIUC) CS573 10 Fall 2013 10 / 24
SLIDE 24
Proof...
. .
1
There are
(n
j
)
input strings with exactly j heads. . .
2
each has probability pj(1 − p)n−j. . .
3
map string s like that to index number in the set Sj =
{
1, . . . ,
(n
j
)}
. . .
4
Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j. . .
5
x ← EncodeBinomCoeff(s) . .
6
x uniform distributed in {1, . . . , N}, N =
(n
j
)
. . .
7
Seen in previous lecture... . .
8
... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N. . .
9
Extract bits using ExtractRandomness(x, N):.
Sariel (UIUC) CS573 10 Fall 2013 10 / 24
SLIDE 25
Proof...
. .
1
There are
(n
j
)
input strings with exactly j heads. . .
2
each has probability pj(1 − p)n−j. . .
3
map string s like that to index number in the set Sj =
{
1, . . . ,
(n
j
)}
. . .
4
Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j. . .
5
x ← EncodeBinomCoeff(s) . .
6
x uniform distributed in {1, . . . , N}, N =
(n
j
)
. . .
7
Seen in previous lecture... . .
8
... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N. . .
9
Extract bits using ExtractRandomness(x, N):.
Sariel (UIUC) CS573 10 Fall 2013 10 / 24
SLIDE 26
Proof...
. .
1
There are
(n
j
)
input strings with exactly j heads. . .
2
each has probability pj(1 − p)n−j. . .
3
map string s like that to index number in the set Sj =
{
1, . . . ,
(n
j
)}
. . .
4
Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j. . .
5
x ← EncodeBinomCoeff(s) . .
6
x uniform distributed in {1, . . . , N}, N =
(n
j
)
. . .
7
Seen in previous lecture... . .
8
... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N. . .
9
Extract bits using ExtractRandomness(x, N):.
Sariel (UIUC) CS573 10 Fall 2013 10 / 24
SLIDE 27
Proof...
. .
1
There are
(n
j
)
input strings with exactly j heads. . .
2
each has probability pj(1 − p)n−j. . .
3
map string s like that to index number in the set Sj =
{
1, . . . ,
(n
j
)}
. . .
4
Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j. . .
5
x ← EncodeBinomCoeff(s) . .
6
x uniform distributed in {1, . . . , N}, N =
(n
j
)
. . .
7
Seen in previous lecture... . .
8
... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N. . .
9
Extract bits using ExtractRandomness(x, N):.
Sariel (UIUC) CS573 10 Fall 2013 10 / 24
SLIDE 28
Proof...
. .
1
There are
(n
j
)
input strings with exactly j heads. . .
2
each has probability pj(1 − p)n−j. . .
3
map string s like that to index number in the set Sj =
{
1, . . . ,
(n
j
)}
. . .
4
Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j. . .
5
x ← EncodeBinomCoeff(s) . .
6
x uniform distributed in {1, . . . , N}, N =
(n
j
)
. . .
7
Seen in previous lecture... . .
8
... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N. . .
9
Extract bits using ExtractRandomness(x, N):.
Sariel (UIUC) CS573 10 Fall 2013 10 / 24
SLIDE 29
Proof...
. .
1
There are
(n
j
)
input strings with exactly j heads. . .
2
each has probability pj(1 − p)n−j. . .
3
map string s like that to index number in the set Sj =
{
1, . . . ,
(n
j
)}
. . .
4
Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j. . .
5
x ← EncodeBinomCoeff(s) . .
6
x uniform distributed in {1, . . . , N}, N =
(n
j
)
. . .
7
Seen in previous lecture... . .
8
... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N. . .
9
Extract bits using ExtractRandomness(x, N):.
Sariel (UIUC) CS573 10 Fall 2013 10 / 24
SLIDE 30
Exciting proof continued...
. .
1
Z: random variable: number of heads in input string s. . .
2
B: number of random bits extracted. E
[
B
]
=
n
∑
k=0
Pr[Z = k] E
[
B
- Z = k
]
, . .
3
Know: E
[
B
- Z = k
]
≥
⌊
lg
(n
k
)⌋
− 1. . .
4
ε < p − 1/2: sufficiently small constant. . .
5
n(p − ε) ≤ k ≤ n(p + ε):
(n
k
)
≥
(
n ⌊n(p + ε)⌋
)
≥ 2nH(p+ε) n + 1 , . .
6
... since 2nH(p) is a good approximation to
( n
np
)
as proved in previous lecture.
Sariel (UIUC) CS573 11 Fall 2013 11 / 24
SLIDE 31
Exciting proof continued...
. .
1
Z: random variable: number of heads in input string s. . .
2
B: number of random bits extracted. E
[
B
]
=
n
∑
k=0
Pr[Z = k] E
[
B
- Z = k
]
, . .
3
Know: E
[
B
- Z = k
]
≥
⌊
lg
(n
k
)⌋
− 1. . .
4
ε < p − 1/2: sufficiently small constant. . .
5
n(p − ε) ≤ k ≤ n(p + ε):
(n
k
)
≥
(
n ⌊n(p + ε)⌋
)
≥ 2nH(p+ε) n + 1 , . .
6
... since 2nH(p) is a good approximation to
( n
np
)
as proved in previous lecture.
Sariel (UIUC) CS573 11 Fall 2013 11 / 24
SLIDE 32
Exciting proof continued...
. .
1
Z: random variable: number of heads in input string s. . .
2
B: number of random bits extracted. E
[
B
]
=
n
∑
k=0
Pr[Z = k] E
[
B
- Z = k
]
, . .
3
Know: E
[
B
- Z = k
]
≥
⌊
lg
(n
k
)⌋
− 1. . .
4
ε < p − 1/2: sufficiently small constant. . .
5
n(p − ε) ≤ k ≤ n(p + ε):
(n
k
)
≥
(
n ⌊n(p + ε)⌋
)
≥ 2nH(p+ε) n + 1 , . .
6
... since 2nH(p) is a good approximation to
( n
np
)
as proved in previous lecture.
Sariel (UIUC) CS573 11 Fall 2013 11 / 24
SLIDE 33
Exciting proof continued...
. .
1
Z: random variable: number of heads in input string s. . .
2
B: number of random bits extracted. E
[
B
]
=
n
∑
k=0
Pr[Z = k] E
[
B
- Z = k
]
, . .
3
Know: E
[
B
- Z = k
]
≥
⌊
lg
(n
k
)⌋
− 1. . .
4
ε < p − 1/2: sufficiently small constant. . .
5
n(p − ε) ≤ k ≤ n(p + ε):
(n
k
)
≥
(
n ⌊n(p + ε)⌋
)
≥ 2nH(p+ε) n + 1 , . .
6
... since 2nH(p) is a good approximation to
( n
np
)
as proved in previous lecture.
Sariel (UIUC) CS573 11 Fall 2013 11 / 24
SLIDE 34
Exciting proof continued...
. .
1
Z: random variable: number of heads in input string s. . .
2
B: number of random bits extracted. E
[
B
]
=
n
∑
k=0
Pr[Z = k] E
[
B
- Z = k
]
, . .
3
Know: E
[
B
- Z = k
]
≥
⌊
lg
(n
k
)⌋
− 1. . .
4
ε < p − 1/2: sufficiently small constant. . .
5
n(p − ε) ≤ k ≤ n(p + ε):
(n
k
)
≥
(
n ⌊n(p + ε)⌋
)
≥ 2nH(p+ε) n + 1 , . .
6
... since 2nH(p) is a good approximation to
( n
np
)
as proved in previous lecture.
Sariel (UIUC) CS573 11 Fall 2013 11 / 24
SLIDE 35
Exciting proof continued...
. .
1
Z: random variable: number of heads in input string s. . .
2
B: number of random bits extracted. E
[
B
]
=
n
∑
k=0
Pr[Z = k] E
[
B
- Z = k
]
, . .
3
Know: E
[
B
- Z = k
]
≥
⌊
lg
(n
k
)⌋
− 1. . .
4
ε < p − 1/2: sufficiently small constant. . .
5
n(p − ε) ≤ k ≤ n(p + ε):
(n
k
)
≥
(
n ⌊n(p + ε)⌋
)
≥ 2nH(p+ε) n + 1 , . .
6
... since 2nH(p) is a good approximation to
( n
np
)
as proved in previous lecture.
Sariel (UIUC) CS573 11 Fall 2013 11 / 24
SLIDE 36
Exciting proof continued...
. .
1
Z: random variable: number of heads in input string s. . .
2
B: number of random bits extracted. E
[
B
]
=
n
∑
k=0
Pr[Z = k] E
[
B
- Z = k
]
, . .
3
Know: E
[
B
- Z = k
]
≥
⌊
lg
(n
k
)⌋
− 1. . .
4
ε < p − 1/2: sufficiently small constant. . .
5
n(p − ε) ≤ k ≤ n(p + ε):
(n
k
)
≥
(
n ⌊n(p + ε)⌋
)
≥ 2nH(p+ε) n + 1 , . .
6
... since 2nH(p) is a good approximation to
( n
np
)
as proved in previous lecture.
Sariel (UIUC) CS573 11 Fall 2013 11 / 24
SLIDE 37
Super exciting proof continued...
E
[
B
]
= ∑n
k=0 Pr[Z = k] E
[
B
- Z = k
]
. E
[
B
]
≥
∑⌈n(p+ε)⌉
k=⌊n(p−ε)⌋ Pr
[
Z = k
]
E
[
B
- Z = k
]
≥
⌈n(p+ε)⌉
∑
k=⌊n(p−ε)⌋
Pr
[
Z = k
] (⌊
lg
(n
k
)⌋
− 1
)
≥
⌈n(p+ε)⌉
∑
k=⌊n(p−ε)⌋
Pr
[
Z = k
](
lg 2nH(p+ε) n + 1 − 2
)
=
(
nH(p + ε) − lg(n + 1) − 2
)
Pr[|Z − np| ≤ εn] ≥
(
nH(p + ε) − lg(n + 1) − 2
)(
1 − 2 exp
(
−nε2 4p
))
, since µ = E[Z] = np and Pr
[
|Z − np| ≥ ε
ppn
]
≤ 2 exp
(
− np
4
(
ε p
)2)
= 2 exp
(
−nε2
4p
)
, by the Chernoff inequality.
Sariel (UIUC) CS573 12 Fall 2013 12 / 24
SLIDE 38
Super exciting proof continued...
E
[
B
]
= ∑n
k=0 Pr[Z = k] E
[
B
- Z = k
]
. E
[
B
]
≥
∑⌈n(p+ε)⌉
k=⌊n(p−ε)⌋ Pr
[
Z = k
]
E
[
B
- Z = k
]
≥
⌈n(p+ε)⌉
∑
k=⌊n(p−ε)⌋
Pr
[
Z = k
] (⌊
lg
(n
k
)⌋
− 1
)
≥
⌈n(p+ε)⌉
∑
k=⌊n(p−ε)⌋
Pr
[
Z = k
](
lg 2nH(p+ε) n + 1 − 2
)
=
(
nH(p + ε) − lg(n + 1) − 2
)
Pr[|Z − np| ≤ εn] ≥
(
nH(p + ε) − lg(n + 1) − 2
)(
1 − 2 exp
(
−nε2 4p
))
, since µ = E[Z] = np and Pr
[
|Z − np| ≥ ε
ppn
]
≤ 2 exp
(
− np
4
(
ε p
)2)
= 2 exp
(
−nε2
4p
)
, by the Chernoff inequality.
Sariel (UIUC) CS573 12 Fall 2013 12 / 24
SLIDE 39
Super exciting proof continued...
E
[
B
]
= ∑n
k=0 Pr[Z = k] E
[
B
- Z = k
]
. E
[
B
]
≥
∑⌈n(p+ε)⌉
k=⌊n(p−ε)⌋ Pr
[
Z = k
]
E
[
B
- Z = k
]
≥
⌈n(p+ε)⌉
∑
k=⌊n(p−ε)⌋
Pr
[
Z = k
] (⌊
lg
(n
k
)⌋
− 1
)
≥
⌈n(p+ε)⌉
∑
k=⌊n(p−ε)⌋
Pr
[
Z = k
](
lg 2nH(p+ε) n + 1 − 2
)
=
(
nH(p + ε) − lg(n + 1) − 2
)
Pr[|Z − np| ≤ εn] ≥
(
nH(p + ε) − lg(n + 1) − 2
)(
1 − 2 exp
(
−nε2 4p
))
, since µ = E[Z] = np and Pr
[
|Z − np| ≥ ε
ppn
]
≤ 2 exp
(
− np
4
(
ε p
)2)
= 2 exp
(
−nε2
4p
)
, by the Chernoff inequality.
Sariel (UIUC) CS573 12 Fall 2013 12 / 24
SLIDE 40
Super exciting proof continued...
E
[
B
]
= ∑n
k=0 Pr[Z = k] E
[
B
- Z = k
]
. E
[
B
]
≥
∑⌈n(p+ε)⌉
k=⌊n(p−ε)⌋ Pr
[
Z = k
]
E
[
B
- Z = k
]
≥
⌈n(p+ε)⌉
∑
k=⌊n(p−ε)⌋
Pr
[
Z = k
] (⌊
lg
(n
k
)⌋
− 1
)
≥
⌈n(p+ε)⌉
∑
k=⌊n(p−ε)⌋
Pr
[
Z = k
](
lg 2nH(p+ε) n + 1 − 2
)
=
(
nH(p + ε) − lg(n + 1) − 2
)
Pr[|Z − np| ≤ εn] ≥
(
nH(p + ε) − lg(n + 1) − 2
)(
1 − 2 exp
(
−nε2 4p
))
, since µ = E[Z] = np and Pr
[
|Z − np| ≥ ε
ppn
]
≤ 2 exp
(
− np
4
(
ε p
)2)
= 2 exp
(
−nε2
4p
)
, by the Chernoff inequality.
Sariel (UIUC) CS573 12 Fall 2013 12 / 24
SLIDE 41
Super exciting proof continued...
E
[
B
]
= ∑n
k=0 Pr[Z = k] E
[
B
- Z = k
]
. E
[
B
]
≥
∑⌈n(p+ε)⌉
k=⌊n(p−ε)⌋ Pr
[
Z = k
]
E
[
B
- Z = k
]
≥
⌈n(p+ε)⌉
∑
k=⌊n(p−ε)⌋
Pr
[
Z = k
] (⌊
lg
(n
k
)⌋
− 1
)
≥
⌈n(p+ε)⌉
∑
k=⌊n(p−ε)⌋
Pr
[
Z = k
](
lg 2nH(p+ε) n + 1 − 2
)
=
(
nH(p + ε) − lg(n + 1) − 2
)
Pr[|Z − np| ≤ εn] ≥
(
nH(p + ε) − lg(n + 1) − 2
)(
1 − 2 exp
(
−nε2 4p
))
, since µ = E[Z] = np and Pr
[
|Z − np| ≥ ε
ppn
]
≤ 2 exp
(
− np
4
(
ε p
)2)
= 2 exp
(
−nε2
4p
)
, by the Chernoff inequality.
Sariel (UIUC) CS573 12 Fall 2013 12 / 24
SLIDE 42
Super exciting proof continued...
E
[
B
]
= ∑n
k=0 Pr[Z = k] E
[
B
- Z = k
]
. E
[
B
]
≥
∑⌈n(p+ε)⌉
k=⌊n(p−ε)⌋ Pr
[
Z = k
]
E
[
B
- Z = k
]
≥
⌈n(p+ε)⌉
∑
k=⌊n(p−ε)⌋
Pr
[
Z = k
] (⌊
lg
(n
k
)⌋
− 1
)
≥
⌈n(p+ε)⌉
∑
k=⌊n(p−ε)⌋
Pr
[
Z = k
](
lg 2nH(p+ε) n + 1 − 2
)
=
(
nH(p + ε) − lg(n + 1) − 2
)
Pr[|Z − np| ≤ εn] ≥
(
nH(p + ε) − lg(n + 1) − 2
)(
1 − 2 exp
(
−nε2 4p
))
, since µ = E[Z] = np and Pr
[
|Z − np| ≥ ε
ppn
]
≤ 2 exp
(
− np
4
(
ε p
)2)
= 2 exp
(
−nε2
4p
)
, by the Chernoff inequality.
Sariel (UIUC) CS573 12 Fall 2013 12 / 24
SLIDE 43
Super exciting proof continued...
E
[
B
]
= ∑n
k=0 Pr[Z = k] E
[
B
- Z = k
]
. E
[
B
]
≥
∑⌈n(p+ε)⌉
k=⌊n(p−ε)⌋ Pr
[
Z = k
]
E
[
B
- Z = k
]
≥
⌈n(p+ε)⌉
∑
k=⌊n(p−ε)⌋
Pr
[
Z = k
] (⌊
lg
(n
k
)⌋
− 1
)
≥
⌈n(p+ε)⌉
∑
k=⌊n(p−ε)⌋
Pr
[
Z = k
](
lg 2nH(p+ε) n + 1 − 2
)
=
(
nH(p + ε) − lg(n + 1) − 2
)
Pr[|Z − np| ≤ εn] ≥
(
nH(p + ε) − lg(n + 1) − 2
)(
1 − 2 exp
(
−nε2 4p
))
, since µ = E[Z] = np and Pr
[
|Z − np| ≥ ε
ppn
]
≤ 2 exp
(
− np
4
(
ε p
)2)
= 2 exp
(
−nε2
4p
)
, by the Chernoff inequality.
Sariel (UIUC) CS573 12 Fall 2013 12 / 24
SLIDE 44
Hyper super exciting proof continued...
. .
1
Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is fixed. . .
2
= ⇒ nH(p) = Ω(n), . .
3
For n sufficiently large: − lg(n + 1) ≥ − δ
10nH(p).
. .
4
... also 2 exp
(
− nε2
4p
)
≤
δ 10.
. .
5
For n large enough; E[B] ≥
(
1 − δ 4 − δ 10
)
nH(p)
(
1 − δ 10
)
≥(1 − δ) nH(p) ,
Sariel (UIUC) CS573 13 Fall 2013 13 / 24
SLIDE 45
Hyper super exciting proof continued...
. .
1
Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is fixed. . .
2
= ⇒ nH(p) = Ω(n), . .
3
For n sufficiently large: − lg(n + 1) ≥ − δ
10nH(p).
. .
4
... also 2 exp
(
− nε2
4p
)
≤
δ 10.
. .
5
For n large enough; E[B] ≥
(
1 − δ 4 − δ 10
)
nH(p)
(
1 − δ 10
)
≥(1 − δ) nH(p) ,
Sariel (UIUC) CS573 13 Fall 2013 13 / 24
SLIDE 46
Hyper super exciting proof continued...
. .
1
Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is fixed. . .
2
= ⇒ nH(p) = Ω(n), . .
3
For n sufficiently large: − lg(n + 1) ≥ − δ
10nH(p).
. .
4
... also 2 exp
(
− nε2
4p
)
≤
δ 10.
. .
5
For n large enough; E[B] ≥
(
1 − δ 4 − δ 10
)
nH(p)
(
1 − δ 10
)
≥(1 − δ) nH(p) ,
Sariel (UIUC) CS573 13 Fall 2013 13 / 24
SLIDE 47
Hyper super exciting proof continued...
. .
1
Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is fixed. . .
2
= ⇒ nH(p) = Ω(n), . .
3
For n sufficiently large: − lg(n + 1) ≥ − δ
10nH(p).
. .
4
... also 2 exp
(
− nε2
4p
)
≤
δ 10.
. .
5
For n large enough; E[B] ≥
(
1 − δ 4 − δ 10
)
nH(p)
(
1 − δ 10
)
≥(1 − δ) nH(p) ,
Sariel (UIUC) CS573 13 Fall 2013 13 / 24
SLIDE 48
Hyper super exciting proof continued...
. .
1
Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is fixed. . .
2
= ⇒ nH(p) = Ω(n), . .
3
For n sufficiently large: − lg(n + 1) ≥ − δ
10nH(p).
. .
4
... also 2 exp
(
− nε2
4p
)
≤
δ 10.
. .
5
For n large enough; E[B] ≥
(
1 − δ 4 − δ 10
)
nH(p)
(
1 − δ 10
)
≥(1 − δ) nH(p) ,
Sariel (UIUC) CS573 13 Fall 2013 13 / 24
SLIDE 49
Hyper super duper exciting proof continued...
. .
1
Need to prove upper bound. . .
2
If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. . .
3
All sequences of length |y| have equal probability to be generated (by definition). . .
4
2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. . .
5
= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) . .
6
E
[
B
]
= ∑
x Pr
[
X = x
]
|Ext(x)| ≤ ∑
x Pr
[
X = x
]
lg
1 Pr [X=x] = H(X) .
Sariel (UIUC) CS573 14 Fall 2013 14 / 24
SLIDE 50
Hyper super duper exciting proof continued...
. .
1
Need to prove upper bound. . .
2
If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. . .
3
All sequences of length |y| have equal probability to be generated (by definition). . .
4
2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. . .
5
= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) . .
6
E
[
B
]
= ∑
x Pr
[
X = x
]
|Ext(x)| ≤ ∑
x Pr
[
X = x
]
lg
1 Pr [X=x] = H(X) .
Sariel (UIUC) CS573 14 Fall 2013 14 / 24
SLIDE 51
Hyper super duper exciting proof continued...
. .
1
Need to prove upper bound. . .
2
If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. . .
3
All sequences of length |y| have equal probability to be generated (by definition). . .
4
2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. . .
5
= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) . .
6
E
[
B
]
= ∑
x Pr
[
X = x
]
|Ext(x)| ≤ ∑
x Pr
[
X = x
]
lg
1 Pr [X=x] = H(X) .
Sariel (UIUC) CS573 14 Fall 2013 14 / 24
SLIDE 52
Hyper super duper exciting proof continued...
. .
1
Need to prove upper bound. . .
2
If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. . .
3
All sequences of length |y| have equal probability to be generated (by definition). . .
4
2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. . .
5
= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) . .
6
E
[
B
]
= ∑
x Pr
[
X = x
]
|Ext(x)| ≤ ∑
x Pr
[
X = x
]
lg
1 Pr [X=x] = H(X) .
Sariel (UIUC) CS573 14 Fall 2013 14 / 24
SLIDE 53
Hyper super duper exciting proof continued...
. .
1
Need to prove upper bound. . .
2
If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. . .
3
All sequences of length |y| have equal probability to be generated (by definition). . .
4
2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. . .
5
= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) . .
6
E
[
B
]
= ∑
x Pr
[
X = x
]
|Ext(x)| ≤ ∑
x Pr
[
X = x
]
lg
1 Pr [X=x] = H(X) .
Sariel (UIUC) CS573 14 Fall 2013 14 / 24
SLIDE 54
Hyper super duper exciting proof continued...
. .
1
Need to prove upper bound. . .
2
If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. . .
3
All sequences of length |y| have equal probability to be generated (by definition). . .
4
2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. . .
5
= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) . .
6
E
[
B
]
= ∑
x Pr
[
X = x
]
|Ext(x)| ≤ ∑
x Pr
[
X = x
]
lg
1 Pr [X=x] = H(X) .
Sariel (UIUC) CS573 14 Fall 2013 14 / 24
SLIDE 55
Hyper super duper exciting proof continued...
. .
1
Need to prove upper bound. . .
2
If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. . .
3
All sequences of length |y| have equal probability to be generated (by definition). . .
4
2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. . .
5
= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) . .
6
E
[
B
]
= ∑
x Pr
[
X = x
]
|Ext(x)| ≤ ∑
x Pr
[
X = x
]
lg
1 Pr [X=x] = H(X) .
Sariel (UIUC) CS573 14 Fall 2013 14 / 24
SLIDE 56
Hyper super duper exciting proof continued...
. .
1
Need to prove upper bound. . .
2
If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. . .
3
All sequences of length |y| have equal probability to be generated (by definition). . .
4
2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. . .
5
= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) . .
6
E
[
B
]
= ∑
x Pr
[
X = x
]
|Ext(x)| ≤ ∑
x Pr
[
X = x
]
lg
1 Pr [X=x] = H(X) .
Sariel (UIUC) CS573 14 Fall 2013 14 / 24
SLIDE 57
Hyper super duper exciting proof continued...
. .
1
Need to prove upper bound. . .
2
If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x]. . .
3
All sequences of length |y| have equal probability to be generated (by definition). . .
4
2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1. . .
5
= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x]) . .
6
E
[
B
]
= ∑
x Pr
[
X = x
]
|Ext(x)| ≤ ∑
x Pr
[
X = x
]
lg
1 Pr [X=x] = H(X) .
Sariel (UIUC) CS573 14 Fall 2013 14 / 24
SLIDE 58
Part III
. .
Coding: Shannon’s Theorem
Sariel (UIUC) CS573 15 Fall 2013 15 / 24
SLIDE 59
Shannon’s Theorem
.
Definition
. . The input to a binary symmetric channel with parameter p is a sequence of bits x1, x2, . . . , and the output is a sequence of bits y1, y2, . . . , such that Pr[xi = yi] = 1 − p independently for each i.
Sariel (UIUC) CS573 16 Fall 2013 16 / 24
SLIDE 60
Encoding/decoding with noise
.
Definition
. . A (k, n) encoding function Enc : {0, 1}k → {0, 1}n takes as input a sequence of k bits and outputs a sequence of n bits. A (k, n) decoding function Dec : {0, 1}n → {0, 1}k takes as input a sequence of n bits and outputs a sequence of k bits.
Sariel (UIUC) CS573 17 Fall 2013 17 / 24
SLIDE 61
Claude Elwood Shannon
Claude Elwood Shannon (April 30, 1916 - February 24, 2001), an American electrical engineer and mathematician, has been called “the father of information theory”. His master thesis was how to building boolean circuits for any boolean function.
Sariel (UIUC) CS573 18 Fall 2013 18 / 24
SLIDE 62
Shannon’s theorem (1948)
.
Theorem (Shannon’s theorem)
. . For a binary symmetric channel with parameter p < 1/2 and for any constants δ, γ > 0, where n is sufficiently large, the following holds: (i) For an k ≤ n(1 − H(p) − δ) there exists (k, n) encoding and decoding functions such that the probability the receiver fails to obtain the correct message is at most γ for every possible k-bit input messages. (ii) There are no (k, n) encoding and decoding functions with k ≥ n(1 − H(p) + δ) such that the probability of decoding correctly is at least γ for a k-bit input message chosen uniformly at random.
Sariel (UIUC) CS573 19 Fall 2013 19 / 24
SLIDE 63
When the sender sends a string...
S = s1s2 . . . sn
S
Sariel (UIUC) CS573 20 Fall 2013 20 / 24
SLIDE 64
When the sender sends a string...
S = s1s2 . . . sn
S
np
Sariel (UIUC) CS573 20 Fall 2013 20 / 24
SLIDE 65
When the sender sends a string...
S = s1s2 . . . sn
S
np
Sariel (UIUC) CS573 20 Fall 2013 20 / 24
SLIDE 66
When the sender sends a string...
S = s1s2 . . . sn
S
np
(1 − δ)np (1 + δ)np
Sariel (UIUC) CS573 20 Fall 2013 20 / 24
SLIDE 67
When the sender sends a string...
S = s1s2 . . . sn
S
np
(1 − δ)np (1 + δ)np
One ring to rule them all!
Sariel (UIUC) CS573 20 Fall 2013 20 / 24
SLIDE 68
Some intuition...
. .
1
senders sent string S = s1s2 . . . sn. . .
2
receiver got string T = t1t2 . . . tn. . .
3
p = Pr[ti ̸= si], for all i. . .
4
U: Hamming distance between S and T: U = ∑
i
[
si ̸= ti
]
. . .
5
By assumption: E[U] = pn, and U is a binomial variable. . .
6
By Chernoff inequality: U ∈
[
(1 − δ)np, (1 + δ)np
]
with high probability, where δ is tiny constant. . .
7
T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np. . .
8
This ring has
(1+δ)np
∑
i=(1−δ)np
(n
i
)
≤ 2
(
n (1 + δ)np
)
≤ α = 2 · 2nH((1+δ)p). strings in it.
Sariel (UIUC) CS573 21 Fall 2013 21 / 24
SLIDE 69
Some intuition...
. .
1
senders sent string S = s1s2 . . . sn. . .
2
receiver got string T = t1t2 . . . tn. . .
3
p = Pr[ti ̸= si], for all i. . .
4
U: Hamming distance between S and T: U = ∑
i
[
si ̸= ti
]
. . .
5
By assumption: E[U] = pn, and U is a binomial variable. . .
6
By Chernoff inequality: U ∈
[
(1 − δ)np, (1 + δ)np
]
with high probability, where δ is tiny constant. . .
7
T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np. . .
8
This ring has
(1+δ)np
∑
i=(1−δ)np
(n
i
)
≤ 2
(
n (1 + δ)np
)
≤ α = 2 · 2nH((1+δ)p). strings in it.
Sariel (UIUC) CS573 21 Fall 2013 21 / 24
SLIDE 70
Some intuition...
. .
1
senders sent string S = s1s2 . . . sn. . .
2
receiver got string T = t1t2 . . . tn. . .
3
p = Pr[ti ̸= si], for all i. . .
4
U: Hamming distance between S and T: U = ∑
i
[
si ̸= ti
]
. . .
5
By assumption: E[U] = pn, and U is a binomial variable. . .
6
By Chernoff inequality: U ∈
[
(1 − δ)np, (1 + δ)np
]
with high probability, where δ is tiny constant. . .
7
T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np. . .
8
This ring has
(1+δ)np
∑
i=(1−δ)np
(n
i
)
≤ 2
(
n (1 + δ)np
)
≤ α = 2 · 2nH((1+δ)p). strings in it.
Sariel (UIUC) CS573 21 Fall 2013 21 / 24
SLIDE 71
Some intuition...
. .
1
senders sent string S = s1s2 . . . sn. . .
2
receiver got string T = t1t2 . . . tn. . .
3
p = Pr[ti ̸= si], for all i. . .
4
U: Hamming distance between S and T: U = ∑
i
[
si ̸= ti
]
. . .
5
By assumption: E[U] = pn, and U is a binomial variable. . .
6
By Chernoff inequality: U ∈
[
(1 − δ)np, (1 + δ)np
]
with high probability, where δ is tiny constant. . .
7
T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np. . .
8
This ring has
(1+δ)np
∑
i=(1−δ)np
(n
i
)
≤ 2
(
n (1 + δ)np
)
≤ α = 2 · 2nH((1+δ)p). strings in it.
Sariel (UIUC) CS573 21 Fall 2013 21 / 24
SLIDE 72
Some intuition...
. .
1
senders sent string S = s1s2 . . . sn. . .
2
receiver got string T = t1t2 . . . tn. . .
3
p = Pr[ti ̸= si], for all i. . .
4
U: Hamming distance between S and T: U = ∑
i
[
si ̸= ti
]
. . .
5
By assumption: E[U] = pn, and U is a binomial variable. . .
6
By Chernoff inequality: U ∈
[
(1 − δ)np, (1 + δ)np
]
with high probability, where δ is tiny constant. . .
7
T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np. . .
8
This ring has
(1+δ)np
∑
i=(1−δ)np
(n
i
)
≤ 2
(
n (1 + δ)np
)
≤ α = 2 · 2nH((1+δ)p). strings in it.
Sariel (UIUC) CS573 21 Fall 2013 21 / 24
SLIDE 73
Some intuition...
. .
1
senders sent string S = s1s2 . . . sn. . .
2
receiver got string T = t1t2 . . . tn. . .
3
p = Pr[ti ̸= si], for all i. . .
4
U: Hamming distance between S and T: U = ∑
i
[
si ̸= ti
]
. . .
5
By assumption: E[U] = pn, and U is a binomial variable. . .
6
By Chernoff inequality: U ∈
[
(1 − δ)np, (1 + δ)np
]
with high probability, where δ is tiny constant. . .
7
T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np. . .
8
This ring has
(1+δ)np
∑
i=(1−δ)np
(n
i
)
≤ 2
(
n (1 + δ)np
)
≤ α = 2 · 2nH((1+δ)p). strings in it.
Sariel (UIUC) CS573 21 Fall 2013 21 / 24
SLIDE 74
Some intuition...
. .
1
senders sent string S = s1s2 . . . sn. . .
2
receiver got string T = t1t2 . . . tn. . .
3
p = Pr[ti ̸= si], for all i. . .
4
U: Hamming distance between S and T: U = ∑
i
[
si ̸= ti
]
. . .
5
By assumption: E[U] = pn, and U is a binomial variable. . .
6
By Chernoff inequality: U ∈
[
(1 − δ)np, (1 + δ)np
]
with high probability, where δ is tiny constant. . .
7
T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np. . .
8
This ring has
(1+δ)np
∑
i=(1−δ)np
(n
i
)
≤ 2
(
n (1 + δ)np
)
≤ α = 2 · 2nH((1+δ)p). strings in it.
Sariel (UIUC) CS573 21 Fall 2013 21 / 24
SLIDE 75
Some intuition...
. .
1
senders sent string S = s1s2 . . . sn. . .
2
receiver got string T = t1t2 . . . tn. . .
3
p = Pr[ti ̸= si], for all i. . .
4
U: Hamming distance between S and T: U = ∑
i
[
si ̸= ti
]
. . .
5
By assumption: E[U] = pn, and U is a binomial variable. . .
6
By Chernoff inequality: U ∈
[
(1 − δ)np, (1 + δ)np
]
with high probability, where δ is tiny constant. . .
7
T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np. . .
8
This ring has
(1+δ)np
∑
i=(1−δ)np
(n
i
)
≤ 2
(
n (1 + δ)np
)
≤ α = 2 · 2nH((1+δ)p). strings in it.
Sariel (UIUC) CS573 21 Fall 2013 21 / 24
SLIDE 76
Many rings for many codewords...
Sariel (UIUC) CS573 22 Fall 2013 22 / 24
SLIDE 77
Some more intuition...
. .
1
Pick as many disjoint rings as possible: R1, . . . , Rκ. . .
2
If every word in the hypercube would be covered... .
3
... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)). . .
4
Consider all possible strings of length k such that 2k ≤ κ. . .
5
Map ith string in {0, 1}k to the center Ci of the ith ring Ri. . .
6
If send Ci = ⇒ receiver gets a string in Ri. . .
7
Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to. . .
8
How many bits? k = ⌊log κ⌋ = n
(
1 − H
(
(1 + δ)p
))
≈ n(1 − H (p)),
Sariel (UIUC) CS573 23 Fall 2013 23 / 24
SLIDE 78
Some more intuition...
. .
1
Pick as many disjoint rings as possible: R1, . . . , Rκ. . .
2
If every word in the hypercube would be covered... .
3
... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)). . .
4
Consider all possible strings of length k such that 2k ≤ κ. . .
5
Map ith string in {0, 1}k to the center Ci of the ith ring Ri. . .
6
If send Ci = ⇒ receiver gets a string in Ri. . .
7
Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to. . .
8
How many bits? k = ⌊log κ⌋ = n
(
1 − H
(
(1 + δ)p
))
≈ n(1 − H (p)),
Sariel (UIUC) CS573 23 Fall 2013 23 / 24
SLIDE 79
Some more intuition...
. .
1
Pick as many disjoint rings as possible: R1, . . . , Rκ. . .
2
If every word in the hypercube would be covered... .
3
... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)). . .
4
Consider all possible strings of length k such that 2k ≤ κ. . .
5
Map ith string in {0, 1}k to the center Ci of the ith ring Ri. . .
6
If send Ci = ⇒ receiver gets a string in Ri. . .
7
Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to. . .
8
How many bits? k = ⌊log κ⌋ = n
(
1 − H
(
(1 + δ)p
))
≈ n(1 − H (p)),
Sariel (UIUC) CS573 23 Fall 2013 23 / 24
SLIDE 80
Some more intuition...
. .
1
Pick as many disjoint rings as possible: R1, . . . , Rκ. . .
2
If every word in the hypercube would be covered... .
3
... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)). . .
4
Consider all possible strings of length k such that 2k ≤ κ. . .
5
Map ith string in {0, 1}k to the center Ci of the ith ring Ri. . .
6
If send Ci = ⇒ receiver gets a string in Ri. . .
7
Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to. . .
8
How many bits? k = ⌊log κ⌋ = n
(
1 − H
(
(1 + δ)p
))
≈ n(1 − H (p)),
Sariel (UIUC) CS573 23 Fall 2013 23 / 24
SLIDE 81
Some more intuition...
. .
1
Pick as many disjoint rings as possible: R1, . . . , Rκ. . .
2
If every word in the hypercube would be covered... .
3
... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)). . .
4
Consider all possible strings of length k such that 2k ≤ κ. . .
5
Map ith string in {0, 1}k to the center Ci of the ith ring Ri. . .
6
If send Ci = ⇒ receiver gets a string in Ri. . .
7
Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to. . .
8
How many bits? k = ⌊log κ⌋ = n
(
1 − H
(
(1 + δ)p
))
≈ n(1 − H (p)),
Sariel (UIUC) CS573 23 Fall 2013 23 / 24
SLIDE 82
Some more intuition...
. .
1
Pick as many disjoint rings as possible: R1, . . . , Rκ. . .
2
If every word in the hypercube would be covered... .
3
... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)). . .
4
Consider all possible strings of length k such that 2k ≤ κ. . .
5
Map ith string in {0, 1}k to the center Ci of the ith ring Ri. . .
6
If send Ci = ⇒ receiver gets a string in Ri. . .
7
Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to. . .
8
How many bits? k = ⌊log κ⌋ = n
(
1 − H
(
(1 + δ)p
))
≈ n(1 − H (p)),
Sariel (UIUC) CS573 23 Fall 2013 23 / 24
SLIDE 83
Some more intuition...
. .
1
Pick as many disjoint rings as possible: R1, . . . , Rκ. . .
2
If every word in the hypercube would be covered... .
3
... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)). . .
4
Consider all possible strings of length k such that 2k ≤ κ. . .
5
Map ith string in {0, 1}k to the center Ci of the ith ring Ri. . .
6
If send Ci = ⇒ receiver gets a string in Ri. . .
7
Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to. . .
8
How many bits? k = ⌊log κ⌋ = n
(
1 − H
(
(1 + δ)p
))
≈ n(1 − H (p)),
Sariel (UIUC) CS573 23 Fall 2013 23 / 24
SLIDE 84
Some more intuition...
. .
1
Pick as many disjoint rings as possible: R1, . . . , Rκ. . .
2
If every word in the hypercube would be covered... .
3
... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)). . .
4
Consider all possible strings of length k such that 2k ≤ κ. . .
5
Map ith string in {0, 1}k to the center Ci of the ith ring Ri. . .
6
If send Ci = ⇒ receiver gets a string in Ri. . .
7
Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to. . .
8
How many bits? k = ⌊log κ⌋ = n
(
1 − H
(
(1 + δ)p
))
≈ n(1 − H (p)),
Sariel (UIUC) CS573 23 Fall 2013 23 / 24
SLIDE 85
Some more intuition...
. .
1
Pick as many disjoint rings as possible: R1, . . . , Rκ. . .
2
If every word in the hypercube would be covered... .
3
... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)). . .
4
Consider all possible strings of length k such that 2k ≤ κ. . .
5
Map ith string in {0, 1}k to the center Ci of the ith ring Ri. . .
6
If send Ci = ⇒ receiver gets a string in Ri. . .
7
Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to. . .
8
How many bits? k = ⌊log κ⌋ = n
(
1 − H
(
(1 + δ)p
))
≈ n(1 − H (p)),
Sariel (UIUC) CS573 23 Fall 2013 23 / 24
SLIDE 86
What is wrong with the above?
. .
1
Can not find such a large set of disjoint rings. . .
2
Reason is that when you pack rings (or balls) you are going to have wasted spaces around. . .
3
Overcome this: allow rings to overlap somewhat. . .
4
Makes things considerably more involved. . .
5
Details in class notes.
Sariel (UIUC) CS573 24 Fall 2013 24 / 24
SLIDE 87
What is wrong with the above?
. .
1
Can not find such a large set of disjoint rings. . .
2
Reason is that when you pack rings (or balls) you are going to have wasted spaces around. . .
3
Overcome this: allow rings to overlap somewhat. . .
4
Makes things considerably more involved. . .
5
Details in class notes.
Sariel (UIUC) CS573 24 Fall 2013 24 / 24
SLIDE 88
What is wrong with the above?
. .
1
Can not find such a large set of disjoint rings. . .
2
Reason is that when you pack rings (or balls) you are going to have wasted spaces around. . .
3
Overcome this: allow rings to overlap somewhat. . .
4
Makes things considerably more involved. . .
5
Details in class notes.
Sariel (UIUC) CS573 24 Fall 2013 24 / 24
SLIDE 89
What is wrong with the above?
. .
1
Can not find such a large set of disjoint rings. . .
2
Reason is that when you pack rings (or balls) you are going to have wasted spaces around. . .
3
Overcome this: allow rings to overlap somewhat. . .
4
Makes things considerably more involved. . .
5
Details in class notes.
Sariel (UIUC) CS573 24 Fall 2013 24 / 24
SLIDE 90
What is wrong with the above?
. .
1
Can not find such a large set of disjoint rings. . .
2
Reason is that when you pack rings (or balls) you are going to have wasted spaces around. . .
3
Overcome this: allow rings to overlap somewhat. . .
4
Makes things considerably more involved. . .
5
Details in class notes.
Sariel (UIUC) CS573 24 Fall 2013 24 / 24
SLIDE 91
Notes
Sariel (UIUC) CS573 25 Fall 2013 25 / 24
SLIDE 92
Notes
Sariel (UIUC) CS573 26 Fall 2013 26 / 24
SLIDE 93
Notes
Sariel (UIUC) CS573 27 Fall 2013 27 / 24
SLIDE 94
Notes
Sariel (UIUC) CS573 28 Fall 2013 28 / 24