[PPT] - Entropy and Shannons Theorem Lecture 24 November 18, 2015 Sariel PowerPoint Presentation

SLIDE 1

NEW CS 473: Theory II, Fall 2015

Entropy and Shannon’s Theorem

Lecture 24

November 18, 2015

Sariel (UIUC) New CS473 1 Fall 2015 1 / 25

SLIDE 2

Part I Entropy

Sariel (UIUC) New CS473 2 Fall 2015 2 / 25

SLIDE 3

Part II Extracting randomness

Sariel (UIUC) New CS473 3 Fall 2015 3 / 25

SLIDE 4

Storing all strings of length n and j bits on

1

Sn,j: set of all strings of length n with j ones in them.

2

Tn,j: prefix tree storing all Sn,j.

T0,0 T1,1 T1,0

Sariel (UIUC) New CS473 4 Fall 2015 4 / 25

SLIDE 5

Binary strings of length 4

1 Sariel (UIUC) New CS473 5 Fall 2015 5 / 25

SLIDE 6

Binary strings of length 4

1

S4,0 = {0000} = ⇒ #(0000) = 0.

2 Sariel (UIUC) New CS473 5 Fall 2015 5 / 25

SLIDE 7

Binary strings of length 4

1

S4,1 = {0001, 0010, 0100, 1000} = ⇒ #(0001) = 0. #(0010) = 1. #(0100) = 2. #(1000) = 3.

2 Sariel (UIUC) New CS473 5 Fall 2015 5 / 25

SLIDE 8

Binary strings of length 4

1

S4,2 = {0011, 0101, 0110, 1001, 1010, 1100} = ⇒ #(0011) = 0. #(0101) = 1. #(0110) = 2. #(1001) = 3. #(1010) = 4. #(1100) = 5.

2 Sariel (UIUC) New CS473 5 Fall 2015 5 / 25

SLIDE 9

Binary strings of length 4

1

S4,3 = {0111, 1011, 1101, 1110} = ⇒ #(0111) = 0. #(1011) = 1. #(1101) = 2. #(1110) = 3.

2 Sariel (UIUC) New CS473 5 Fall 2015 5 / 25

SLIDE 10

Binary strings of length 4

1

S4,4 = {1111} = ⇒ #(1111) = 0.

2 Sariel (UIUC) New CS473 5 Fall 2015 5 / 25

SLIDE 11

Prefix tree ∀ binary strings of length n with j ones

Tn,j

1

Tn−1,j−1 Tn−1,j

Tn,0

Tn−1,0

Tn,n

1

Tn−1,n−1

Sariel (UIUC) New CS473 6 Fall 2015 6 / 25

SLIDE 12

Prefix tree ∀ binary strings of length n with j ones

Tn,j

1

Tn−1,j−1 Tn−1,j

# of leafs: |Tn,j| = |Tn−1,j| + |Tn−1,j−1| Tn,0

Tn−1,0

Tn,n

1

Tn−1,n−1

Sariel (UIUC) New CS473 6 Fall 2015 6 / 25

SLIDE 13

Prefix tree ∀ binary strings of length n with j ones

Tn,j

1

Tn−1,j−1 Tn−1,j

# of leafs: |Tn,j| = |Tn−1,j| + |Tn−1,j−1| n

j

=

n−1

j

+

n−1

j−1

Tn,0

Tn−1,0

Tn,n

1

Tn−1,n−1

Sariel (UIUC) New CS473 6 Fall 2015 6 / 25

SLIDE 14

Prefix tree ∀ binary strings of length n with j ones

Tn,j

1

Tn−1,j−1 Tn−1,j

# of leafs: |Tn,j| = |Tn−1,j| + |Tn−1,j−1| n

j

=

n−1

j

+

n−1

j−1

=

⇒ |Tn,j| = n

j

.

Tn,0

Tn−1,0

Tn,n

1

Tn−1,n−1

Sariel (UIUC) New CS473 6 Fall 2015 6 / 25

SLIDE 15

Encoding a string in Sn,j

1

Tn,j leafs corresponds to strings of Sn,j.

2

Order all strings of Sn,j order in lexicographical ordering

3

≡ ordering leafs of Tn,j from left to right.

4

Input: s ∈ Sn,j: compute index of s in sorted set Sn,j.

5

EncodeBinomCoeff(s) denote this polytime procedure.

Sariel (UIUC) New CS473 7 Fall 2015 7 / 25

SLIDE 16

Encoding a string in Sn,j

1

Tn,j leafs corresponds to strings of Sn,j.

2

Order all strings of Sn,j order in lexicographical ordering

3

≡ ordering leafs of Tn,j from left to right.

4

Input: s ∈ Sn,j: compute index of s in sorted set Sn,j.

5

EncodeBinomCoeff(s) denote this polytime procedure.

Sariel (UIUC) New CS473 7 Fall 2015 7 / 25

SLIDE 17

Encoding a string in Sn,j

1

Tn,j leafs corresponds to strings of Sn,j.

2

Order all strings of Sn,j order in lexicographical ordering

3

≡ ordering leafs of Tn,j from left to right. Tn,j

1

Tn−1,j−1 Tn−1,j

4

Input: s ∈ Sn,j: compute index of s in sorted set Sn,j.

5

EncodeBinomCoeff(s) denote this polytime procedure.

Sariel (UIUC) New CS473 7 Fall 2015 7 / 25

SLIDE 18

Encoding a string in Sn,j

1

Tn,j leafs corresponds to strings of Sn,j.

2

Order all strings of Sn,j order in lexicographical ordering

3

≡ ordering leafs of Tn,j from left to right. Tn,j

1

Tn−1,j−1 Tn−1,j

4

Input: s ∈ Sn,j: compute index of s in sorted set Sn,j.

5

EncodeBinomCoeff(s) denote this polytime procedure.

Sariel (UIUC) New CS473 7 Fall 2015 7 / 25

SLIDE 19

Encoding a string in Sn,j

1

Tn,j leafs corresponds to strings of Sn,j.

2

Order all strings of Sn,j order in lexicographical ordering

3

≡ ordering leafs of Tn,j from left to right. Tn,j

1

Tn−1,j−1 Tn−1,j

4

Input: s ∈ Sn,j: compute index of s in sorted set Sn,j.

5

EncodeBinomCoeff(s) denote this polytime procedure.

Sariel (UIUC) New CS473 7 Fall 2015 7 / 25

SLIDE 20

Decoding a string in Sn,j

1

Tn,j leafs corresponds to strings of Sn,j.

2

Order all strings of Sn,j order in lexicographical ordering

3

≡ ordering leafs of Tn,j from left to right.

4

x ∈

1, . . . ,

n

j

: compute xth string in Sn,j in polytime.

5

DecodeBinomCoeff (x) denote this procedure.

Sariel (UIUC) New CS473 8 Fall 2015 8 / 25

SLIDE 21

Decoding a string in Sn,j

1

Tn,j leafs corresponds to strings of Sn,j.

2

Order all strings of Sn,j order in lexicographical ordering

3

≡ ordering leafs of Tn,j from left to right.

4

x ∈

1, . . . ,

n

j

: compute xth string in Sn,j in polytime.

5

DecodeBinomCoeff (x) denote this procedure.

Sariel (UIUC) New CS473 8 Fall 2015 8 / 25

SLIDE 22

Decoding a string in Sn,j

1

Tn,j leafs corresponds to strings of Sn,j.

2

Order all strings of Sn,j order in lexicographical ordering

3

≡ ordering leafs of Tn,j from left to right. Tn,j

1

Tn−1,j−1 Tn−1,j

4

x ∈

1, . . . ,

n

j

: compute xth string in Sn,j in polytime.

5

DecodeBinomCoeff (x) denote this procedure.

Sariel (UIUC) New CS473 8 Fall 2015 8 / 25

SLIDE 23

Decoding a string in Sn,j

1

Tn,j leafs corresponds to strings of Sn,j.

2

Order all strings of Sn,j order in lexicographical ordering

3

≡ ordering leafs of Tn,j from left to right. Tn,j

1

Tn−1,j−1 Tn−1,j

4

x ∈

1, . . . ,

n

j

: compute xth string in Sn,j in polytime.

5

DecodeBinomCoeff (x) denote this procedure.

Sariel (UIUC) New CS473 8 Fall 2015 8 / 25

SLIDE 24

Decoding a string in Sn,j

1

Tn,j leafs corresponds to strings of Sn,j.

2

Order all strings of Sn,j order in lexicographical ordering

3

≡ ordering leafs of Tn,j from left to right. Tn,j

1

Tn−1,j−1 Tn−1,j

4

x ∈

1, . . . ,

n

j

: compute xth string in Sn,j in polytime.

5

DecodeBinomCoeff (x) denote this procedure.

Sariel (UIUC) New CS473 8 Fall 2015 8 / 25

SLIDE 25

Encoding/decoding strings of Sn,j

Lemma

Sn,j: Set of binary strings of length n with j ones, sorted lexicographically.

1

EncodeBinomCoeff(α): Input is string α ∈ Sn,j, compute index x of α in Sn,j in polynomial time in n.

2

DecodeBinomCoeff(x): Input index x ∈

1, . . . ,

n

j

.

Output xth string α in Sn,j, in time O(polylog n + n).

Sariel (UIUC) New CS473 9 Fall 2015 9 / 25

SLIDE 26

Extracting randomness

Theorem

Consider a coin that comes up heads with probability p > 1/2. For any constant δ > 0 and for n sufficiently large: (A) One can extract, from an input of a sequence of n flips, an

utput sequence of (1 − δ)nH(p) (unbiased) independent

random bits. (B) One can not extract more than nH(p) bits from such a sequence.

Sariel (UIUC) New CS473 10 Fall 2015 10 / 25

SLIDE 27

Proof...

1

There are n

j

input strings with exactly j heads.

2

each has probability pj(1 − p)n−j.

3

map string s like that to index number in the set Sj =

1, . . . ,

n

j

.

4

Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j.

5

x ← EncodeBinomCoeff(s)

6

x uniform distributed in {1, . . . , N}, N = n

j

.

7

Seen in previous lecture...

8

... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N.

9

Extract bits using ExtractRandomness(x, N):.

Sariel (UIUC) New CS473 11 Fall 2015 11 / 25

SLIDE 28

Proof...

1

There are n

j

input strings with exactly j heads.

2

each has probability pj(1 − p)n−j.

3

map string s like that to index number in the set Sj =

1, . . . ,

n

j

.

4

Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j.

5

x ← EncodeBinomCoeff(s)

6

x uniform distributed in {1, . . . , N}, N = n

j

.

7

Seen in previous lecture...

8

... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N.

9

Extract bits using ExtractRandomness(x, N):.

Sariel (UIUC) New CS473 11 Fall 2015 11 / 25

SLIDE 29

Proof...

1

There are n

j

input strings with exactly j heads.

2

each has probability pj(1 − p)n−j.

3

map string s like that to index number in the set Sj =

1, . . . ,

n

j

.

4

Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j.

5

x ← EncodeBinomCoeff(s)

6

x uniform distributed in {1, . . . , N}, N = n

j

.

7

Seen in previous lecture...

8

... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N.

9

Extract bits using ExtractRandomness(x, N):.

Sariel (UIUC) New CS473 11 Fall 2015 11 / 25

SLIDE 30

Proof...

1

There are n

j

input strings with exactly j heads.

2

each has probability pj(1 − p)n−j.

3

map string s like that to index number in the set Sj =

1, . . . ,

n

j

.

4

Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j.

5

x ← EncodeBinomCoeff(s)

6

x uniform distributed in {1, . . . , N}, N = n

j

.

7

Seen in previous lecture...

8

... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N.

9

Extract bits using ExtractRandomness(x, N):.

Sariel (UIUC) New CS473 11 Fall 2015 11 / 25

SLIDE 31

Proof...

1

There are n

j

input strings with exactly j heads.

2

each has probability pj(1 − p)n−j.

3

map string s like that to index number in the set Sj =

1, . . . ,

n

j

.

4

Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j.

5

x ← EncodeBinomCoeff(s)

6

x uniform distributed in {1, . . . , N}, N = n

j

.

7

Seen in previous lecture...

8

... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N.

9

Extract bits using ExtractRandomness(x, N):.

Sariel (UIUC) New CS473 11 Fall 2015 11 / 25

SLIDE 32

Proof...

1

There are n

j

input strings with exactly j heads.

2

each has probability pj(1 − p)n−j.

3

map string s like that to index number in the set Sj =

1, . . . ,

n

j

.

4

Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j.

5

x ← EncodeBinomCoeff(s)

6

x uniform distributed in {1, . . . , N}, N = n

j

.

7

Seen in previous lecture...

8

... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N.

9

Extract bits using ExtractRandomness(x, N):.

Sariel (UIUC) New CS473 11 Fall 2015 11 / 25

SLIDE 33

Proof...

1

There are n

j

input strings with exactly j heads.

2

each has probability pj(1 − p)n−j.

3

map string s like that to index number in the set Sj =

1, . . . ,

n

j

.

4

Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j.

5

x ← EncodeBinomCoeff(s)

6

x uniform distributed in {1, . . . , N}, N = n

j

.

7

Seen in previous lecture...

8

... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N.

9

Extract bits using ExtractRandomness(x, N):.

Sariel (UIUC) New CS473 11 Fall 2015 11 / 25

SLIDE 34

Proof...

1

There are n

j

input strings with exactly j heads.

2

each has probability pj(1 − p)n−j.

3

map string s like that to index number in the set Sj =

1, . . . ,

n

j

.

4

Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j.

5

x ← EncodeBinomCoeff(s)

6

x uniform distributed in {1, . . . , N}, N = n

j

.

7

Seen in previous lecture...

8

... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N.

9

Extract bits using ExtractRandomness(x, N):.

Sariel (UIUC) New CS473 11 Fall 2015 11 / 25

SLIDE 35

Proof...

1

There are n

j

input strings with exactly j heads.

2

each has probability pj(1 − p)n−j.

3

map string s like that to index number in the set Sj =

1, . . . ,

n

j

.

4

Given that input string s has j ones (out of n bits) defines a uniform distribution on Sn,j.

5

x ← EncodeBinomCoeff(s)

6

x uniform distributed in {1, . . . , N}, N = n

j

.

7

Seen in previous lecture...

8

... extract in expectation, ⌊lg N⌋ − 1 bits from uniform random variable in the range 1, . . . , N.

9

Extract bits using ExtractRandomness(x, N):.

Sariel (UIUC) New CS473 11 Fall 2015 11 / 25

SLIDE 36

Exciting proof continued...

1

Z: random variable: number of heads in input string s.

2

B: number of random bits extracted. E

B
=

n

k=0

Pr[Z = k] E

B
Z = k
,

3

Know: E

B
Z = k
≥
lg

n k

− 1.

4

ε < p − 1/2: sufficiently small constant.

5

n(p − ε) ≤ k ≤ n(p + ε): n k

≥
n

⌊n(p + ε)⌋

≥ 2nH(p+ε)

n + 1 ,

6

... since 2nH(p) is a good approximation to n

np

as proved in

previous lecture.

Sariel (UIUC) New CS473 12 Fall 2015 12 / 25

SLIDE 37

Exciting proof continued...

1

Z: random variable: number of heads in input string s.

2

B: number of random bits extracted. E

B
=

n

k=0

Pr[Z = k] E

B
Z = k
,

3

Know: E

B
Z = k
≥
lg

n k

− 1.

4

ε < p − 1/2: sufficiently small constant.

5

n(p − ε) ≤ k ≤ n(p + ε): n k

≥
n

⌊n(p + ε)⌋

≥ 2nH(p+ε)

n + 1 ,

6

... since 2nH(p) is a good approximation to n

np

as proved in

previous lecture.

Sariel (UIUC) New CS473 12 Fall 2015 12 / 25

SLIDE 38

Exciting proof continued...

1

Z: random variable: number of heads in input string s.

2

B: number of random bits extracted. E

B
=

n

k=0

Pr[Z = k] E

B
Z = k
,

3

Know: E

B
Z = k
≥
lg

n k

− 1.

4

ε < p − 1/2: sufficiently small constant.

5

n(p − ε) ≤ k ≤ n(p + ε): n k

≥
n

⌊n(p + ε)⌋

≥ 2nH(p+ε)

n + 1 ,

6

... since 2nH(p) is a good approximation to n

np

as proved in

previous lecture.

Sariel (UIUC) New CS473 12 Fall 2015 12 / 25

SLIDE 39

Exciting proof continued...

1

Z: random variable: number of heads in input string s.

2

B: number of random bits extracted. E

B
=

n

k=0

Pr[Z = k] E

B
Z = k
,

3

Know: E

B
Z = k
≥
lg

n k

− 1.

4

ε < p − 1/2: sufficiently small constant.

5

n(p − ε) ≤ k ≤ n(p + ε): n k

≥
n

⌊n(p + ε)⌋

≥ 2nH(p+ε)

n + 1 ,

6

... since 2nH(p) is a good approximation to n

np

as proved in

previous lecture.

Sariel (UIUC) New CS473 12 Fall 2015 12 / 25

SLIDE 40

Exciting proof continued...

1

Z: random variable: number of heads in input string s.

2

B: number of random bits extracted. E

B
=

n

k=0

Pr[Z = k] E

B
Z = k
,

3

Know: E

B
Z = k
≥
lg

n k

− 1.

4

ε < p − 1/2: sufficiently small constant.

5

n(p − ε) ≤ k ≤ n(p + ε): n k

≥
n

⌊n(p + ε)⌋

≥ 2nH(p+ε)

n + 1 ,

6

... since 2nH(p) is a good approximation to n

np

as proved in

previous lecture.

Sariel (UIUC) New CS473 12 Fall 2015 12 / 25

SLIDE 41

Exciting proof continued...

1

Z: random variable: number of heads in input string s.

2

B: number of random bits extracted. E

B
=

n

k=0

Pr[Z = k] E

B
Z = k
,

3

Know: E

B
Z = k
≥
lg

n k

− 1.

4

ε < p − 1/2: sufficiently small constant.

5

n(p − ε) ≤ k ≤ n(p + ε): n k

≥
n

⌊n(p + ε)⌋

≥ 2nH(p+ε)

n + 1 ,

6

... since 2nH(p) is a good approximation to n

np

as proved in

previous lecture.

Sariel (UIUC) New CS473 12 Fall 2015 12 / 25

SLIDE 42

Exciting proof continued...

1

Z: random variable: number of heads in input string s.

2

B: number of random bits extracted. E

B
=

n

k=0

Pr[Z = k] E

B
Z = k
,

3

Know: E

B
Z = k
≥
lg

n k

− 1.

4

ε < p − 1/2: sufficiently small constant.

5

n(p − ε) ≤ k ≤ n(p + ε): n k

≥
n

⌊n(p + ε)⌋

≥ 2nH(p+ε)

n + 1 ,

6

... since 2nH(p) is a good approximation to n

np

as proved in

previous lecture.

Sariel (UIUC) New CS473 12 Fall 2015 12 / 25

SLIDE 43

Super exciting proof continued...

E

B
= n

k=0 Pr[Z = k] E

B
Z = k
.

E

B
≥ ⌈n(p+ε)⌉

k=⌊n(p−ε)⌋ Pr

Z = k
E
B
Z = k
≥

⌈n(p+ε)⌉

k=⌊n(p−ε)⌋

Pr

Z = k

lg n k

− 1
≥

⌈n(p+ε)⌉

k=⌊n(p−ε)⌋

Pr

Z = k

lg 2nH(p+ε) n + 1 − 2

=
nH(p + ε) − lg(n + 1) − 2
Pr[|Z − np| ≤ εn]

≥

nH(p + ε) − lg(n + 1) − 2
1 − 2 exp
−nε2

4p

,

since µ = E[Z] = np and Pr

|Z − np| ≥ ε

ppn

≤ 2 exp
− np

4

ε

p

2 = 2 exp

− nε2

4p

,

by the Chernoff inequality.

Sariel (UIUC) New CS473 13 Fall 2015 13 / 25

SLIDE 44

Super exciting proof continued...

E

B
= n

k=0 Pr[Z = k] E

B
Z = k
.

E

B
≥ ⌈n(p+ε)⌉

k=⌊n(p−ε)⌋ Pr

Z = k
E
B
Z = k
≥

⌈n(p+ε)⌉

k=⌊n(p−ε)⌋

Pr

Z = k

lg n k

− 1
≥

⌈n(p+ε)⌉

k=⌊n(p−ε)⌋

Pr

Z = k

lg 2nH(p+ε) n + 1 − 2

=
nH(p + ε) − lg(n + 1) − 2
Pr[|Z − np| ≤ εn]

≥

nH(p + ε) − lg(n + 1) − 2
1 − 2 exp
−nε2

4p

,

since µ = E[Z] = np and Pr

|Z − np| ≥ ε

ppn

≤ 2 exp
− np

4

ε

p

2 = 2 exp

− nε2

4p

,

by the Chernoff inequality.

Sariel (UIUC) New CS473 13 Fall 2015 13 / 25

SLIDE 45

Super exciting proof continued...

E

B
= n

k=0 Pr[Z = k] E

B
Z = k
.

E

B
≥ ⌈n(p+ε)⌉

k=⌊n(p−ε)⌋ Pr

Z = k
E
B
Z = k
≥

⌈n(p+ε)⌉

k=⌊n(p−ε)⌋

Pr

Z = k

lg n k

− 1
≥

⌈n(p+ε)⌉

k=⌊n(p−ε)⌋

Pr

Z = k

lg 2nH(p+ε) n + 1 − 2

=
nH(p + ε) − lg(n + 1) − 2
Pr[|Z − np| ≤ εn]

≥

nH(p + ε) − lg(n + 1) − 2
1 − 2 exp
−nε2

4p

,

since µ = E[Z] = np and Pr

|Z − np| ≥ ε

ppn

≤ 2 exp
− np

4

ε

p

2 = 2 exp

− nε2

4p

,

by the Chernoff inequality.

Sariel (UIUC) New CS473 13 Fall 2015 13 / 25

SLIDE 46

Super exciting proof continued...

E

B
= n

k=0 Pr[Z = k] E

B
Z = k
.

E

B
≥ ⌈n(p+ε)⌉

k=⌊n(p−ε)⌋ Pr

Z = k
E
B
Z = k
≥

⌈n(p+ε)⌉

k=⌊n(p−ε)⌋

Pr

Z = k

lg n k

− 1
≥

⌈n(p+ε)⌉

k=⌊n(p−ε)⌋

Pr

Z = k

lg 2nH(p+ε) n + 1 − 2

=
nH(p + ε) − lg(n + 1) − 2
Pr[|Z − np| ≤ εn]

≥

nH(p + ε) − lg(n + 1) − 2
1 − 2 exp
−nε2

4p

,

since µ = E[Z] = np and Pr

|Z − np| ≥ ε

ppn

≤ 2 exp
− np

4

ε

p

2 = 2 exp

− nε2

4p

,

by the Chernoff inequality.

Sariel (UIUC) New CS473 13 Fall 2015 13 / 25

SLIDE 47

Super exciting proof continued...

E

B
= n

k=0 Pr[Z = k] E

B
Z = k
.

E

B
≥ ⌈n(p+ε)⌉

k=⌊n(p−ε)⌋ Pr

Z = k
E
B
Z = k
≥

⌈n(p+ε)⌉

k=⌊n(p−ε)⌋

Pr

Z = k

lg n k

− 1
≥

⌈n(p+ε)⌉

k=⌊n(p−ε)⌋

Pr

Z = k

lg 2nH(p+ε) n + 1 − 2

=
nH(p + ε) − lg(n + 1) − 2
Pr[|Z − np| ≤ εn]

≥

nH(p + ε) − lg(n + 1) − 2
1 − 2 exp
−nε2

4p

,

since µ = E[Z] = np and Pr

|Z − np| ≥ ε

ppn

≤ 2 exp
− np

4

ε

p

2 = 2 exp

− nε2

4p

,

by the Chernoff inequality.

Sariel (UIUC) New CS473 13 Fall 2015 13 / 25

SLIDE 48

Super exciting proof continued...

E

B
= n

k=0 Pr[Z = k] E

B
Z = k
.

E

B
≥ ⌈n(p+ε)⌉

k=⌊n(p−ε)⌋ Pr

Z = k
E
B
Z = k
≥

⌈n(p+ε)⌉

k=⌊n(p−ε)⌋

Pr

Z = k

lg n k

− 1
≥

⌈n(p+ε)⌉

k=⌊n(p−ε)⌋

Pr

Z = k

lg 2nH(p+ε) n + 1 − 2

=
nH(p + ε) − lg(n + 1) − 2
Pr[|Z − np| ≤ εn]

≥

nH(p + ε) − lg(n + 1) − 2
1 − 2 exp
−nε2

4p

,

since µ = E[Z] = np and Pr

|Z − np| ≥ ε

ppn

≤ 2 exp
− np

4

ε

p

2 = 2 exp

− nε2

4p

,

by the Chernoff inequality.

Sariel (UIUC) New CS473 13 Fall 2015 13 / 25

SLIDE 49

Super exciting proof continued...

E

B
= n

k=0 Pr[Z = k] E

B
Z = k
.

E

B
≥ ⌈n(p+ε)⌉

k=⌊n(p−ε)⌋ Pr

Z = k
E
B
Z = k
≥

⌈n(p+ε)⌉

k=⌊n(p−ε)⌋

Pr

Z = k

lg n k

− 1
≥

⌈n(p+ε)⌉

k=⌊n(p−ε)⌋

Pr

Z = k

lg 2nH(p+ε) n + 1 − 2

=
nH(p + ε) − lg(n + 1) − 2
Pr[|Z − np| ≤ εn]

≥

nH(p + ε) − lg(n + 1) − 2
1 − 2 exp
−nε2

4p

,

since µ = E[Z] = np and Pr

|Z − np| ≥ ε

ppn

≤ 2 exp
− np

4

ε

p

2 = 2 exp

− nε2

4p

,

by the Chernoff inequality.

Sariel (UIUC) New CS473 13 Fall 2015 13 / 25

SLIDE 50

Hyper super exciting proof continued...

1

Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is fixed.

2

= ⇒ nH(p) = Ω(n),

3

For n sufficiently large: − lg(n + 1) ≥ − δ

10nH(p).

4

... also 2 exp

− nε2

4p

≤

δ 10.

5

For n large enough; E[B] ≥

1 − δ

4 − δ 10

nH(p)
1 − δ

10

≥ (1 − δ)nH(p),

Sariel (UIUC) New CS473 14 Fall 2015 14 / 25

SLIDE 51

Hyper super exciting proof continued...

1

Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is fixed.

2

= ⇒ nH(p) = Ω(n),

3

For n sufficiently large: − lg(n + 1) ≥ − δ

10nH(p).

4

... also 2 exp

− nε2

4p

≤

δ 10.

5

For n large enough; E[B] ≥

1 − δ

4 − δ 10

nH(p)
1 − δ

10

≥ (1 − δ)nH(p),

Sariel (UIUC) New CS473 14 Fall 2015 14 / 25

SLIDE 52

Hyper super exciting proof continued...

1

Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is fixed.

2

= ⇒ nH(p) = Ω(n),

3

For n sufficiently large: − lg(n + 1) ≥ − δ

10nH(p).

4

... also 2 exp

− nε2

4p

≤

δ 10.

5

For n large enough; E[B] ≥

1 − δ

4 − δ 10

nH(p)
1 − δ

10

≥ (1 − δ)nH(p),

Sariel (UIUC) New CS473 14 Fall 2015 14 / 25

SLIDE 53

Hyper super exciting proof continued...

1

Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is fixed.

2

= ⇒ nH(p) = Ω(n),

3

For n sufficiently large: − lg(n + 1) ≥ − δ

10nH(p).

4

... also 2 exp

− nε2

4p

≤

δ 10.

5

For n large enough; E[B] ≥

1 − δ

4 − δ 10

nH(p)
1 − δ

10

≥ (1 − δ)nH(p),

Sariel (UIUC) New CS473 14 Fall 2015 14 / 25

SLIDE 54

Hyper super exciting proof continued...

1

Fix ε > 0, such that H(p + ε) > (1 − δ/4)H(p), p is fixed.

2

= ⇒ nH(p) = Ω(n),

3

For n sufficiently large: − lg(n + 1) ≥ − δ

10nH(p).

4

... also 2 exp

− nε2

4p

≤

δ 10.

5

For n large enough; E[B] ≥

1 − δ

4 − δ 10

nH(p)
1 − δ

10

≥ (1 − δ)nH(p),

Sariel (UIUC) New CS473 14 Fall 2015 14 / 25

SLIDE 55

Hyper super duper exciting proof continued...

1

Need to prove upper bound.

2

If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x].

3

All sequences of length |y| have equal probability to be generated (by definition).

4

2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.

5

= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])

6

E

B
=

x Pr

X = x
|Ext(x)|

≤

x Pr

X = x
lg

1 Pr [X=x] = H(X).

Sariel (UIUC) New CS473 15 Fall 2015 15 / 25

SLIDE 56

Hyper super duper exciting proof continued...

1

Need to prove upper bound.

2

If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x].

3

All sequences of length |y| have equal probability to be generated (by definition).

4

2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.

5

= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])

6

E

B
=

x Pr

X = x
|Ext(x)|

≤

x Pr

X = x
lg

1 Pr [X=x] = H(X).

Sariel (UIUC) New CS473 15 Fall 2015 15 / 25

SLIDE 57

Hyper super duper exciting proof continued...

1

Need to prove upper bound.

2

If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x].

3

All sequences of length |y| have equal probability to be generated (by definition).

4

2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.

5

= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])

6

E

B
=

x Pr

X = x
|Ext(x)|

≤

x Pr

X = x
lg

1 Pr [X=x] = H(X).

Sariel (UIUC) New CS473 15 Fall 2015 15 / 25

SLIDE 58

Hyper super duper exciting proof continued...

1

Need to prove upper bound.

2

If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x].

3

All sequences of length |y| have equal probability to be generated (by definition).

4

2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.

5

= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])

6

E

B
=

x Pr

X = x
|Ext(x)|

≤

x Pr

X = x
lg

1 Pr [X=x] = H(X).

Sariel (UIUC) New CS473 15 Fall 2015 15 / 25

SLIDE 59

Hyper super duper exciting proof continued...

1

Need to prove upper bound.

2

If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x].

3

All sequences of length |y| have equal probability to be generated (by definition).

4

2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.

5

= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])

6

E

B
=

x Pr

X = x
|Ext(x)|

≤

x Pr

X = x
lg

1 Pr [X=x] = H(X).

Sariel (UIUC) New CS473 15 Fall 2015 15 / 25

SLIDE 60

Hyper super duper exciting proof continued...

1

Need to prove upper bound.

2

If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x].

3

All sequences of length |y| have equal probability to be generated (by definition).

4

2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.

5

= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])

6

E

B
=

x Pr

X = x
|Ext(x)|

≤

x Pr

X = x
lg

1 Pr [X=x] = H(X).

Sariel (UIUC) New CS473 15 Fall 2015 15 / 25

SLIDE 61

Hyper super duper exciting proof continued...

1

Need to prove upper bound.

2

If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x].

3

All sequences of length |y| have equal probability to be generated (by definition).

4

2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.

5

= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])

6

E

B
=

x Pr

X = x
|Ext(x)|

≤

x Pr

X = x
lg

1 Pr [X=x] = H(X).

Sariel (UIUC) New CS473 15 Fall 2015 15 / 25

SLIDE 62

Hyper super duper exciting proof continued...

1

Need to prove upper bound.

2

If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x].

3

All sequences of length |y| have equal probability to be generated (by definition).

4

2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.

5

= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])

6

E

B
=

x Pr

X = x
|Ext(x)|

≤

x Pr

X = x
lg

1 Pr [X=x] = H(X).

Sariel (UIUC) New CS473 15 Fall 2015 15 / 25

SLIDE 63

Hyper super duper exciting proof continued...

1

Need to prove upper bound.

2

If input sequence x has probability Pr[X = x], then y = Ext(x) has probability to be generated ≥ Pr[X = x].

3

All sequences of length |y| have equal probability to be generated (by definition).

4

2|Ext(x)| Pr[X = x] ≤ 2|Ext(x)| Pr[y = Ext(x)] ≤ 1.

5

= ⇒ |Ext(x)| ≤ lg(1/ Pr[X = x])

6

E

B
=

x Pr

X = x
|Ext(x)|

≤

x Pr

X = x
lg

1 Pr [X=x] = H(X).

Sariel (UIUC) New CS473 15 Fall 2015 15 / 25

SLIDE 64

Part III Coding: Shannon’s Theorem

Sariel (UIUC) New CS473 16 Fall 2015 16 / 25

SLIDE 65

Shannon’s Theorem

Definition

1

binary symmetric channel with parameter p

2

sequence of bits x1, x2, . . ., an

3

utput: y1, y2, . . . ,

a sequence of bits such that...

4

Pr[xi = yi] = 1 − p independently for each i.

Sariel (UIUC) New CS473 17 Fall 2015 17 / 25

SLIDE 66

Shannon’s Theorem

Definition

1

binary symmetric channel with parameter p

2

sequence of bits x1, x2, . . ., an

3

utput: y1, y2, . . . ,

a sequence of bits such that...

4

Pr[xi = yi] = 1 − p independently for each i.

Sariel (UIUC) New CS473 17 Fall 2015 17 / 25

SLIDE 67

Shannon’s Theorem

Definition

1

binary symmetric channel with parameter p

2

sequence of bits x1, x2, . . ., an

3

utput: y1, y2, . . . ,

a sequence of bits such that...

4

Pr[xi = yi] = 1 − p independently for each i.

Sariel (UIUC) New CS473 17 Fall 2015 17 / 25

SLIDE 68

Shannon’s Theorem

Definition

1

binary symmetric channel with parameter p

2

sequence of bits x1, x2, . . ., an

3

utput: y1, y2, . . . ,

a sequence of bits such that...

4

Pr[xi = yi] = 1 − p independently for each i.

Sariel (UIUC) New CS473 17 Fall 2015 17 / 25

SLIDE 69

Encoding/decoding with noise

Definition

1

(k, n) encoding function Enc : {0, 1}k → {0, 1}n takes as input a sequence of k bits and outputs a sequence of n bits.

2

(k, n) decoding function Dec : {0, 1}n → {0, 1}k takes as input a sequence of n bits and outputs a sequence of k bits.

Sariel (UIUC) New CS473 18 Fall 2015 18 / 25

SLIDE 70

Encoding/decoding with noise

Definition

1

(k, n) encoding function Enc : {0, 1}k → {0, 1}n takes as input a sequence of k bits and outputs a sequence of n bits.

2

(k, n) decoding function Dec : {0, 1}n → {0, 1}k takes as input a sequence of n bits and outputs a sequence of k bits.

Sariel (UIUC) New CS473 18 Fall 2015 18 / 25

SLIDE 71

Claude Elwood Shannon

Claude Elwood Shannon (April 30, 1916 - February 24, 2001), an American electrical engineer and mathematician, has been called “the father of information theory”. His master thesis was how to building boolean circuits for any boolean function.

Sariel (UIUC) New CS473 19 Fall 2015 19 / 25

SLIDE 72

Shannon’s theorem (1948)

Theorem (Shannon’s theorem)

For a binary symmetric channel with parameter p < 1/2 and for any constants δ, γ > 0, where n is sufficiently large, the following holds: (i) For an k ≤ n(1 − H(p) − δ) there exists (k, n) encoding and decoding functions such that the probability the receiver fails to obtain the correct message is at most γ for every possible k-bit input messages. (ii) There are no (k, n) encoding and decoding functions with k ≥ n(1 − H(p) + δ) such that the probability of decoding correctly is at least γ for a k-bit input message chosen uniformly at random.

Sariel (UIUC) New CS473 20 Fall 2015 20 / 25

SLIDE 73

When the sender sends a string...

S = s1s2 . . . sn

S

Sariel (UIUC) New CS473 21 Fall 2015 21 / 25

SLIDE 74

When the sender sends a string...

S = s1s2 . . . sn

S

np

Sariel (UIUC) New CS473 21 Fall 2015 21 / 25

SLIDE 75

When the sender sends a string...

S = s1s2 . . . sn

S

np

Sariel (UIUC) New CS473 21 Fall 2015 21 / 25

SLIDE 76

When the sender sends a string...

S = s1s2 . . . sn

S

np

(1 − δ)np (1 + δ)np

Sariel (UIUC) New CS473 21 Fall 2015 21 / 25

SLIDE 77

When the sender sends a string...

S = s1s2 . . . sn

S

np

(1 − δ)np (1 + δ)np

One ring to rule them all!

Sariel (UIUC) New CS473 21 Fall 2015 21 / 25

SLIDE 78

Some intuition...

1

senders sent string S = s1s2 . . . sn.

2

receiver got string T = t1t2 . . . tn.

3

p = Pr[ti = si], for all i.

4

U: Hamming distance between S and T : U =

i

si = ti
.

5

By assumption: E[U] = pn, and U is a binomial variable.

6

By Chernoff inequality: U ∈

(1 − δ)np, (1 + δ)np
with

high probability, where δ is tiny constant.

7

T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np.

8

This ring has

(1+δ)np

i=(1−δ)np

n i

≤ 2
n

(1 + δ)np

≤ α = 2 · 2nH((1+δ)p).

strings in it.

Sariel (UIUC) New CS473 22 Fall 2015 22 / 25

SLIDE 79

Some intuition...

1

senders sent string S = s1s2 . . . sn.

2

receiver got string T = t1t2 . . . tn.

3

p = Pr[ti = si], for all i.

4

U: Hamming distance between S and T : U =

i

si = ti
.

5

By assumption: E[U] = pn, and U is a binomial variable.

6

By Chernoff inequality: U ∈

(1 − δ)np, (1 + δ)np
with

high probability, where δ is tiny constant.

7

T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np.

8

This ring has

(1+δ)np

i=(1−δ)np

n i

≤ 2
n

(1 + δ)np

≤ α = 2 · 2nH((1+δ)p).

strings in it.

Sariel (UIUC) New CS473 22 Fall 2015 22 / 25

SLIDE 80

Some intuition...

1

senders sent string S = s1s2 . . . sn.

2

receiver got string T = t1t2 . . . tn.

3

p = Pr[ti = si], for all i.

4

U: Hamming distance between S and T : U =

i

si = ti
.

5

By assumption: E[U] = pn, and U is a binomial variable.

6

By Chernoff inequality: U ∈

(1 − δ)np, (1 + δ)np
with

high probability, where δ is tiny constant.

7

T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np.

8

This ring has

(1+δ)np

i=(1−δ)np

n i

≤ 2
n

(1 + δ)np

≤ α = 2 · 2nH((1+δ)p).

strings in it.

Sariel (UIUC) New CS473 22 Fall 2015 22 / 25

SLIDE 81

Some intuition...

1

senders sent string S = s1s2 . . . sn.

2

receiver got string T = t1t2 . . . tn.

3

p = Pr[ti = si], for all i.

4

U: Hamming distance between S and T : U =

i

si = ti
.

5

By assumption: E[U] = pn, and U is a binomial variable.

6

By Chernoff inequality: U ∈

(1 − δ)np, (1 + δ)np
with

high probability, where δ is tiny constant.

7

T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np.

8

This ring has

(1+δ)np

i=(1−δ)np

n i

≤ 2
n

(1 + δ)np

≤ α = 2 · 2nH((1+δ)p).

strings in it.

Sariel (UIUC) New CS473 22 Fall 2015 22 / 25

SLIDE 82

Some intuition...

1

senders sent string S = s1s2 . . . sn.

2

receiver got string T = t1t2 . . . tn.

3

p = Pr[ti = si], for all i.

4

U: Hamming distance between S and T : U =

i

si = ti
.

5

By assumption: E[U] = pn, and U is a binomial variable.

6

By Chernoff inequality: U ∈

(1 − δ)np, (1 + δ)np
with

high probability, where δ is tiny constant.

7

T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np.

8

This ring has

(1+δ)np

i=(1−δ)np

n i

≤ 2
n

(1 + δ)np

≤ α = 2 · 2nH((1+δ)p).

strings in it.

Sariel (UIUC) New CS473 22 Fall 2015 22 / 25

SLIDE 83

Some intuition...

1

senders sent string S = s1s2 . . . sn.

2

receiver got string T = t1t2 . . . tn.

3

p = Pr[ti = si], for all i.

4

U: Hamming distance between S and T : U =

i

si = ti
.

5

By assumption: E[U] = pn, and U is a binomial variable.

6

By Chernoff inequality: U ∈

(1 − δ)np, (1 + δ)np
with

high probability, where δ is tiny constant.

7

T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np.

8

This ring has

(1+δ)np

i=(1−δ)np

n i

≤ 2
n

(1 + δ)np

≤ α = 2 · 2nH((1+δ)p).

strings in it.

Sariel (UIUC) New CS473 22 Fall 2015 22 / 25

SLIDE 84

Some intuition...

1

senders sent string S = s1s2 . . . sn.

2

receiver got string T = t1t2 . . . tn.

3

p = Pr[ti = si], for all i.

4

U: Hamming distance between S and T : U =

i

si = ti
.

5

By assumption: E[U] = pn, and U is a binomial variable.

6

By Chernoff inequality: U ∈

(1 − δ)np, (1 + δ)np
with

high probability, where δ is tiny constant.

7

T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np.

8

This ring has

(1+δ)np

i=(1−δ)np

n i

≤ 2
n

(1 + δ)np

≤ α = 2 · 2nH((1+δ)p).

strings in it.

Sariel (UIUC) New CS473 22 Fall 2015 22 / 25

SLIDE 85

Some intuition...

1

senders sent string S = s1s2 . . . sn.

2

receiver got string T = t1t2 . . . tn.

3

p = Pr[ti = si], for all i.

4

U: Hamming distance between S and T : U =

i

si = ti
.

5

By assumption: E[U] = pn, and U is a binomial variable.

6

By Chernoff inequality: U ∈

(1 − δ)np, (1 + δ)np
with

high probability, where δ is tiny constant.

7

T is in a ring R centered at S, with inner radius (1 − δ)np and outer radius (1 + δ)np.

8

This ring has

(1+δ)np

i=(1−δ)np

n i

≤ 2
n

(1 + δ)np

≤ α = 2 · 2nH((1+δ)p).

strings in it.

Sariel (UIUC) New CS473 22 Fall 2015 22 / 25

SLIDE 86

Many rings for many codewords...

Sariel (UIUC) New CS473 23 Fall 2015 23 / 25

SLIDE 87

Some more intuition...

1

Pick as many disjoint rings as possible: R1, . . . , Rκ.

2

If every word in the hypercube would be covered...

3

... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)).

4

Consider all possible strings of length k such that 2k ≤ κ.

5

Map ith string in {0, 1}k to the center Ci of the ith ring Ri.

6

If send Ci = ⇒ receiver gets a string in Ri.

7

Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to.

8

How many bits? k = ⌊log κ⌋ = n

1 − H
(1 + δ)p
≈ n(1 − H(p)),

Sariel (UIUC) New CS473 24 Fall 2015 24 / 25

SLIDE 88

Some more intuition...

1

Pick as many disjoint rings as possible: R1, . . . , Rκ.

2

If every word in the hypercube would be covered...

3

... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)).

4

Consider all possible strings of length k such that 2k ≤ κ.

5

Map ith string in {0, 1}k to the center Ci of the ith ring Ri.

6

If send Ci = ⇒ receiver gets a string in Ri.

7

Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to.

8

How many bits? k = ⌊log κ⌋ = n

1 − H
(1 + δ)p
≈ n(1 − H(p)),

Sariel (UIUC) New CS473 24 Fall 2015 24 / 25

SLIDE 89

Some more intuition...

1

Pick as many disjoint rings as possible: R1, . . . , Rκ.

2

If every word in the hypercube would be covered...

3

... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)).

4

Consider all possible strings of length k such that 2k ≤ κ.

5

Map ith string in {0, 1}k to the center Ci of the ith ring Ri.

6

If send Ci = ⇒ receiver gets a string in Ri.

7

Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to.

8

How many bits? k = ⌊log κ⌋ = n

1 − H
(1 + δ)p
≈ n(1 − H(p)),

Sariel (UIUC) New CS473 24 Fall 2015 24 / 25

SLIDE 90

Some more intuition...

1

Pick as many disjoint rings as possible: R1, . . . , Rκ.

2

If every word in the hypercube would be covered...

3

... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)).

4

Consider all possible strings of length k such that 2k ≤ κ.

5

Map ith string in {0, 1}k to the center Ci of the ith ring Ri.

6

If send Ci = ⇒ receiver gets a string in Ri.

7

Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to.

8

How many bits? k = ⌊log κ⌋ = n

1 − H
(1 + δ)p
≈ n(1 − H(p)),

Sariel (UIUC) New CS473 24 Fall 2015 24 / 25

SLIDE 91

Some more intuition...

1

Pick as many disjoint rings as possible: R1, . . . , Rκ.

2

If every word in the hypercube would be covered...

3

... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)).

4

Consider all possible strings of length k such that 2k ≤ κ.

5

Map ith string in {0, 1}k to the center Ci of the ith ring Ri.

6

If send Ci = ⇒ receiver gets a string in Ri.

7

Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to.

8

How many bits? k = ⌊log κ⌋ = n

1 − H
(1 + δ)p
≈ n(1 − H(p)),

Sariel (UIUC) New CS473 24 Fall 2015 24 / 25

SLIDE 92

Some more intuition...

1

Pick as many disjoint rings as possible: R1, . . . , Rκ.

2

If every word in the hypercube would be covered...

3

... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)).

4

Consider all possible strings of length k such that 2k ≤ κ.

5

Map ith string in {0, 1}k to the center Ci of the ith ring Ri.

6

If send Ci = ⇒ receiver gets a string in Ri.

7

Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to.

8

How many bits? k = ⌊log κ⌋ = n

1 − H
(1 + δ)p
≈ n(1 − H(p)),

Sariel (UIUC) New CS473 24 Fall 2015 24 / 25

SLIDE 93

Some more intuition...

1

Pick as many disjoint rings as possible: R1, . . . , Rκ.

2

If every word in the hypercube would be covered...

3

... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)).

4

Consider all possible strings of length k such that 2k ≤ κ.

5

Map ith string in {0, 1}k to the center Ci of the ith ring Ri.

6

If send Ci = ⇒ receiver gets a string in Ri.

7

Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to.

8

How many bits? k = ⌊log κ⌋ = n

1 − H
(1 + δ)p
≈ n(1 − H(p)),

Sariel (UIUC) New CS473 24 Fall 2015 24 / 25

SLIDE 94

Some more intuition...

1

Pick as many disjoint rings as possible: R1, . . . , Rκ.

2

If every word in the hypercube would be covered...

3

... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)).

4

Consider all possible strings of length k such that 2k ≤ κ.

5

Map ith string in {0, 1}k to the center Ci of the ith ring Ri.

6

If send Ci = ⇒ receiver gets a string in Ri.

7

Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to.

8

How many bits? k = ⌊log κ⌋ = n

1 − H
(1 + δ)p
≈ n(1 − H(p)),

Sariel (UIUC) New CS473 24 Fall 2015 24 / 25

SLIDE 95

Some more intuition...

1

Pick as many disjoint rings as possible: R1, . . . , Rκ.

2

If every word in the hypercube would be covered...

3

... use 2n codewords = ⇒ κ ≥ κ ≥ 2n |R| ≥ 2n 2 · 2nH((1+δ)p) ≈ 2n(1−H((1+δ)p)).

4

Consider all possible strings of length k such that 2k ≤ κ.

5

Map ith string in {0, 1}k to the center Ci of the ith ring Ri.

6

If send Ci = ⇒ receiver gets a string in Ri.

7

Decoding is easy - find the ring Ri containing the received string, take its center string Ci, and output the original string it was mapped to.

8

How many bits? k = ⌊log κ⌋ = n

1 − H
(1 + δ)p
≈ n(1 − H(p)),

Sariel (UIUC) New CS473 24 Fall 2015 24 / 25

SLIDE 96

What is wrong with the above?

1

Can not find such a large set of disjoint rings.

2

Reason is that when you pack rings (or balls) you are going to have wasted spaces around.

3

Overcome this: allow rings to overlap somewhat.

4

Makes things considerably more involved.

5

Details in class notes.

Sariel (UIUC) New CS473 25 Fall 2015 25 / 25

SLIDE 97

What is wrong with the above?

1

Can not find such a large set of disjoint rings.

2

Reason is that when you pack rings (or balls) you are going to have wasted spaces around.

3

Overcome this: allow rings to overlap somewhat.

4

Makes things considerably more involved.

5

Details in class notes.

Sariel (UIUC) New CS473 25 Fall 2015 25 / 25

SLIDE 98

What is wrong with the above?

1

Can not find such a large set of disjoint rings.

2

Reason is that when you pack rings (or balls) you are going to have wasted spaces around.

3

Overcome this: allow rings to overlap somewhat.

4

Makes things considerably more involved.

5

Details in class notes.

Sariel (UIUC) New CS473 25 Fall 2015 25 / 25

SLIDE 99

What is wrong with the above?

1

Can not find such a large set of disjoint rings.

2

Reason is that when you pack rings (or balls) you are going to have wasted spaces around.

3

Overcome this: allow rings to overlap somewhat.

4

Makes things considerably more involved.

5

Details in class notes.

Sariel (UIUC) New CS473 25 Fall 2015 25 / 25

SLIDE 100

What is wrong with the above?

1

Can not find such a large set of disjoint rings.

2

Reason is that when you pack rings (or balls) you are going to have wasted spaces around.

3

Overcome this: allow rings to overlap somewhat.

4

Makes things considerably more involved.

5

Details in class notes.

Sariel (UIUC) New CS473 25 Fall 2015 25 / 25

SLIDE 101

Notes

Sariel (UIUC) New CS473 26 Fall 2015 26 / 25

SLIDE 102

Notes

Sariel (UIUC) New CS473 27 Fall 2015 27 / 25

SLIDE 103

Notes

Sariel (UIUC) New CS473 28 Fall 2015 28 / 25

SLIDE 104

Notes

Sariel (UIUC) New CS473 29 Fall 2015 29 / 25