[PPT] - On Third-Order Asymptotics for DMCs Vincent Y. F. Tan Institute for PowerPoint Presentation

SLIDE 1

On Third-Order Asymptotics for DMCs

Vincent Y. F. Tan

Institute for Infocomm Research (I2R) National University of Singapore (NUS)

January 20, 2013

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 1 / 29

SLIDE 2

Acknowledgements

This is joint work with Marco Tomamichel Centre for Quantum Technologies National University of Singapore

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 2 / 29

SLIDE 3

Transmission of Information

TRANSMITTER MESSAGE SIGNAL RECEIVED SIGNAL RECEIVER DESTINATION MESSAGE NOISE SOURCE INFORMATION SOURCE

Shannon’s Figure 1

Information theory ≡ Finding fundamental limits for reliable information transmission

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 3 / 29

SLIDE 4

Transmission of Information

TRANSMITTER MESSAGE SIGNAL RECEIVED SIGNAL RECEIVER DESTINATION MESSAGE NOISE SOURCE INFORMATION SOURCE

Shannon’s Figure 1

Information theory ≡ Finding fundamental limits for reliable information transmission Channel coding: Concerned with the maximum rate of communication in bits/channel use

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 3 / 29

SLIDE 5

Channel Coding (One-Shot)

✲ ✲ ✲ ✲

M X Y e W d

M

A code is an triple C = {M, e, d} where M is the message set

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 4 / 29

SLIDE 6

Channel Coding (One-Shot)

✲ ✲ ✲ ✲

M X Y e W d

M

A code is an triple C = {M, e, d} where M is the message set The average error probability perr(C) is perr(C) := Pr [ M = M] where M is uniform on M

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 4 / 29

SLIDE 7

Channel Coding (One-Shot)

✲ ✲ ✲ ✲

M X Y e W d

M

A code is an triple C = {M, e, d} where M is the message set The average error probability perr(C) is perr(C) := Pr [ M = M] where M is uniform on M ε-Error Capacity is M∗(W, ε) := sup

m ∈ N
∃ C s.t. m = |M|, perr(C) ≤ ε
Vincent Tan (I2R and NUS)

Third-Order Asymptotics for DMCs HKTW Workshop 2013 4 / 29

SLIDE 8

Channel Coding (n-Shot)

✲ ✲ ✲ ✲

M Xn Yn e Wn d

M

Consider n independent uses of a channel

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 5 / 29

SLIDE 9

Channel Coding (n-Shot)

✲ ✲ ✲ ✲

M Xn Yn e Wn d

M

Consider n independent uses of a channel Assume W is a discrete memoryless channel

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 5 / 29

SLIDE 10

Channel Coding (n-Shot)

✲ ✲ ✲ ✲

M Xn Yn e Wn d

M

Consider n independent uses of a channel Assume W is a discrete memoryless channel For vectors x = (x1, . . . , xn) ∈ X n and y := (y1, . . . , yn) ∈ Yn, Wn(y|x) =

n

i=1

W(yi|xi)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 5 / 29

SLIDE 11

Channel Coding (n-Shot)

✲ ✲ ✲ ✲

M Xn Yn e Wn d

M

Consider n independent uses of a channel Assume W is a discrete memoryless channel For vectors x = (x1, . . . , xn) ∈ X n and y := (y1, . . . , yn) ∈ Yn, Wn(y|x) =

n

i=1

W(yi|xi) Blocklength n, ε-Error Capacity is M∗(Wn, ε)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 5 / 29

SLIDE 12

Main Contribution

Upper bound log M∗(Wn, ε) for n large (converse)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 6 / 29

SLIDE 13

Main Contribution

Upper bound log M∗(Wn, ε) for n large (converse) Concerned with the third-order term of the asymptotic expansion

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 6 / 29

SLIDE 14

Main Contribution

Upper bound log M∗(Wn, ε) for n large (converse) Concerned with the third-order term of the asymptotic expansion Going beyond the normal approximation terms

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 6 / 29

SLIDE 15

Main Contribution

Upper bound log M∗(Wn, ε) for n large (converse) Concerned with the third-order term of the asymptotic expansion Going beyond the normal approximation terms Theorem (Tomamichel-Tan (2013)) For all DMCs with positive ε-dispersion Vε, log M∗(Wn, ε) ≤ nC − √ nVεQ−1(ε) + 1 2 log n + O(1) where Q(a) := +∞

a 1 √ 2π exp

− 1

2x2

dx

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 6 / 29

SLIDE 16

Main Contribution

Upper bound log M∗(Wn, ε) for n large (converse) Concerned with the third-order term of the asymptotic expansion Going beyond the normal approximation terms Theorem (Tomamichel-Tan (2013)) For all DMCs with positive ε-dispersion Vε, log M∗(Wn, ε) ≤ nC − √ nVεQ−1(ε) + 1 2 log n + O(1) where Q(a) := +∞

a 1 √ 2π exp

− 1

2x2

dx The 1

2 log n term is our main contribution

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 6 / 29

SLIDE 17

Main Contribution: Remarks

Our bound log M∗(Wn, ε) ≤ nC − √ nVεQ−1(ε) + 1 2 log n + O(1)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 7 / 29

SLIDE 18

Main Contribution: Remarks

Our bound log M∗(Wn, ε) ≤ nC − √ nVεQ−1(ε) + 1 2 log n + O(1) Best upper bound till date: log M∗(Wn, ε) ≤ nC − √ nVεQ−1(ε) +

|X| − 1

2

log n + O(1)
V. Strassen (1964)

Polyanskiy-Poor-Verdú or PPV (2010)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 7 / 29

SLIDE 19

Main Contribution: Remarks

Our bound log M∗(Wn, ε) ≤ nC − √ nVεQ−1(ε) + 1 2 log n + O(1) Best upper bound till date: log M∗(Wn, ε) ≤ nC − √ nVεQ−1(ε) +

|X| − 1

2

log n + O(1)
V. Strassen (1964)

Polyanskiy-Poor-Verdú or PPV (2010) Requires new converse techniques

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 7 / 29

SLIDE 20

Outline

1 Background 2 Related work 3 Main result 4 New converse 5 Proof sketch 6 Summary and open problems

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 8 / 29

SLIDE 21

Background: Shannon’s Channel Coding Theorem

Shannon’s noisy channel coding theorem and Wolfowitz’s strong converse state that

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 9 / 29

SLIDE 22

Background: Shannon’s Channel Coding Theorem

Shannon’s noisy channel coding theorem and Wolfowitz’s strong converse state that Theorem (Shannon (1949), Wolfowitz (1959)) lim

n→∞

1 n log M∗(Wn, ε) = C, ∀ ε ∈ (0, 1) where C is the channel capacity defined as C = C(W) = max

P

I(P, W)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 9 / 29

SLIDE 23

Background: Shannon’s Channel Coding Theorem

lim

n→∞

1 n log M∗(Wn, ε) = C bits/channel use Noisy channel coding theorem is independent of ε ∈ (0, 1)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 10 / 29

SLIDE 24

Background: Shannon’s Channel Coding Theorem

lim

n→∞

1 n log M∗(Wn, ε) = C bits/channel use Noisy channel coding theorem is independent of ε ∈ (0, 1)

✲ ✻

C R 1 lim

n→∞ perr(C)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 10 / 29

SLIDE 25

Background: Shannon’s Channel Coding Theorem

lim

n→∞

1 n log M∗(Wn, ε) = C bits/channel use Noisy channel coding theorem is independent of ε ∈ (0, 1)

✲ ✻

C R 1 lim

n→∞ perr(C)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 10 / 29

SLIDE 26

Background: Shannon’s Channel Coding Theorem

lim

n→∞

1 n log M∗(Wn, ε) = C bits/channel use Noisy channel coding theorem is independent of ε ∈ (0, 1)

✲ ✻

C R 1 lim

n→∞ perr(C)

Phase transition at capacity

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 10 / 29

SLIDE 27

Background: ε-Dispersion

What happens at capacity?

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 11 / 29

SLIDE 28

Background: ε-Dispersion

What happens at capacity? More precisely, what happens when log |M| ≈ nC + a√n for some a ∈ R?

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 11 / 29

SLIDE 29

Background: ε-Dispersion

What happens at capacity? More precisely, what happens when log |M| ≈ nC + a√n for some a ∈ R? Assume capacity-achieving input distribution (CAID) P∗ is unique

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 11 / 29

SLIDE 30

Background: ε-Dispersion

What happens at capacity? More precisely, what happens when log |M| ≈ nC + a√n for some a ∈ R? Assume capacity-achieving input distribution (CAID) P∗ is unique The ε-dispersion is an operational quantity that is equal to Vε = V(P∗, W) = EP∗

VarW(·|X)
log W(·|X)

Q∗(·)

X
where (X, Y) ∼ P∗ × W and Q∗(y) =

x P∗(x)W(y|x)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 11 / 29

SLIDE 31

Background: ε-Dispersion

What happens at capacity? More precisely, what happens when log |M| ≈ nC + a√n for some a ∈ R? Assume capacity-achieving input distribution (CAID) P∗ is unique The ε-dispersion is an operational quantity that is equal to Vε = V(P∗, W) = EP∗

VarW(·|X)
log W(·|X)

Q∗(·)

X
where (X, Y) ∼ P∗ × W and Q∗(y) =

x P∗(x)W(y|x)

Since CAID is unique, Vε = V

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 11 / 29

SLIDE 32

Background: ε-Dispersion

Assume rate of the code satisfies 1 n log |M| = C + a √n

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 12 / 29

SLIDE 33

Background: ε-Dispersion

Assume rate of the code satisfies 1 n log |M| = C + a √n

✲ ✻

0.5 1 a lim

n→∞ perr(C) Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 12 / 29

SLIDE 34

Background: ε-Dispersion

Assume rate of the code satisfies 1 n log |M| = C + a √n

✲ ✻

0.5 1 a lim

n→∞ perr(C)

perr(C) ≈ Φ

a

√ V

Vincent Tan (I2R and NUS)

Third-Order Asymptotics for DMCs HKTW Workshop 2013 12 / 29

SLIDE 35

Background: ε-Dispersion

Assume rate of the code satisfies 1 n log |M| = C + a √n

✲ ✻

0.5 1 a lim

n→∞ perr(C)

perr(C) ≈ Φ

a

√ V

Here, we have fixed a, the second-order coding rate [Hayashi (2009)]

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 12 / 29

SLIDE 36

Background: ε-Dispersion

Theorem (Strassen (1964), Hayashi (2009), Polyanskiy-Poor-Verdú (2010)) For every ε ∈ (0, 1), and if Vε > 0, we have log M∗(Wn, ε) = nC − √ nVQ−1(ε) + O(log n)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 13 / 29

SLIDE 37

Background: ε-Dispersion

Theorem (Strassen (1964), Hayashi (2009), Polyanskiy-Poor-Verdú (2010)) For every ε ∈ (0, 1), and if Vε > 0, we have log M∗(Wn, ε) = nC − √ nVQ−1(ε) + O(log n)

V. Strassen

(1964)

M. Hayashi

(2009) Polyanskiy-Poor-Verdú (2010)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 13 / 29

SLIDE 38

Background: ε-Dispersion

Berry-Esséen theorem: For independent Xi with zero-mean and variances σ2

i ,

P

1

√n

n

i=1

Xi ≥ a

= Q

a ¯ σ

± 6 B

√n where ¯ σ2 = 1

n

i=1 σ2 i and B is related to the third moment

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 14 / 29

SLIDE 39

Background: ε-Dispersion

Berry-Esséen theorem: For independent Xi with zero-mean and variances σ2

i ,

P

1

√n

n

i=1

Xi ≥ a

= Q

a ¯ σ

± 6 B

√n where ¯ σ2 = 1

n

i=1 σ2 i and B is related to the third moment

PPV showed that the normal approximation log M∗(Wn, ε) ≈ nC − √ nVQ−1(ε) is very accurate even at moderate blocklengths of ≈ 100

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 14 / 29

SLIDE 40

Background: ε-Dispersion for the BSC

For a BSC with crossover probability p = 0.11, the normal approximation yields:

100 200 300 400 500 600 700 800 900 1000 0.3 0.35 0.4 0.45 0.5 Blocklength n Bits per channel use Normal approximation Capacity ε = 0.01 ε = 0.1 Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 15 / 29

SLIDE 41

Related Work: Third-Order Term

Recall that we are interested in quantifying the third-order term ρn ρn = log M∗(Wn, ε) −

nC −

√ nVQ−1(ε)

ρn = O(log n) if channel is non-exotic

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 16 / 29

SLIDE 42

Related Work: Third-Order Term

Recall that we are interested in quantifying the third-order term ρn ρn = log M∗(Wn, ε) −

nC −

√ nVQ−1(ε)

ρn = O(log n) if channel is non-exotic

Motivation 1: ρn may be important at very short blocklengths

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 16 / 29

SLIDE 43

Related Work: Third-Order Term

Recall that we are interested in quantifying the third-order term ρn ρn = log M∗(Wn, ε) −

nC −

√ nVQ−1(ε)

ρn = O(log n) if channel is non-exotic

Motivation 1: ρn may be important at very short blocklengths Motivation 2: Because we’re information theorists Wir müssen wissen – wir werden wissen (David Hilbert)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 16 / 29

SLIDE 44

Related Work: Third-Order Term

ρn = log M∗(Wn, ε) −

nC −

√ nVQ−1(ε)

For the BSC [PPV10]

ρn = 1 2 log n + O(1)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 17 / 29

SLIDE 45

Related Work: Third-Order Term

ρn = log M∗(Wn, ε) −

nC −

√ nVQ−1(ε)

For the BSC [PPV10]

ρn = 1 2 log n + O(1) For the BEC [PPV10] ρn = O(1)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 17 / 29

SLIDE 46

Related Work: Third-Order Term

ρn = log M∗(Wn, ε) −

nC −

√ nVQ−1(ε)

For the BSC [PPV10]

ρn = 1 2 log n + O(1) For the BEC [PPV10] ρn = O(1) For the AWGN under maximum-power constraints [PPV10] O(1) ≤ ρn ≤ 1 2 log n + O(1)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 17 / 29

SLIDE 47

Related Work: Third-Order Term

ρn = log M∗(Wn, ε) −

nC −

√ nVQ−1(ε)

For the BSC [PPV10]

ρn = 1 2 log n + O(1) For the BEC [PPV10] ρn = O(1) For the AWGN under maximum-power constraints [PPV10] O(1) ≤ ρn ≤ 1 2 log n + O(1) Our converse technique can be applied to the AWGN channel

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 17 / 29

SLIDE 48

Related Work: Achievability for Third-Order Term

Proposition (Polyanskiy (2010)) Assume that all elements of {W(y|x) : x ∈ X, y ∈ Y} are positive and C > 0. Then, ρn ≥ 1 2 log n + O(1)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 18 / 29

SLIDE 49

Related Work: Achievability for Third-Order Term

Proposition (Polyanskiy (2010)) Assume that all elements of {W(y|x) : x ∈ X, y ∈ Y} are positive and C > 0. Then, ρn ≥ 1 2 log n + O(1) This is an achievability result

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 18 / 29

SLIDE 50

Related Work: Achievability for Third-Order Term

Proposition (Polyanskiy (2010)) Assume that all elements of {W(y|x) : x ∈ X, y ∈ Y} are positive and C > 0. Then, ρn ≥ 1 2 log n + O(1) This is an achievability result BEC doesn’t satisfy assumptions

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 18 / 29

SLIDE 51

Related Work: Achievability for Third-Order Term

Proposition (Polyanskiy (2010)) Assume that all elements of {W(y|x) : x ∈ X, y ∈ Y} are positive and C > 0. Then, ρn ≥ 1 2 log n + O(1) This is an achievability result BEC doesn’t satisfy assumptions We will not try to improve on it

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 18 / 29

SLIDE 52

Related Work: Converse for Third-Order Term

Proposition (Polyanskiy (2010)) If W is weakly input-symmetric ρn ≤ 1 2 log n + O(1)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 19 / 29

SLIDE 53

Related Work: Converse for Third-Order Term

Proposition (Polyanskiy (2010)) If W is weakly input-symmetric ρn ≤ 1 2 log n + O(1) This is a converse result

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 19 / 29

SLIDE 54

Related Work: Converse for Third-Order Term

Proposition (Polyanskiy (2010)) If W is weakly input-symmetric ρn ≤ 1 2 log n + O(1) This is a converse result Gallager-symmetric channels are weakly input-symmetric

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 19 / 29

SLIDE 55

Related Work: Converse for Third-Order Term

Proposition (Polyanskiy (2010)) If W is weakly input-symmetric ρn ≤ 1 2 log n + O(1) This is a converse result Gallager-symmetric channels are weakly input-symmetric The set of weakly input-symmetric channels is very thin

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 19 / 29

SLIDE 56

Related Work: Converse for Third-Order Term

Proposition (Polyanskiy (2010)) If W is weakly input-symmetric ρn ≤ 1 2 log n + O(1) This is a converse result Gallager-symmetric channels are weakly input-symmetric The set of weakly input-symmetric channels is very thin We dispense of this symmetry assumption

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 19 / 29

SLIDE 57

Related Work: Converse for Third-Order Term

Proposition (Strassen (1964), PPV (2010)) If W is a DMC with positive ε-dispersion, ρn ≤

|X| − 1

2

log n + O(1)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 20 / 29

SLIDE 58

Related Work: Converse for Third-Order Term

Proposition (Strassen (1964), PPV (2010)) If W is a DMC with positive ε-dispersion, ρn ≤

|X| − 1

2

log n + O(1)

Every code can be partitioned into no more than (n + 1)|X|−1 constant-composition subcodes

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 20 / 29

SLIDE 59

Related Work: Converse for Third-Order Term

Proposition (Strassen (1964), PPV (2010)) If W is a DMC with positive ε-dispersion, ρn ≤

|X| − 1

2

log n + O(1)

Every code can be partitioned into no more than (n + 1)|X|−1 constant-composition subcodes M∗

P(Wn, ε): Max size of a constant-composition code with type P

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 20 / 29

SLIDE 60

Related Work: Converse for Third-Order Term

Proposition (Strassen (1964), PPV (2010)) If W is a DMC with positive ε-dispersion, ρn ≤

|X| − 1

2

log n + O(1)

Every code can be partitioned into no more than (n + 1)|X|−1 constant-composition subcodes M∗

P(Wn, ε): Max size of a constant-composition code with type P

As such, M∗(Wn, ε) ≤ (n + 1)|X|−1 max

P∈Pn(X) M∗ P(Wn, ε)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 20 / 29

SLIDE 61

Related Work: Converse for Third-Order Term

Proposition (Strassen (1964), PPV (2010)) If W is a DMC with positive ε-dispersion, ρn ≤

|X| − 1

2

log n + O(1)

Every code can be partitioned into no more than (n + 1)|X|−1 constant-composition subcodes M∗

P(Wn, ε): Max size of a constant-composition code with type P

As such, M∗(Wn, ε) ≤ (n + 1)|X|−1 max

P∈Pn(X) M∗ P(Wn, ε)

This is where the dependence on |X| comes in

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 20 / 29

SLIDE 62

Main Result: Tight Third-Order Term

Theorem (Tomamichel-Tan (2013)) If W is a DMC with positive ε-dispersion, ρn ≤ 1 2 log n + O(1)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 21 / 29

SLIDE 63

Main Result: Tight Third-Order Term

Theorem (Tomamichel-Tan (2013)) If W is a DMC with positive ε-dispersion, ρn ≤ 1 2 log n + O(1) The 1

2 cannot be improved without further assumptions

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 21 / 29

SLIDE 64

Main Result: Tight Third-Order Term

Theorem (Tomamichel-Tan (2013)) If W is a DMC with positive ε-dispersion, ρn ≤ 1 2 log n + O(1) The 1

2 cannot be improved without further assumptions

For BSC ρn = 1 2 log n + O(1)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 21 / 29

SLIDE 65

Main Result: Tight Third-Order Term

Theorem (Tomamichel-Tan (2013)) If W is a DMC with positive ε-dispersion, ρn ≤ 1 2 log n + O(1) The 1

2 cannot be improved without further assumptions

For BSC ρn = 1 2 log n + O(1) We can dispense of the positive ε-dispersion assumption as well

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 21 / 29

SLIDE 66

Main Result: Tight Third-Order Term

Theorem (Tomamichel-Tan (2013)) If W is a DMC with positive ε-dispersion, ρn ≤ 1 2 log n + O(1) The 1

2 cannot be improved without further assumptions

For BSC ρn = 1 2 log n + O(1) We can dispense of the positive ε-dispersion assumption as well No need for unique CAID

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 21 / 29

SLIDE 67

Main Result: Tight Third-Order Term

All cases are covered

✚✚✚✚ ❃ ❩❩❩❩ ⑦

Yes No Vε > 0 ≤nC−√nVεQ−1(ε)+ 1

2 log n+O(1)

✚✚✚✚ ❃ ❩❩❩❩ ⑦

Yes No not exotic

r ε< 1

2

≤nC+O(1)

✚✚✚✚ ❃ ❩❩❩❩ ⑦

Yes No exotic and ε= 1

2

≤nC+ 1

2 log n+O(1)

≤nC+O

n

1 3

[PPV10]

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 22 / 29

SLIDE 68

Proof Technique for Tight Third-Order Term

For the regular case, ρn ≤ 1

2 log n + O(1)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 23 / 29

SLIDE 69

Proof Technique for Tight Third-Order Term

For the regular case, ρn ≤ 1

2 log n + O(1)

The type-counting trick and upper bounds on M∗

P(Wn, ε) are not

sufficiently tight

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 23 / 29

SLIDE 70

Proof Technique for Tight Third-Order Term

For the regular case, ρn ≤ 1

2 log n + O(1)

The type-counting trick and upper bounds on M∗

P(Wn, ε) are not

sufficiently tight We need a new converse bound for general DMCs

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 23 / 29

SLIDE 71

Proof Technique for Tight Third-Order Term

For the regular case, ρn ≤ 1

2 log n + O(1)

The type-counting trick and upper bounds on M∗

P(Wn, ε) are not

sufficiently tight We need a new converse bound for general DMCs Information spectrum divergence Dε

s(PQ) := sup

R ∈ R
P
log P(X)

Q(X) ≤ R

≤ ε
“Information Spectrum Methods in Information Theory”

by T. S. Han (2003)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 23 / 29

SLIDE 72

Proof Technique: Information Spectrum Divergence

Dε

s(PQ) := sup

R ∈ R
P
log P(X)

Q(X) ≤ R

≤ ε
t

t ✲

“Density” of log P(X)

Q(X)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 24 / 29

SLIDE 73

Proof Technique: Information Spectrum Divergence

Dε

s(PQ) := sup

R ∈ R
P
log P(X)

Q(X) ≤ R

≤ ε
t

t ✲

“Density” of log P(X)

Q(X)

R∗ ε

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 24 / 29

SLIDE 74

Proof Technique: Information Spectrum Divergence

Dε

s(PQ) := sup

R ∈ R
P
log P(X)

Q(X) ≤ R

≤ ε
t

t ✲

“Density” of log P(X)

Q(X)

R∗ ε 1 − ε

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 24 / 29

SLIDE 75

Proof Technique: Information Spectrum Divergence

Dε

s(PQ) := sup

R ∈ R
P
log P(X)

Q(X) ≤ R

≤ ε
t

t ✲

“Density” of log P(X)

Q(X)

R∗ ε 1 − ε If Xn is i.i.d. P, the central limit theorem yields Dε

s(PnQn) ≈ nD(PQ) −

nV(PQ)Q−1(ε)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 24 / 29

SLIDE 76

Proof Technique: The New Converse Bound

Lemma (Tomamichel-Tan (2013)) For every channel W, every ε ∈ (0, 1) and δ ∈ (0, 1 − ε), we have log M∗(W, ε) ≤ min

Q∈P(Y) max x∈X Dε+δ s

(W(·|x)Q) + log 1 δ

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 25 / 29

SLIDE 77

Proof Technique: The New Converse Bound

Lemma (Tomamichel-Tan (2013)) For every channel W, every ε ∈ (0, 1) and δ ∈ (0, 1 − ε), we have log M∗(W, ε) ≤ min

Q∈P(Y) max x∈X Dε+δ s

(W(·|x)Q) + log 1 δ When DMC is used n times, log M∗(Wn, ε) ≤ min

Q(n)∈P(Yn) max x∈X n Dε+δ s

(Wn(·|x)Q(n)) + log 1 δ

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 25 / 29

SLIDE 78

Proof Technique: The New Converse Bound

Lemma (Tomamichel-Tan (2013)) For every channel W, every ε ∈ (0, 1) and δ ∈ (0, 1 − ε), we have log M∗(W, ε) ≤ min

Q∈P(Y) max x∈X Dε+δ s

(W(·|x)Q) + log 1 δ When DMC is used n times, log M∗(Wn, ε) ≤ min

Q(n)∈P(Yn) max x∈X n Dε+δ s

(Wn(·|x)Q(n)) + log 1 δ Choose δ = n− 1

2 so log 1

δ = 1 2 log n

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 25 / 29

SLIDE 79

Proof Technique: The New Converse Bound

Lemma (Tomamichel-Tan (2013)) For every channel W, every ε ∈ (0, 1) and δ ∈ (0, 1 − ε), we have log M∗(W, ε) ≤ min

Q∈P(Y) max x∈X Dε+δ s

(W(·|x)Q) + log 1 δ When DMC is used n times, log M∗(Wn, ε) ≤ min

Q(n)∈P(Yn) max x∈X n Dε+δ s

(Wn(·|x)Q(n)) + log 1 δ Choose δ = n− 1

2 so log 1

δ = 1 2 log n

Since all x within a type class result in the same Dε+δ

s

(if Q(n) is permutation invariant), it’s really a max over types Px ∈ Pn(X)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 25 / 29

SLIDE 80

Proof Technique: Choice of Output Distribution

log M∗(Wn, ε) ≤ max

x∈X n Dε+δ s

(Wn(·|x)Q(n)) + log 1 δ , ∀ Q(n) ∈ P(Yn) Q(n)(y): invariant to permutations of the n channel uses

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 26 / 29

SLIDE 81

Proof Technique: Choice of Output Distribution

log M∗(Wn, ε) ≤ max

x∈X n Dε+δ s

(Wn(·|x)Q(n)) + log 1 δ , ∀ Q(n) ∈ P(Yn) Q(n)(y): invariant to permutations of the n channel uses Q(n)(y) := 1 2

k∈K

λ(k)Qn

k(y) + 1

2

P∈Pn(X)

1 |Pn(X)|(PW)n(y)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 26 / 29

SLIDE 82

Proof Technique: Choice of Output Distribution

log M∗(Wn, ε) ≤ max

x∈X n Dε+δ s

(Wn(·|x)Q(n)) + log 1 δ , ∀ Q(n) ∈ P(Yn) Q(n)(y): invariant to permutations of the n channel uses Q(n)(y) := 1 2

k∈K

λ(k)Qn

k(y) + 1

2

P∈Pn(X)

1 |Pn(X)|(PW)n(y) First term: Qk’s and λ(k)’s designed to form an n− 1

2 -cover of P(Y):

∀ Q ∈ P(Y), ∃ k ∈ K s.t. Q − Qk2 ≤ n− 1

2 . Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 26 / 29

SLIDE 83

Proof Technique: Choice of Output Distribution

log M∗(Wn, ε) ≤ max

x∈X n Dε+δ s

(Wn(·|x)Q(n)) + log 1 δ , ∀ Q(n) ∈ P(Yn) Q(n)(y): invariant to permutations of the n channel uses Q(n)(y) := 1 2

k∈K

λ(k)Qn

k(y) + 1

2

P∈Pn(X)

1 |Pn(X)|(PW)n(y) First term: Qk’s and λ(k)’s designed to form an n− 1

2 -cover of P(Y):

∀ Q ∈ P(Y), ∃ k ∈ K s.t. Q − Qk2 ≤ n− 1

2 .

Second term: Mixture over output distributions induced by input types [Hayashi (2009)]

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 26 / 29

SLIDE 84

Proof Technique: Choice of Output Distribution

Q(n)(y) := 1 2

k∈K

λ(k)Qn

k(y) + 1

2

P∈Pn(X)

1 |Pn(X)|(PW)n(y)

✲ ✻

Q(0) Q(1)

(0, 1) (1, 0) P(Y)

✏ ✏ ✏ ✮ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 27 / 29

SLIDE 85

Proof Technique: Choice of Output Distribution

Q(n)(y) := 1 2

k∈K

λ(k)Qn

k(y) + 1

2

P∈Pn(X)

1 |Pn(X)|(PW)n(y)

✲ ✻

Q(0) Q(1)

(0, 1) (1, 0) P(Y)

✏ ✏ ✏ ✮ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ sQ∗

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 27 / 29

SLIDE 86

Proof Technique: Choice of Output Distribution

Q(n)(y) := 1 2

k∈K

λ(k)Qn

k(y) + 1

2

P∈Pn(X)

1 |Pn(X)|(PW)n(y)

✲ ✻

Q(0) Q(1)

(0, 1) (1, 0) P(Y)

✏ ✏ ✏ ✮ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ sQ∗ s s s s s s

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 27 / 29

SLIDE 87

Proof Technique: Choice of Output Distribution

Q(n)(y) := 1 2

k∈K

λ(k)Qn

k(y) + 1

2

P∈Pn(X)

1 |Pn(X)|(PW)n(y)

✲ ✻

Q(0) Q(1)

(0, 1) (1, 0) P(Y)

✏ ✏ ✏ ✮ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ sQ∗ s s s s s s

Q[−1,1] Q[1,−1] Q[2,−2] Q[−2,2]

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 27 / 29

SLIDE 88

Proof Technique: Choice of Output Distribution

Q(n)(y) := 1 2

k∈K

λ(k)Qn

k(y) + 1

2

P∈Pn(X)

1 |Pn(X)|(PW)n(y)

✲ ✻

Q(0) Q(1)

(0, 1) (1, 0) P(Y)

✏ ✏ ✏ ✮ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ ❅ sQ∗ s s s s s s

Q[−1,1] Q[1,−1] Q[2,−2] Q[−2,2]

1 √ 2n 1 √ 2n Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 27 / 29

SLIDE 89

Proof Technique: Summary

Q(n)(y) := 1 2

k∈K

λ(k)Qn

k(y) + 1

2

P∈Pn(X)

1 |Pn(X)|(PW)n(y) This construction ensures that for every type Px near the CAID is well-approximated by by a Qk(x)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 28 / 29

SLIDE 90

Proof Technique: Summary

Q(n)(y) := 1 2

k∈K

λ(k)Qn

k(y) + 1

2

P∈Pn(X)

1 |Pn(X)|(PW)n(y) This construction ensures that for every type Px near the CAID is well-approximated by by a Qk(x) Well in the sense that the loss is − log λ(k) = O(1) for every x such that Px is near the CAID

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 28 / 29

SLIDE 91

Proof Technique: Summary

Q(n)(y) := 1 2

k∈K

λ(k)Qn

k(y) + 1

2

P∈Pn(X)

1 |Pn(X)|(PW)n(y) This construction ensures that for every type Px near the CAID is well-approximated by by a Qk(x) Well in the sense that the loss is − log λ(k) = O(1) for every x such that Px is near the CAID For types Px far from the CAID, use the second part and I(Px, W) ≤ C′ < C

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 28 / 29

SLIDE 92

Summary and Food for Thought

We showed that for DMCs with positive ε-dispersion, log M∗(Wn, ε) ≤ nC − √ nVεQ−1(ε) + 1 2 log n + O(1)

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 29 / 29

SLIDE 93

Summary and Food for Thought

We showed that for DMCs with positive ε-dispersion, log M∗(Wn, ε) ≤ nC − √ nVεQ−1(ε) + 1 2 log n + O(1) How important is the assumption of discreteness?

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 29 / 29

SLIDE 94

Summary and Food for Thought

We showed that for DMCs with positive ε-dispersion, log M∗(Wn, ε) ≤ nC − √ nVεQ−1(ε) + 1 2 log n + O(1) How important is the assumption of discreteness? Does our uniform quantization technique extend to lossy source coding? [Ingber-Kochman (2010), Kostina-Verdú (2012)]

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 29 / 29

SLIDE 95

Summary and Food for Thought

We showed that for DMCs with positive ε-dispersion, log M∗(Wn, ε) ≤ nC − √ nVεQ−1(ε) + 1 2 log n + O(1) How important is the assumption of discreteness? Does our uniform quantization technique extend to lossy source coding? [Ingber-Kochman (2010), Kostina-Verdú (2012)] Alternate proof using Bahadur-Ranga Rao [Moulin (2012)]? P

1

n

i=1

Xi ≥ c

= Θ

exp(−nI(c)) √n

Vincent Tan (I2R and NUS)

Third-Order Asymptotics for DMCs HKTW Workshop 2013 29 / 29

SLIDE 96

Summary and Food for Thought

We showed that for DMCs with positive ε-dispersion, log M∗(Wn, ε) ≤ nC − √ nVεQ−1(ε) + 1 2 log n + O(1) How important is the assumption of discreteness? Does our uniform quantization technique extend to lossy source coding? [Ingber-Kochman (2010), Kostina-Verdú (2012)] Alternate proof using Bahadur-Ranga Rao [Moulin (2012)]? P

1

n

i=1

Xi ≥ c

= Θ

exp(−nI(c)) √n

This result has been used to refine the sphere-packing bound

[Altug-Wagner (2012)]

Vincent Tan (I2R and NUS) Third-Order Asymptotics for DMCs HKTW Workshop 2013 29 / 29