Empirical Properties of Good Channel Codes Qinghua (Devon) Ding - - PowerPoint PPT Presentation

Empirical Properties of Good Channel Codes

Qinghua (Devon) Ding June 8, 2020

The Chinese University of Hong Kong

1

Introduction

Shannon’s Channel Coding Theorem

2

Shannon’s Channel Coding Theorem

C = maxPX I(X; Y) { R > C ⇒ ∀C, P(m ̸= ˆ m) → 1. R < C ⇒ ∃C, P(m ̸= ˆ m) → 0.

2

Shannon’s Channel Coding Theorem

C = maxPX I(X; Y) { R > C ⇒ ∀C, P(m ̸= ˆ m) → 1. R < C ⇒ ∃C, P(m ̸= ˆ m) → 0. P∗

X = arg max PX I(X; Y) need not be unique. 2

Result I: Characterization of the P∗

X

Consider a channel W = (p1, ..., p|X|), denote r = (H(p1), ..., H(pm)).

3

Result I: Characterization of the P∗

X

Consider a channel W = (p1, ..., p|X|), denote r = (H(p1), ..., H(pm)). Given some P∗

X ∈ arg maxPX I(X; Y) (e.g. by Blahut-Amiroto algorithm). 3

Result I: Characterization of the P∗

X

Consider a channel W = (p1, ..., p|X|), denote r = (H(p1), ..., H(pm)). The whole set of capacity-achieving input distribution is P∗

X =

{ P∗

X + ker

(W r )} ∩ Rm

+.1

1A non-linear equation system for P∗ X is developed in [Mur53] and its followup works.

There’s no analytical solutions in general.

3

Optimizing Input Distribution

Suppose W has unique CAID P∗

X with supp(P∗ X) = X ′ ⊂ X. 4

Optimizing Input Distribution

Suppose W has unique CAID P∗

X with supp(P∗ X) = X ′ ⊂ X.

• Claim. The distributions {pi, i ∈ X ′} should be linearly independent.

4

Optimizing Input Distribution

Suppose W has unique CAID P∗

X with supp(P∗ X) = X ′ ⊂ X.

• Claim. The distributions {pi, i ∈ X ′} should be linearly independent.

Proof by contrapositive (details later).

4

Property of Random Code Ensemble

C = maxPX I(X; Y) { R > C ⇒ ∀C, P(W ̸= ˆ W) → 1. R < C ⇒ ∃C, P(W ̸= ˆ W) → 0.

5

Property of Random Code Ensemble

C = maxPX I(X; Y) { R > C ⇒ ∀C, P(W ̸= ˆ W) → 1. R < C ⇒ ∃C, P(W ̸= ˆ W) → 0. Random code ensemble is capacity-achieving.

5

Property of Random Code Ensemble

Random codes: each alphabet i.i.d. from P∗

X.

P∗

X ∈ arg maxPX I(X; Y) 6

Property of Random Code Ensemble

Random codes: each alphabet i.i.d. from P∗

X.

P∗

X ∈ arg maxPX I(X; Y)

Empirical independence: #{(xi, x′

i) = (a, b)} ≈ nP∗ X(a)P∗ X(b).2

2This condition is difgerent from [HV93, PV13, SV97].

6

Property of Random Code Ensemble

Random codes: each alphabet i.i.d. from P∗

X.

P∗

X ∈ arg maxPX I(X; Y)

Empirical independence: #{(xi, x′

i) = (a, b)} ≈ nP∗ X(a)P∗ X(b).2

• Observation. Random codes have “most” (1 − o(1) fraction)

codeword pairs that’s empirically independent, w.h.p.

2This condition is difgerent from [HV93, PV13, SV97].

6

Property of Random Code Ensemble

Random codes: each alphabet i.i.d. from P∗

X.

P∗

X ∈ arg maxPX I(X; Y)

#{(xi, x′

i, x′′ i ) = (a, b, c)} ≈ nPX∗(a)PX∗(b)PX∗(c).

Generalization to k = O(1). Random codes have “most” codeword k-tuples that’s empirically independent, w.h.p.

6

Result II: Necessary Conditions for Good Codes

Capacity-achieving code (or good code) { R = C − ϵ, P(m ̸= ˆ m) → 0.

7

Result II: Necessary Conditions for Good Codes

Capacity-achieving code (or good code) { R = C − ϵ, P(m ̸= ˆ m) → 0. Theorem (Property of Good Codes) For any DMC with unique P∗

X, any good code for it should have

1 − o(1) fraction of codeword k-tuples empirically independent.

7

Result II: Necessary Conditions for Good Codes

Capacity-achieving code (or good code) { R = C − ϵ, P(m ̸= ˆ m) → 0. Theorem (Property of Good Codes) For any DMC with unique P∗

X, any good code for it should have

1 − o(1) fraction of codeword k-tuples empirically independent. Similar results holds for AWGN channel.

7

Advertisement Parallel work [ZVJ20] on Quadratically Constrained Two-Way Adversarial Channels ISIT 2020 https://sites.google.com/view/yihan/

8

Advertisement Parallel work [ZVJ20] on Quadratically Constrained Two-Way Adversarial Channels ISIT 2020 https://sites.google.com/view/yihan/

8

Result III: Non-universality of Good Codes

Two channels W and W′ are similar ifg P∗

X = P∗ X′ and C = C′. 9

Result III: Non-universality of Good Codes

Two channels W and W′ are similar ifg P∗

X = P∗ X′ and C = C′.

• Observation. Random code ensemble with alphabet

i.i.d.

∼ P∗

X achieves

capacity under vanishing error probability for all similar channels.

9

Result III: Non-universality of Good Codes

Two channels W and W′ are similar ifg P∗

X = P∗ X′ and C = C′.

• Observation. Random code ensemble with alphabet

i.i.d.

∼ P∗

X achieves

capacity under vanishing error probability for all similar channels. Theorem (Non-universality of Good Codes) There exists similar DMCs W, W′ and code C that’s capacity-achieving for W, s.t. no expurgation of C with the same rate is good for W′.

9

Result III: Non-universality of Good Codes

10

Result III: Non-universality of Good Codes

10

Result III: Non-universality of Good Codes

10

Proof Ideas

Proof to Characterization Result

For DMC W, given some PX∗ ∈ arg maxPX I(X; Y), we have P∗

X =

{ P∗

X + ker

(W r )} ∩ Rm

+. 11

Proof to Characterization Result

For DMC W, given some PX∗ ∈ arg maxPX I(X; Y), we have P∗

X =

{ P∗

X + ker

(W r )} ∩ Rm

+.

Proof by standard linear algebra.

11

Proof to Characterization Result

Generalizing to k-use channel, we have P∗

Xk =

{ P∗⊗k

X

+ ker (W⊗k r(k) )} ∩ Rmk

+ . 12

Proof to Characterization Result

Generalizing to k-use channel, we have P∗

Xk =

{ P∗⊗k

X

+ ker (W⊗k r(k) )} ∩ Rmk

+ .

Consider the following noisy typewritter channel. W =      1/2 1/2 1/2 1/2 1/2 1/2 1/2 1/2      Although C and P∗

Y tensorize, P∗ Xk does not tensorize. 12

Proof to Linear Indepence Lemma

Suppose W has unique CAID P∗

X with supp(P∗ X) = X ′ ⊂ X.

• Claim. The distributions {pi, i ∈ X ′} should be linearly independent.

Proof by contrapositive.

13

Proof to Linear Indepence Lemma

Suppose linear independence does not hold.

14

Proof to Linear Indepence Lemma

Suppose linear independence does not hold. We can find feasible direction δ ̸= 0 such that ⟨r, δ⟩ = 0, Wδ = 0.

14

Proof to Linear Indepence Lemma

Suppose linear independence does not hold. We can find feasible direction δ ̸= 0 such that ⟨r, δ⟩ = 0, Wδ = 0. Then I(PX∗ + ϵδ; PY) = C for small enough ϵ, contradiction!

14

Proof to Empirical Independence Property

Consider a discrete memoryless channel W with unique PX∗.

15

Proof to Empirical Independence Property

Consider a discrete memoryless channel W with unique PX∗.

• Claim. Any good code C for W has the property that ∀δ > 0,

Px1,...,xk∼C(∥τx1,...,xk − P∗⊗k

X

∥1 > δ) → 0.

15

Proof to Empirical Independence Property

Consider a discrete memoryless channel W with unique PX∗.

• Claim. Any good code C for W has the property that ∀δ > 0,

Px1,...,xk∼C(∥τx1,...,xk − P∗⊗k

X

∥1 > δ) → 0. Proof by considering the k-use channel.

15

Proof to Empirical Independece Property

16

Proof to Empirical Independece Property

16

Proof to Empirical Independece Property

16

Proof to Empirical Independece Property

16

Proof to Empirical Independence Property (Cont’d)

Consider the AWGN(P, N) channel denoted as W.

17

Proof to Empirical Independence Property (Cont’d)

Consider the AWGN(P, N) channel denoted as W.

• Claim. Any good code for W has the property that ∀δ > 0,

PX1,...,Xk∼C(∃i ̸= j, |⟨Xi, Xj⟩| > δn) → 0.

17

Proof to Empirical Independence Property (Cont’d)

Consider the AWGN(P, N) channel denoted as W.

• Claim. Any good code for W has the property that ∀δ > 0,

PX1,...,Xk∼C(∃i ̸= j, |⟨Xi, Xj⟩| > δn) → 0. Proof by contrapositive.

17

Properties of Good Channel Codess

Suppose the codewords are empirically correlated.

18

Properties of Good Channel Codess

Suppose the codewords are empirically correlated. Then we can extract a large subcode good for another channel with P′ < P, contradiction!

18

Non-universality of Good Channel Codes

Non-universality of Good Channel Codes

Consider similar channels BEC(H(p)) and BSC(p). 2

2The figures are from [CT12].

19

Non-universality of Good Channel Codes

Consider similar channels BEC(H(p)) and BSC(p). 2

• Claim. There exists good code for BEC(H(p)) such that no expurgated

subcode of the same rate can be good for BSC(p).

2The figures are from [CT12].

19

Non-universality of Good Channel Codes

Consider similar channels BEC(H(p)) and BSC(p). 2

• Claim. There exists good code for BEC(H(p)) such that no expurgated

subcode of the same rate can be good for BSC(p). Proof by construction.

2The figures are from [CT12].

19

Non-universality of good codes

We can show certain superposition code good for BEC(H(p)) can be bad for BSC(p) even for more powerful decoders.

20

Conclusions

Conclusions

• 1. Capacity-achieving input distributions.

21

Conclusions

• 1. Capacity-achieving input distributions.
• 2. Necessary conditions on good codes.

21

Conclusions

• 1. Capacity-achieving input distributions.
• 2. Necessary conditions on good codes.
• 3. Non-universality of good codes.

21

References i

References

[CT12] Thomas M Cover and Joy A Thomas. Elements of information

• theory. John Wiley & Sons, 2012.

[HV93] Te Sun Han and Sergio Verdú. Approximation theory of

• utput statistics. IEEE Transactions on Information Theory,

39(3):752–772, 1993. [Mur53] Saburo Muroga. On the capacity of a discrete channel. i mathematical expression of capacity of a channel which is disturbed by noise in its every one symbol and expressible in

• ne state diagram. Journal of the Physical Society of Japan,

8(4):484–494, 1953.

22

References ii

[PV13] Yury Polyanskiy and Sergio Verdú. Empirical distribution of good channel codes with nonvanishing error probability. IEEE transactions on information theory, 60(1):5–21, 2013. [SV97] Shlomo Shamai and Sergio Verdú. The empirical distribution

• f good codes. IEEE Transactions on Information Theory,

43(3):836–846, 1997. [ZVJ20] Yihan Zhang, Shashank Vatedka, and Sidharth Jaggi. Quadratically constrained two-way adversarial channels. arXiv preprint arXiv:2001.02575, 2020.

23