Zero-Error Coding with a Generator Set of Variable-Length Words - - PowerPoint PPT Presentation

Zero-Error Coding with a Generator Set of Variable-Length Words

Nicolas Charpenay, Maël le Treust 2020 IEEE International Symposium on Information Theory

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 1 / 25

Introduction

Zero-error information theory (channel coding)

⬩ Transmission over a discrete memoryless channel ⬩ Correct decoding of each codeword with probability 1 ▶ Zero error capacity : Maximal asymptotic feasible rate with these constraints ?

State of the art

⬩ Shannon (1956) defined zero-error capacity and built the first tools ⬩ Lovasz (1979) built the θ function (bound on zero-error capacity) ⬩ Answers for specific cases (channels with 4 or less inputs, bipartite channel graphs, etc...) ▶ For an arbitrary DMC, determining the zero-error capacity is a wide

• pen problem

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 2 / 25

Zero-error information theory

Definition [channel graph]

Given a discrete memoryless channel with transition probabilities W = (pY

y (x))x∈X,y∈Y, we define the channel graph :

GW ≐ (V(GW ), E(GW )) with V(GW ) = X and xx′ ∈ E(GW ) iff ∃y ∈ Y, Wx,y, Wx′,y > 0 (1) ▶ Examples : If ǫ ∈ (0, 1) then 1 2 A B X Y and

ǫ 1 − ǫ

1 2 A C B X Y both have the channel graph 1 2

1 − ǫ ǫ ǫ/2 ǫ/2 1 − ǫ

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 3 / 25

Zero-error information theory

Definition [independent set]

Consider a graph G = (V, E) : ⬩ An independent set S of G is a subset of V verifying : ∀s, s′ ∈ Q, ss′ ∉ E. ⬩ The independence number : α(G) ≐ max

S independent set of G #S

(2) ▶ Interpretation : α(GW ) is the maximum zero-error rate for one use

• f the channel W .

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 4 / 25

Zero-error information theory

Definition [strong product ⊠]

Let G = (V, E) and G′ = (V′, E′), we define : G ⊠ G′ ≐ (V(G ⊠ G′), E(G ⊠ G′)) where V(G ⊠ G′) = V × V′ and (v1, v′

1)(v2, v′ 2) ∈ E(G ⊠ G′) iff ( v1v2 ∈ E

• r v1 = v2

) AND ( v′

1v′ 2 ∈ E′

• r v′

1 = v′ 2

) (3) ▶ Example : Take G = G′ = 0 1 2 , then G ⊠ G′ is the king’s graph :

0,0 0,1 0,2 1,0 1,1 1,2 2,0 2,1 2,2

( Here it gives the channel graph for 2 channel uses. )

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 5 / 25

Zero-error information theory

▶ Interpretation : G⊠T

W

= GW ⊠ ... ⊠ GW

• T times

is the channel graph for T channel uses. Thus the following are equivalent : ⬩ xx′ ∈ E(G⊠T

W )

⬩ (xt)t≤T and (x′

t)t≤T lead to the same output sequence with positive

probability ⬩ For all t ≤ T, there exists yt such that Wxt,yt > 0 and Wx′

t,yt > 0 Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 6 / 25

Zero-error capacity

Definition (cf. Shannon [4])

Given a channel W we define its zero-error capacity : C0(W ) = lim

T→∞

1 T log (α(G⊠T

W ))

=

[Fekete] sup T∈N

1 T log (α(G⊠T

W ))

(4)

Fekete’s Lemma

▶ For all subadditive sequence u (i.e. : un+m ≤ un + um for all m, n), lim

n→∞ un n exists and is equal to inf n un n (it can be −∞).

▶ Remark : Note that log (α(G⊠T

W )) is superadditive. Idea : independent

sets generate a product independent set in the product graph. ▶ Interpretation : C0(W ) is the asymptotic maximum zero-error rate : limit of the rates

• f (St)t∈N, where St is a max independent set of G⊠T

W

for all t.

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 7 / 25

Our main example

▶ Generated by a 5-symbol noisy typewriter union a 1-symbol perfect channel. 1 2 3 4 5 1 2 3 4 5 X Y has the channel graph C5 ⊞ 1 : 1 2 3 4 5 ▶ Known zero-error capacity : log( √ 5 + 1) (consequence of Shannon’s theorem for ⊞ and Lovasz’ theorem : C0(C5) = log √ 5 )

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 8 / 25

Codes with variable length

Set of words and slicing

For a given finite set S, we define the set of words over S : S∗ ≐ ⋃

L∈N

SL with S0 = {ǫ} (ǫ denotes the empty word) (5) For w ∈ C, ∣w∣ is the length of w. Given S′ ⊆ S∗, let : S′

[L] ≐ {w ∈ S′ ∣ ∣w∣ = L}

(6)

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 9 / 25

Example of words concatenation

Take C = {0} ∪ {11, 23, 35, 42, 54} (⊆ 0, 5∗) : Possible words (concatenation) : ⋮ ⋮ ⋮ #C∗

[3] = 11

#C∗

[2] = 6

#C∗

[1] = 1

#C∗

[0] = 1

ǫ 11 23 35 42 54

054 042 035 023 011 000

00

110 230 350 420 540

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 10 / 25

First contribution

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 11 / 25

Codes with variable length

Definition [variable length code]

⬩ A variable length code for a channel (Wx,y)x∈X,y∈Y is a finite subset C of X ∗ (called generator set). ⬩ C is zero-error over W if it verifies : ∀c, c′ ∈ C such that c ≠ c′ and ∣c∣ ≤ ∣c′∣, cc′

∶∣c∣ ∉ E(G ⊠∣c∣ W

) with auto-adjacency convention.

Definition [variable length code rate]

Let C be a zero-error variable length code over W , we define its rate : r(C) ≐ lim

L→∞

1 L log #C∗

[L]

(7) Then we define the average number of distinguishable inputs : ν(C) ≐ lim

L→∞

L

√ #C∗

[L] = 2r(C)

(8)

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 12 / 25

Codes with variable length

Proposition

C0(W ) = supC r(C) ▶ Interpretation : Considering variable-length coding is considering families of fixed-length codes → same supremum rate.

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 13 / 25

Codes with variable length

Theorem [rate computation]

Consider a zero-error code with : ⬩ Generator set C nonempty ⬩ Max (resp. min) reached length l (resp. l) ⬩ r(C) = log ν(C) > 0 its zero-error rate Then ν(C) is the unique positive solution of X l = ∑l

i=l #C[i]X l−i, where

C[i] = {c ∈ C ∣ ∣c∣ = i}. ▶ Remark : An equivalent formulation is the linear recursive sequence for #C∗

[i] :

#C∗

[L] = l

∑

l=l

#C[l]#C∗

[L−l]

(9) The rate is the maximum eigenvalue of the transition matrix.

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 14 / 25

Codes with variable length

▶ Examples : Consider the channel C5 ⊞ 1, and two possible zero-error variable-length codes C ≐ {0, 11, 23, 35, 42, 54}, C′ ≐ {11, 23, 35, 42, 54, 001, 003}. Here are the rates of some generated fixed-length codes : 5 1 2 3 4 L = 3 ∶ #C∗

[3] = 11; rate = log(11)/3 ≃ 1.153

L = 4 ∶ #C∗

[4] = 41; rate = log(41)/4 ≃ 1.339

L = 5 ∶ #C∗

[5] = 96; rate = log(96)/5 ≃ 1.317

L = 3 ∶ #C′∗

[3] = 2; rate = log(2)/3 ≃ 0.333

L = 4 ∶ #C′∗

[4] = 25; rate = log(25)/4 ≃ 1.161

L = 5 ∶ #C′∗

[5] = 20; rate = log(20)/5 ≃ 1.864

Generator sets : C = {0, 11, 23, 35, 42, 54} C′ = {11, 23, 35, 42, 54, 001, 003} Polynomial equation : X 2 = X + 5 X 3 = 5X + 2 Value of ν : ν(C) = 1+

√ 21 2

ν(C′) = 1 + √ 2 Asymptotic rate : log ( 1+

√ 21 2

) ≃ 1.481 log(1 + √ 2) ≃ 1.272 These rates are inferior to the known capacity log(1 + √ 5) ≃ 1.694

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 15 / 25

Codes with variable length

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 16 / 25

Codes with variable length

▶ f1, f2, f3 obtained by studying secondary eigenvalues.

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 17 / 25

Second contribution

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 18 / 25

Intermingled codes

▶ Based on Shannon’s "intermingling patterns" (cf. [4, proof of Th. 4])

Definition [Intermingled codes]

An intermingled code is : ▶ A finite subset C ⊆ X ∗ with max. (resp. min.) reached length l (resp. l), ▶ A succession rule ρ ∶ ∏c∈C0, ∣c∣ − 1 → P(C) such that ρ(z) is nonempty for all z ∈ ∏c∈C0, ∣c∣ − 1

Coding patterns

The encoder chooses a time horizon T then maps the message to some sequence (σt)t≤T ∈ X T based on the following algorithm : ⬩ Initialize z ← (0, ..., 0) vector of size #C and t ← 0 ⬩ While t ≤ T : ⋄ Choose a word w ∈ ρ(z) (⊆ C) ⋄ Transmit σt = wzw+1 over the channel ⋄ zw ← zw + 1 mod ∣w∣ ⋄ t ← t + 1

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 19 / 25

Intermingled codes

▶ Example : Consider C5 ⊞ 1 and the intermingled code (C, ρ), where C = {0} ∪ {11, 23, 35, 42, 54}, ρ ∶ (z0, z11, z23, z35, z42, z54) ↦ {C if (z11, z23, z35, z42, z54) = (0, 0, 0, 0, 0) {0} ∪ {c ∈ C ∣ zc ≠ 0} otherwise (10)

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 20 / 25

An illustration of the coding patterns

Take C = {0} ∪ {11, 23, 35, 42, 54} on C5 ⊞ 1 : Possible words (intermingling) : ⋮ ⋮ ⋮ #S3 = 16 #S2 = 6 #S1 = 1 #S0 = 1 ǫ 1 2 3 4 5

10 20 30 40 50 01 02 03 04 05 11 23 35 42 54 00

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 21 / 25

Intermingled codes

Definition [zero-error property]

(C, ρ) is zero-error over W if for all generated sequences σ and σ′ from X T, there exists a time step t ≤ T such that σtσ′

t ≠ E(GW ).

Definition [rate of intermingled codes]

r(C, ρ) ≐ lim

T,∞

1 T log #ST (11) with ST = {σ ∈ X T

• σ can be generated by the coding patterns algorithm with

time horizon T, with zw = (0, ..., 0) at the last time step } ▶ Rate still evaluated with the generated fixed-length codes. ▶ Remark : Shannon’s theorem for ⊠ and ⊞ gives the optimality of such codes in particular cases (cf. [4]).

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 22 / 25

Intermingled codes

Definition [transition graph]

Let (C, ρ) be a zero-error intermingled code, the transition graph G = (V, E) of the transmission states is : V = ∏

c∈C

0, ∣c∣ − 1 ∀v, v′ ∈ V, vv′ ∈ E iff ∃i ∈ ρ(v), ∀j ≤ l, v′

j = vj + 1i=j

mod j (12) where (ei)i≤#V is the canonical basis of R#V.

Theorem [rate computation]

Let (C, ρ) be a zero-error intermingled code over the channel W with positive rate. Then : r(C, ρ) = log maxi ∣λi(MG)∣, where (λi(MG))i≤#V are the eigenvalues of the adjacency matrix of the transition graph.

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 23 / 25

Intermingled codes

The adjacency matrix of the transition graph with the possible transmission states (0, e2, e3, e4, e5, e6) (where (ei) denotes the canonical basis of R6) is : MG = ⎛ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠ Rate equal to log(maximal eigenvalue) : log( √ 5 + 1), it reaches the channel capacity. e2 e3 e4 e5 e6 ▶ Since ( √ 5 + 1)n ∉ N for all n > 0, any fixed-length code is suboptimal.

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 24 / 25

Conclusion

Contributions

⬩ A new approach for zero-error channel coding, based on concatenation of words with variable length ⬩ Theorem 1 : asymptotic rate characterized using a polynomial which depends on the generator set data ; still sub-optimal in some cases ⬩ A new approach based on intermingled coding with a generator set ⬩ Theorem 2 : asymptotic rate of intermingled coding characterized with the spectral radius of the adjacency matrix of its transition graph ; optimality on particular cases

Future work

⬩ Study more general classes of codes for this problem (e.g. automata-based codes)

Thanks for your attention !

Nicolas Charpenay, Maël le Treust Zero-Error Coding with a Generator Set of Variable-Length Words 25 / 25