Lecture 2 Matthieu Bloch 1 Channel coding problem As illustrated - - PDF document

Revised December 5, 2019 Information Theoretic Security

Lecture 2

Matthieu Bloch

1 Channel coding problem As illustrated in Figure 1, the problem of channel coding consists in designing an encoder and decoder to reliably transmit messages over a noisy channel. Tie noisy channel is characterized by a triplet (X, {WY n|Xn}n⩾1, Y}; X and Y represent the input and output alphabets of transmitted and received symbols, respectively; {WY n|Xn}n⩾1 is the set of transition probabilities characterizing the channel noise affecting sequence of n input symbols for every n ∈ N∗. Messages are represented by the random variable W, which the encoder maps to codewords of n symbols X and transmits

• ver the channel; the decoder estimates

W from the noisy received sequence Y. We assume that messages are uniformly distributed and that the statistics of the channel are known ahead of time.

W

ENCODER X

WY n|Xn Y

DECODER

ˆ W

Figure 1: Channel coding over a noisy channel. An (M, n) channel code consists of two stochastic maps f : 1, M → X n, to encode messages into codewords, and g : Yn → 1, M∪{?}, which outputs an estimate of the transmitted message

• r an error symbol “?.” Tie parameter n is called the blocklength and 1

n log2 M is the rate of the

code, which measures the number of bits transmitted per symbol. Tie set {f(m) : m ∈ 1, M} forms the codebook and its component are the codewords. With a slight abuse of notation, we use C to denote both an (M, n) channel code with its associated encoder and decoder and its codebook. Tie performance of a code C is measured in terms of the average probability of error Pe(C) ≜ P

• W = W|C
• = 1

M

M

• m=1

P(g(Y) = m|W = m)

• r in terms of the maximal probability of error

P max

e

(C) ≜ max

m∈1,M P(g(Y) = m|W = m).

Remark 1.1. Our definition allows the encoder and the decoder to be stochastic, even though we will

• ften show the existence or construct channel codes with deterministic encoder and decoders. Note that a

stochastic encoder allows possibly several codewords to represent the same message. Definition 1.2 (Achievable channel coding rate and channel capacity). A rate R is an achievable channel coding rate if there exists a sequence {Cn}n⩾1 of (Mn, n) channel codes with increasing block- length such that lim inf

n→∞

1 n log Mn ⩾ R and lim sup

n→∞

Pe(Cn) = 0. Tie supremum of all achievable channel coding rates is called the channel capacity, denoted by C({WY n|Xn}n⩾1). 1

Revised December 5, 2019 Information Theoretic Security

Tie notion of achievable rate is asymptotic in the blocklength n and disregards any complexity constraints at the encoder and decoder; the notion of capacity is therefore a fundamental limit independent of any technological constraint. In general, one does not worry about identifying explicit low-complexity channel codes approaching the limits when characterizing channel capacity. Tie design of low-complexity encoders and decoders for codes operating at rates approaching the capacity is a challenging problem in itself, and several families of channel codes will be discussed later. 2 Random coding for channel reliability Tie proof of existence of codes with rates achieving capacity that we develop here relies on a tech- nique known as random coding. Intuitively, the idea is to randomly generate a codebook by inde- pendently sampling its codewords according to a prescribed probability distribution, and to analyze the probability of error averaged over the set of all possible codebooks. Averaging over a set of codes instead of studying a specific code considerably simplifies the analysis by making the exact structure

• f the codebook disappear from the analysis.

Formally, we consider a generic channel

• U, WV |U, V

, in which the alphabets U and V are arbitrary, and we construct an (M, 1) code. Let C = {ui : i ∈ 1, M} be a codebook of M codewords obtained by independently sampling the same distribution pU ∈ P(U). Tie distribution

• f the random variable C representing the random codebook is then

∀u ∈ UM pC (u) =

M

• i=1

pU(ui), and for any function φ : UM → R : C = (u1, . . . , uM) → φ(u1, . . . , uM), we have EC(φ(C)) =

• u1

pU(u1) · · ·

• ui

pU(ui) · · ·

• uM

pU(uM) φ(u1, . . . , ui, . . . , uM). Let pV (v) ≜

u WV |U(v|u)pU(u) and for γ > 0

Aγ ≜

• (u, v) ∈ U × V : log WV |U(v|u)

pV (v) ⩾ γ

• .

Define the encoder as the mapping f : 1, M → U : i → ui. Define the decoder g : V → 1, M∪{?} : v → i∗, where i∗ = j if uj is the unique codeword such that (uj, v) ∈ Aγ; otherwise, an error i∗ =? is declared. Tiis decoding operation is called threshold decoding. Although threshold decoding is suboptimal, it turns out to be sufficient to obtain optimal asymptotic results. Note that both encoder and decoder are deterministic. Tie probability of decoding error Pe(C) under threshold decoding averaged over the randomly generated codebook C satisfies the following. Lemma 2.1 (Random coding for channel reliability). EC(Pe(C)) ⩽ PpUWV |U ((U, V ) / ∈ Aγ) + M2−γ.

• Proof. Let us first explicit Pe(C) for any code C = {ui : i ∈ 1, M}. When transmitting message

i, the channel output is distributed according to WV |U(v|ui). Consequently, using the definition 2

Revised December 5, 2019 Information Theoretic Security

• f the threshold decoder, we obtain

Pe(C) = 1 M

M

• i=1

P(g(V ) = i|W = i) = 1 M

M

• i=1
• v

WV |U(v|ui)1{g(v) = i|W = i} = 1 M

M

• i=1
• v

WV |U(v|ui)1{(ui, v) / ∈ Aγ or ∃j = isuch that (uj, v) ∈ Aγ} . It is convenient to split the two predicates in the indicator function and bound Pe(C) as Pe(C) ⩽ 1 M

M

• i=1
• v

WV |U(v|ui)1{(ui, v) / ∈ Aγ} + 1 M

M

• i=1
• v

WV |U(v|ui)

• j∈1,M,j̸=i

1{(uj, v) ∈ Aγ} . (1) Let us study separately the expected value over C of the two terms in the right-hand side of (1). Denote {Ui}i∈1,Mn the random variables representing the randomly generated codewords in the random codebook C. First, EC

• 1

M

M

• i=1
• v

WV |U(v|Ui)1{(Ui, v) / ∈ Aγ}

• = 1

M

M

• i=1
• v

EUi

• WV |U(v|Ui)1{(Ui, v) /

∈ Aγ}

• = 1

M

M

• i=1
• v
• ui

pU(ui)WV |U(v|ui)1{(ui, v) / ∈ Aγ} =

• v
• u

pU(u)WV |U(v|u)1{(u, v) / ∈ Aγ} , = PpUWV |U ((U, V ) / ∈ Aγ) (2) where we have remarked that ui is merely a dummy index that does not depend on i, which we can replace by a generic index u. Next, EC   1 M

M

• i=1
• v

WV |U(v|Ui)

• j∈1,M,j̸=i

1{(Uj, v) ∈ Aγ}   = 1 M

M

• i=1
• j∈1,M\{i}
• v

EUiUj

• WV |U(v|Ui)1{(Uj, v) ∈ Aγ}
• = 1

M

M

• i=1
• j∈1,M\{i}
• v
• ui

pU(ui)

• uj

pU(uj)WV |U(v|ui)1{(uj, v) ∈ Aγ} = (M − 1)

• u
• u′
• v

WV |U(v|u)pU(u)pU(u′)1{(u′, v) ∈ Aγ} , (3) 3

Revised December 5, 2019 Information Theoretic Security

where we have replaced the dummy indices ui and uj by generic indices u and u′, respectively. Note that M − 1 ⩽ M and

u WV |U(v|u)pU(u) = pV (v). In addition, for (u′, v) ∈ Aγ, we have

pV (v) ⩽ WV |U(v|u′)2−γ. Using these facts with (3), we obtain EC   1 M

M

• i=1
• v

WV |U(v|Ui)

• j∈1,M,j̸=i

1{(Uj, v) ∈ Aγ}   ⩽ M

• u′

pV (v)pU(u′)1{(u′, v) ∈ Aγ} ⩽ M

• u′
• v

W(v|u′)2−γpU(u′)1{(u′, v) ∈ Aγ} ⩽ M2−γ, (4) since 1{(u′, v) ∈ Aγ} ⩽ 1 and

u′

• v W(v|u′)pU(u′) = 1. Finally, we obtain the desired result

by combining the bounds (2) and (4) with (1).

■

Tie upper bound given in Lemma 2.1 is by no means the best one since the decoding procedure is sub-optimal. Tiere exist alternative techniques to develop bounds, which are explored as exercises. Nevertheless, Lemma 2.1 is “good enough” to recover the first order fundamental limits, and we will therefore content ourself with the result. Tie form of Lemma 2.1 is often referred to as a “one-shot” result, since it considers codes with blocklength 1. With a direct application of Markov’s inequality, we obtain the following result. Proposition 2.2. Let

• U, WV |U, V

be a channel and let pU ∈ P(U). For any M ∈ N∗ and γ > 0, there exists an (M, 1) channel code C with deterministic encoder and decoder such that Pe(C) ⩽ PpUWV |U ((U, V ) / ∈ Aγ) + M2−γ. 4