Lecture 4 Channel Coding I-Hsiang Wang Department of Electrical - - PowerPoint PPT Presentation

Channel Capacity and the Weak Converse Achievability Proof Summary

Lecture 4 Channel Coding

I-Hsiang Wang

Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw

October 15, 2014

1 / 16 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse Achievability Proof Summary

The Channel Coding Problem

Channel Encoder Channel Decoder

xN yN

Noisy Channel

w b w

Meta Description

1 Message: Random message W ∼ Unif [1 : 2K]. 2 Channel: Consist of an input alphabet X, an output alphabet Y,

and a family of conditional distributions { p ( yk|xk, yk−1) | k ∈ N } determining the stochastic relationship between the output symbol yk and the input symbol xk along with all past signals ( xk−1, yk−1) .

3 Encoder: Encode the message w by a length N codeword xN ∈ X N. 4 Decoder: Reconstruct message

w from the channel output yN.

5 Efficiency: Maximize the code rate R := K N bits/channel use, given

certain decoding criterion.

2 / 16 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse Achievability Proof Summary

Decoding Criterion: Vanishing Error Probability

Channel Encoder Channel Decoder

xN yN

Noisy Channel

w b w

A key performance measure: Error Probability P(N)

e

:= Pr { W ̸= W } . Question: Is it possible to get zero error probability? Ans: Probably not, unless the channel noise has some special structure. Following the development of lossless source coding, Shannon turned the attention to answering the following question: Is it possible to have a sequence of encoder/decoder pairs such that P(N)

e

→ 0 as N → ∞? If so, what is the largest possible code rate R where vanishing error probability is possible? Note: In lossless source coding, we see that the infimum of compression rates where vanishing error probability is possible is H ({Si}).

3 / 16 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse Achievability Proof Summary

Rate

R

Block Length

N P (N)

e

Probability

• f Error

Take N → ∞, Require P(N)

e

→ 0: sup R = C. capacity Take N → ∞: min P(N)

e

≈ 2−NE(R). error exponent Fix N, Require P(N)

e

≤ ϵ: sup R ≈ C − √

V n Q−1 (ϵ).

finite length

4 / 16 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse Achievability Proof Summary

In this course we only focus on capacity. In other words, we ignore the issue of delay and do not pursue finer analysis of the error probability via large deviation techniques.

5 / 16 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse Achievability Proof Summary

Discrete Memoryless Channel (DMC)

In order to demonstrate the key ideas in channel coding, in this lecture we shall focus on discrete memoryless channels (DMC) defined below. Definition 1 (Discrete Memoryless Channel) A discrete channel ( X, { p ( yk|xk, yk−1) | k ∈ N } , Y ) is memoryless if ∀ k ∈ N, p ( yk|xk, yk−1) = pY|X (yk|xk) . In other words, Yk − Xk − ( Xk−1, Yk−1) . Here the conditional p.m.f. pY|X is called the channel law or channel transition function. Question: is our definition of a channel sufficient to specify p ( yN|xN) , the stochastic relationship between the channel input (codeword) xN and the channel output yN?

6 / 16 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse Achievability Proof Summary

p ( yN|xN) = p ( xN, yN) p (xN) p ( xN, yN) =

N

∏

k=1

p ( xk, yk|xk−1, yk−1) =

N

∏

k=1

p ( yk|xk, yk−1) p ( xk|xk−1, yk−1) Hence, we need to further specify { p ( xk|xk−1, yk−1) | k ∈ N } , which cannot be obtained from p ( xN) . Interpretation: { p ( xk|xk−1, yk−1) | k ∈ N } is induced by the encoding function, which implies that the encoder can potentially make use of the past channel output, i.e., feedback.

7 / 16 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse Achievability Proof Summary

DMC without Feedback

Channel Encoder

xk yk

Noisy Channel

w

(a) No Feedback

Channel Encoder

xk yk

Noisy Channel

w

D

yk−1

(b) With Feedback

Suppose in the system, the encoder has no knowledge about the realization of the channel output, then, p ( xk|xk−1, yk−1) = p ( xk|xk−1) for all k ∈ N, and it is said the the channel has no feedback. In this case, specifying { p ( yk|xk, yk−1) | k ∈ N } suffices to specify p ( yN|xN) . Proposition 1 (DMC without Feedback) For a DMC ( X, pY|X, Y ) without feedback, p ( yN|xN) =

N

∏

k=1

pY|X (yi|xi).

8 / 16 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse Achievability Proof Summary

Overview

In this lecture, we would like to establish the following (informally described) noisy channel coding theorem due to Shannon: For a DMC ( X, pY|X, Y ) , the maximum code rate with vanishing error probability is the channel capacity C := max

pX(·) I (X; Y) .

The above holds regardless of the availability of feedback. To demonstrate this beautiful result, we organize this lecture as follows:

1 Give the problem formulation, state the main theorem, and visit a

couple of examples to show how to compute channel capacity.

2 Prove the converse part: an achievable rate cannot exceed C. 3 Prove the achievability part with a random coding argument.

9 / 16 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse Achievability Proof Summary

1 Channel Capacity and the Weak Converse 2 Achievability Proof 3 Summary

10 / 16 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse Achievability Proof Summary

Channel Coding: Problem Setup

Channel Encoder Channel Decoder

xN yN

Noisy Channel

w b w

1 A

( 2NR, N ) channel code consists of

an encoding function (encoder) encN : [1 : 2K] → X N that maps each message w to a length N codeword xN, where K := ⌈NR⌉. a decoding function (decoder) decN : YN → [1 : 2K] ∪ {∗} that maps a channel output sequence yN to a reconstructed message w or an error message ∗.

2 The error probability is defined as P(N) e

:= Pr { W ̸= W } .

3 A rate R is said to be achievable if there exist a sequence of

( 2NR, N ) codes such that P(N)

e

→ 0 as N → ∞. The channel capacity is defined as C := sup {R | R : achievable}.

11 / 16 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse Achievability Proof Summary

Channel Coding Theorem for Discrete Memoryless Channel

Channel Encoder Channel Decoder

xN yN

Noisy Channel

w b w

Theorem 1 (Channel Coding Theorem for DMC) The capacity of the DMC p (y|x) is given by C = max

p(x) I (X; Y) ,

regardless of the availability of feedback.

12 / 16 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse Achievability Proof Summary

Proof of the (Weak) Converse (1)

We would like to show that for every sequence of ( 2NR, N ) codes such that P(N)

e

→ 0 as N → ∞, the rate R ≤ max

p(x) I (X; Y).

pf: Note that W ∼ Unif [1 : 2K] and hence K = H (W). NR ≤ H (W) = I ( W; W ) + H ( W

• W

) (1) ≤ I ( W; YN) + ( 1 + P(N)

e

log ( 2K + 1 )) (2) ≤

N

∑

k=1

I ( W; Yk|Yk−1) + ( 1 + P(N)

e

(NR + 2) ) (3) (1) is due to K = ⌈NR⌉ ≥ NR and chain rule. (2) is due to W − YN − W and Fano’s inequality. (3) is due to chain rule and 2K + 1 ≤ 2NR+1 + 1 ≤ 2 × 2NR+1.

13 / 16 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse Achievability Proof Summary

Proof of the (Weak) Converse (2)

Set ϵN := 1

N

( 1 + P(N)

e

(NR + 2) ) , we see that ϵN → 0 as N → ∞ because limN→∞ P(N)

e

= 0. The next step is to relate ∑N

k=1 I

( W; Yk|Yk−1) to I (X; Y), by the following manipulation: I ( W; Yk|Yk−1) ≤ I ( W, Yk−1; Yk ) ≤ I ( W, Yk−1, Xk; Yk ) (4) = I (Xk; Yk) ≤ max

p(x) I (X; Y)

(5) (4) is due to the fact that conditioning reduces entropy. (5) is due to DMC: Yk − Xk − ( Xk−1, Yk−1) . Hence, we have R ≤ maxp(x) I (X; Y) + ϵN for all N. Taking N → ∞, we conclude that R ≤ maxp(x) I (X; Y) if it is achievable.

14 / 16 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse Achievability Proof Summary

1 Channel Capacity and the Weak Converse 2 Achievability Proof 3 Summary

15 / 16 I-Hsiang Wang NIT Lecture 4

Channel Capacity and the Weak Converse Achievability Proof Summary

1 Channel Capacity and the Weak Converse 2 Achievability Proof 3 Summary

16 / 16 I-Hsiang Wang NIT Lecture 4