Lecture 9 Polar Coding I-Hsiang Wang Department of Electrical - - PowerPoint PPT Presentation

Polarization

Lecture 9 Polar Coding

I-Hsiang Wang

Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw

December 29, 2015

1 / 25 I-Hsiang Wang IT Lecture 9

Polarization

In Pursuit of Shannon’s Limit

Since 1948, Shannon’s theory has drawn the sharp boundary between the possible and the impossible in data compression and data transmission. Once fundamental limits are characterized, the next natural question is:

How to achieve these limits with acceptable complexity?

For source coding, soon after Shannon’s 1948 paper, information and coding theorists found optimal compression schemes with low complexity: Huffman Code (1952): optimal for memoryless source Lempel-Ziv (1977): optimal for stationary ergodic source On the other hand, for channel coding, it turns out be a much harder

• problem. It has been the holy grail for coding theorist to find a coding

scheme that achieves Shannon’s limit with low complexity.

2 / 25 I-Hsiang Wang IT Lecture 9

Polarization

In Pursuit of Capacity-Achieving Codes

Two barriers in pursuing a low-complexity capacity-achieving codes:

1 Lack of explicit construction. In Shannon’s proof, it is only

proved that there exists coding schemes that achieve capacity.

2 Lack of structure to reduce complexity. In the proof of coding

theorems, complexity issues are often neglected, while codes with structures are hard to prove to achieve capacity. Since 1990’s, there are several practical codes found to approach capacity, including turbo code, low-density parity-check (LDPC) code, etc. These codes perform very well empirically, but still in lack of theoretical investigation on the performances and even proof of optimality. The first provably capacity-achieving coding scheme with acceptable complexity is polar code, introduced by Erdal Arıkan in 2007. Later in 2012, spatially coupled LDPC codes were also shown to achieve capacity (Shrinivas Kudekar, Tom Richardson, and Rüediger Urbanke).

3 / 25 I-Hsiang Wang IT Lecture 9

Polarization

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 55, NO. 7, JULY 2009 3051

Channel Polarization: A Method for Constructing Capacity-Achieving Codes for Symmetric Binary-Input Memoryless Channels

Erdal Arıkan, Senior Member, IEEE

The paper wins the 2010 Information Theory Society Best Paper Award.

4 / 25 I-Hsiang Wang IT Lecture 9

Polarization

Overview

When Arıkan introduced polar codes in 2007, he focus on achieving capacity for the general binary-input memoryless symmetric channels (BMS), including BSC, BEC, etc. Later, polar codes are shown to be optimal in many other settings, including lossy source coding, non-binary-input channels, multiple access channels, source coding with side information (Wyner-Ziv problem), etc. Instead of giving a comprehensive introduction, we shall introduce channel polarization and polar coding for BMS, in the following order:

1 First we introduce the concept of channel polarization. 2 Then we explore polar coding.

5 / 25 I-Hsiang Wang IT Lecture 9

Polarization

Notations

Recall in channel coding, we use the DMC N times with N being the blocklength of the coding scheme. Since the channel is the main focus, we shall use the following notations throughout this lecture: W to denote the channel pY|X P to denote the input distribution pX I (P, W ) to denote I (X ; Y ). Beside, since we focus on BMS channels, and it is not difficult to prove that X ∼ Ber ( 1

2

) achieves the channel capacity of any BMS, we shall use I (W ) (abuse of notation) to denote I (P, W ) when the input P is Ber ( 1

2

) . In other words, the channel capacity of the BMS channel W is I (W ).

6 / 25 I-Hsiang Wang IT Lecture 9

Polarization Basic Channel Transformation Channel Polarization

1 Polarization

Basic Channel Transformation Channel Polarization

7 / 25 I-Hsiang Wang IT Lecture 9

Polarization Basic Channel Transformation Channel Polarization

Single Usage of Channel W X Y W N Usage of Channel W . . . ENC DEC W W W M ˆ M X1 X2 XN Y1 Y2 YN

8 / 25 I-Hsiang Wang IT Lecture 9

Polarization Basic Channel Transformation Channel Polarization

Arıkan’s Idea

. . . Pre- Processing W W W X1 X2 XN Y1 Y2 YN UN U2 U1 Post- Processing V1 V2 VN Apply special transforms to both input and output

9 / 25 I-Hsiang Wang IT Lecture 9

Polarization Basic Channel Transformation Channel Polarization

Arıkan’s Idea

W1 . . . W2 WN UN U2 U1 V1 V2 VN

10 / 25 I-Hsiang Wang IT Lecture 9

Polarization Basic Channel Transformation Channel Polarization

Arıkan’s Idea

W1 . . . W2 WN UN U2 U1 V1 V2 VN

Roughly NI (W ) channels with capacity ≈ 1

11 / 25 I-Hsiang Wang IT Lecture 9

Polarization Basic Channel Transformation Channel Polarization

Arıkan’s Idea

W1 . . . W2 WN UN U2 U1 V1 V2 VN

Roughly NI (W ) channels with capacity ≈ 1 Roughly N (1 − I (W )) channels with capacity ≈ 0

Equivalently some perfect channels and some useless channels − → Polarization Coding becomes extremely simple: simply use those perfect channels for uncoded transmission, and throw those useless channels away.

12 / 25 I-Hsiang Wang IT Lecture 9

Polarization Basic Channel Transformation Channel Polarization

1 Polarization

Basic Channel Transformation Channel Polarization

13 / 25 I-Hsiang Wang IT Lecture 9

Polarization Basic Channel Transformation Channel Polarization

Arıkan’s Basic Channel Transformation

Consider two channel uses of W:

14 / 25 I-Hsiang Wang IT Lecture 9

X1 X2 W W Y1 Y2

Polarization Basic Channel Transformation Channel Polarization

Arıkan’s Basic Channel Transformation

Consider two channel uses of W: Apply the pre-processor: X1 = U1 ⊕ U2, X2 = U2, where U1 ⊥ ⊥ U2, U1, U2 ∼ Ber ( 1

2

) . We now have two synthetic channels induced by the above procedure: W− : U1 → V1 ≜ (Y1, Y2) W+ : U2 → V2 ≜ (Y1, Y2, U1) The above transform yields the following two crucial phenomenon: I (W− ) ≤ I (W ) ≤ I (W+ ) (Polarization) I (W− ) + I (W+ ) = 2I (W ) (Conservation of Information)

15 / 25 I-Hsiang Wang IT Lecture 9

W W Y1 Y2 U2 U1

Polarization Basic Channel Transformation Channel Polarization

Example: Binary Erasure Channel

Example 1 Let W be a BEC with erasure probability ε ∈ (0, 1), and I (W ) = 1 − ε. Find the values of I (W− ) and I (W+ ), and verify the above properties. sol: Intuitively W− is worse than W and W+ is better than W: For W−, input is U1, output is (Y1, Y2). Only when both Y1 and Y2 are not erased, one can figure out U1! = ⇒ W− is BEC with erasure probability 1 − (1 − ε)2 = 2ε − ε2. For W+, input is U2, output is (Y1, Y2, U1). As long as one of Y1 and Y2 are not erased, one can figure out U2! = ⇒ W+ is BEC with erasure probability ε2. Hence, I (W− ) = 1 − 2ε + ε2 and I (W+ ) = 1 − ε2.

16 / 25 I-Hsiang Wang IT Lecture 9

Polarization Basic Channel Transformation Channel Polarization

Example: Binary Symmetric Channel

Example 2 Let W be a BSC with crossover probability p ∈ (0, 1), and I (W ) = 1 − Hb (p). Find the values of I (W− ) and I (W+ ).

17 / 25 I-Hsiang Wang IT Lecture 9

Polarization Basic Channel Transformation Channel Polarization

Basic Properties

Theorem 1 For any BMS channel W and the induced channels {W−, W+} from Arıkan’s basic transformation, we have I (W− ) ≤ I (W ) ≤ I (W+ ) with equality iff I (W ) = 0 or 1. I (W− ) + I (W+ ) = 2I (W ) pf: We prove the conservation of information first: I ( W− ) + I ( W+ ) = I (U1 ; Y1, Y2 ) + I (U2 ; Y1, Y2, U1 ) = I (U1 ; Y1, Y2 ) + I (U2 ; Y1, Y2 |U1 ) = I (U1, U2 ; Y1, Y2 ) = I (X1, X2 ; Y1, Y2 ) = I (X1 ; Y1 ) + I (X2 ; Y2 ) = 2I (W ) . I (W+ ) = I (X2 ; Y1, Y2, U1 ) ≥ I (X2 ; Y2 ) = I (W ), and hence the first property holds. (Proof of the condition for equality is left as exercise.)

18 / 25 I-Hsiang Wang IT Lecture 9

Polarization Basic Channel Transformation Channel Polarization

Extremal Channels

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5

I(W+) − I(W−) [bits] I(W) [bits] BEC BSC

(Taken from Chap. 12.1 of Moser[4].)

If we plot the “information stretch” I (W+ ) − I (W− ) versus the original information I (W ), it can be shown that among all BMS channels: BEC maximizes the stretch BSC minimizes the stretch Lower boundary: 2Hb (2p(1 − p)) − 2Hb (p) , where p = Hb

−1 (1 − I (W )).

Upper boundary: 2I (W ) (1 − I (W )) .

19 / 25 I-Hsiang Wang IT Lecture 9

Polarization Basic Channel Transformation Channel Polarization

1 Polarization

Basic Channel Transformation Channel Polarization

20 / 25 I-Hsiang Wang IT Lecture 9

Polarization Basic Channel Transformation Channel Polarization

Recursive Application of Arıkan’s Transformation

Duplicate W, apply the transformation, and get W− and W+.

21 / 25 I-Hsiang Wang IT Lecture 9

W W

Polarization Basic Channel Transformation Channel Polarization

Recursive Application of Arıkan’s Transformation

Duplicate W, apply the transformation, and get W− and W+. Duplicate W− (and W+).

22 / 25 I-Hsiang Wang IT Lecture 9

W W W W

Polarization Basic Channel Transformation Channel Polarization

Recursive Application of Arıkan’s Transformation

Duplicate W, apply the transformation, and get W− and W+. Duplicate W− (and W+). Apply the transformation on W−, and get W−− and W−+.

23 / 25 I-Hsiang Wang IT Lecture 9

W W W W

Polarization Basic Channel Transformation Channel Polarization

Recursive Application of Arıkan’s Transformation

Duplicate W, apply the transformation, and get W− and W+. Duplicate W− (and W+). Apply the transformation on W−, and get W−− and W−+. Apply the transformation on W+, and get W+− and W++.

24 / 25 I-Hsiang Wang IT Lecture 9

W W W W

Polarization Basic Channel Transformation Channel Polarization

Recursive Application of Arıkan’s Transformation

Duplicate W, apply the transformation, and get W− and W+. Duplicate W− (and W+). Apply the transformation on W−, and get W−− and W−+. Apply the transformation on W+, and get W+− and W++.

. . .

We can keep going and going, until the desired blocklength is reached.

25 / 25 I-Hsiang Wang IT Lecture 9

W W W W W W W W