Constructing Error- -Correction Codes Correction Codes - - PowerPoint PPT Presentation

Constructing Error Constructing Error-

• Correction Codes

Correction Codes from Scale from Scale-

• Free Networks

Free Networks

Francis C.M. Lau Francis C.M. Lau

Department of Electronic and Information Engineering Hong Kong Polytechnic University International Workshop on Complex Systems and Networks 2007 Guilin, China

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

IWCSN' 2007, Guilin, China 2

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Part 1: Communications and Coding Part 1: Communications and Coding

IWCSN' 2007, Guilin, China 3

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Communications without Coding

How are you today ? Hxw au& u%$ wqo . Welf affi zv iol bxg. How aru yox tuday ?

Information can be easily corrupted when sent through a channel !

IWCSN' 2007, Guilin, China 4

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

More Reliable Communications

How are you today ? How are you today ? How are you today ? How aru yox tuday ? How aee yeu todey ? Hoe are you toxak ? How are you today ? Error-correction capability

channel

IWCSN' 2007, Guilin, China 5

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Bit-level Communications

111 100 011 010 101 001 110 000 1

Without coding schemes: 0,1 0,0

Info Source

channel

Info Sink noise

0,1 0,1 000, 111 000, 110

1 information bit 2 check bits

Code Rate=number of information bits/ block length=1/3 Block Length =1+2=3

With coding schemes:

Info Source Encoder

channel

Decoder Info Sink noise

IWCSN' 2007, Guilin, China 6

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Reliable Communications

Add redundant information at transmitter Decode information intelligently at receiver

Error-Correction Capability

Any better ways than to repeat the information several times? Any performance bounds?

IWCSN' 2007, Guilin, China 7

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Shannon’s Capacity Theorem

Channel

Additive White Gaussian Noise Bandwidth W Average Received Signal Power S Average Noise Power N

⎟ ⎠ ⎞ ⎜ ⎝ ⎛ + = N S W C 1 log2

System Capacity of the channel

• C. E. Shannon “A mathematical theory of communications,” Bell Syst. Tech. J., vol. 27, pp.

379–423, 623–656, 1948.

IWCSN' 2007, Guilin, China 8

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Shannon’s Capacity Theorem

possible theoretically to transmit information at any rate R ≤ C with an arbitrarily small error probability (with coding) if R > C, not possible to transmit information with an arbitrarily small error probability (even with coding)

⎟ ⎠ ⎞ ⎜ ⎝ ⎛ + = N S W C 1 log2

Channel

Information with rate R

IWCSN' 2007, Guilin, China 9

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Normalized channel capacity versus SNR

⎟ ⎠ ⎞ ⎜ ⎝ ⎛ + = N S W C 1 log2

IWCSN' 2007, Guilin, China 10

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Normalized channel bandwidth versus SNR

⎟ ⎠ ⎞ ⎜ ⎝ ⎛ + = N S C W 1 log 1

2

The graph is not telling the whole story !

IWCSN' 2007, Guilin, China 11

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Because …

noise power is proportional to bandwidth

W N N =

⎟ ⎠ ⎞ ⎜ ⎝ ⎛ + = N S C W 1 log 1

2

IWCSN' 2007, Guilin, China 12

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Further …

when bit rate R equals channel capacity C

C S R S Eb = =

energy per bit

IWCSN' 2007, Guilin, China 13

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Shannon Limit

⎟ ⎠ ⎞ ⎜ ⎝ ⎛ + = N S W C 1 log2

( )

1 2

/

− =

W C b

C W N E

W N N =

C S R S Eb = =

rearrangement

IWCSN' 2007, Guilin, China 14

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

( )

1 2

/

− =

W C b

C W N E

Shannon Limit

/ / dB 59 . 1 / → ⇔ ∞ → ⇒ − → W C C W N Eb

Channel capacity approaches zero, regardless of the channel bandwidth No error-free communications below

dB 59 . 1 / − = N Eb

IWCSN' 2007, Guilin, China 15

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Typical error performance of coded and uncoded modulations

IWCSN' 2007, Guilin, China 16

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Improved error performance; more bandwidth required to add redundancy bits

IWCSN' 2007, Guilin, China 17

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Reduction in requirement; more bandwidth required to add redundancy bits

/ N Eb

coding gain

IWCSN' 2007, Guilin, China 18

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Significance/Conclusions of Shannon’s work

proved theoretically that there exists codes that could improve the error probability performance from uncoded modulation schemes there is a minimum requirement

/ N Eb

IWCSN' 2007, Guilin, China 19

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BUT ….

IWCSN' 2007, Guilin, China 20

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

How should we design coding schemes, with reasonable complexity, that work as close to the Shannon limit as possible?

IWCSN' 2007, Guilin, China 21

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Solutions

Not provided by Shannon ! So do research on Coding !

IWCSN' 2007, Guilin, China 22

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Part 2: Parity Part 2: Parity-

• Check Codes

Check Codes

IWCSN' 2007, Guilin, China 23

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Parity-Check Codes

single-parity-check code

1 1 1 1 1 1

parity bit message bits

IWCSN' 2007, Guilin, China 24

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Parity-Check Codes

single-parity- check code

even-parity code

can detect all single-and triple-error patterns (e.g. 0100 or 0010) but cannot correct errors

IWCSN' 2007, Guilin, China 25

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Parity-Check Codes

rectangular code (or product code)

horizontal parity check vertical parity check

message

IWCSN' 2007, Guilin, China 26

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Parity-Check Codes

rectangular code (or product code)

can correct a single error pattern

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

horizontal parity check fails vertical parity check fails

bit in error channel

IWCSN' 2007, Guilin, China 27

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Linear Block Codes

a class of parity-check codes denoted by (n, k)

codeword length message length

maps k-bit messages (k-tuples) linearly and uniquely to n-bit codewords (n-tuples)

IWCSN' 2007, Guilin, China 28

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Subset S of a vector space is a subspace if

it contains the all-zeros vector sum of any two vectors in S is also in S

IWCSN' 2007, Guilin, China 29

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

• Packing as many

Packing as many codewords codewords in the entire in the entire space as possible improves space as possible improves coding efficiency coding efficiency

• Putting the

Putting the codewords codewords as as far apart from one far apart from one another as possible another as possible increases the chance of increases the chance of decoding the decoding the codewords codewords correctly correctly

IWCSN' 2007, Guilin, China 30

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

(6, 3) Code Example

form a subspace

1 1 1 1

+

Modulo-2 Addition all-zeros vector

IWCSN' 2007, Guilin, China 31

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Modulo-2 Multiplication

• 1

1 1

Modulo-2 Addition and Multiplication

addition can be accomplished electronically

using an Exclusive-OR gate multiplication can be accomplished using an AND gate

Modulo-2 Addition + 1 1 1 1

IWCSN' 2007, Guilin, China 32

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Encoding the Messages

Table look-up is possible for small k For large k, table look-up may become extremely difficult

e.g.,

30

10 26 . 1 2 100 × ≈ ⇒ =

k

k

Use of Generator Matrix

IWCSN' 2007, Guilin, China 33

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Generator Matrix G (size k x n)

a basis set of k linearly independent n-tuples that spans the subspace

n-tuple n-tuple n-tuple

] [

2 1 k

m m m L = m

message

mG U =

codeword (size 1 x n)

IWCSN' 2007, Guilin, China 34

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

(6, 3) Code Example

] [

6 5 4 3 2 1

u u u u u u =

IWCSN' 2007, Guilin, China 35

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Parity-Check Matrix H (size (n-k) x n)

For each generator matrix G, there exists an (n-k) x n matrix H such that rows of G are orthogonal to rows of H.

GH =

T

k x (n-k) all-zeros matrix

mGH UH = =

T T

H can be used to test whether a received vector is a valid codeword.

H G ↔

IWCSN' 2007, Guilin, China 36

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Parity-Check Matrix H (size (n-k) x n)

GH =

T

⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎣ ⎡ = 1 1 1 1 1 1 1 1 1 H

rows are

• rthogonal

IWCSN' 2007, Guilin, China 37

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Parity-Check Matrix H

1 1 1 1 1 1 1 1 1 ] [

6 5 3 5 4 2 6 4 1 6 5 4 3 2 1

= + + = + + = + + ⇒ = ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ ⇒ = u u u u u u u u u u u u u u u

T

UH

IWCSN' 2007, Guilin, China 38

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Bipartite Graph

) (

5 4 2 2

= + + = u u u c

) (

6 5 3 3

= + + = u u u c

1

u

2

u

3

u

4

u

5

u

6

u

variable nodes check nodes

⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎣ ⎡ = 1 1 1 1 1 1 1 1 1 H

) (

6 4 1 1

= + + = u u u c

IWCSN' 2007, Guilin, China 39

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Decoding

received vector r’ Codeword U

Additive White Gaussian Noise Channel

0 +1 volt 1 −1 volt

IWCSN' 2007, Guilin, China 40

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Eight codewords in a 6-tuple space

Hard Decoding

received vector r’ (AWGN channel) decoded codeword after error correction after making hard decision on each bit ri > 0 volt 0 ri < 0 volt 1

IWCSN' 2007, Guilin, China 41

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

r’

) ' | 011101 ( r U = P

) ' | 101110 ( r U = P ) ' | 110100 ( r U = P

maximum a posteriori (MAP) decision rule: Select codeword U that has the largest

) ' | ( r U P

) ' | 101001 ( r U = P

a posteriori probability (APP)

Soft Decoding

IWCSN' 2007, Guilin, China 42

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Performance of Some Well- known Block Codes (Coherent BPSK

• ver an AWGN

channel)

t = maximum number of guaranteed correctable errors per codeword

IWCSN' 2007, Guilin, China 43

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Performance of BCH Codes (Coherent BPSK

• ver an AWGN

channel)

IWCSN' 2007, Guilin, China 44

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Part 3: Low Part 3: Low-

• Density

Density-

• Parity

Parity-

• Check (LDPC) Codes

Check (LDPC) Codes

IWCSN' 2007, Guilin, China 45

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Parity-Check Matrix H (size (n-k) x n)

GH =

T

⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎣ ⎡ = 1 1 1 1 1 1 1 1 1 H

IWCSN' 2007, Guilin, China 46

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Low-Density-Parity-Check Codes

proposed by Gallager (1960) parity-check matrix H

sparse (most elements are zeros) fraction of 1’s ~ O(n)

elements of H determine the connections between variable nodes and check nodes

degree of variable node u6 = 2 degree of check node c3 = 3

IWCSN' 2007, Guilin, China 47

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Low-Density-Parity-Check Codes

sparse (low-density)

parity-check matrix H implies that all variable nodes and check nodes have very few connections

H G ↔

IWCSN' 2007, Guilin, China 48

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Error rates achieved by different coding schemes under the binary AWGN channel. Codeword length = 106. Rate =0.5.

IWCSN' 2007, Guilin, China 49

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Types of LDPC Codes

Regular LDPC

all nodes of the same type (variable node or check node) have the same degree

A (3, 6)-regular LDPC code of length 10 and rate one-half.

check node degree variable node degree

IWCSN' 2007, Guilin, China 50

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Types of LDPC Codes

Irregular LDPC: the degrees of each set of nodes are chosen according to some distribution

IWCSN' 2007, Guilin, China 51

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Degree Distribution of Nodes

Degree distribution of variable nodes Degree distribution of check nodes

fraction of edges connected to the variable nodes with degree k

∑

= −

=

v

d k k kx

x

2 1

) ( λ λ

∑

= −

=

c

d k k kx

x

2 1

) ( ρ ρ

fraction of edges connected to the check nodes with degree k maximum variable node degree maximum check node degree

IWCSN' 2007, Guilin, China 52

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Code Rate

∫ ∫

− =

1 1

d ) ( d ) ( 1 x x x x R λ ρ

IWCSN' 2007, Guilin, China 53

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Question

Given specific

∑

= −

=

v

d k k kx

x

2 1

) ( λ λ

∑

= −

=

c

d k k kx

x

2 1

) ( ρ ρ and . How would the LDPC code perform?

IWCSN' 2007, Guilin, China 54

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Answer

Dependent on the actual design (connections), the channel type, e.g. AWGN, binary symmetric channel (BSC), binary erasure channel (BEC) and the decoding algorithm.

IWCSN' 2007, Guilin, China 55

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

More Question

Any idea on the optimal performance of practical LDPC decoders?

IWCSN' 2007, Guilin, China 56

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Part 4: Part 4: Belief Propagation (BP) Decoding

Belief Propagation (BP) Decoding Algorithm Algorithm

IWCSN' 2007, Guilin, China 57

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Belief Propagation (BP) Decoding Algorithm

A kind of message-passing decoding algorithm Applicable to both regular and irregular LDPC codes Produces very good error performance

IWCSN' 2007, Guilin, China 58

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Belief Propagation (BP) Decoding Algorithm

Define Log Likelihood Ratio (LLR): ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ = = info) | 1 bit ( info) | bit ( log P P

variable nodes check nodes

IWCSN' 2007, Guilin, China 59

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BP Decoding Algorithm

Compute initial Log Likelihood Ratio (LLR) for each variable node based on the received signal vector r (real number elements) ⎥ ⎦ ⎤ ⎢ ⎣ ⎡ = = ) | 1 bit ( ) | bit ( log

i i

r P r P

1

r ) ( LLR

1 0 r

iteration number

i

r

IWCSN' 2007, Guilin, China 60

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BP Decoding Algorithm

Set iteration number k = 1 Pass the LLR messages from variable nodes to the connected check nodes ) ( LLR

1 0 r

IWCSN' 2007, Guilin, China 61

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BP Decoding Algorithm

Check nodes received the LLR messages from the connected variable nodes

IWCSN' 2007, Guilin, China 62

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BP Decoding Algorithm

Each check node computes a message for each of its connected variable nodes, based on the messages from all

• ther variables nodes

IWCSN' 2007, Guilin, China 63

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BP Decoding Algorithm

Each check node computes a message for each of its connected variable nodes, based on the messages from all

• ther variables nodes

IWCSN' 2007, Guilin, China 64

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BP Decoding Algorithm

Each check node computes a message for each of its connected variable nodes, based on the messages from all

• ther variables nodes

IWCSN' 2007, Guilin, China 65

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BP Decoding Algorithm

Each check node computes a message for each of its connected variable nodes, based on the messages from all

• ther variables nodes

IWCSN' 2007, Guilin, China 66

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BP Decoding Algorithm

Each check node computes a message for each of its connected variable nodes, based on the messages from all

• ther variables nodes

IWCSN' 2007, Guilin, China 67

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BP Decoding Algorithm

Each check node computes a message for each of its connected variable nodes, based on the messages from all

• ther variables nodes

IWCSN' 2007, Guilin, China 68

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BP Decoding Algorithm

Each variable node update its LLR based on the messages passed from the check nodes and the initial LLR Based on the updated LLR, estimate the codeword

) ( LLR ) ( LLR

1 1 1

r r →

IWCSN' 2007, Guilin, China 69

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BP Decoding Algorithm

Estimate the codeword as

) ( LLR ) ( LLR

1 1 1

r r →

[ ]

n

c c c ˆ ˆ ˆ ˆ

2 1

L = c

where

⎭ ⎬ ⎫ ⎩ ⎨ ⎧ < > = ) ( LLR if 1 ) ( LLR if ˆ

1 1

r r c

k k i

If , is the decoded codeword.

H c =

T

ˆ c ˆ

IWCSN' 2007, Guilin, China 70

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BP Decoding Algorithm

) ( LLR

1 0 r

If , increment the iteration number k. Each variable node computes a message for each of its connected check nodes, based

• n its initial LLR and the

messages from all other connected check nodes

H c ≠

T

ˆ

IWCSN' 2007, Guilin, China 71

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BP Decoding Algorithm

) ( LLR

1 0 r

Each variable node computes a message for each of its connected check nodes, based

• n its initial LLR and the

messages from all other connected check nodes

IWCSN' 2007, Guilin, China 72

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BP Decoding Algorithm

) ( LLR

1 0 r

Each variable node computes a message for each of its connected check nodes, based

• n its initial LLR and the

messages from all other connected check nodes

IWCSN' 2007, Guilin, China 73

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BP Decoding Algorithm

Check nodes received the LLR messages from the connected variable nodes

IWCSN' 2007, Guilin, China 74

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

BP Decoding Algorithm

Each check node computes a message for each of its connected variable nodes, based on the messages from all other variables nodes Same iterative process repeated …. until convergence to a valid codeword or maximum number of iterations exceeded

IWCSN' 2007, Guilin, China 75

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Capacity (Threshold) of LDPC codes

Given the degree distributions and the channel type (AWGN, BSC or BEC) and the use of BP decoding algorithm.

∑

= −

=

v

d k k kx

x

2 1

) ( λ λ

∑

= −

=

c

d k k kx

x

2 1

) ( ρ ρ

Richard and Urbanke (2001) proposed an effective algorithm – density evolution – to determine the capacity (threshold) of LDPC codes.

• T. J. Richardson and R. Urbanke, “The capacity of low-density parity-check codes under message-

passing decoding,” IEEE Trans. Inform. Theory, vol. 47, pp. 599–618, Feb. 2001.

IWCSN' 2007, Guilin, China 76

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Density Evolution Algorithm

channel type

∑

= −

=

v

d k k kx

x

2 1

) ( λ λ

∑

= −

=

c

d k k kx

x

2 1

) ( ρ ρ

Density Evolution Algorithm

iterations threshold value

*

σ

a higher threshold value indicates a higher achievable achievable performance of the code

IWCSN' 2007, Guilin, China 77

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Good Degree Distribution Pairs (Rate = 0.5)

• T. J. Richardson et al., “Design of capacity-approaching irregular low-density parity-check codes,”

IEEE Trans. Inform. Theory, vol. 47, pp. 619–637, Feb.2001.

IWCSN' 2007, Guilin, China 78

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Good Degree Distribution Pairs (Rate = 0.5)

IWCSN' 2007, Guilin, China 79

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

the threshold value , for example indicates the maximum noise power that can be tolerated for error-free communication in AWGN channels the threshold value can be achieved achieved if

the message-passing process does not contain any cycles number of iterations tends to infinity

• codeword length is infinite

codeword length is infinite

Density Evolution Algorithm

*

σ

*

σ

IWCSN' 2007, Guilin, China 80

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Density Evolution Algorithm

Problems:

• ptimizing the codes based on DE algorithm is not a

simple task codeword length cannot be infinite

• ptimizing the threshold value may

give a more complex code

number of connections

∑

= −

=

v

d k k kx

x

2 1

) ( λ λ

∑

= −

=

c

d k k kx

x

2 1

) ( ρ ρ

vary to maximize the threshold value

IWCSN' 2007, Guilin, China 81

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Part 5: Review of Complex Networks Part 5: Review of Complex Networks

IWCSN' 2007, Guilin, China 82

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Basic properties

Path length: the distance between two nodes, which is defined as the number of edges along the shortest path connecting them

IWCSN' 2007, Guilin, China 83

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Basic properties

Betweenness centrality: the fraction of shortest paths going through a given node Assortative mixing: preference of high-degree nodes attach to other high-degree nodes Disassortative mixing: preference of high- degree nodes attach to low-degree nodes

IWCSN' 2007, Guilin, China 84

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Examples of Complex Networks

Random Networks

Given a network with N nodes. Each pair of nodes are connected with a probability of p.

Poisson distribution

( ) !

ke

p k k

μ

μ

−

=

μ =

0.1

ER

p =

IWCSN' 2007, Guilin, China 85

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Examples of Complex Networks

Regular Coupled Networks

high clustering large average path length

Fully-connected Networks

2 2.5 3 3.5 4 4.5 0.2 0.4 0.6 0.8 1 <k> P(k)

IWCSN' 2007, Guilin, China 86

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Examples of Complex Networks

Small-World Networks

Each edge of a regular coupled network is re-wired with a probability of p high clustering small average path length

WS

p =

0.2

WS

p =

IWCSN' 2007, Guilin, China 87

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Examples of Complex Networks

Scale-Free Networks

IWCSN' 2007, Guilin, China 88

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Examples of Complex Networks

Scale-Free (SF) Networks

γ i i

n n

−

~ ) Pr(

γ

x x f

−

~ ) (

IWCSN' 2007, Guilin, China 89

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Characteristics of Typical Complex Networks

High Uniform Long Regular Coupled Network

• High

Low Clustering Coefficient Power-Law Very Short Scale Free Networks

• Short

Small World Networks Poisson Short Random Networks Degree Distribution Average Distance (log( )) N Ο

(log(log( )))* N Ο

*The exponent parameter should be valued between 2 and 3. See reference “Scale-Free Networks Are Ultrasmall”, PRL, vol. 90, no. 5

(log( )) N Ο

( ) N Ο

Fast to disseminate information

IWCSN' 2007, Guilin, China 90

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Part 6: Scale Part 6: Scale-

• free Networks to

free Networks to LDPC Codes LDPC Codes

IWCSN' 2007, Guilin, China 91

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Scale-free Networks meet LDPC Codes

Can the “very short distance” property

• f scale-free network helps

passing/spreading messages quickly in the decoding of LDPC codes?

If so, how?

IWCSN' 2007, Guilin, China 92

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Scale-free Networks meet LDPC Codes

? ?

IWCSN' 2007, Guilin, China 93

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

From Bipartite Graph to Unipartite Graph

IWCSN' 2007, Guilin, China 94

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

From Bipartite Graph to Unipartite Graph

power-law degree distribution Power-law degree distribution !

IWCSN' 2007, Guilin, China 95

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Typical node degree distribution of the corresponding unipartite graph. Codeword length = 10000 and maximum variable node degree = 20.

• X. Zheng, F.C.M. Lau and C.K. Tse, " Study of LDPC Codes Built on Scale-Free Networks,"

Proceedings, NOLTA'06, Bologna, Italy, September 2006, pp. 563-566.

IWCSN' 2007, Guilin, China 96

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Building LDPC Codes From SF Networks

Assume that the variable nodes have the power-law degree distribution and the check nodes

• bey the Poisson-law degree distribution

Use the (Density Evolution) DE to select the

• ptimized parameters

and

.

( ) ~ P k k γ

λ −

γ

μ

! ) ( l e l P

l μ ρ

μ

−

=

IWCSN' 2007, Guilin, China 97

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Threshold value and average variable node degrees <k> of LDPC codes built from scale-free networks and the optimized ones reported in [1] for an AWGN

• channel. Rate equals 0.5.

* σ

[1]

[1] T. J. Richardson et al., “Design of capacity-approaching irregular low-density parity-check codes,” IEEE Trans. Inform. Theory, vol. 47, pp. 619–637, Feb.2001.

IWCSN' 2007, Guilin, China 98

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Typical node degree distribution of the corresponding unipartite graph. Codeword length = 1000 and maximum variable node degree = 15.

• X. Zheng, F.C.M. Lau and C.K. Tse, “Error Performance of Short-Block-Length LDPC Code Built on Scale-Free

Networks,” Proceedings, The Third Shanghai International Symposium on Nonlinear Sciences and Applications, Shanghai, China, June 2007, pp. 55-57.

IWCSN' 2007, Guilin, China 99

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Building LDPC Codes From SF Networks

Threshold values lower compared with those reported in the literature Average variable node degrees <k> lower compared with those reported in the literature

* σ

Which one is better in practice ?

IWCSN' 2007, Guilin, China 100

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

The threshold value can be achieved achieved if

the message-passing process does not contain any cycles and number of iterations tends to infinity and

• codeword length is infinite

codeword length is infinite

Threshold Value

*

σ

IWCSN' 2007, Guilin, China 101

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

PEG and Enhanced PEG

Progressive Edge-Growth algorithm (PEG)

an effective method to construct codes with girth average as large as possible based on the given degree distributions

Enhanced PEG (E-PEG) proposed by us

stopping set and the near codeword are also checked after each variable node is added.

• X. Y. Hu, E. Eleftheriou and D. M. Arnold, “Regular and irregular progressive edge-

growth tanner graphs,” IEEE Trans. Inform. Theory, vol. 51, no. 1, pp. 386–398, 2005.

IWCSN' 2007, Guilin, China 102

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Part 7: Simulation Results Part 7: Simulation Results

IWCSN' 2007, Guilin, China 103

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Block Error Rates

Block length=1008 Code rate=0.5

• Max. no. of iterations = 50

IWCSN' 2007, Guilin, China 104

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Bit Error Rates

Block length=1008 Code rate=0.5

• Max. no. of iterations = 50

IWCSN' 2007, Guilin, China 105

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

0.5 1 1.5 2 2.5 3 10

• 6

10

• 5

10

• 4

10

• 3

10

• 2

10

• 1

10 SNR(dB) Block Error Rate DE10 (PEG), <k>=3.66 DE10 (E-PEG), <k>=3.66

PEG and E-PEG Algorithms

Block length=1008 Code rate=0.5

• Max. no. of iterations = 50

IWCSN' 2007, Guilin, China 106

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

0.5 1 1.5 2 2.5 3 10

• 8

10

• 6

10

• 4

10

• 2

10 SNR(dB) Bit Error Rate DE10 (PEG), <k>=3.66 DE10 (E-PEG), <k>=3.66

PEG and E-PEG Algorithms

Block length=1008 Code rate=0.5

• Max. no. of iterations = 50

IWCSN' 2007, Guilin, China 107

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

0.5 1 1.5 2 2.5 3 10

• 7

10

• 6

10

• 5

10

• 4

10

• 3

10

• 2

10

• 1

10 SNR(dB) Block Error Rate SF20 (PEG), <k>=3.72 SF20 (E-PEG), <k>=3.72

PEG and E-PEG Algorithms

Block length=1008 Code rate=0.5

• Max. no. of iterations = 50

IWCSN' 2007, Guilin, China 108

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

0.5 1 1.5 2 2.5 3 10

• 8

10

• 6

10

• 4

10

• 2

10 SNR(dB) Bit Error Rate SF20 (PEG), <k>=3.72 SF20 (E-PEG), <k>=3.72

PEG and E-PEG Algorithms

Block length=1008 Code rate=0.5

• Max. no. of iterations = 50

IWCSN' 2007, Guilin, China 109

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Part 8: Summary Part 8: Summary

IWCSN' 2007, Guilin, China 110

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Summary

Coding for a reliable communication Operation principles of parity-check codes Low-density-parity-check (LPDC) codes Belief propagation decoding algorithm Density evolution

IWCSN' 2007, Guilin, China 111

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Summary

LPDC codes with scale-free variable-node-degree distribution achieve very good theoretical threshold (error correction performance) Short LPDC codes built with scale-free variable- node-degree distribution outperform other well- known LPDC codes with similar complexity

IWCSN' 2007, Guilin, China 112

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Some of Our Related Work

1.

• X. Zheng, F.C.M. Lau and C.K. Tse, “Error Performance of Short-Block-Length

LDPC Code Built on Scale-Free Networks,” Proceedings, The Third Shanghai International Symposium on Nonlinear Sciences and Applications, Shanghai, China, June 2007, pp. 55-57. 2.

• X. Zheng, F.C.M. Lau, Chi K. Tse and S.C. Wong, “Study of Bifurcation

Behavior of LDPC Decoders", International Journal of Bifurcation and Chaos,

• vol. 16, no. 11, pp. 3435-3449, Nov. 2006.

3.

• X. Zheng, F.C.M. Lau and Chi K. Tse, “Study of LDPC Codes Built on Scale-

Free Networks,” Proceedings, International Symposium on Nonlinear Theory and Its Applications (NOLTA'06), Bologna, Italy, September 2006, pp. 563-566. 4.

• X. Zheng, F.C.M. Lau, C.K. Tse and S.C. Wong, “Techniques for Improving

Block Error Rate of LDPC Decoders,” Proceedings, IEEE International Symposium on Circuits and Systems (ISCAS'06), Kos, Greece, May 2006, pp. 2261-2264. 5.

• X. Zheng, F.C.M. Lau, Chi K. Tse and S.C. Wong, “Study of Nonlinear

Dynamics of LDPC Decoders", Proceedings, European Conference on Circuit Theory and Design (ECCTD ‘2005), Dublin, Ireland, August 2005, paper 207. (CD version)

IWCSN' 2007, Guilin, China 113

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Collaborators

Dr Wai-man TAM

• Prof. Chi K. TSE

Dr Siu C. WONG Miss Xia ZHENG

Constructing Error Constructing Error-

• Correction Codes

Correction Codes from Scale from Scale-

• Free Networks

Free Networks

Francis C.M. Lau Francis C.M. Lau

Department of Electronic and Information Engineering Hong Kong Polytechnic University

International Workshop on Complex Systems and Networks 2007 Guilin, China

T THE

HE H

HONG

ONG K

KONG

ONG

P POLYTECHNIC

OLYTECHNIC U

UNIVERSITY

NIVERSITY

Thank You !