More on the Reliability Function of the BSC Alexander Barg Andrew - - PowerPoint PPT Presentation

More on the Reliability Function of the BSC

Andrew McGregor University of Pennsylvania Alexander Barg DIMACS, Rutgers University

ISIT 2003, Yokohama

Some Definitions

Some Definitions

 Communicating over a binary symmetric

channel with cross-over probability p.

Some Definitions

 Communicating over a binary symmetric

channel with cross-over probability p.

 We use a length n binary code C={x1, x2, …

x|C|} with rate ≥ R ie.

Some Definitions

 Communicating over a binary symmetric

channel with cross-over probability p.

 We use a length n binary code C={x1, x2, …

x|C|} with rate ≥ R ie. |C|≥2nR

Some Definitions

 Communicating over a binary symmetric

channel with cross-over probability p.

 We use a length n binary code C={x1, x2, …

x|C|} with rate ≥ R ie. |C|≥2nR

 No matter what code we use there is the

possibility of making errors - for a given rate

• f transmission there is some degree of error

that is inherent to the channel itself.

Making Decoding Errors



Maximum Likelihood Decoding: When we receive a word y we’ll guess that the sent codeword is the codeword that lies closest to it.



For each codeword x we define the Voronoi region:



Let Pe(x) be the probability that, when codeword x is transmitted, this decoding procedure leads to an error. Therefore we have

Making Decoding Errors



Maximum Likelihood Decoding: When we receive a word y we’ll guess that the sent codeword is the codeword that lies closest to it.



For each codeword x we define the Voronoi region:



Let Pe(x) be the probability that, when codeword x is transmitted, this decoding procedure leads to an error. Therefore we have

D(x) = {y ∈ {0,1}n : d(x,y) < d(x j,y)∀x j ∈ C \ x}

Making Decoding Errors



Maximum Likelihood Decoding: When we receive a word y we’ll guess that the sent codeword is the codeword that lies closest to it.



For each codeword x we define the Voronoi region:



Let Pe(x) be the probability that, when codeword x is transmitted, this decoding procedure leads to an error. Therefore we have

D(x) = {y ∈ {0,1}n : d(x,y) < d(x j,y)∀x j ∈ C \ x}

P

e(x) = P x({0,1}n \ D(x))

The Reliability Function

 The average error probability of decoding is  We’re interested in  We present a new lower bound for this

quantity, or equivalently, an upper bound on the reliability function or error exponent of the channel:

The Reliability Function

 The average error probability of decoding is  We’re interested in  We present a new lower bound for this

quantity, or equivalently, an upper bound on the reliability function or error exponent of the channel:

P

e(C) = 1

|C | P

e(x) x∈C

∑

The Reliability Function

 The average error probability of decoding is  We’re interested in  We present a new lower bound for this

quantity, or equivalently, an upper bound on the reliability function or error exponent of the channel:

P

e(C) = 1

|C | P

e(x) x∈C

∑

P

e(R) =

min

C:Rate(C )>R P e(C)

The Reliability Function

 The average error probability of decoding is  We’re interested in  We present a new lower bound for this

quantity, or equivalently, an upper bound on the reliability function or error exponent of the channel:

P

e(C) = 1

|C | P

e(x) x∈C

∑

E(R, p) = −lim

n→∞

1 n log min

C:R(C )>R P e(C)

[ ]

P

e(R) =

min

C:Rate(C )>R P e(C)

Bounds on the Error Exponent:

• Combination of Best Lower Bounds:

[Gallager, 63] & [Elias, ‘56]

• Combination of Best Upper Bounds

prior to 1999: [Elias, ‘56] & [McEliece et al, ‘77]

• Litsyn’s Bound: [Litsyn ‘99]
• Our New Bound

E(R,p) R p=0.01

Bounds on the Error Exponent:

• Combination of Best Lower Bounds:

[Gallager, 63] & [Elias, ‘56]

• Combination of Best Upper Bounds

prior to 1999: [Elias, ‘56], [Shannon et al, ‘67] & [McEliece et al, ‘77]

• Litsyn’s Bound: [Litsyn ‘99]
• Our New Bound

0.2 0.4 0.6 0.8 0.2 0.4 0.6 0.8 1

E(R,p) R p=0.01

Bounds on the Error Exponent:

• Combination of Best Lower Bounds:

[Gallager, 63] & [Elias, ‘56]

• Combination of Best Upper Bounds

prior to 1999: [Elias, ‘56], [Shannon et al, ‘67] & [McEliece et al, ‘77]

• Litsyn’s Bound: [Litsyn ‘99]
• Our New Bound

0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1

E(R,p) R p=0.01

Bounds on the Error Exponent:

• Combination of Best Lower Bounds:

[Gallager, 63] & [Elias, ‘56]

• Combination of Best Upper Bounds

prior to 1999: [Elias, ‘56], [Shannon et al, ‘67] & [McEliece et al, ‘77]

• Litsyn’s Bound: [Litsyn ‘99]
• Our New Bound

0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1

E(R,p) R p=0.01

Litsyn’s Distance Distribution Bound

 Define  Litsyn’s Distance Distribution Bound:

For any code C of rate R there exists a w such that

Litsyn’s Distance Distribution Bound

 Define  Litsyn’s Distance Distribution Bound:

For any code C of rate R there exists a w such that

Bw(x) =|{x j : d(x ,x j) = w} |

Litsyn’s Distance Distribution Bound

 Define  Litsyn’s Distance Distribution Bound:

For any code C of rate R there exists a w such that

Bw(x) =|{x j : d(x ,x j) = w} | Bw(x) ≥ µ(R,w)

Estimating Pe(x)

x

P

e(x) = P x({0,1}n \ D(x))

Estimating Pe(x)

The Voronoi Region

P

e(x) =

pd(y,x)(1− p)n−d(y,x)

y∈C: d (y,x j )≤d(y,x) for some x j ∈C

∑

x

Estimating Pe(x)

Use the distance distribution result…

x w

P

e(x) =

pd(y,x)(1− p)n−d(y,x)

y∈C: d (y,x j )≤d(y,x) for some x j ∈C

∑

Estimating Pe(x)

Approximating the Voronoi Region…

x

P

e(x) ≥

pd(y,x)(1− p)n−d(y,x)

y∈C: d (y,x j )≤d(y,x) for some x j ∈C where d(x,x j )= w

∑

Estimating Pe(x)

Introducing the Xj…

x

P

e(x) ≥ P x(

X j

j:d (x,x j )= w

U

)

For each neighbour xj define a set Xj such that

y ∈ X j ⇒ d(y,x j) ≤ d(y,x)

Estimating Pe(x)

“Pruning” the Xj…

P

e(x) ≥

P

x(Y j) j:d (x,x j )= w

∑

For each neighbour xj assign a priority nj at random. Let

Y j = X j \ Xk

k:nk >n j

U

x

Estimating Pe(x)

Applying the Reverse Union Bound…

The Reverse Union Bound: Giving us our final shape of our bound:

Estimating Pe(x)

Applying the Reverse Union Bound…

The Reverse Union Bound: Giving us our final shape of our bound:

P

x(Y j) = P x(X j \

Xk)

k:nk >n j

U

≥ P

x(X j)(1−

P

x(Xk | X j) k:nk >n j

∑

)

Estimating Pe(x)

Applying the Reverse Union Bound…

The Reverse Union Bound: Giving us our final shape of our bound:

P

e(x) ≥

P

x(X j)(1− j:d (x,x j )= w

∑

P

x(Xk | X j) k:nk >n j

∑

) P

x(Y j) = P x(X j \

Xk)

k:nk >n j

U

≥ P

x(X j)(1−

P

x(Xk | X j) k:nk >n j

∑

)

 Now look across the entire code. Let Xij

and Yij be the sets for the neighbourhood of codeword xi.

 Therefore we have:

and where, the amount of “pruning” is

 What we do now depends on the values of

the Kij…

 Now look across the entire code. Let Xij

and Yij be the sets for the neighbourhood of codeword xi.

 Therefore we have:

and where, the amount of “pruning” is

 What we do now depends on the values of

the Kij…

P

e(xi) ≥

P

i(Yij) j:d (xi ,x j )= w

∑

 Now look across the entire code. Let Xij

and Yij be the sets for the neighbourhood of codeword xi.

 Therefore we have:

and where, the amount of “pruning” is

 What we do now depends on the values of

the Kij…

P(Yij) ≥ P

i(Xij)(1− Kij)

P

e(xi) ≥

P

i(Yij) j:d (xi ,x j )= w

∑

 Now look across the entire code. Let Xij

and Yij be the sets for the neighbourhood of codeword xi.

 Therefore we have:

and where, the amount of “pruning” is

 What we do now depends on the values of

the Kij…

P(Yij) ≥ P

i(Xij)(1− Kij)

P

e(xi) ≥

P

i(Yij) j:d (xi ,x j )= w

∑

Kij = P

i(Xik | Xij) k:nik > nij

∑

 Consider the set of codewords

 Consider the set of codewords

S={xj : Kij > 1/2 for some i}

 Consider the set of codewords

S={xj : Kij > 1/2 for some i}

 Either this is a “substantially” sized

subcode or it isn’t.

 Consider the set of codewords

S={xj : Kij > 1/2 for some i}

 Either this is a “substantially” sized

subcode or it isn’t.

 Ie, either we had to do a lot of pruning

• r we didn’t have to do a lot of pruning.

If S was not substantially sized…

 Just remove codewords in S from the code!  Then in the remaining code we have for all Yij

Pi(Yij ) ≥ Pi(Xij )/2

 Hence, modulo constant factors, the average

error probability satisfies Pe(C,p ) ≥ A(w)µ(w)

 where A(w)= Pi(Xij )

If S was substantially sized…

 Consider

where

 Consider a codeword xj such that Kij>1/2. Then there

exists an l’ such that Bl’(xj)> 1/(2nB(w,l’))

 The upshot of S being substantial is that we discover a

nuisance level l1, such that Pe(xj) ≥ A(w)/B(w,l1 ) and a substantial number of codewords have the Bl1(xj)> 1/B(w,l1)

If S was substantially sized…

 Consider

where

 Consider a codeword xj such that Kij>1/2. Then there

exists an l’ such that Bl’(xj)> 1/(2nB(w,l’))

 The upshot of S being substantial is that we discover a

nuisance level l1, such that Pe(xj) ≥ A(w)/B(w,l1 ) and a substantial number of codewords have the Bl1(xj)> 1/B(w,l1)

Kij = P

i(Xik | Xij) k:nik > nij

∑

= B(w,l)

k:nik > nij ,d(x j ,xk )= l

∑

       

l= 0 n

∑

If S was substantially sized…

 Consider

where

 Consider a codeword xj such that Kij>1/2. Then there

exists an l’ such that Bl’(xj)> 1/(2nB(w,l’))

 The upshot of S being substantial is that we discover a

nuisance level l1, such that Pe(xj) ≥ A(w)/B(w,l1 ) and a substantial number of codewords have the Bl1(xj)> 1/B(w,l1)

Kij = P

i(Xik | Xij) k:nik > nij

∑

= B(w,l)

k:nik > nij ,d(x j ,xk )= l

∑

       

l= 0 n

∑

B(w,l) = P

i(Xik | Xij) where d(xi,x j) = d(xi,xk) = w, d(x j,xk) = l

 A priori we don’t know whether we required a

lot or a little pruning. We therefore take the weaker of the two bounds:

 But if there existed a nuisance level l1 then

we know that for a substantial number codewords such that

 Hence we can repeat the process with this

new bound on the distribution.

 A priori we don’t know whether we required a

lot or a little pruning. We therefore take the weaker of the two bounds:

 But if there existed a nuisance level l1 then

we know that for a substantial number codewords such that

 Hence we can repeat the process with this

new bound on the distribution.

P

e(C, p) ≥ min A(w)µ(w), A(w) B(w,l1 )

[ ]

 A priori we don’t know whether we required a

lot or a little pruning. We therefore take the weaker of the two bounds:

 But if there existed a nuisance level l1 then

we know that for a substantial number codewords such that

 Hence we can repeat the process with this

new bound on the distribution.

P

e(C, p) ≥ min A(w)µ(w), A(w) B(w,l1 )

[ ]

Bl1(x) ≥ 1 B(w,l

1)

Our Bound

 Continuing in this way we eventually get  Minimizing over l and w gives us our

final bound.

P

e(C, p) ≥ min A(w)µ(w), A(l ) B(w,l )

[ ]

where 0 ≤ l ≤ w ≤ δLPn

Random Linear Codes

 It can be shown that, with high probability, the

weight distribution of a random linear code converges to Bw=exp[n(R+h(w)-1)]

 Using this instead of Litsyn’s expression µ

leads us to believe that the expurgation bound E(R,p)≥-δGV(p)/2 log 2p(1-p) is tight for a random linear code for very low rates.

0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1

The End The End