ECEN 5682 Theory and Practice of Error Control Codes Introduction - - PowerPoint PPT Presentation

Introduction to Block Codes

ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Peter Mathys

University of Colorado

Spring 2007

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Basic Definitions

Definition: A block code of length n and size M over an alphabet with q symbols is a set of M q-ary n-tuples called codewords. Example: Code #1. Binary code of length n = 5 with M = 4 codewords given by C = {00000, 01011, 10101, 11110} . Definition: The rate R of a q-ary block code of length n with M codewords is given by R = logq M n .

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Definition: The redundancy r of a q-ary block code of length n with M codewords is given by r = n − logq M . Example: Code #1 has rate R = log2 4/5 = 2/5 = 0.4 and redundancy r = 5 − log2 4 = 3 bits. Example: Code #2. 5-ary code of length n = 4 with M = 5 codewords given by C = {0000, 1342, 2134, 3421, 4213} . This code has rate R = log5 5/4 = 1/4 = 0.25 and redundancy r = 4 − log5 5 = 3 symbols.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

The goal when using error control codes is to detect and/or correct transmission errors. Suppose code #1 is used and the (corrupted) codeword v = (00101) is received. Comparing v with all “legal” codewords and marking the discrepancies with * yields: 00000 01011 10101 11110 00101 00101 00101 00101

• ..*.*

.***. *.... **.*. The discrepancies are the result of transmission errors. If all error positions are marked with one and all other positions with zero, then the received codeword v = (00101) corresponds to the set of possible errors E = {00101, 01110, 10000, 11010} , when code #1 is used. But which of these 4 errors is the “right”

• ne?

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

To decide which error out of a set of errors is the “right” one, one needs to make additional assumptions about the likelihood of errors and error patterns. The two most common models for the

• ccurrence of errors are:

(i) Independent and identically distributed (iid) errors with probablity ǫ. This requires a memoryless transmission channel model. (ii) Burst errors of length L. If an error occurs, it is very likely that it is followed by L − 1 more errors. Burst errors occur for instance in mobile communications due to fading and in magnetic recording due to media defects. Burst errors can be converted to iid errors by the use of an interleaver.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

More generally, and especially for non-binary codes, one also needs a model for the error amplitudes. Two possibilities are (i) Uniformly distributed non-zero error amplitudes. This is a good model for orthogonal signaling. (ii) Non-uniformly distributed non-zero error amplitudes with smaller error magnitudes more likely than larger ones. This is a good model for QAM signaling.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

In addition to the models that describe which error pattern e is most likely, a transmission channel model is also needed that specifies how codewords c and error patterns e are combined to form the received codeword v = f (c, e). The most prevalent model is the additive model shown in the following figure. Note that addition is often assumed to be modulo q addition for a q-ary code.

+ error e c codeword v = c + e received codeword vector addition (often modulo q)

A concise graphical way to describe simple error models is in the form of a discrete channel model that shows all possible transitions from the channel input X to the channel output Y , together with the associated transition probabilities pY |X(y|x).

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Example: The simplest discrete channel model is the memoryless binary symmetric channel (BSC) shown in the following figure.

• 1

1 Input X Output Y 1 − ǫ ǫ ǫ 1 − ǫ

This channel is completely described by the set of four transition probabilities: pY |X(0|0) = 1 − ǫ , pY |X(1|0) = ǫ , pY |X(0|1) = ǫ , pY |X(1|1) = 1 − ǫ . Clearly P{Y = X} = ǫ and thus the (uncoded) probability of a bit error is Pb(E) = ǫ.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Thus, if ǫ < 0.5 on a memoryless BSC, fewer errors are more likely and the “right” error pattern is the one with the fewest number of 1’s in it. Note that, since all symbols are binary here, only errors of amplitude 1 are possible and no specification for the distribution of error amplitudes is needed. Example: Suppose q = 5, errors occur iid with Pr{Y = X} < 0.5, and uniformly distributed amplitudes. The corresponding channel model is a memoryless 5-ary symmetric channel (5SC) with transition probabilities pY |X(y|x) = ǫ/4 , if y = x , x, y ∈ {0, 1, 2, 3, 4} , 1 − ǫ , if y = x , x, y ∈ {0, 1, 2, 3, 4} . In this case the decoding rule assumes again that the “right” error pattern is the one with the fewest nonzero symbols in it.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Example: Suppose again q = 5 and errors occur iid with P{Y = X} < 0.5. But now assume that only errors of magnitude 1 occur, with +1 and −1 being equally likely. This leads to another memoryless 5SC with transition probabilities pY |X(y|x) =    ǫ/2 , if y = x ± 1 (mod 5) , x, y ∈ {0, 1, 2, 3, 4} , 1 − ǫ , if y = x , x, y ∈ {0, 1, 2, 3, 4} , 0 ,

• therwise .

Now the decoder decides on the error pattern with the fewest number of ±1 (mod 5) symbols as the “right” error.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Once an error and a channel model are defined, a distance measure between codewords can be defined. Then one can determine the minimum distance between any two distinct codewords and this in turn determines how many errors a code can detect and/or correct under the given error and channel models. For the iid error model with (discrete) uniform error amplitude distribution the most appropriate measure is Hamming distance which is defined as follows. Definition: The Hamming distance d(H)(x, y) (or simply d(x, y)) between two q-ary n-tuples x and y is the number of places in which they differ. Example: d(10221, 20122) = 3.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

The Hamming distance is probably the most popular distance measure for error control codes. Another measure that is more suitable in cases where smaller error magnitudes are more likely than larger ones is the Lee distance which is defined next. Definition: The Lee distance d(L)(x, y) between two q-ary n-tuples x and y is defined as d(L)(x, y) = |x0 − y0| + |x1 − y1| + . . . + |xn−1 − yn−1| , where the magnitude |v| of a q-ary symbol v is computed modulo q as |v| = min{v, q − v} . Definition: The minimum distance dmin of a code C = {ci, i = 0, 1, . . . M − 1} is the smallest distance between any two distinct codewords of the code, i.e., dmin = min

i=j d(ci, cj) ,

ci, cj ∈ C .

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Example: Code #2 has has the following Hamming distances between pairs of distinct codewords:

d(H)(x, y) 0000 1342 2134 3421 4213 0000 − 4 4 4 4 1342 4 − 4 4 4 2134 4 4 − 4 4 3421 4 4 4 − 4 4213 4 4 4 4 −

The Lee distances between pairs of distinct codewords are:

d(L)(x, y) 0000 1342 2134 3421 4213 0000 − 6 6 6 6 1342 6 − 6 6 6 2134 6 6 − 6 6 3421 6 6 6 − 6 4213 6 6 6 6 −

Thus, for code #2, d(H) min = 4 and d(L) min = 6.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Example: Code #3. Binary code with n = 10 and the following set of M = 4 codewords C = {0010010111, 0100101110, 1001011100, 1110001001} . For this code the Hamming distances between pairs of codewords are

d(H)(x, y) 0010010111 0100101110 1001011100 1110001001 0010010111 − 6 6 6 0100101110 6 − 6 6 1001011100 6 6 − 6 1110001001 6 6 6 −

That is, code #3 has minimum Hamming distance 6.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Theorem: A code with minimum Hamming distance dmin can detect all error patterns with dmin − 1 or fewer nonzero components. Proof: The only error patterns that cannot be detected are those that make the transmitted codeword look like another codeword. But because the smallest Hamming distance between any two distinct codewords is dmin, this can only happen if the error pattern affects dmin or more coordinates of the transmitted codeword.QED Definition: The sphere of radius t about codeword c is the set St(c) = {v | d(c, v) ≤ t} , where d(., .) is the distance measure used (e.g., Hamming distance).

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Example: Consider the codeword c = 01011 from binary code #1. Then, using Hamming distance as the distance measure,

S0(01011) = {01011} S1(01011) = {01011, 11011, 00011, 01111, 01001, 01010} S2(01011) = {01011, 11011, 00011, 01111, 01001, 01010, 10011, 11111, 11001, 11010, 00111, 00001, 00010, 01101, 01110, 01000}

Theorem: The Hamming distance satisfies the triangle inequality, i.e., for any 3 n-tuples x, y, z d(x, y) + d(y, z) ≥ d(x, z) .

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

d(x, y) + d(y, z) ≥ d(x, z) Proof: First note that for any u = (u0, u1, . . . , un−1) and v = (v0, v1, . . . , vn−1), the Hamming distance satisfies d(u, v) = d(u0, v0)+d(u1, v1)+. . .+d(un−1, vn−1), d(ui, vi) ∈ {0, 1}, where the addition is over the reals. Consider now a coordinate, say j, where x and z differ, i.e., xj = zj. Then there are three possible cases for yj: (i) yj = xj, which implies d(yj, zj) = 1. (ii) yj = zj, which implies d(yj, xj) = 1. (iii) yj = xj and yj = zj, which implies d(yj, xj) = 1 and d(yj, zj) = 1. Thus, in all three cases d(x, y) + d(y, z) increases by at least one while d(x, z) increases by exactly one. QED

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Theorem: A code with minimum Hamming distance dmin can correct all error patterns with t or fewer nonzero components as long as 2t < dmin. Proof: Because of the triangle inequality, the spheres St(ci) and St(cj) of any two distinct codewords ci and cj contain no common elements as long as 2t < dmin. Theorem: The Hamming or sphere-packing bound for q-ary codes with dmin = 2t + 1 states that the redundancy r must satisfy r ≥ logq  

t

• j=0

n j

• (q − 1)j

  . Proof: Left as an exercise.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Definition: A code which satisfies the Hamming bound with equality is called a perfect code. Example: Code #4. The binary code with blocklength n = 7 and the following set of M = 16 codewords

C = {0000000, 0001011, 0010101, 0011110, 0100110, 0101101, 0110011, 0111000, 1000111, 1001100, 1010010, 1011001, 1100001, 1101010, 1110100, 1111111} ,

is a perfect code. By inspection dmin = 3 is found and thus all patterns of t = 1 errors are correctable. Therefore r ≥ log2 7

• +

7 1

• = log2(1 + 7) = 3 .

But r = n − log2 M = 7 − 4 = 3, i.e., the code satisfies the Hamming bound with equality. This code is known as binary (7, 4, 3) Hamming code.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Example: Probably the most prominent and celebrated perfect code is the binary Golay code with blocklength n = 23, M = 212 codewords and minimum Hamming distance dmin = 7. It can correct all error patterns with up to t = 3 errors and thus the Hamming bound requires that

r ≥ log2 »“23 ” + “23 1 ” + “23 2 ” + “23 3 ”– = log2(1+23+253+1771) = log2(2048) = 11.

The actual redundancy of the code is r = 23 − log2 212 = 23 − 12 = 11 and therefore it is perfect. Note: Don’t take the name “perfect code” too literally. Perfect just simply means that the spheres of radius t = (dmin − 1)/2 around all codewords fill out the whole codewordspace perfectly. It does not necessarily mean that perfect codes are the best error detecting and/or correcting codes.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

How easy is it to find a block code by trial and error? Example: Binary rate R = 0.8 code of length n = 10. There are 210

28

• ways to choose 256 codwords out of 1024 possibilities. Here

is the actual number:

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Linear Block Codes

Definition: A q-ary linear (n, k) blockcode C is defined as the set

• f all linear combinations, taken modulo q, of k independent

vectors from V , where V is the set of all q-ary n-tuples. If C has minimum distance dmin, C is called a q-ary (n, k, dmin) code. Definition: A generator matrix G of a linear (n, k) code C is a k × n matrix whose rows form a basis for the k-dimensional subspace C of V . Definition: The q-ary k-tuple u = (u0, u1, . . . uk−1) is used to denote a dataword. In general, it is assumed that there are no restrictions on u, i.e., it may take on all possible qk values and, unless otherwise specified, all these values are equally likely. Other names for u are message or information word.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Definition: For the encoding operation, any one-to-one association between datawords u and codewords c may be used. For a linear code with generator matrix G, the most natural encoding procedure is to use c = u G . Example: Code #5. Ternary (n = 5, k = 2) code with generator matrix G = 1 2 1 2 1 2 2 1

• .

This defines a linear code C whose codewords

C = {00000, 01221, 02112, 10212, 11100, 12021, 20121, 21012, 22200} ,

lie in a 2-dimensional subspace of V , where V is the set of all 3-ary 5-tuples.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Example: Code #1 is a linear binary (n, k, dmin) = (5, 2, 3) code with generator matrix G = 1 1 1 1 1 1

• .

To verify this, generate the set of codewords by taking all possible linear combinations (modulo 2, or modulo q in general) of the rows

• f G as follows:

C =

• (0, 0) · G = (0, 0, 0, 0, 0) ,

(0, 1) · G = (0, 1, 0, 1, 1) , (1, 0) · G = (1, 0, 1, 0, 1) , (1, 1) · G = (1, 1, 1, 1, 0)

• .

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Example: Code #2 is a linear 5-ary (4,1,4) code with generator matrix G =

• 1

3 4 2

• .

The set of codewords is obtained as

C = n 0 · G = (0, 0, 0, 0) , 1 · G = (1, 3, 4, 2) , 2 · G = (2, 1, 3, 4) , 3 · G = (3, 4, 2, 1) , 4 · G = (4, 2, 1, 3)

• .

Example: Code #3 is a nonlinear binary code with n = 10. The sum of codewords

(0, 1, 0, 0, 1, 0, 1, 1, 1, 0) + (1, 0, 0, 1, 0, 1, 1, 1, 0, 0) = (1, 1, 0, 1, 1, 1, 0, 0, 1, 0) ,

for instance, is not a codeword of the code. In fact, the 4 codewords C = {0010010111, 0100101110, 1001011100, 1110001001} , are linearly independent and form the basis of a 4-dimensional subspace of the space of all binary 10-tuples.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Definition: The Hamming weight w(c) of a codeword c ∈ C is equal to the number of nonzero components of c. The minimum Hamming weight wmin of a code C is equal to the smallest Hamming weight of any nonzero codeword in C. Definition: The Lee weight w (L)(c) of a codeword c ∈ C is defined as w(L)(c) = |c0| + |c1| + . . . + |cn−1| , where the magnitude |v| of a q-ary symbol v is computed modulo q as |v| = min{v, q − v} . The minimum Lee weight w (L)

min of a code C is equal to the smallest

Lee weight of any nonzero codeword in C.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Theorem: For a linear code dmin = wmin. Proof: For any x, y ∈ C dmin = min x=y d(x, y) = min x=y d(x − y, 0) = min c=0 w(c) , where c = x − y ∈ C because the code is linear. QED

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Standard Array for Decoding. A good conceptual, but practically inefficient way to visualize the decoding operation for a q-ary linear code under the Hamming distance measure is the so-called standard array. It is set up as follows: (1) The first row of the array consists of all the codewords of the code, starting on the left with the all-zero codeword that must be present in every linear code. (2) The first column starts out with all q-ary n-tuples that are in the decoding sphere St(0) of radius t about the all-zero codeword c0 = 0, where t is the maximum number of errors the code can correct. There are P = |St(0)| n-tuples in this sphere, and, assuming an additive (modulo q) error model, all are correctable error patterns (including the the all-zero “error”). The elements in this column are called the coset leaders.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Standard Array for Decoding (contd.) (3) Making use of the linearity of the code, anything that applies to the all-zero codeword c0 also applies to any other codeword cj by simply translating the origin. Thus, the first P entries of the j-th column make up the decoding sphere St(cj). All entries in the j-th column are obtained by adding (modulo q) the error pattern on the left to the codeword above. (4) Each of the rows in the standard array is called a coset. Altogether, the first P rows contain the M = qk distinct decoding spheres St(cj) for j = 0, 1, . . . M − 1. If the code is a perfect code, then P M = qn, else there are (qn − P M) = (Q − P)M, where Q = qn−k, distinct q-ary n-tuples that do not yet appear in the array. These correspond to error patterns with more than t errors, but in general only a few of these are correctable.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Standard Array for Decoding (contd.) (5) To complete the standard array, organize the (Q − P)M q-ary n-tuples that do not yet appear in the array into Q − P

• cosets. For each coset, select an error pattern of smallest

Hamming weight that has not yet appeared anywhere in the array as a coset leader, and complete the coset by adding the error pattern on the left to the codeword above. Because of the linearity of the code it can be shown that it is always possible to fill the bottom Q − P rows in this way with distinct n-tuples and that the set of all Q rows contains all qn possible q-ary n-tuples.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Decoding using the standard array consists of looking up the received n-tuple in the array and returning the codeword above it as the result. Definition: A decoder which decodes only received n-tuples within decoding spheres of radius t or less, but not the whole decoding region, is called an incomplete decoder. If one of the n-tuples in the bottom Q − P rows of the standard array is received, an incomplete decoder declares a detected but uncorrectable error

• pattern. Conversely, a complete decoder assigns a nearby

codeword to every received n-tuple. Note: For a perfect t-error correcting code P = Q, and no coset leader has weight greater than t.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

The following figure shows the contents of the standard array graphically.

decoding sphere decoding

• f radius t

region ........... ............. codewords ----> c0= : c1 : c2 ... : cM-1 : 00..0 : : : :

• /

e1 : c1+e1 : c2+e1 ... : cM-1+e1 : | ....................................................... coset

• ----:->

e2 : c1+e2 : c2+e2 ... : cM-1+e2 : : coset leader | ....................................................... | e3 : c1+e3 : c2+e3 ... : cM-1+e3 : 1,2,..,t | . : . : . : . : error | . : . : . : . : patterns | eP-1 : c1+eP-1 : c2+eP-1 ... : cM-1+eP-1 : P rows \ ........... : : above line

• - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

/ eP c1+eP c2+eP : cM-1+eP : Q-P rows more than | . . . : . : below line t errors | . . . : . : \ eQ-1 c1+eQ-1 c2+eQ-1 ... : cM-1+eQ-1 : .............

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Example: Code #1 has minimum (Hamming) distance dmin = 3 and thus is a single error correcting code. The standard array for this code is:

decoding sphere decoding

• f radius 1

region ......... ......... c0 : c1 : c2 : c3 : codewords ----> 00000 : 01011 : 10101 : 11110 :

• /

00001 : 01010 : 10100 : 11111 : single | 00010 : 01001 : 10111 : 11100 : errors < 00100 : 01111 : 10001 : 11010 : | ........................................ coset

• ----:-> 01000

: 00011 : 11101 : 10110 : : coset leader | ........................................ \ 10000 : 11011 : 00101 : 01110 : ......... : :

• - - - - - - - - - - - - - - - - - - - - - -

00110 01101 10011 : 11000 : 01100 00111 11001 : 10010 : .........

The selection of the coset leaders for the last two rows is not

• unique. For example, 11000 and 10010 could have been used

instead of 00110 and 01100.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Because the size of the standard array increases exponentially in the block length n, it is quite clearly not very practical even for moderately large values of n. Therefore the concept of a parity check matrix is introduced, which will lead to a more compact decoding method using syndromes. Definition: Let C ⊂ V , where V is the set of all q-ary n-tuples, be a linear (n, k) code. Then the dual or orthogonal code of C, denoted C⊥ (“C perp”), is defined by C⊥ = {u ∈ V | u · w = 0 for all w ∈ C} . The orthogonal complement of C has dimension n − k and thus C⊥ is a linear (n, n − k) code.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Definition: A parity check matrix H of a linear (n, k) code C is a (n − k) × n matrix whose rows form a basis for the (n − k)-dimensional subspace C⊥ of the set of all q-ary n-tuples V . That is, any parity check matrix of C is a generator matrix of C⊥. Theorem: Let C be a q-ary linear (n, k) code with generator matrix G and parity check matrix H. Then, using arithmetic modulo q, ci HT = 0, where T denotes transpose, for all ci ∈ C, and G HT = 0. Proof: Follows directly from the definition of C⊥ and the definition of H. QED

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Example: Code #5 (ternary (5,2,3) code) has generator matrix G and parity check matrix H given by G = 1 2 1 2 1 2 2 1

• ,

H =   1 1 1 2 1 1 1 2 1   . Thus, C⊥ is the set

C⊥ = {00000, 12001, 21002, 21010, 00011, 12012, 12020, 21021, 00022, 11100, 20101, 02102, 02110, 11111, 20112, 20120, 02121, 11122, 22200, 01201, 10202, 10210, 22211, 01212, 01220, 10221, 22222}

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Theorem: A linear code C contains a nonzero codeword of Hamming weight w iff a linearly dependent set of w columns of H exists. Proof: Let c ∈ C have weight w. From c HT = 0 we can thus find a set of w linearly dependent columns of H. Conversely, if H contains a linearly dependent set of w columns, then we can construct a codeword c with nonzero coefficients corresponding to the w columns, such that c HT = 0. QED Corollary: A linear code C has minimum weight wmin iff every set

• f wmin − 1 columns of H is linearly independent and at least one

set of wmin columns of H is linearly dependent.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Definition: Elementary row operations on a matrix are the following: (i) Interchange of any two rows. (ii) Multiplication of any row by a nonzero scalar. (iii) Replacement of any row by the sum of itself and a multiple of any other row. Definition: A matrix is said to be in row-echelon form if it satisfies the following conditions: (i) The leading term of every nonzero row is a one. (ii) Every column with a leading term has all other entries zero. (iii) The leading term of any row is to the right of the leading term in every previous row. All-zero rows (if any) are placed at the bottom. Note: Under modulo q arithmetic, any matrix can be put into row-echelon form by elementary row operations if q is a prime number.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Example: Let q = 11 and consider the matrix A = 2 4 6 9 1 10 6 10 2 1 7 4 9 10 10 8 6 2 3 9 7 8 4 7 4 3 5 . Multiply the first row by 6−1 = 2. Then replace the second row by the difference of the second row minus the new first row. Next, subtract 2 times the new first row from the third row and replace the third row with the result to obtain A′ = 2 4 1 7 2 9 1 9 4 2 9 1 4 6 5 6 8 10 4 3 5 . Now start by multiplying the second row by 2−1 = 6. Then subtract 2 times this row from the first row. Finally, note that the third row is just a multiple of the second row, so that it can be replaced by an all zero row. The result is A in row-echelon form: A′′ = 2 4 1 7 9 3 8 5 1 10 6 2 3 3 5 .

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Definition: Two codes which are the same except for a permutation of codeword components are called equivalent. The generator matrices G and G′ of equivalent codes are related as

• follows. The code corresponding to G is the set of all linear

combinations of rows of G and is thus unchanged under elementary row operations. Permutation of the columns of G corresponds to permutation of codeword components and therefore two codes are equivalent if their generator matrices G and G′ are related by (i) Column permutations, and (ii) Elementary row operations.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

From this it follows that every generator matrix G of a linear code is equivalent to one in row-echelon form. Because G is a k × n matrix whose rows span a k-dimensional subspace, all rows of G must be linearly independent. This proves the following Theorem: Every generator matrix of a q-ary linear code, where q is a prime (or a prime power), is equivalent to one of the form G = [Ik | P] ,

• r

G = [P | Ik] , where Ik is a k × k identity matrix and P is a k × (n − k) matrix. Definition: A code C with codewords whose first (or last) k components are the unmodified information symbols is called a systematic code. The remaining n − k codeword symbols are called parity symbols. A systematic code has generator matrix G = [Ik | P] (or G = [P | Ik]).

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Let u = (u0, u1, . . . , uk−1) be the information word and let c = (c0, c1, . . . , cn−1) be the corresponding codeword. If G is in systematic form, then G =        1 . . . p0,k p0,k+1 . . . p0,n−1 1 . . . p1,k p1,k+1 . . . p1,n−1 1 . . . p2,k p2,k+1 . . . p2,n−1 . . . . . . . . . ... . . . . . . . . . ... . . . . . . 1 pk−1,k pk−1,k+1 . . . pk−1,n−1        , then the components of c = u G are cj = uj, for 0 ≤ j ≤ k − 1, and cj = u0 p0,j + u1 p1,j + · · · + uk−1 pk−1,j , k ≤ j ≤ n − 1 . This latter set of equations is known as the set of parity-check equations of the code.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Theorem: The systematic form of H corresponding to G = [Ik | P] is H =

• − PT | In−k
• .

Proof: Multiplying corresponding submatrices in G and HT together yields G HT = [Ik | P] −P In−k

• = −Ik P + P In−k = 0 .

Thus, H = [−PT | In−k] satisfies G HT = 0. QED

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Written out explicitly, the systematic form of H is H =        −p0,k −p1,k . . . −pk−1,k 1 . . . −p0,k+1 −p1,k+1 . . . −pk−1,k+1 1 . . . −p0,k+2 −p1,k+2 . . . −pk−1,k+2 1 . . . . . . . . . ... . . . . . . . . . . . . ... . . . −p0,n−1 −p1,n−1 . . . −pk−1,n−1 . . . 1        . From c HT = 0 it thus follows for the m-th row of H that

−c0 p0,k+m − c1 p1,k+m − · · · − ck−1 pk−1,k+m + ck+m = 0 , 0 ≤ m ≤ n − k − 1 .

Letting j = k + m and using the fact that for a systematic code ci = ui, 0, ≤ i ≤ k − 1, one obtains cj = u0 p0,j + u1 p1,j + · · · + uk−1 pk−1,j , k ≤ j ≤ n − 1 , which is the same set of parity check equations as before. This shows that a systematic linear (n, k) code is completely specified either by its generator matrix G or by its parity check matrix H.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Example: Let q = 11 and consider the (8, 2) code with generator matrix G = 3 10 5 9 2 4 1 7 1 9 2 3 2 8

• .

Put G into row-echelon form which yields G′ = 1 7 9 3 8 5 1 10 6 2 3

• .

Note that G and G′ produce exactly the same set of codewords, but using a different set of basis vectors and therefore a different mapping from datawords u to codewords c. To obtain a generator matrix in systematic form, permute the second and third columns

• f G′ so that

Gsys = 1 7 9 3 8 5 1 10 6 2 3

• .

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Now, using Hsys = [−PT | In−k], one easily finds

Hsys = 2 6 6 6 6 6 6 4 4 1 2 1 8 1 1 3 5 1 9 1 6 8 1 3 7 7 7 7 7 7 5 .

Finally, to obtain a parity check matrix H for the original generator matrix G, all the column permutations that were necessary to

• btain Gsys from G need to be undone. Here only columns two

and three need to be permuted to obtain

H = 2 6 6 6 6 6 6 4 4 1 2 1 8 1 1 3 5 1 9 1 6 8 1 3 7 7 7 7 7 7 5 .

A quick check shows that indeed G HT = 0 modulo 11.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Theorem: Singleton bound. The minimum distance of any linear (n, k) code satisfies dmin ≤ n − k + 1 . Proof: Any linear code can be converted to systematic form (possibly permuting coordinates which does not affect dmin) and thus G = [Ik | P]. Since P is a k × (n − k) matrix and systematic codewords with only one nonzero information symbol exist, the result follows. QED Note: It can be shown that the Singleton bound also applies to non-linear codes.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Definition: Any linear code whose dmin satisfies dmin = n − k + 1 , is called maximum distance separable (MDS).

Note: The name maximum distance separable code comes from the fact that such a code has the maximum possible (Hamming) distance between codewords and that the codeword symbols can be separated into data symbols and parity check symbols (i.e., the code has a systematic encoder). Example: Code #6. The ternary (4, 2) code with generator matrix G = »1 2 2 1 2 1 – , is an MDS code. The set of codewords is C = {0000, 0121, 0212, 1022, 1110, 1201, 2011, 2102, 2220} . From this it is easily seen that dmin = 3 = n − k + 1, which proves the claim that this code is MDS.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Definition: Let C be a linear (n, k) code and let v = c + e, where c ∈ C is a codeword and e is an error vector of length n, be a received n-tuple. The syndrome s of v is defined by s = v HT . Theorem: All vectors in the same coset (cf. standard array decomposition of a linear code) have the same syndrome, unique to that coset.

Proof: If v and v′ are in the same coset, then v = ci + e and v′ = cj + e for some e (coset leader) and codewords ci, cj. But, for any codeword c, c HT = 0 and therefore s = v HT = ci HT | {z }

=0

+e HT = e HT s′ = v′ HT = cj HT | {z }

=0

+e HT = e HT = ⇒ s = s′ . Conversely, suppose that s = s′. Then s − s′ = (v − v′) HT = 0, which implies that v − v′ is a codeword. But that further implies that v and v′ are in the same coset. QED

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Note: In practice, this theorem has some important consequences. To decode a linear code, one does not need to store the whole standard array. Only the mapping from the syndrome to the most likely error pattern needs to be stored. Example: Syndrome decoding for code #1. This is a binary code with parity check matrix H =   1 1 1 1 1 1 1   . Computing s = e HT for e = 0, all single error error patterns, and some double error patterns, the following table that uniquely relates syndromes to error patterns is obtained.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Error e Syndrome s 00000 000 00001 001 00010 010 00100 100 01000 011 01000 101 00110 110 01100 111 H = 2 4 1 1 1 1 1 1 1 3 5 Note that an incomplete (or bounded distance) decoder would only use the first six entries (above the dividing line at the bottom) for decoding. The choice of the error patterns for the last two entries is somewhat arbitrary (just as it was in the case of the standard array), and other double error patterns that yield the same syndromes could have been used. Suppose now that v = (11101) was

• received. To decode v, compute s = v HT = (011). From the syndrome lookup

table the corresponding e = (01000). Finally, the corrected codeword c is

• btained as c = v − e = (10101).

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Note: To construct a linear q-ary single error correcting code with redundancy r = n − k, one can start from a parity check matrix H whose columns are q-ary r-tuples with the property that they are all distinct, even when multiplied by an arbitrary nonzero q-ary

• scalar. The resulting codes are called Hamming codes.

Example: Natural parameters (n, k, dmin) of commonly used small binary linear codes are: (7, 4, 3) (15, 11, 3) (15, 7, 5) (15, 5, 7) (23, 12, 7) (31, 26, 3) (31, 21, 5) (31, 16, 7) (31, 11, 11) (63, 57, 3) (63, 51, 5) (63, 45, 7) (63, 39, 9) (63, 36, 11) (127, 120, 3) (127, 113, 5) (127, 106, 7) (127, 99, 9) (127, 92, 11) (255, 247, 3) (255, 239, 5) (255, 231, 7) (255, 223, 9) (255, 215, 11) Note that most of the block lengths of these codes are of the form 2m − 1 for some integer m. Such block lengths are called primitive block lengths.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Modified Linear Blockcodes. Often the natural parameters of block codes are not suitable for a particular application, e.g., for computer storage applications the data length is typically a multiple of 8, whereas the natural parameter k of a code may be a “crummy” number like 113 (= 14 × 8 + 1). Thus, it may be necessary to change either k or n or both. To explain the different procedures, code #7 which is a linear binary (6, 3, 3) code with G and H as shown below, is used. G =   1 1 1 1 1 1 1 1 1   , H =   1 1 1 1 1 1 1 1 1   . The six different modifications that can be applied to the parameters n and k of a linear block code are: Lenghtening (n+,k+), shortening (n−,k−), extending (n+,k =), puncturing (n−,k =), augmenting (n =,k+), and expurgating (n =,k−).

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

• Lengthening. Increase blocklength n by adding more data symbols

while keepig redundancy r = n − k fixed. The result is a code that has n and k increased by the same amount. In the best case dmin will be unchanged, but it can drop to as low as 1 and needs to be reexamined carefully. Example: Code #7 lengthened by 1 results in a (7, 4, 3) code with generator and parity check matrices G =     1 1 1 1 1 1 1 1 1 1 1 1 1     , H =   1 1 1 1 1 1 1 1 1 1 1 1   .

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

• Shortening. Decrease blocklength n by dropping data symbols

while keeping the redundancy r fixed. The resulting code has n and k reduced by the same amount. In most cases dmin will be unchanged, occasionally dmin may increase. Example: Code #7 shortened by 1 results in a (5, 2, 3) code with generator and parity check matrices G = 1 1 1 1 1 1

• ,

H =   1 1 1 1 1 1 1   .

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

• Extending. Increase blocklength n by adding more parity check

symbols while keeping k fixed. The result is a code that has n and r = n − k increased by the same amount. The minimum distance may or may not increase and needs to be reexamined. A common method to extend a code from n to n + 1 is to add an overall parity check. Example: Code #7 extended by 1 results in a (7, 3, 4) code with generator and parity check matrices G =   1 1 1 1 1 1 1 1 1 1 1 1   , H =     1 1 1 1 1 1 1 1 1 1 1 1 1     .

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

• Puncturing. Decrease blocklength n by dropping parity check

symbols while keeping k fixed. The resulting code has n and r = n − k decreased by the same amount. Except for the trivial case of removing all zero columns from G, the minimum distance

• decreases. Example: Code #7 punctured by 1 yields a (5, 3, 2)

code with generator and parity check matrices G =   1 1 1 1 1 1 1   , H = 1 1 1 1 1 1

• .

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

• Augmenting. Increase datalength k while keeping n fixed by

reducing the redundancy r = n − k. The result is a code which has k increased by the same amount as r = n − k is decreased. Because of the reduction of r, the minimum distance generally

• decreases. Example: Code #7 augmented by 1 gives a (6, 4, 2)

code with generator and parity check matrices G =     1 1 1 1 1 1 1 1 1 1 1 1     , H = 1 1 1 1 1 1 1 1

• .

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

• Expurgating. Decrease datalength k while keeping n fixed by

increasing the redundancy r = n − k. The resulting code has k decreased by the same amount as r = n − k is increased. The increase in r may or may not lead to an increase in dmin. Example: Code #7 expurgated by 1 gives a (6, 2, 4) code with generator and parity check matrices G = 1 1 1 1 1 1 1 1

• ,

H =     1 1 1 1 1 1 1 1 1 1 1 1     .

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Definition: |u|u + v|-construction. Let u = (u0, u1, . . . , un−1) and v = (v0, v1, . . . , vn−1) be two q-ary n-tuples and define |u|u + v| = (u0, u1, . . . , un−1, u0 + v0, u1 + v1, . . . un−1 + vn−1) , where the addition is modulo q addition. Let C1 be a q-ary linear (n, k1, dmin = d1) code and let C2 be a q-ary linear (n, k2, dmin = d2) code. A new q-ary code C of length 2n is then defined by C =

• |u|u + v| : u ∈ C1, v ∈ C2
• .

The generator matrix of the (2n, k1 + k2) code C is G = G1 G1 G2

• ,

where 0 is a k2 × n all-zero matrix, G1 is the generator matrix of C1 and G2 is the generator matrix of C2.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes

Introduction to Block Codes Basic Definitions Linear Block Codes

Theorem: The minimum distance of the code C obtained from the |u|u + v|-construction is dmin(C) = min{2d1, d2} . Proof: Let x = |u|u + v| and y = |u′|u′ + v′| be two distinct codewords of C. Then d(x, y) = w(u − u′) + w(u − u′ + v − v′) , where d(., .) denotes Hamming distance and w(.) denotes Hamming weight. Case (i): v = v′. Case (ii): v = v′.

Peter Mathys ECEN 5682 Theory and Practice of Error Control Codes