[PPT] - I n f o r m a t i o n T r a n s m i s s i o n PowerPoint Presentation

SLIDE 1

OVE EDFORS ELECTRICAL AND INFORMATION TECHNOLOGY

I n f

r

m a t i

n

T r a n s m i s s i

n

C h a p t e r 5 , C h a n n e l c

d

i n g

SLIDE 2

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 2

L e a r n i n g

u

t c

m

e s

A

f t e r t h i s l e c t u r e t h e s t u d e n t s h

u

l d

– u

n d e r s t a n d t h e p r i n c i p l e s

f

c h a n n e l c

d

i n g ,

– u

n d e r s t a n d h

w

t y p i c a l s e q u e n c e s c a n b e u s e d t

fi

n d

u

t h

w

” f a s t ” w e c a n s e n d i n f

r

m a t i

n
v

e r a c h a n n e l ,

– h

a v e a b a s i c k n

w

l e d g e a b

u

t h

w

c h a n n e l c a p a c i t y i s r e l a t e d t

m

u t u a l i n f

r

m a t i

n

a n d i t s m a x i m i z a t i

n
v

e r t h e c h a n n e l i n p u t d i s t r i b u t i

n

– k

n

w

h

w

t

c

a l c u l a t e t h e c h a n n e l c a p a c i t y f

r

t h e b i n a r y s y m m e t r i c c h a n n e l a n d t h e a d d i t i v e w h i t e G a u s s i a n n

i

s e ( A WG N ) c h a n n e l

SLIDE 3

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 3

Wh e r e a r e w e i n t h e B I G P I C T U R E ?

Channel coding Lecture relates to pages 166-179 in textbook. Channel capacity

SLIDE 4

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 4

Wh a t d i d S h a n n

n

p r

m

i s e ?

As long as the SNR is above
1.6 dB in an AWGN channel

we can provide reliable communication

SLIDE 5

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 5

A s c h e m a t i c c

m

m u n i c a t i

n

s y s t e m

OUR FOCUS

SLIDE 6

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 6

T y p i c a l s e q u e n c e s

All typical long sequences have approximately the same probability and from the law of large numbers it follows that the set of these typical sequences is overwhelmingly probable. The probability that a long source output sequence is typical is close to one, and, there are approximately typical long sequences.

REP.

SLIDE 7

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 7

E x a m p l e f r

m

t e x t b

k

( d r a w f r

m

u r n )

- probability 1/3

○ - probability 2/3 Number of typical sequences should be about: Sequences with “observed uncertainty” within 15% of h(1/3) (probability between 0.027 and 0.068): (the ones marked with stars) Why the large discrepancy? Only valid for “long” sequences. … but the 15 sequences are less than 1/2 of all sequences and contain about 2/3 of all probability.

REP.

SLIDE 8

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 8

P r

p

e r t i e s

f

t y p i c a l s e q u e n c e s

REP.

SLIDE 9

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 9

L

n

g e r t y p i c a l s e q u e n c e s

Let us now choose a smaller namely (5% of h(1/3)), and increase the length of the sequences. Then we obtain the following table:

Note: In the first example with length-five sequences we had a wider tolerance

f 15% of h(1/3), and

captured 2/3 of the probability in our typical sequences. With this tighter tolerance we need sequences of length 100 to capture 2/3

f the total probability

in the typical sequences.

REP.

SLIDE 10

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 1

T y p i c a l s e q u e n c e s i n t e x t

If we have L letters in our alphabet, then we can compose Ln different sequences that are n letters long. Only approximately , where H(X) is the uncertainty

f the language, of these are “meaningful”.

What is meant by “meaningful” is determined by the structure of the language; that is, by its grammar, spelling rules etc.

REP.

SLIDE 11

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 1 1

T y p i c a l s e q u e n c e s i n t e x t

Only a fraction which vanishes when n grows provided that is ”meaningful” text of length n letters. For the English language H(X) is typically 1.5 bits/letter and bits/letter.

REP.

SLIDE 12

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 1 2

S t r u c t u r e i n t e x t

Shannon illustrated how increasing structure between letters will give better approximations of the English language. Assuming an alphabet with 27 symbols – 26 letters and one space – he started with an approximation of the first order. The symbols are chosen independently of each other but with the actual probability distribution (12 % E, 2 % W, etc.): OCRO HLI RGWR NMIELWIS EU LL NBNESEBYA TH EEI ALHENHTTPA OOBTTVA NAH BRL

REP.

SLIDE 13

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 1 3

S t r u c t u r e i n t e x t

Then Shannon continued with the approximation of the second order. The symbols are chosen with the actual bigram statistics – when a symbol has been chosen, the next symbol is chosen according to the actual conditional probability distribution: ON IE ANTSOUTINYS ARE T INCTORE ST BE S DEAMY ACHIN D ILONASIVE TUCOOWE AT TEASONARE FUSO TIZIN ANDY TOBE SEACE CTISBE

REP.

SLIDE 14

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 1 4

S t r u c t u r e i n t e x t

The approximation of the third order is based on the trigram statistics – when two successive symbols have been chosen, the next symbol is chosen according to the actual conditional probability distribution: IN NO IST LAT WHEY CRATICT FROURE BIRS GROCID PONDENOME OF DEMONSTRURES OF THE REPTAGIN IS REGOACTIONA OF CRE

REP.

SLIDE 15

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 1 5

Tie p r i n c i p l e

f

s

u

r c e c

d

i n g

Consider the set of typical long output sequences of n symbols from a source with uncertainty H(X) bits per source symbol. Since there are fewer than typical long sequences in this set, they can be represented by binary digits; that is, by binary digits per source symbol.

REP.

SLIDE 16

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 1 6

Channel coding

SLIDE 17

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 1 7

There are 2K different blocks u of K information bits (here 16).

B l

c

k c

d

i n g b a s i c s

… 1001|1110|1010|0011|1010|1111|1110 … Divide the information-sequence to be transmitted into blocks u = [ u1 u1 … uK ] of K bits. Divided into blocks of 4 bits here For each unique block of information bits, assign a unique code word x = [ x1 x2 … xN ] of length N > K bits. Let's use N = 7. Note that this is a subset of all possible sequences of length N. Encode your information sequence by replacing each information block u with the corresponding code word x. … 0011001|0010110|0100101|1000011|1011010|1111111|0010110 … 7 bit code words here This is called an (N,K) block code, with code rate and in this case it is a (7,4) code with rate

SLIDE 18

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 1 8

D i g i t a l c h a n n e l – s y m b

l

s i n a n d

u

t

Symbols X as input to channel Symbols Y as

utput from channel

Our code words Our code words, corrupted by the noisy channel

SLIDE 19

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 1 9

F a n s (

f

a t y p i c a l i n p u t s e q u e n c e a n d i t s t y p i c a l

u

t p u t s e q u e n c e s )

Consider a channel with input X and output Y. Then we have approximately and typical input and output sequences of length N, respectively. Furthermore, for each typical long input sequence we have approximately typical long output sequences that are jointly typical with the given input sequence, we call such an input sequence together with its jointly typical

utput sequences a fan.

SLIDE 20

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 2

We can have at most non-overlapping fans

Input sequences of length N Output sequences of length N

SLIDE 21

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 2 1

Ma x i m u m r a t e

Each fan can represent a message. Hence, the number of distinguishable messages, can be at most, , that is Equivalently, the largest value of the rate R for non-

verlapping fans is

SLIDE 22

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 2 2

C h a n n e l c a p a c i t y

Since we would like to communicate with as high code rate R as possible we choose the input symbols according to the probability distribution that maximizes the mutual information I(X;Y). This maximum value is defined as the capacity of the channel,

SLIDE 23

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 2 3

C h a n n e l c a p a c i t y

Let the encoder map the messages to the typical long input sequences that represent non-overlappling fans, which requires that the code rate R is at most equal to the capacity of the channel, that is, Then the received typical long output sequence is used to identify the corresponding fan and, hence, the corresponding typical long input sequence, or, equivalently, the message, and this can be done correctly with a probability arbitrarily close to 1.

SLIDE 24

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 2 4

C h a n n e l c

d

i n g t h e

r

e m

Suppose we transmit information symbols at rate R=K/N bits per channel using a block code via a channel with capacity C. Provided that R<C we can achieve arbitrary reliability, that is, we can transmit the symbols virtually error-free, by choosing N large enough. Conversely, if R>C, then significant distortion must occur and reliable communication is not possible.

SLIDE 25

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 2 5

B i n a r y s y m m e t r i c c h a n n e l ( B S C ) B i n a r y e r a s u r e c h a n n e l ( B E C )

SLIDE 26

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 2 6

C h a n n e l c a p a c i t y

f

t h e B S C

Since the channel is symmetric (behaves the same for 0 and 1) we can assume that the maximizing input distribution is

SLIDE 27

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 2 7

C h a n n e l c a p a c i t y f

r

t h e B S C

SLIDE 28

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 2 8

Tie a d d i t i v e w h i t e G a u s s i a n n

i

s e ( A WG N ) c h a n n e l

So far we have considered only channels with binary

inputs. Now we shall introduce the time-discrete Gaussian

channel whose output Yi at time i is the sum of the input Xi and the noise Zi where Xi and Yi are real numbers and Zi is a Gaussian random variable with mean 0 and variance .

SLIDE 29

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 2 9

C a p a c i t y

f

t h e A WG N

A natural limitation on the inputs is an average energy constraint; assuming a codeword of N symbols being transmitted, we require that where E is the signaling energy per symbol. It can be shown that the capacity of a Gaussian channel with energy constraint E and noise variance is

SLIDE 30

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 3

C a p a c i t y

f

b a n d l i m i t e d A WG N c h a n n e l

The channel capacity of the bandwidth limited Gaussian channel with two-sided noise spectral density where W denotes the bandwidth in Hz and Ps is the signaling power in Watts.

SLIDE 31

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 3 1

S h a n n

n

' s c h a n n e l c

d

i n g t h e

r

e m

In any system that provides reliable communication

ver a Gaussian channel the signal-to-noise ratio

Eb/N0 must exceed the Shannon limit, -1.6 dB! So long as Eb/N0 > -1.6 dB, Shannon's channel coding theorem guarantees the existence of a system – although it might be very complex – for reliable communication over the channel.

SLIDE 32

O v e E d f

r

s E I T A 3

C

h a p t e r 5 ( P a r t 2 ) 3 2

S u m m a r y

D

i g i t a l c h a n n e l s a r e c h a r a c t e r i z e d b y t h e t r a n s i t i

n

p r

b

a b i l i t i e s ( X → Y )

T

y p i c a l s e q u e n c e s c a n h e l p u s fi n d

u

t h

w

f a s t w e c a n c

m

m u n i c a t e

n

a c h a n n e l

C

h a n n e l c a p a c i t y i s d e fi n e d a s t h e m a x i m a l m u t u a l i n f

r

m a t i

n

b e t w e e n i n p u t ( X ) a n d

u

t p u t ( Y ) a n d i t s h

w

s h

w

f a s t w e c a n c

m

m u n i c a t e r e l i a b l y

v

e r t h e c h a n n e l

C

a p a c i t y

f

t h e b i n a r y s y m m e t r i c c h a n n e l ( B S C ) i s

C

a p a c i t y

f

t h e a d d i t i v e w h i t e G a u s s i a n n

i

s e ( A WG N ) c h a n n e l i s

– (

t i m e

d

i s c r e t e )

– (

c

n

t i n u

u

s b a n d l i m i t e d )

[bit/channel use] [bit/channel use] [bit/sec]

SLIDE 33