The Source Coding Theorem Mathias Winther Madsen - - PowerPoint PPT Presentation

The Source Coding Theorem

Mathias Winther Madsen mathias.winther@gmail.com

Institute for Logic, Language, and Computation University of Amsterdam

March 2015

The Convergence of Averages

Problem

Which of the following is more probable?

• 1. an average of 4,000 in 1,000 dice rolls;
• 2. an average of 4,000,000 in 1,000,000 dice rolls.

The Convergence of Averages

The Weak Law of Large Numbers

For every ε > 0 and α > 0 there is a t such that Pr

• n

i=1 Xi

n − E[X]

• > ε
• ≤ α.

10 20 2 4 6 10 20 50 100

Sequence Probabilities

Problem

With the point probabilities x t s e p(x) .25 .50 .25 Given that we draw 10 letters from this distribution,

• 1. what is Pr(stetsesses)?
• 2. what is the most probable sequence?

Sequence Probabilities · s t e t s e s s e s

0.2 0.4 0.6 0.8 1 Probability

Sequence Probabilities · s t e t s e s s e s

10−6 10−5 10−4 10−3 10−2 10−1 100 Probability

Sequence Probabilities · s t e t s e s s e s

−20 −15 −10 −5 Logarithmic probability

Sequence Probabilities · s t e t s e s s e s

−3 −2 −1 Logarithmic probability

Typical Sequences

Definition

The entropy of a random variable X is H = E

• log

1 p(X)

• = −E [log p(X)] .

Definition

An ε-typical sequence of length n is a sequence for which

• log

1 p(x1, x2, . . . , xn) − Hn

• < ε.

Typical Sequences

The Asymptotic Equipartition Property

Eventually, everything has the same probability.

The Source Coding Theorem

For large n, there are only 2Hn sequences worth caring about.

Typical Sequences