Lecture 3 Lossless Source Coding I-Hsiang Wang Department of - PowerPoint PPT Presentation

Typical Sequences and a Lossless Source Coding Theorem Weakly Typical Sequences and Sources with Memory Lecture 3 Lossless Source Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw September 25, 2015 1 / 47 I-Hsiang Wang IT Lecture 3

Typical Sequences and a Lossless Source Coding Theorem Meta Description I-Hsiang Wang 2 / 47 3 Efficiency : losslessly or within a certain distortion. From the source codeword w , reconstruct the source sequence either 2 Decoder : , with K as small as possible. Weakly Typical Sequences and Sources with Memory 1 Encoder : IT Lecture 3 The Source Coding Problem s [1 : N ] b [1 : K ] s [1 : N ] b Source Source Encoder Decoder Source Destination Represent the source sequence s [1 : N ] by a binary source codeword { } 0 , 1 , . . . , 2 K − 1 w ≜ b [1 : K ] ∈ Determined by the code rate R ≜ K N bits/symbol time

Typical Sequences and a Lossless Source Coding Theorem Weakly Typical Sequences and Sources with Memory I-Hsiang Wang 3 / 47 prescribed distortion. source coding problem. Naturally, one would think of two different decoding criteria for the IT Lecture 3 Decoding Criteria s [1 : N ] b [1 : K ] s [1 : N ] b Source Source Encoder Decoder Source Destination 1 Exact: the reconstructed sequence � s [1 : N ] = s [1 : N ] . 2 Lossy: the reconstructed sequence � s [1 : N ] ̸ = s [1 : N ] , but is within a

Typical Sequences and a Lossless Source Coding Theorem Why? I-Hsiang Wang 4 / 47 What is going wrong? impossible to have data compression. As a consequence, it seems that if we require exact reconstruction, it is Because every possible sequence has to be uniquely represented by K bits! IT Lecture 3 Weakly Typical Sequences and Sources with Memory must satisfy recovery criterion. Let us begin with some simple analysis of the system with the exact For N fixed, if the decoder would like to reconstruct s [1 : N ] exactly for all possible s [1 : N ] ∈ S N , then it is simple to see that the smallest K 2 K − 1 < |S| N ≤ 2 K = ⇒ K = ⌈ N log |S|⌉ .

Typical Sequences and a Lossless Source Coding Theorem With a random source model, immediately there are two approaches one I-Hsiang Wang 5 / 47 Allow (almost) lossless reconstruction rather than exact recovery probabilities, rather than fixing it to be K Allow variable codeword length for different symbols with different can take to demonstrate data compression: different probabilities to be drawn . Weakly Typical Sequences and Sources with Memory engineering reasons, as mentioned in Lecture 1.) One of the simplest ways to capture redundancy is to model the source source sequence. Recall: data compression is possible because there is redundancy in the Random Source IT Lecture 3 as a random process. (Another reason to use a random source model is due to Redundancy comes from the fact that different symbols in S take

Typical Sequences and a Lossless Source Coding Theorem Weakly Typical Sequences and Sources with Memory I-Hsiang Wang 6 / 47 Note : the decoding criterion here is exact reconstruction. minimum compression rate for a given distribution of the random source. In a later lecture on Data Compression, we will introduce an optimal probability, we tend to use shorter codewords to represent it. Using variable codeword length is intuitive – for symbols with higher The key difference here is that we allow K to depend on the realization of IT Lecture 3 Block-to-Variable Source Coding s [1 : N ] b [1 : K ] b s [1 : N ] Source Source Encoder Decoder Variable Length Source Destination the source, s [1 : N ] . The definition of the code rate is modified to R ≜ E [ K ] N . block-to-variable source code, Huffman code, which can achieve the

Typical Sequences and a Lossless Source Coding Theorem Weakly Typical Sequences and Sources with Memory I-Hsiang Wang 7 / 47 combinatorial , the analysis here is majorly probabilistic . Compared to the previous approach where the analysis is mainly Key features of this approach: IT Lecture 3 N e Another way to let the randomness kick in: allow non-exact recovery. (Almost) Lossless Decoding Criterion To be precise, we turn our focus to finding the smallest possible R = K given that the error probability { } S [1 : N ] ̸ = � P ( N ) ≜ P S [1 : N ] → 0 as N → ∞ . Focus on the asymptotic regime where N → ∞ ; instead of error-free reconstruction, the criterion is relaxed to vanishing error probability.

Typical Sequences and a Lossless Source Coding Theorem Weakly Typical Sequences and Sources with Memory I-Hsiang Wang 8 / 47 memoryless source (DMS). i.i.d. of the random source. IT Lecture 3 memory, and prove a similar lossless source coding theorem there. 2 Second, extend the typical sequence framework to sources with lossless source coding theorem called typical sequences, and use typical sequences to prove a 1 First, focusing on memoryless sources, introduce a powerful tool In this lecture, we shall Outline We will show that the minimum compression rate is equal to the entropy Let us begin with the simplest case where the source { S [ t ] | t = 1 , 2 , . . . } consists of i.i.d. random variables S [ t ] ∼ p S , which is called a discrete

Typical Sequences and a Lossless Source Coding Theorem Weakly Typical Sequences and Sources with Memory Typicality and AEP Lossless Source Coding Theorem 1 Typical Sequences and a Lossless Source Coding Theorem Typicality and AEP Lossless Source Coding Theorem 2 Weakly Typical Sequences and Sources with Memory Entropy Rate of Random Processes Typicality for Sources with Memory 9 / 47 I-Hsiang Wang IT Lecture 3

Typical Sequences and a Lossless Source Coding Theorem Weakly Typical Sequences and Sources with Memory Typicality and AEP Lossless Source Coding Theorem 1 Typical Sequences and a Lossless Source Coding Theorem Typicality and AEP Lossless Source Coding Theorem 2 Weakly Typical Sequences and Sources with Memory Entropy Rate of Random Processes Typicality for Sources with Memory 10 / 47 I-Hsiang Wang IT Lecture 3

Typical Sequences and a Lossless Source Coding Theorem (cf. Lecture 2 “Operational Meaning of Entropy”) I-Hsiang Wang 11 / 47 Notation : For notational convenience, we shall use the following and weak typicality. the literature. In this lecture, we give two definitions: (robust) typicality Note : There are several notions of typicality and various definitions in Weakly Typical Sequences and Sources with Memory For lossless reconstruction with vanishing error probability, we can use whole probability, while others become “atypical”. Goal : Understand and exploit the probabilistic asymptotic properties of a Overview of Typicality Methods Lossless Source Coding Theorem Typicality and AEP IT Lecture 3 i.i.d. randomly generated sequence S [1 : N ] for coding. Key Observation : When N → ∞ , one often observe that a substantially small set of sequences become “typical”, which contribute almost the shorter codewords to label “typical” sequences and ignore “atypical” ones. interchangeably: x [ t ] ← → x t , and x [1 : N ] ← → x N .

Typical Sequences and a Lossless Source Coding Theorem n I-Hsiang Wang 12 / 47 Definition 1 (Typical Sequence) empirical p.m.f. does not deviate too much from the actual p.m.f. Weakly Typical Sequences and Sources with Memory . p IT Lecture 3 Roughly speaking, a (robust) typical sequence is a sequence whose Typicality and AEP Lossless Source Coding Theorem Typical Sequence empirical distribution is close to the actual distribution. For a sequence x n , its empirical p.m.f. is given by the frequency of ∑ n i =1 1 { x i = a } occurrence of a symbol in the sequence: π ( a | x n ) ≜ Due the law of large numbers, π ( a | x n ) → p X ( a ) for all a ∈ X as n → ∞ , if x n is drawn i.i.d. based on p X . With high probability, the For X ∼ p X and ε ∈ (0 , 1) , a sequence x n is called ε -typical if | π ( a | x n ) − p X ( a ) | ≤ ε p X ( a ) , ∀ a ∈ X . The typical set is defined as the collection of all ε -typical length- n sequences, denoted by T ( n ) ( X ) . ε

Typical Sequences and a Lossless Source Coding Theorem Consider a random bit sequence generated i.i.d. based on Ber I-Hsiang Wang 13 / 47 is Weakly Typical Sequences and Sources with Memory ? How large is the typical set? . Let 2 IT Lecture 3 ” Example 1 Typicality and AEP Lossless Source Coding Theorem Note : In the following, if the context is clear, we will write “ T ( n ) ε instead of “ T ( n ) ( X ) ”. ε ( 1 ) us set ε = 0 . 2 and n = 10 . What is T ( n ) ε sol : Based on the definition, a n -sequence x n is ε -typical iff π (0 | x n ) ∈ [0 . 4 , 0 . 6] and π (1 | x n ) ∈ [0 . 4 , 0 . 6] . In other words, the # of “ 0 ”s in the sequence should be 4, 5, or 6. Hence, T ( n ) consists of all length-10 sequences with 4, 5, or 6 0 ”s. ε ( 10 ) ( 10 ) ( 10 ) The size of T ( n ) + + = 714 . ε 4 5 6