Lossless compression in lossy compression systems Almost every lossy compression system contains a lossless compression system Lossy compression system Dequantizer Transform Lossless Lossless Inverse Quantizer Encoder Decoder Transform Lossless compression system We discuss the basics of lossless compression first, then move on to lossy compression Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 1
Topics in lossless compression Binary decision trees and variable length coding Entropy and bit-rate Prefix codes, Huffman codes, Golomb codes Joint entropy, conditional entropy, sources with memory Fax compression standards Arithmetic coding Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 2
Example: 20 Questions Alice thinks of an outcome (from a finite set), but does not disclose her selection. Bob asks a series of yes/no questions to uniquely determine the outcome chosen. The goal of the game is to ask as few questions as possible on average. Our goal: Design the best strategy for Bob. Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 3
Example: 20 Questions (cont.) Which strategy is better? 0 (=no) 1 (=yes) 0 1 0 0 1 0 1 0 1 1 A B C C D 0 0 1 0 1 F 0 1 E F A B D E Observation: The collection of questions and answers yield a binary code for each outcome. Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 4
Fixed length codes 0 1 0 1 0 1 0 0 1 0 0 1 A B C D E F G H l av log 2 K Average description length for K outcomes Optimum for equally likely outcomes Verify by modifying tree Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 5
Variable length codes If outcomes are NOT equally probable: Use shorter descriptions for likely outcomes Use longer descriptions for less likely outcomes Intuition: Optimum balanced code trees, i.e., with equally likely outcomes, can be pruned to yield unbalanced trees with unequal probabilities. The unbalanced code trees such obtained are also optimum. Hence, an outcome of probability p should require about 1 log 2 bits p Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 6
Entropy of a random variable Consider a discrete, finite-alphabet random variable X Alphabet X { 0 , 1 , 2 ,..., K 1 } P X x for each x X PMF f X x Information associated with the event X=x h x log f x X 2 X Entropy of X is the expected value of that information H X E h X f x log f x X X 2 X x X Unit: bits Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 7
Information and entropy: properties 0 h X x Information Information h X ( x ) strictly increases with decreasing probability f X ( x ) Boundedness of entropy 0 H X ( ) log 2 X Equality if only one Equality if all outcomes outcome can occur are equally likely Very likely and very unlikely events do not substantially change entropy p log p 0 for p 0 or p 1 2 Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 8
Example: Binary random variable H X p log p (1 p )log (1 p ) 2 2 1 0.9 H X 0.8 Equally likely 0.7 0.6 0.5 0.4 0.3 deterministic 0.2 0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 p Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 9
Entropy and bit-rate X n Consider IID random process (or “source”) where each sample (or “symbol”) possesses identical entropy H ( X ) X n H ( X ) is called “entropy rate” of the random process. Noiseless Source Coding Theorem [Shannon, 1948] The entropy H ( X ) is a lower bound for the average word length R of a decodable variable-length code for the symbols. Conversely, the average word length R can approach H ( X ) , if sufficiently large blocks of symbols are encoded jointly. Redundancy of a code: R H X 0 Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 10
Variable length codes X n Given IID random process with alphabet and X PMF X f x Task: assign a distinct code word, c x , to each element, x c c , where is a string of bits, such that each x X x c x symbol can be determined, even if the codewords n x n are directly concatenated in a bitstream Codes with the above property are said to be “uniquely decodable.” Prefix codes No code word is a prefix of any other codeword Uniquely decodable, symbol by symbol, in natural order 0, 1, 2, . . . , n, . . . Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 11
Example of non-decodable code Encode sequence of source symbols , , , , 0 2 3 0 1 Resulting bit-stream 0 10 11 0 01 Encode sequence of source symbols 1 , 0 , 3 , 0 , 1 Resulting bit-stream 01 0 11 0 01 Same bit-stream for different sequences of source symbols: ambiguous, not uniquely decodable BTW: Not a prefix code. Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 12
Unique decodability: McMillan and Kraft conditions Necessary condition for unique decodability [McMillan] c 2 1 x x X Given a set of code word lengths || c x || satisfying McMillan condition, a corresponding prefix code always exists [Kraft] Hence, McMillan inequality is both necessary and sufficient. Also known as Kraft inequality or Kraft-McMillan inequality. No loss by only considering prefix codes. Prefix code is not unique. Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 13
Prefix Decoder Code word LUT Shift register to hold longest code word Input buffer . . . . . . Advance ||c x || bits Code word length LUT Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 14
Binary trees and prefix codes Any binary tree can be 0 1 converted into a prefix code 1 0 0 1 by traversing the tree from 00 01 10 1 root to leaves. 0 111 0 1 1100 1101 Any prefix code corresponding to a binary tree meets McMillan 2 4 3 3 2 2 2 2 1 condition with equality c 2 1 x x X Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 15
Binary trees and prefix codes (cont.) Augmenting binary tree by two 0 1 new nodes does not change McMillan sum. 1 0 0 1 Pruning binary tree does not 1 0 change McMillan sum. 1 2 l l 1 l 1 l 2 2 2 McMillan sum for simplest binary tree 1 1 2 2 1 Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 16
Instantaneous variable length encoding without redundancy Example A code without redundancy, i.e. R H X ( ) requires all individual code word lengths l log f 2 X k k All probabilities would have to be binary fractions: H X 1.75 bits l f ( ) 2 k R 1.75 bits X k 0 Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 17
Huffman Code Design algorithm for variable length codes proposed by Huffman (1952) always finds a code with minimum redundancy. Obtain code tree as follows: 1 Pick the two symbols with lowest probabilities and merge them into a new auxiliary symbol. 2 Calculate the probability of the auxiliary symbol. 3 If more than one symbol remains, repeat steps 1 and 2 for the new auxiliary alphabet. 4 Convert the code tree into a prefix code. Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 18
Huffman Code - Example R fixed 4 bits/symbol Fixed length coding: R Huffman 2.77 bits/symbol Huffman code: H ( X ) 2.69 bits/symbol Entropy 0.08 bits/symbol Redundancy of the Huffman code: Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 19
Redundancy of prefix code for general distribution Huffman code redundancy 0 1 bit/symbol Theorem: For any distribution f X , a prefix code can be found, whose rate R satisfies 1 H X R H X Proof Left hand inequality: Shannon’s noiseless coding theorem Right hand inequality: Choose code word lengths c log f x x 2 X Resulting rate R f x log f x X 2 X x X f x 1 log f x X 2 X x X H X 1 Bernd Girod: EE398A Image and Video Compression Entropy and Lossless Coding no. 20
Recommend
More recommend
