Compression, Information and Entropy Huffmans coding Lecture 26 - PowerPoint PPT Presentation

Codes... . . Morse code is binary+prefix code but not prefix-free. 1 . . ... code for S ( · · · ) includes the code for E ( · ) as a prefix. 2 . . Prefix codes are binary trees... 3 0 1 d 0 1 a 0 1 c b . ...characters in leafs, code word is path from root. 4 . prefix treestree!prefix tree or code trees . 5 . . Decoding/encoding is easy. 6 Sariel (UIUC) CS573 4 Fall 2013 4 / 22

Codes... . . Encoding: given frequency table: 1 f [1 . . . n ] . . . f [ i ] : frequency of i th character. 2 . . code( i ) : binary string for i th character. 3 len( s ) : length (in bits) of binary string s . . . Compute tree T that minimizes 4 n ∑ cost( T ) = f [ i ] ∗ len(code( i )) , (1) i =1 Sariel (UIUC) CS573 5 Fall 2013 5 / 22

Codes... . . Encoding: given frequency table: 1 f [1 . . . n ] . . . f [ i ] : frequency of i th character. 2 . . code( i ) : binary string for i th character. 3 len( s ) : length (in bits) of binary string s . . . Compute tree T that minimizes 4 n ∑ cost( T ) = f [ i ] ∗ len(code( i )) , (1) i =1 Sariel (UIUC) CS573 5 Fall 2013 5 / 22

Codes... . . Encoding: given frequency table: 1 f [1 . . . n ] . . . f [ i ] : frequency of i th character. 2 . . code( i ) : binary string for i th character. 3 len( s ) : length (in bits) of binary string s . . . Compute tree T that minimizes 4 n ∑ cost( T ) = f [ i ] ∗ len(code( i )) , (1) i =1 Sariel (UIUC) CS573 5 Fall 2013 5 / 22

Codes... . . Encoding: given frequency table: 1 f [1 . . . n ] . . . f [ i ] : frequency of i th character. 2 . . code( i ) : binary string for i th character. 3 len( s ) : length (in bits) of binary string s . . . Compute tree T that minimizes 4 n ∑ cost( T ) = f [ i ] ∗ len(code( i )) , (1) i =1 Sariel (UIUC) CS573 5 Fall 2013 5 / 22

Frequency table for... “A tale of two cities” by Dickens \ n 16,492 ‘1’ 61 ‘C’ 13,896 ‘Q’ 667 ’ ’ 130,376 ‘2’ 10 ‘D’ 28,041 ‘R’ 37,187 ‘!’ 955 ‘3’ 12 ‘E’ 74,809 ‘S’ 37,575 ‘”’ 5,681 ‘4’ 10 ‘F’ 13,559 ‘T’ 54,024 ‘$’ 2 ‘5’ 14 ‘G’ 12,530 ‘U’ 16,726 ‘%’ 1 ‘6’ 11 ‘H’ 38,961 ‘V’ 5,199 ‘” 1,174 ‘7’ 13 ‘I’ 41,005 ‘W’ 14,113 ‘(’ 151 ‘8’ 13 ‘J’ 710 ‘X’ 724 ‘)’ 151 ‘9’ 14 ‘K’ 4,782 ‘Y’ 12,177 ‘*’ 70 ‘:’ 267 ‘L’ 22,030 ‘Z’ 215 ‘,’ 13,276 ‘;’ 1,108 ‘M’ 15,298 ‘ ’ 182 ‘–’ 2,430 ‘?’ 913 ‘N’ 42,380 ’‘’ 93 ‘.’ 6,769 ‘A’ 48,165 ‘O’ 46,499 ‘@’ 2 ‘0’ 20 ‘B’ 8,414 ‘P’ 9,957 ‘/’ 26 Sariel (UIUC) CS573 6 Fall 2013 6 / 22

Computed prefix codes... char code char code frequency freq ‘A’ 48165 1110 ‘N’ 42380 1100 ‘B’ 8414 101000 ‘O’ 46499 1101 ‘C’ 13896 00100 ‘P’ 9957 101001 ‘D’ 28041 0011 ‘Q’ 667 1111011001 ‘E’ 74809 011 ‘R’ 37187 0101 ‘F’ 13559 111111 ‘S’ 37575 1000 ‘G’ 12530 111110 ‘T’ 54024 000 ‘H’ 38961 1001 ‘U’ 16726 01001 ‘V’ 5199 1111010 ‘I’ 41005 1011 ‘J’ 710 1111011010 ‘W’ 14113 00101 ‘K’ 4782 11110111 ‘X’ 724 1111011011 ‘L’ 22030 10101 ‘Y’ 12177 111100 ‘M’ 15298 01000 ‘Z’ 215 1111011000 Sariel (UIUC) CS573 7 Fall 2013 7 / 22

The Huffman tree generating the code Build only on A-Z for clarity. • . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • . . . . . . . . . . . . . . . . . . . . . . . . . . • . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • . . . . . . . . . . . . . . . . • • . . . . . . . . . . . . . . . . . . • . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • • . . . . • . . . . . . . . . . . . • • . . . . . . . . . . . . • T . . . . . . . . . . E . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • . . . . . . • . . . • . . . . . . . . . . . . . . . . . . . . • . . . . D . . . . R S H . . . . I N O A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • . . . • . . . . . . . . . . . . . . . . . . • C W M U . . . . L . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • B P Y . . . . . G F . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • V . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • . . . . . . . . . . K . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • . . . . . . . . . . • . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Z Q J X Sariel (UIUC) CS573 8 Fall 2013 8 / 22

Mergeablity of code trees . . two trees for some disjoint parts of the alphabet... 1 . . Merge into larger tree by creating a new node and hanging the 2 trees from this common node. • . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M U ⇒ . . . . . . . . . . 3 M U . . ...put together two subtrees. 4 • A B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ⇒ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A . . . . . . . . B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sariel (UIUC) CS573 9 Fall 2013 9 / 22

Mergeablity of code trees . . two trees for some disjoint parts of the alphabet... 1 . . Merge into larger tree by creating a new node and hanging the 2 trees from this common node. • . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M U ⇒ . . . . . . . . . . 3 M U . . ...put together two subtrees. 4 • A B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ⇒ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A . . . . . . . . B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sariel (UIUC) CS573 9 Fall 2013 9 / 22

Mergeablity of code trees . . two trees for some disjoint parts of the alphabet... 1 . . Merge into larger tree by creating a new node and hanging the 2 trees from this common node. • . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M U ⇒ . . . . . . . . . . 3 M U . . ...put together two subtrees. 4 • A B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ⇒ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A . . . . . . . . B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sariel (UIUC) CS573 9 Fall 2013 9 / 22

Mergeablity of code trees . . two trees for some disjoint parts of the alphabet... 1 . . Merge into larger tree by creating a new node and hanging the 2 trees from this common node. • . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M U ⇒ . . . . . . . . . . 3 M U . . ...put together two subtrees. 4 • A B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ⇒ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A . . . . . . . . B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sariel (UIUC) CS573 9 Fall 2013 9 / 22

Building optimal prefix code trees . . take two least frequent characters in frequency table... 1 . . ... merge them into a tree, and put the root of merged tree back 2 into table. . . ...instead of the two old trees. 3 . . Algorithm stops when there is a single tree. 4 . . Intuition: infrequent characters participate in a large number of 5 merges. Long code words. . . Algorithm is due to David Huffman (1952). 6 . . Resulting code is best one can do. 7 . Huffman coding : building block used by numerous other 8 compression algorithms. Sariel (UIUC) CS573 10 Fall 2013 10 / 22

Building optimal prefix code trees . . take two least frequent characters in frequency table... 1 . . ... merge them into a tree, and put the root of merged tree back 2 into table. . . ...instead of the two old trees. 3 . . Algorithm stops when there is a single tree. 4 . . Intuition: infrequent characters participate in a large number of 5 merges. Long code words. . . Algorithm is due to David Huffman (1952). 6 . . Resulting code is best one can do. 7 . Huffman coding : building block used by numerous other 8 compression algorithms. Sariel (UIUC) CS573 10 Fall 2013 10 / 22

Building optimal prefix code trees . . take two least frequent characters in frequency table... 1 . . ... merge them into a tree, and put the root of merged tree back 2 into table. . . ...instead of the two old trees. 3 . . Algorithm stops when there is a single tree. 4 . . Intuition: infrequent characters participate in a large number of 5 merges. Long code words. . . Algorithm is due to David Huffman (1952). 6 . . Resulting code is best one can do. 7 . Huffman coding : building block used by numerous other 8 compression algorithms. Sariel (UIUC) CS573 10 Fall 2013 10 / 22

Building optimal prefix code trees . . take two least frequent characters in frequency table... 1 . . ... merge them into a tree, and put the root of merged tree back 2 into table. . . ...instead of the two old trees. 3 . . Algorithm stops when there is a single tree. 4 . . Intuition: infrequent characters participate in a large number of 5 merges. Long code words. . . Algorithm is due to David Huffman (1952). 6 . . Resulting code is best one can do. 7 . Huffman coding : building block used by numerous other 8 compression algorithms. Sariel (UIUC) CS573 10 Fall 2013 10 / 22

Building optimal prefix code trees . . take two least frequent characters in frequency table... 1 . . ... merge them into a tree, and put the root of merged tree back 2 into table. . . ...instead of the two old trees. 3 . . Algorithm stops when there is a single tree. 4 . . Intuition: infrequent characters participate in a large number of 5 merges. Long code words. . . Algorithm is due to David Huffman (1952). 6 . . Resulting code is best one can do. 7 . Huffman coding : building block used by numerous other 8 compression algorithms. Sariel (UIUC) CS573 10 Fall 2013 10 / 22

Building optimal prefix code trees . . take two least frequent characters in frequency table... 1 . . ... merge them into a tree, and put the root of merged tree back 2 into table. . . ...instead of the two old trees. 3 . . Algorithm stops when there is a single tree. 4 . . Intuition: infrequent characters participate in a large number of 5 merges. Long code words. . . Algorithm is due to David Huffman (1952). 6 . . Resulting code is best one can do. 7 . Huffman coding : building block used by numerous other 8 compression algorithms. Sariel (UIUC) CS573 10 Fall 2013 10 / 22

Building optimal prefix code trees . . take two least frequent characters in frequency table... 1 . . ... merge them into a tree, and put the root of merged tree back 2 into table. . . ...instead of the two old trees. 3 . . Algorithm stops when there is a single tree. 4 . . Intuition: infrequent characters participate in a large number of 5 merges. Long code words. . . Algorithm is due to David Huffman (1952). 6 . . Resulting code is best one can do. 7 . Huffman coding : building block used by numerous other 8 compression algorithms. Sariel (UIUC) CS573 10 Fall 2013 10 / 22

Building optimal prefix code trees . . take two least frequent characters in frequency table... 1 . . ... merge them into a tree, and put the root of merged tree back 2 into table. . . ...instead of the two old trees. 3 . . Algorithm stops when there is a single tree. 4 . . Intuition: infrequent characters participate in a large number of 5 merges. Long code words. . . Algorithm is due to David Huffman (1952). 6 . . Resulting code is best one can do. 7 . Huffman coding : building block used by numerous other 8 compression algorithms. Sariel (UIUC) CS573 10 Fall 2013 10 / 22

Analysis... . Lemma . . T : optimal code tree (prefix free!). 1 . . Then T is a full binary tree. 2 . . ... every node of T has either 0 or 2 children. 3 . . If height of T is d , then there are leafs nodes of height d that 4 are sibling. . Sariel (UIUC) CS573 11 Fall 2013 11 / 22

Proof... . . If there is an internal node in T that has one child, we can 1 remove this node from T , by connecting its only child directly with its parent. The resulting code tree is clearly a better compressor, in the sense of cost( T ) = ∑ n i =1 f [ i ] ∗ len(code( i )) . . . u : leaf u with maximum depth d in T . Consider parent 2 v = p( u ) . . . = ⇒ v : has two children, both leafs 3 Sariel (UIUC) CS573 12 Fall 2013 12 / 22

Proof... . . If there is an internal node in T that has one child, we can 1 remove this node from T , by connecting its only child directly with its parent. The resulting code tree is clearly a better compressor, in the sense of cost( T ) = ∑ n i =1 f [ i ] ∗ len(code( i )) . . . u : leaf u with maximum depth d in T . Consider parent 2 v = p( u ) . . . = ⇒ v : has two children, both leafs 3 Sariel (UIUC) CS573 12 Fall 2013 12 / 22

Proof... . . If there is an internal node in T that has one child, we can 1 remove this node from T , by connecting its only child directly with its parent. The resulting code tree is clearly a better compressor, in the sense of cost( T ) = ∑ n i =1 f [ i ] ∗ len(code( i )) . . . u : leaf u with maximum depth d in T . Consider parent 2 v = p( u ) . . . = ⇒ v : has two children, both leafs 3 Sariel (UIUC) CS573 12 Fall 2013 12 / 22

Proof... . . If there is an internal node in T that has one child, we can 1 remove this node from T , by connecting its only child directly with its parent. The resulting code tree is clearly a better compressor, in the sense of cost( T ) = ∑ n i =1 f [ i ] ∗ len(code( i )) . . . u : leaf u with maximum depth d in T . Consider parent 2 v = p( u ) . . . = ⇒ v : has two children, both leafs 3 Sariel (UIUC) CS573 12 Fall 2013 12 / 22

Infrequence characters are stuck together... . Lemma . Let x and y be the two least frequent characters (breaking ties between equally frequent characters arbitrarily). There is an optimal code tree in which x and y are siblings. . Sariel (UIUC) CS573 13 Fall 2013 13 / 22

Proof... . . Claim: ∃ optimal code s.t. x and y are siblings + deepest. 1 . . T : optimal code tree with depth d . 2 . . By lemma... T has two leafs at depth d that are siblings, 3 . . If not x and y , but some other characters α and β . 4 . . T ′ : swap x and α . 5 . . x depth inc by ∆ , and depth of α decreases by ∆ . 6 . . ( ) cost( T ′ ) = cost( T ) − f [ α ] − f [ x ] ∆ . 7 . . x : one of the two least frequent characters. 8 ...but α is not. . = ⇒ f [ α ] > f [ x ] . 9 . . 10 Swapping x and α does not increase cost. . . 11 T : optimal code tree, swapping x and α does not decrease cost. 12 T ′ is also an optimal code tree ( f [ α ] = f [ x ] ). . . . . 13 Swapping y and b must give yet another optimal code tree. . . 14 Final opt code tree, x , y are max-depth siblings. Sariel (UIUC) CS573 14 Fall 2013 14 / 22

Proof... . . Claim: ∃ optimal code s.t. x and y are siblings + deepest. 1 . . T : optimal code tree with depth d . 2 . . By lemma... T has two leafs at depth d that are siblings, 3 . . If not x and y , but some other characters α and β . 4 . . T ′ : swap x and α . 5 . . x depth inc by ∆ , and depth of α decreases by ∆ . 6 . . ( ) cost( T ′ ) = cost( T ) − f [ α ] − f [ x ] ∆ . 7 . . x : one of the two least frequent characters. 8 ...but α is not. . = ⇒ f [ α ] > f [ x ] . 9 . . 10 Swapping x and α does not increase cost. . . 11 T : optimal code tree, swapping x and α does not decrease cost. 12 T ′ is also an optimal code tree ( f [ α ] = f [ x ] ). . . . . 13 Swapping y and b must give yet another optimal code tree. . . 14 Final opt code tree, x , y are max-depth siblings. Sariel (UIUC) CS573 14 Fall 2013 14 / 22

Proof... . . Claim: ∃ optimal code s.t. x and y are siblings + deepest. 1 . . T : optimal code tree with depth d . 2 . . By lemma... T has two leafs at depth d that are siblings, 3 . . If not x and y , but some other characters α and β . 4 . . T ′ : swap x and α . 5 . . x depth inc by ∆ , and depth of α decreases by ∆ . 6 . . ( ) cost( T ′ ) = cost( T ) − f [ α ] − f [ x ] ∆ . 7 . . x : one of the two least frequent characters. 8 ...but α is not. . = ⇒ f [ α ] > f [ x ] . 9 . . 10 Swapping x and α does not increase cost. . . 11 T : optimal code tree, swapping x and α does not decrease cost. 12 T ′ is also an optimal code tree ( f [ α ] = f [ x ] ). . . . . 13 Swapping y and b must give yet another optimal code tree. . . 14 Final opt code tree, x , y are max-depth siblings. Sariel (UIUC) CS573 14 Fall 2013 14 / 22

Proof... . . Claim: ∃ optimal code s.t. x and y are siblings + deepest. 1 . . T : optimal code tree with depth d . 2 . . By lemma... T has two leafs at depth d that are siblings, 3 . . If not x and y , but some other characters α and β . 4 . . T ′ : swap x and α . 5 . . x depth inc by ∆ , and depth of α decreases by ∆ . 6 . . ( ) cost( T ′ ) = cost( T ) − f [ α ] − f [ x ] ∆ . 7 . . x : one of the two least frequent characters. 8 ...but α is not. . = ⇒ f [ α ] > f [ x ] . 9 . . 10 Swapping x and α does not increase cost. . . 11 T : optimal code tree, swapping x and α does not decrease cost. 12 T ′ is also an optimal code tree ( f [ α ] = f [ x ] ). . . . . 13 Swapping y and b must give yet another optimal code tree. . . 14 Final opt code tree, x , y are max-depth siblings. Sariel (UIUC) CS573 14 Fall 2013 14 / 22

Proof... . . Claim: ∃ optimal code s.t. x and y are siblings + deepest. 1 . . T : optimal code tree with depth d . 2 . . By lemma... T has two leafs at depth d that are siblings, 3 . . If not x and y , but some other characters α and β . 4 . . T ′ : swap x and α . 5 . . x depth inc by ∆ , and depth of α decreases by ∆ . 6 . . ( ) cost( T ′ ) = cost( T ) − f [ α ] − f [ x ] ∆ . 7 . . x : one of the two least frequent characters. 8 ...but α is not. . = ⇒ f [ α ] > f [ x ] . 9 . . 10 Swapping x and α does not increase cost. . . 11 T : optimal code tree, swapping x and α does not decrease cost. 12 T ′ is also an optimal code tree ( f [ α ] = f [ x ] ). . . . . 13 Swapping y and b must give yet another optimal code tree. . . 14 Final opt code tree, x , y are max-depth siblings. Sariel (UIUC) CS573 14 Fall 2013 14 / 22

Proof... . . Claim: ∃ optimal code s.t. x and y are siblings + deepest. 1 . . T : optimal code tree with depth d . 2 . . By lemma... T has two leafs at depth d that are siblings, 3 . . If not x and y , but some other characters α and β . 4 . . T ′ : swap x and α . 5 . . x depth inc by ∆ , and depth of α decreases by ∆ . 6 . . ( ) cost( T ′ ) = cost( T ) − f [ α ] − f [ x ] ∆ . 7 . . x : one of the two least frequent characters. 8 ...but α is not. . = ⇒ f [ α ] > f [ x ] . 9 . . 10 Swapping x and α does not increase cost. . . 11 T : optimal code tree, swapping x and α does not decrease cost. 12 T ′ is also an optimal code tree ( f [ α ] = f [ x ] ). . . . . 13 Swapping y and b must give yet another optimal code tree. . . 14 Final opt code tree, x , y are max-depth siblings. Sariel (UIUC) CS573 14 Fall 2013 14 / 22

Proof... . . Claim: ∃ optimal code s.t. x and y are siblings + deepest. 1 . . T : optimal code tree with depth d . 2 . . By lemma... T has two leafs at depth d that are siblings, 3 . . If not x and y , but some other characters α and β . 4 . . T ′ : swap x and α . 5 . . x depth inc by ∆ , and depth of α decreases by ∆ . 6 . . ( ) cost( T ′ ) = cost( T ) − f [ α ] − f [ x ] ∆ . 7 . . x : one of the two least frequent characters. 8 ...but α is not. . = ⇒ f [ α ] > f [ x ] . 9 . . 10 Swapping x and α does not increase cost. . . 11 T : optimal code tree, swapping x and α does not decrease cost. 12 T ′ is also an optimal code tree ( f [ α ] = f [ x ] ). . . . . 13 Swapping y and b must give yet another optimal code tree. . . 14 Final opt code tree, x , y are max-depth siblings. Sariel (UIUC) CS573 14 Fall 2013 14 / 22

Proof... . . Claim: ∃ optimal code s.t. x and y are siblings + deepest. 1 . . T : optimal code tree with depth d . 2 . . By lemma... T has two leafs at depth d that are siblings, 3 . . If not x and y , but some other characters α and β . 4 . . T ′ : swap x and α . 5 . . x depth inc by ∆ , and depth of α decreases by ∆ . 6 . . ( ) cost( T ′ ) = cost( T ) − f [ α ] − f [ x ] ∆ . 7 . . x : one of the two least frequent characters. 8 ...but α is not. . = ⇒ f [ α ] > f [ x ] . 9 . . 10 Swapping x and α does not increase cost. . . 11 T : optimal code tree, swapping x and α does not decrease cost. 12 T ′ is also an optimal code tree ( f [ α ] = f [ x ] ). . . . . 13 Swapping y and b must give yet another optimal code tree. . . 14 Final opt code tree, x , y are max-depth siblings. Sariel (UIUC) CS573 14 Fall 2013 14 / 22

Proof... . . Claim: ∃ optimal code s.t. x and y are siblings + deepest. 1 . . T : optimal code tree with depth d . 2 . . By lemma... T has two leafs at depth d that are siblings, 3 . . If not x and y , but some other characters α and β . 4 . . T ′ : swap x and α . 5 . . x depth inc by ∆ , and depth of α decreases by ∆ . 6 . . ( ) cost( T ′ ) = cost( T ) − f [ α ] − f [ x ] ∆ . 7 . . x : one of the two least frequent characters. 8 ...but α is not. . = ⇒ f [ α ] > f [ x ] . 9 . . 10 Swapping x and α does not increase cost. . . 11 T : optimal code tree, swapping x and α does not decrease cost. 12 T ′ is also an optimal code tree ( f [ α ] = f [ x ] ). . . . . 13 Swapping y and b must give yet another optimal code tree. . . 14 Final opt code tree, x , y are max-depth siblings. Sariel (UIUC) CS573 14 Fall 2013 14 / 22

Proof... . . Claim: ∃ optimal code s.t. x and y are siblings + deepest. 1 . . T : optimal code tree with depth d . 2 . . By lemma... T has two leafs at depth d that are siblings, 3 . . If not x and y , but some other characters α and β . 4 . . T ′ : swap x and α . 5 . . x depth inc by ∆ , and depth of α decreases by ∆ . 6 . . ( ) cost( T ′ ) = cost( T ) − f [ α ] − f [ x ] ∆ . 7 . . x : one of the two least frequent characters. 8 ...but α is not. . = ⇒ f [ α ] > f [ x ] . 9 . . 10 Swapping x and α does not increase cost. . . 11 T : optimal code tree, swapping x and α does not decrease cost. 12 T ′ is also an optimal code tree ( f [ α ] = f [ x ] ). . . . . 13 Swapping y and b must give yet another optimal code tree. . . 14 Final opt code tree, x , y are max-depth siblings. Sariel (UIUC) CS573 14 Fall 2013 14 / 22

Proof... . . Claim: ∃ optimal code s.t. x and y are siblings + deepest. 1 . . T : optimal code tree with depth d . 2 . . By lemma... T has two leafs at depth d that are siblings, 3 . . If not x and y , but some other characters α and β . 4 . . T ′ : swap x and α . 5 . . x depth inc by ∆ , and depth of α decreases by ∆ . 6 . . ( ) cost( T ′ ) = cost( T ) − f [ α ] − f [ x ] ∆ . 7 . . x : one of the two least frequent characters. 8 ...but α is not. . = ⇒ f [ α ] > f [ x ] . 9 . . 10 Swapping x and α does not increase cost. . . 11 T : optimal code tree, swapping x and α does not decrease cost. 12 T ′ is also an optimal code tree ( f [ α ] = f [ x ] ). . . . . 13 Swapping y and b must give yet another optimal code tree. . . 14 Final opt code tree, x , y are max-depth siblings. Sariel (UIUC) CS573 14 Fall 2013 14 / 22

Proof... . . Claim: ∃ optimal code s.t. x and y are siblings + deepest. 1 . . T : optimal code tree with depth d . 2 . . By lemma... T has two leafs at depth d that are siblings, 3 . . If not x and y , but some other characters α and β . 4 . . T ′ : swap x and α . 5 . . x depth inc by ∆ , and depth of α decreases by ∆ . 6 . . ( ) cost( T ′ ) = cost( T ) − f [ α ] − f [ x ] ∆ . 7 . . x : one of the two least frequent characters. 8 ...but α is not. . = ⇒ f [ α ] > f [ x ] . 9 . . 10 Swapping x and α does not increase cost. . . 11 T : optimal code tree, swapping x and α does not decrease cost. 12 T ′ is also an optimal code tree ( f [ α ] = f [ x ] ). . . . . 13 Swapping y and b must give yet another optimal code tree. . . 14 Final opt code tree, x , y are max-depth siblings. Sariel (UIUC) CS573 14 Fall 2013 14 / 22

Proof... . . Claim: ∃ optimal code s.t. x and y are siblings + deepest. 1 . . T : optimal code tree with depth d . 2 . . By lemma... T has two leafs at depth d that are siblings, 3 . . If not x and y , but some other characters α and β . 4 . . T ′ : swap x and α . 5 . . x depth inc by ∆ , and depth of α decreases by ∆ . 6 . . ( ) cost( T ′ ) = cost( T ) − f [ α ] − f [ x ] ∆ . 7 . . x : one of the two least frequent characters. 8 ...but α is not. . = ⇒ f [ α ] > f [ x ] . 9 . . 10 Swapping x and α does not increase cost. . . 11 T : optimal code tree, swapping x and α does not decrease cost. 12 T ′ is also an optimal code tree ( f [ α ] = f [ x ] ). . . . . 13 Swapping y and b must give yet another optimal code tree. . . 14 Final opt code tree, x , y are max-depth siblings. Sariel (UIUC) CS573 14 Fall 2013 14 / 22

Proof... . . Claim: ∃ optimal code s.t. x and y are siblings + deepest. 1 . . T : optimal code tree with depth d . 2 . . By lemma... T has two leafs at depth d that are siblings, 3 . . If not x and y , but some other characters α and β . 4 . . T ′ : swap x and α . 5 . . x depth inc by ∆ , and depth of α decreases by ∆ . 6 . . ( ) cost( T ′ ) = cost( T ) − f [ α ] − f [ x ] ∆ . 7 . . x : one of the two least frequent characters. 8 ...but α is not. . = ⇒ f [ α ] > f [ x ] . 9 . . 10 Swapping x and α does not increase cost. . . 11 T : optimal code tree, swapping x and α does not decrease cost. 12 T ′ is also an optimal code tree ( f [ α ] = f [ x ] ). . . . . 13 Swapping y and b must give yet another optimal code tree. . . 14 Final opt code tree, x , y are max-depth siblings. Sariel (UIUC) CS573 14 Fall 2013 14 / 22

Huffman’s codes are optimal . Theorem . Huffman codes are optimal prefix-free binary codes. . Sariel (UIUC) CS573 15 Fall 2013 15 / 22

Proof... . . If message has 1 or 2 diff characters, then theorem easy. 1 . . f [1 . . . n ] be original input frequencies. 2 . . Assume f [1] and f [2] are the two smallest. 3 . . Let f [ n + 1] = f [1] + f [2] . 4 . . lemma = ⇒ ∃ opt. code tree T opt for f [1 .. n ] 5 . . T opt has 1 and 2 as siblings. 6 . . Remove 1 and 2 from T opt . 7 . . T ′ opt : Remaining tree has 3 , . . . , n as leafs and “special” 8 character n + 1 (i.e., parent 1 , 2 in T opt ) Sariel (UIUC) CS573 16 Fall 2013 16 / 22

Proof... . . If message has 1 or 2 diff characters, then theorem easy. 1 . . f [1 . . . n ] be original input frequencies. 2 . . Assume f [1] and f [2] are the two smallest. 3 . . Let f [ n + 1] = f [1] + f [2] . 4 . . lemma = ⇒ ∃ opt. code tree T opt for f [1 .. n ] 5 . . T opt has 1 and 2 as siblings. 6 . . Remove 1 and 2 from T opt . 7 . . T ′ opt : Remaining tree has 3 , . . . , n as leafs and “special” 8 character n + 1 (i.e., parent 1 , 2 in T opt ) Sariel (UIUC) CS573 16 Fall 2013 16 / 22

Proof... . . If message has 1 or 2 diff characters, then theorem easy. 1 . . f [1 . . . n ] be original input frequencies. 2 . . Assume f [1] and f [2] are the two smallest. 3 . . Let f [ n + 1] = f [1] + f [2] . 4 . . lemma = ⇒ ∃ opt. code tree T opt for f [1 .. n ] 5 . . T opt has 1 and 2 as siblings. 6 . . Remove 1 and 2 from T opt . 7 . . T ′ opt : Remaining tree has 3 , . . . , n as leafs and “special” 8 character n + 1 (i.e., parent 1 , 2 in T opt ) Sariel (UIUC) CS573 16 Fall 2013 16 / 22

Proof... . . If message has 1 or 2 diff characters, then theorem easy. 1 . . f [1 . . . n ] be original input frequencies. 2 . . Assume f [1] and f [2] are the two smallest. 3 . . Let f [ n + 1] = f [1] + f [2] . 4 . . lemma = ⇒ ∃ opt. code tree T opt for f [1 .. n ] 5 . . T opt has 1 and 2 as siblings. 6 . . Remove 1 and 2 from T opt . 7 . . T ′ opt : Remaining tree has 3 , . . . , n as leafs and “special” 8 character n + 1 (i.e., parent 1 , 2 in T opt ) Sariel (UIUC) CS573 16 Fall 2013 16 / 22

Proof... . . If message has 1 or 2 diff characters, then theorem easy. 1 . . f [1 . . . n ] be original input frequencies. 2 . . Assume f [1] and f [2] are the two smallest. 3 . . Let f [ n + 1] = f [1] + f [2] . 4 . . lemma = ⇒ ∃ opt. code tree T opt for f [1 .. n ] 5 . . T opt has 1 and 2 as siblings. 6 . . Remove 1 and 2 from T opt . 7 . . T ′ opt : Remaining tree has 3 , . . . , n as leafs and “special” 8 character n + 1 (i.e., parent 1 , 2 in T opt ) Sariel (UIUC) CS573 16 Fall 2013 16 / 22

Proof... . . If message has 1 or 2 diff characters, then theorem easy. 1 . . f [1 . . . n ] be original input frequencies. 2 . . Assume f [1] and f [2] are the two smallest. 3 . . Let f [ n + 1] = f [1] + f [2] . 4 . . lemma = ⇒ ∃ opt. code tree T opt for f [1 .. n ] 5 . . T opt has 1 and 2 as siblings. 6 . . Remove 1 and 2 from T opt . 7 . . T ′ opt : Remaining tree has 3 , . . . , n as leafs and “special” 8 character n + 1 (i.e., parent 1 , 2 in T opt ) Sariel (UIUC) CS573 16 Fall 2013 16 / 22

Proof... . . If message has 1 or 2 diff characters, then theorem easy. 1 . . f [1 . . . n ] be original input frequencies. 2 . . Assume f [1] and f [2] are the two smallest. 3 . . Let f [ n + 1] = f [1] + f [2] . 4 . . lemma = ⇒ ∃ opt. code tree T opt for f [1 .. n ] 5 . . T opt has 1 and 2 as siblings. 6 . . Remove 1 and 2 from T opt . 7 . . T ′ opt : Remaining tree has 3 , . . . , n as leafs and “special” 8 character n + 1 (i.e., parent 1 , 2 in T opt ) Sariel (UIUC) CS573 16 Fall 2013 16 / 22

Proof... . . If message has 1 or 2 diff characters, then theorem easy. 1 . . f [1 . . . n ] be original input frequencies. 2 . . Assume f [1] and f [2] are the two smallest. 3 . . Let f [ n + 1] = f [1] + f [2] . 4 . . lemma = ⇒ ∃ opt. code tree T opt for f [1 .. n ] 5 . . T opt has 1 and 2 as siblings. 6 . . Remove 1 and 2 from T opt . 7 . . T ′ opt : Remaining tree has 3 , . . . , n as leafs and “special” 8 character n + 1 (i.e., parent 1 , 2 in T opt ) Sariel (UIUC) CS573 16 Fall 2013 16 / 22

La proof continued... . . character n + 1 : has frequency f [ n + 1] . 1 Now, f [ n + 1] = f [1] + f [2] , we have n ∑ cost( T opt ) = f [ i ]depth T opt ( i ) i =1 n +1 ∑ = f [ i ]depth T opt ( i ) + f [1]depth T opt (1) i =3 + f [2]depth T opt (2) − f [ n + 1]depth T opt ( n + 1) ( ) ( ) T ′ = cost + f [1] + f [2] depth( T opt ) opt ( ) − f [1] + f [2] (depth( T opt ) − 1) ( ) T ′ = cost + f [1] + f [2] . opt Sariel (UIUC) CS573 17 Fall 2013 17 / 22

La proof continued... . . character n + 1 : has frequency f [ n + 1] . 1 Now, f [ n + 1] = f [1] + f [2] , we have n ∑ cost( T opt ) = f [ i ]depth T opt ( i ) i =1 n +1 ∑ = f [ i ]depth T opt ( i ) + f [1]depth T opt (1) i =3 + f [2]depth T opt (2) − f [ n + 1]depth T opt ( n + 1) ( ) ( ) T ′ = cost + f [1] + f [2] depth( T opt ) opt ( ) − f [1] + f [2] (depth( T opt ) − 1) ( ) T ′ = cost + f [1] + f [2] . opt Sariel (UIUC) CS573 17 Fall 2013 17 / 22

La proof continued... . . character n + 1 : has frequency f [ n + 1] . 1 Now, f [ n + 1] = f [1] + f [2] , we have n ∑ cost( T opt ) = f [ i ]depth T opt ( i ) i =1 n +1 ∑ = f [ i ]depth T opt ( i ) + f [1]depth T opt (1) i =3 + f [2]depth T opt (2) − f [ n + 1]depth T opt ( n + 1) ( ) ( ) T ′ = cost + f [1] + f [2] depth( T opt ) opt ( ) − f [1] + f [2] (depth( T opt ) − 1) ( ) T ′ = cost + f [1] + f [2] . opt Sariel (UIUC) CS573 17 Fall 2013 17 / 22

La proof continued... . . character n + 1 : has frequency f [ n + 1] . 1 Now, f [ n + 1] = f [1] + f [2] , we have n ∑ cost( T opt ) = f [ i ]depth T opt ( i ) i =1 n +1 ∑ = f [ i ]depth T opt ( i ) + f [1]depth T opt (1) i =3 + f [2]depth T opt (2) − f [ n + 1]depth T opt ( n + 1) ( ) ( ) T ′ = cost + f [1] + f [2] depth( T opt ) opt ( ) − f [1] + f [2] (depth( T opt ) − 1) ( ) T ′ = cost + f [1] + f [2] . opt Sariel (UIUC) CS573 17 Fall 2013 17 / 22

La proof continued... . . implies min cost of T opt ≡ min cost T ′ opt . 1 . . T ′ opt : must be optimal coding tree for f [3 . . . n + 1] . 2 . . T ′ H : Huffman tree for f [3 , . . . , n + 1] 3 T H : overall Huffman tree constructed for f [1 , . . . , n ] . . . By construction: T ′ H formed by removing leafs 1 and 2 from 4 T H . . . By induction: Huffman tree generated for f [3 , . . . , n + 1] is 5 optimal. . . ( ) ( ) T ′ T ′ cost = cost . 6 opt H . . ( ) T ′ = ⇒ cost( T H ) = cost + f [1] + f [2] = 7 H ( ) T ′ cost + f [1] + f [2] = cost( T opt ) , opt . . = ⇒ Huffman tree has the same cost as the optimal tree. 8 Sariel (UIUC) CS573 18 Fall 2013 18 / 22

La proof continued... . . implies min cost of T opt ≡ min cost T ′ opt . 1 . . T ′ opt : must be optimal coding tree for f [3 . . . n + 1] . 2 . . T ′ H : Huffman tree for f [3 , . . . , n + 1] 3 T H : overall Huffman tree constructed for f [1 , . . . , n ] . . . By construction: T ′ H formed by removing leafs 1 and 2 from 4 T H . . . By induction: Huffman tree generated for f [3 , . . . , n + 1] is 5 optimal. . . ( ) ( ) T ′ T ′ cost = cost . 6 opt H . . ( ) T ′ = ⇒ cost( T H ) = cost + f [1] + f [2] = 7 H ( ) T ′ cost + f [1] + f [2] = cost( T opt ) , opt . . = ⇒ Huffman tree has the same cost as the optimal tree. 8 Sariel (UIUC) CS573 18 Fall 2013 18 / 22

La proof continued... . . implies min cost of T opt ≡ min cost T ′ opt . 1 . . T ′ opt : must be optimal coding tree for f [3 . . . n + 1] . 2 . . T ′ H : Huffman tree for f [3 , . . . , n + 1] 3 T H : overall Huffman tree constructed for f [1 , . . . , n ] . . . By construction: T ′ H formed by removing leafs 1 and 2 from 4 T H . . . By induction: Huffman tree generated for f [3 , . . . , n + 1] is 5 optimal. . . ( ) ( ) T ′ T ′ cost = cost . 6 opt H . . ( ) T ′ = ⇒ cost( T H ) = cost + f [1] + f [2] = 7 H ( ) T ′ cost + f [1] + f [2] = cost( T opt ) , opt . . = ⇒ Huffman tree has the same cost as the optimal tree. 8 Sariel (UIUC) CS573 18 Fall 2013 18 / 22

La proof continued... . . implies min cost of T opt ≡ min cost T ′ opt . 1 . . T ′ opt : must be optimal coding tree for f [3 . . . n + 1] . 2 . . T ′ H : Huffman tree for f [3 , . . . , n + 1] 3 T H : overall Huffman tree constructed for f [1 , . . . , n ] . . . By construction: T ′ H formed by removing leafs 1 and 2 from 4 T H . . . By induction: Huffman tree generated for f [3 , . . . , n + 1] is 5 optimal. . . ( ) ( ) T ′ T ′ cost = cost . 6 opt H . . ( ) T ′ = ⇒ cost( T H ) = cost + f [1] + f [2] = 7 H ( ) T ′ cost + f [1] + f [2] = cost( T opt ) , opt . . = ⇒ Huffman tree has the same cost as the optimal tree. 8 Sariel (UIUC) CS573 18 Fall 2013 18 / 22

La proof continued... . . implies min cost of T opt ≡ min cost T ′ opt . 1 . . T ′ opt : must be optimal coding tree for f [3 . . . n + 1] . 2 . . T ′ H : Huffman tree for f [3 , . . . , n + 1] 3 T H : overall Huffman tree constructed for f [1 , . . . , n ] . . . By construction: T ′ H formed by removing leafs 1 and 2 from 4 T H . . . By induction: Huffman tree generated for f [3 , . . . , n + 1] is 5 optimal. . . ( ) ( ) T ′ T ′ cost = cost . 6 opt H . . ( ) T ′ = ⇒ cost( T H ) = cost + f [1] + f [2] = 7 H ( ) T ′ cost + f [1] + f [2] = cost( T opt ) , opt . . = ⇒ Huffman tree has the same cost as the optimal tree. 8 Sariel (UIUC) CS573 18 Fall 2013 18 / 22

La proof continued... . . implies min cost of T opt ≡ min cost T ′ opt . 1 . . T ′ opt : must be optimal coding tree for f [3 . . . n + 1] . 2 . . T ′ H : Huffman tree for f [3 , . . . , n + 1] 3 T H : overall Huffman tree constructed for f [1 , . . . , n ] . . . By construction: T ′ H formed by removing leafs 1 and 2 from 4 T H . . . By induction: Huffman tree generated for f [3 , . . . , n + 1] is 5 optimal. . . ( ) ( ) T ′ T ′ cost = cost . 6 opt H . . ( ) T ′ = ⇒ cost( T H ) = cost + f [1] + f [2] = 7 H ( ) T ′ cost + f [1] + f [2] = cost( T opt ) , opt . . = ⇒ Huffman tree has the same cost as the optimal tree. 8 Sariel (UIUC) CS573 18 Fall 2013 18 / 22

La proof continued... . . implies min cost of T opt ≡ min cost T ′ opt . 1 . . T ′ opt : must be optimal coding tree for f [3 . . . n + 1] . 2 . . T ′ H : Huffman tree for f [3 , . . . , n + 1] 3 T H : overall Huffman tree constructed for f [1 , . . . , n ] . . . By construction: T ′ H formed by removing leafs 1 and 2 from 4 T H . . . By induction: Huffman tree generated for f [3 , . . . , n + 1] is 5 optimal. . . ( ) ( ) T ′ T ′ cost = cost . 6 opt H . . ( ) T ′ = ⇒ cost( T H ) = cost + f [1] + f [2] = 7 H ( ) T ′ cost + f [1] + f [2] = cost( T opt ) , opt . . = ⇒ Huffman tree has the same cost as the optimal tree. 8 Sariel (UIUC) CS573 18 Fall 2013 18 / 22

La proof continued... . . implies min cost of T opt ≡ min cost T ′ opt . 1 . . T ′ opt : must be optimal coding tree for f [3 . . . n + 1] . 2 . . T ′ H : Huffman tree for f [3 , . . . , n + 1] 3 T H : overall Huffman tree constructed for f [1 , . . . , n ] . . . By construction: T ′ H formed by removing leafs 1 and 2 from 4 T H . . . By induction: Huffman tree generated for f [3 , . . . , n + 1] is 5 optimal. . . ( ) ( ) T ′ T ′ cost = cost . 6 opt H . . ( ) T ′ = ⇒ cost( T H ) = cost + f [1] + f [2] = 7 H ( ) T ′ cost + f [1] + f [2] = cost( T opt ) , opt . . = ⇒ Huffman tree has the same cost as the optimal tree. 8 Sariel (UIUC) CS573 18 Fall 2013 18 / 22

What we get... . . A tale of two cities: 779,940 bytes. 1 . . using above Huffman compression results in a compression to a 2 file of size 439,688 bytes. . . Ignoring space to store tree. 3 . . gzip : 301,295 bytes 4 bzip2 : 220,156 bytes! . . Huffman encoder can be easily written in a few hours of work! 5 . . All later compressors use it as a black box... 6 Sariel (UIUC) CS573 19 Fall 2013 19 / 22

What we get... . . A tale of two cities: 779,940 bytes. 1 . . using above Huffman compression results in a compression to a 2 file of size 439,688 bytes. . . Ignoring space to store tree. 3 . . gzip : 301,295 bytes 4 bzip2 : 220,156 bytes! . . Huffman encoder can be easily written in a few hours of work! 5 . . All later compressors use it as a black box... 6 Sariel (UIUC) CS573 19 Fall 2013 19 / 22

What we get... . . A tale of two cities: 779,940 bytes. 1 . . using above Huffman compression results in a compression to a 2 file of size 439,688 bytes. . . Ignoring space to store tree. 3 . . gzip : 301,295 bytes 4 bzip2 : 220,156 bytes! . . Huffman encoder can be easily written in a few hours of work! 5 . . All later compressors use it as a black box... 6 Sariel (UIUC) CS573 19 Fall 2013 19 / 22

What we get... . . A tale of two cities: 779,940 bytes. 1 . . using above Huffman compression results in a compression to a 2 file of size 439,688 bytes. . . Ignoring space to store tree. 3 . . gzip : 301,295 bytes 4 bzip2 : 220,156 bytes! . . Huffman encoder can be easily written in a few hours of work! 5 . . All later compressors use it as a black box... 6 Sariel (UIUC) CS573 19 Fall 2013 19 / 22

What we get... . . A tale of two cities: 779,940 bytes. 1 . . using above Huffman compression results in a compression to a 2 file of size 439,688 bytes. . . Ignoring space to store tree. 3 . . gzip : 301,295 bytes 4 bzip2 : 220,156 bytes! . . Huffman encoder can be easily written in a few hours of work! 5 . . All later compressors use it as a black box... 6 Sariel (UIUC) CS573 19 Fall 2013 19 / 22

What we get... . . A tale of two cities: 779,940 bytes. 1 . . using above Huffman compression results in a compression to a 2 file of size 439,688 bytes. . . Ignoring space to store tree. 3 . . gzip : 301,295 bytes 4 bzip2 : 220,156 bytes! . . Huffman encoder can be easily written in a few hours of work! 5 . . All later compressors use it as a black box... 6 Sariel (UIUC) CS573 19 Fall 2013 19 / 22

Average size of code word . . input is made out of n characters. 1 . . p i : fraction of input that is i th char (probability). 2 . . use probabilities to build Huffman tree. 3 . . Q: What is the length of the codewords assigned to characters 4 as function of probabilities? . . special case... 5 . Lemma . Let 1 , . . . , n be n symbols, such that the probability for the i th symbol is p i , and furthermore, there is an integer l i ≥ 0 , such that p i = 1 / 2 l i . Then, in the Huffman coding for this input, the code for i is of length l i . . Sariel (UIUC) CS573 20 Fall 2013 20 / 22

Average size of code word . . input is made out of n characters. 1 . . p i : fraction of input that is i th char (probability). 2 . . use probabilities to build Huffman tree. 3 . . Q: What is the length of the codewords assigned to characters 4 as function of probabilities? . . special case... 5 . Lemma . Let 1 , . . . , n be n symbols, such that the probability for the i th symbol is p i , and furthermore, there is an integer l i ≥ 0 , such that p i = 1 / 2 l i . Then, in the Huffman coding for this input, the code for i is of length l i . . Sariel (UIUC) CS573 20 Fall 2013 20 / 22

Average size of code word . . input is made out of n characters. 1 . . p i : fraction of input that is i th char (probability). 2 . . use probabilities to build Huffman tree. 3 . . Q: What is the length of the codewords assigned to characters 4 as function of probabilities? . . special case... 5 . Lemma . Let 1 , . . . , n be n symbols, such that the probability for the i th symbol is p i , and furthermore, there is an integer l i ≥ 0 , such that p i = 1 / 2 l i . Then, in the Huffman coding for this input, the code for i is of length l i . . Sariel (UIUC) CS573 20 Fall 2013 20 / 22

Average size of code word . . input is made out of n characters. 1 . . p i : fraction of input that is i th char (probability). 2 . . use probabilities to build Huffman tree. 3 . . Q: What is the length of the codewords assigned to characters 4 as function of probabilities? . . special case... 5 . Lemma . Let 1 , . . . , n be n symbols, such that the probability for the i th symbol is p i , and furthermore, there is an integer l i ≥ 0 , such that p i = 1 / 2 l i . Then, in the Huffman coding for this input, the code for i is of length l i . . Sariel (UIUC) CS573 20 Fall 2013 20 / 22

Average size of code word . . input is made out of n characters. 1 . . p i : fraction of input that is i th char (probability). 2 . . use probabilities to build Huffman tree. 3 . . Q: What is the length of the codewords assigned to characters 4 as function of probabilities? . . special case... 5 . Lemma . Let 1 , . . . , n be n symbols, such that the probability for the i th symbol is p i , and furthermore, there is an integer l i ≥ 0 , such that p i = 1 / 2 l i . Then, in the Huffman coding for this input, the code for i is of length l i . . Sariel (UIUC) CS573 20 Fall 2013 20 / 22

Proof . . induction of the Huffman algorithm. 1 . . n = 2 : claim holds since there are only two characters with 2 probability 1 / 2 . . . Let i and j be the two characters with lowest probability. 3 . . Must be that p i = p j (otherwise, ∑ k p k can not be equal to 4 one). . . Huffman’s tree merges this two letters, into a single “character” 5 that have probability 2 p i . . . New “character” has encoding of length l i − 1 , by induction 6 (on remaining n − 1 symbols). . . resulting tree encodes i and j by code words of length 7 ( l i − 1) + 1 = l i . Sariel (UIUC) CS573 21 Fall 2013 21 / 22

Proof . . induction of the Huffman algorithm. 1 . . n = 2 : claim holds since there are only two characters with 2 probability 1 / 2 . . . Let i and j be the two characters with lowest probability. 3 . . Must be that p i = p j (otherwise, ∑ k p k can not be equal to 4 one). . . Huffman’s tree merges this two letters, into a single “character” 5 that have probability 2 p i . . . New “character” has encoding of length l i − 1 , by induction 6 (on remaining n − 1 symbols). . . resulting tree encodes i and j by code words of length 7 ( l i − 1) + 1 = l i . Sariel (UIUC) CS573 21 Fall 2013 21 / 22

Proof . . induction of the Huffman algorithm. 1 . . n = 2 : claim holds since there are only two characters with 2 probability 1 / 2 . . . Let i and j be the two characters with lowest probability. 3 . . Must be that p i = p j (otherwise, ∑ k p k can not be equal to 4 one). . . Huffman’s tree merges this two letters, into a single “character” 5 that have probability 2 p i . . . New “character” has encoding of length l i − 1 , by induction 6 (on remaining n − 1 symbols). . . resulting tree encodes i and j by code words of length 7 ( l i − 1) + 1 = l i . Sariel (UIUC) CS573 21 Fall 2013 21 / 22

Proof . . induction of the Huffman algorithm. 1 . . n = 2 : claim holds since there are only two characters with 2 probability 1 / 2 . . . Let i and j be the two characters with lowest probability. 3 . . Must be that p i = p j (otherwise, ∑ k p k can not be equal to 4 one). . . Huffman’s tree merges this two letters, into a single “character” 5 that have probability 2 p i . . . New “character” has encoding of length l i − 1 , by induction 6 (on remaining n − 1 symbols). . . resulting tree encodes i and j by code words of length 7 ( l i − 1) + 1 = l i . Sariel (UIUC) CS573 21 Fall 2013 21 / 22

Proof . . induction of the Huffman algorithm. 1 . . n = 2 : claim holds since there are only two characters with 2 probability 1 / 2 . . . Let i and j be the two characters with lowest probability. 3 . . Must be that p i = p j (otherwise, ∑ k p k can not be equal to 4 one). . . Huffman’s tree merges this two letters, into a single “character” 5 that have probability 2 p i . . . New “character” has encoding of length l i − 1 , by induction 6 (on remaining n − 1 symbols). . . resulting tree encodes i and j by code words of length 7 ( l i − 1) + 1 = l i . Sariel (UIUC) CS573 21 Fall 2013 21 / 22

Proof . . induction of the Huffman algorithm. 1 . . n = 2 : claim holds since there are only two characters with 2 probability 1 / 2 . . . Let i and j be the two characters with lowest probability. 3 . . Must be that p i = p j (otherwise, ∑ k p k can not be equal to 4 one). . . Huffman’s tree merges this two letters, into a single “character” 5 that have probability 2 p i . . . New “character” has encoding of length l i − 1 , by induction 6 (on remaining n − 1 symbols). . . resulting tree encodes i and j by code words of length 7 ( l i − 1) + 1 = l i . Sariel (UIUC) CS573 21 Fall 2013 21 / 22

Proof . . induction of the Huffman algorithm. 1 . . n = 2 : claim holds since there are only two characters with 2 probability 1 / 2 . . . Let i and j be the two characters with lowest probability. 3 . . Must be that p i = p j (otherwise, ∑ k p k can not be equal to 4 one). . . Huffman’s tree merges this two letters, into a single “character” 5 that have probability 2 p i . . . New “character” has encoding of length l i − 1 , by induction 6 (on remaining n − 1 symbols). . . resulting tree encodes i and j by code words of length 7 ( l i − 1) + 1 = l i . Sariel (UIUC) CS573 21 Fall 2013 21 / 22

Translating lemma... . . p i = 1 / 2 l i 1 . . l i = lg 1 / p i . 2 . . Average length of a code word is 3 p i lg 1 ∑ . p i i . . X is a random variable that takes a value i with probability p i , 4 then this formula is 1 ∑ H ( X ) = Pr[ X = i ] lg Pr[ X = i ] , i which is the entropy of X . Sariel (UIUC) CS573 22 Fall 2013 22 / 22

Translating lemma... . . p i = 1 / 2 l i 1 . . l i = lg 1 / p i . 2 . . Average length of a code word is 3 p i lg 1 ∑ . p i i . . X is a random variable that takes a value i with probability p i , 4 then this formula is 1 ∑ H ( X ) = Pr[ X = i ] lg Pr[ X = i ] , i which is the entropy of X . Sariel (UIUC) CS573 22 Fall 2013 22 / 22

Translating lemma... . . p i = 1 / 2 l i 1 . . l i = lg 1 / p i . 2 . . Average length of a code word is 3 p i lg 1 ∑ . p i i . . X is a random variable that takes a value i with probability p i , 4 then this formula is 1 ∑ H ( X ) = Pr[ X = i ] lg Pr[ X = i ] , i which is the entropy of X . Sariel (UIUC) CS573 22 Fall 2013 22 / 22

Translating lemma... . . p i = 1 / 2 l i 1 . . l i = lg 1 / p i . 2 . . Average length of a code word is 3 p i lg 1 ∑ . p i i . . X is a random variable that takes a value i with probability p i , 4 then this formula is 1 ∑ H ( X ) = Pr[ X = i ] lg Pr[ X = i ] , i which is the entropy of X . Sariel (UIUC) CS573 22 Fall 2013 22 / 22