basic definitions and facts
play

Basic Definitions and Facts Iftach Haitner Tel Aviv University. - PowerPoint PPT Presentation

Nov 19, 2023 •161 likes •1.34k views

Application of Information Theory, Lecture 1 Basic Definitions and Facts Iftach Haitner Tel Aviv University. October 28, 2014 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 1 / 12 The entropy function X

  1. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  2. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  3. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  4. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 ◮ H ( X ) = H ( p , q ) = − p log p − q log q Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  5. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 ◮ H ( X ) = H ( p , q ) = − p log p − q log q ◮ H ( 1 , 0 ) = ( 0 , 1 ) = 0 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  6. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 ◮ H ( X ) = H ( p , q ) = − p log p − q log q ◮ H ( 1 , 0 ) = ( 0 , 1 ) = 0 ◮ H ( 1 2 , 1 2 ) = 1 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  7. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 ◮ H ( X ) = H ( p , q ) = − p log p − q log q ◮ H ( 1 , 0 ) = ( 0 , 1 ) = 0 ◮ H ( 1 2 , 1 2 ) = 1 ◮ h ( p ) := H ( p , 1 − p ) is continuous Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  8. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 ◮ H ( X ) = H ( p , q ) = − p log p − q log q ◮ H ( 1 , 0 ) = ( 0 , 1 ) = 0 ◮ H ( 1 2 , 1 2 ) = 1 ◮ h ( p ) := H ( p , 1 − p ) is continuous Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  9. Examples 1. X ∼ ( 1 2 , 1 4 , 1 4 ) : (i.e., for some x 1 � = x 2 � = x 3 , P X ( x 1 ) = 1 2 , P X ( x 2 ) = 1 4 , P X ( x 3 ) = 1 4 ) H ( X ) = − 1 2 log 1 2 − 1 4 log 1 4 − 1 4 log 1 4 = 1 2 + 1 4 · 2 + 1 4 · 2 = 1 1 2 . 2. H ( X ) = H ( 1 2 , 1 4 , 1 4 ) . 3. X is uniformly distributed over { 0 , 1 } n : H ( X ) = − � 2 n 2 n log 1 1 2 n = − log 1 2 n = n . i = 1 ◮ n bits are needed to describe X ◮ n bits are needed to create X 4. X = X 1 , . . . , X n where X i ’s iid over { 0 , 1 } , with P X i ( 1 ) = 1 3 . H ( X ) =? 5. X ∼ ( p , q ) , p + q = 1 ◮ H ( X ) = H ( p , q ) = − p log p − q log q ◮ H ( 1 , 0 ) = ( 0 , 1 ) = 0 ◮ H ( 1 2 , 1 2 ) = 1 ◮ h ( p ) := H ( p , 1 − p ) is continuous Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 3 / 12

  10. Axiomatic derivation of the entropy function Any other choices for defining entropy? Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 4 / 12

  11. Axiomatic derivation of the entropy function Any other choices for defining entropy? Shannon function is the only symmetric function (over probability distributions) satisfying the following three axioms: A1 Continuity: H ( p , 1 − p ) is continuous function of p . A2 Normalization: H ( 1 2 , 1 2 ) = 1 A3 Grouping axiom: p 1 p 2 H ( p 1 , p 2 , . . . , p m ) = H ( p 1 + p 2 , p 3 , . . . , p m ) + ( p 1 + p 2 ) H ( p 1 + p 2 , p 1 + p 2 ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 4 / 12

  12. Axiomatic derivation of the entropy function Any other choices for defining entropy? Shannon function is the only symmetric function (over probability distributions) satisfying the following three axioms: A1 Continuity: H ( p , 1 − p ) is continuous function of p . A2 Normalization: H ( 1 2 , 1 2 ) = 1 A3 Grouping axiom: p 1 p 2 H ( p 1 , p 2 , . . . , p m ) = H ( p 1 + p 2 , p 3 , . . . , p m ) + ( p 1 + p 2 ) H ( p 1 + p 2 , p 1 + p 2 ) Why A 3? Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 4 / 12

  13. Axiomatic derivation of the entropy function Any other choices for defining entropy? Shannon function is the only symmetric function (over probability distributions) satisfying the following three axioms: A1 Continuity: H ( p , 1 − p ) is continuous function of p . A2 Normalization: H ( 1 2 , 1 2 ) = 1 A3 Grouping axiom: p 1 p 2 H ( p 1 , p 2 , . . . , p m ) = H ( p 1 + p 2 , p 3 , . . . , p m ) + ( p 1 + p 2 ) H ( p 1 + p 2 , p 1 + p 2 ) Why A 3? Not hard to prove that Shannon’s entropy function satisfies above axioms, proving this is the only such function is more challenging. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 4 / 12

  14. Axiomatic derivation of the entropy function Any other choices for defining entropy? Shannon function is the only symmetric function (over probability distributions) satisfying the following three axioms: A1 Continuity: H ( p , 1 − p ) is continuous function of p . A2 Normalization: H ( 1 2 , 1 2 ) = 1 A3 Grouping axiom: p 1 p 2 H ( p 1 , p 2 , . . . , p m ) = H ( p 1 + p 2 , p 3 , . . . , p m ) + ( p 1 + p 2 ) H ( p 1 + p 2 , p 1 + p 2 ) Why A 3? Not hard to prove that Shannon’s entropy function satisfies above axioms, proving this is the only such function is more challenging. Let H be a symmetric function that satisfying the above axioms. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 4 / 12

  15. Axiomatic derivation of the entropy function Any other choices for defining entropy? Shannon function is the only symmetric function (over probability distributions) satisfying the following three axioms: A1 Continuity: H ( p , 1 − p ) is continuous function of p . A2 Normalization: H ( 1 2 , 1 2 ) = 1 A3 Grouping axiom: p 1 p 2 H ( p 1 , p 2 , . . . , p m ) = H ( p 1 + p 2 , p 3 , . . . , p m ) + ( p 1 + p 2 ) H ( p 1 + p 2 , p 1 + p 2 ) Why A 3? Not hard to prove that Shannon’s entropy function satisfies above axioms, proving this is the only such function is more challenging. Let H be a symmetric function that satisfying the above axioms. We prove (assuming additional axiom) that H is the Shannon function. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 4 / 12

  16. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  17. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  18. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  19. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Proof : Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  20. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Proof : Let h ( q ) = H ( q , 1 − q ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  21. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Proof : Let h ( q ) = H ( q , 1 − q ) . H ( S 2 , p 3 , . . . , p m ) + S 2 h ( p 2 H ( p 1 , p 2 , . . . , p m ) = ) (1) S 2 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  22. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Proof : Let h ( q ) = H ( q , 1 − q ) . H ( S 2 , p 3 , . . . , p m ) + S 2 h ( p 2 H ( p 1 , p 2 , . . . , p m ) = ) (1) S 2 H ( S 3 , p 4 , . . . , p m ) + S 3 h ( p 3 ) + S 2 h ( p 2 = ) S 3 S 2 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  23. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Proof : Let h ( q ) = H ( q , 1 − q ) . H ( S 2 , p 3 , . . . , p m ) + S 2 h ( p 2 H ( p 1 , p 2 , . . . , p m ) = ) (1) S 2 H ( S 3 , p 4 , . . . , p m ) + S 3 h ( p 3 ) + S 2 h ( p 2 = ) S 3 S 2 . . . k S i h ( p i � = H ( S k , p k + 1 , . . . , p m ) + ) S i i = 2 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  24. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Proof : Let h ( q ) = H ( q , 1 − q ) . H ( S 2 , p 3 , . . . , p m ) + S 2 h ( p 2 H ( p 1 , p 2 , . . . , p m ) = ) (1) S 2 H ( S 3 , p 4 , . . . , p m ) + S 3 h ( p 3 ) + S 2 h ( p 2 = ) S 3 S 2 . . . k S i h ( p i � = H ( S k , p k + 1 , . . . , p m ) + ) S i i = 2 Hence, k − 1 H ( p 1 , . . . , p k ) = H ( S k − 1 , p k S i h ( p i / S k � ) + ) S k S k S k S k S k S i / S k i = 2 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  25. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Proof : Let h ( q ) = H ( q , 1 − q ) . H ( S 2 , p 3 , . . . , p m ) + S 2 h ( p 2 H ( p 1 , p 2 , . . . , p m ) = ) (1) S 2 H ( S 3 , p 4 , . . . , p m ) + S 3 h ( p 3 ) + S 2 h ( p 2 = ) S 3 S 2 . . . k S i h ( p i � = H ( S k , p k + 1 , . . . , p m ) + ) S i i = 2 Hence, k − 1 k H ( p 1 , . . . , p k ) = H ( S k − 1 , p k S i h ( p i / S k ) = 1 S i h ( p i � � ) + ) (2) S k S k S k S k S k S i / S k S k S i i = 2 i = 2 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  26. Generalization of the grouping axiom Fix p = ( p 1 , . . . , p m ) and let S k = � k i = 1 p i . Grouping axiom: H ( p 1 , p 2 , . . . , p m ) = H ( S 2 , p 3 , . . . , p m ) + S 2 H ( p 1 S 2 , p 2 S 2 ) . Claim 1 (Generalized grouping axiom) H ( p 1 , p 2 , . . . , p m ) = H ( S k , p k + 1 , . . . , p m ) + S k · H ( p 1 S k , . . . , p k S k ) Proof : Let h ( q ) = H ( q , 1 − q ) . H ( S 2 , p 3 , . . . , p m ) + S 2 h ( p 2 H ( p 1 , p 2 , . . . , p m ) = ) (1) S 2 H ( S 3 , p 4 , . . . , p m ) + S 3 h ( p 3 ) + S 2 h ( p 2 = ) S 3 S 2 . . . k S i h ( p i � = H ( S k , p k + 1 , . . . , p m ) + ) S i i = 2 Hence, k − 1 k H ( p 1 , . . . , p k ) = H ( S k − 1 , p k S i h ( p i / S k ) = 1 S i h ( p i � � ) + ) (2) S k S k S k S k S k S i / S k S k S i i = 2 i = 2 Claim follows by combining the above equations. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 5 / 12

  27. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  28. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  29. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Follow by the extended group axiom and the symmetry of H Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  30. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Follow by the extended group axiom and the symmetry of H Implication: Let f ( m ) = H ( 1 m , . . . , 1 ) m � �� � m Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  31. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Follow by the extended group axiom and the symmetry of H Implication: Let f ( m ) = H ( 1 m , . . . , 1 ) m � �� � m ◮ f ( 3 2 ) = Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  32. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Follow by the extended group axiom and the symmetry of H Implication: Let f ( m ) = H ( 1 m , . . . , 1 ) m � �� � m ◮ f ( 3 2 ) = Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  33. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Follow by the extended group axiom and the symmetry of H Implication: Let f ( m ) = H ( 1 m , . . . , 1 ) m � �� � m ◮ f ( 3 2 ) = 2 f ( 3 ) = 2 H ( 1 3 , 1 3 , 1 3 ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  34. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Follow by the extended group axiom and the symmetry of H Implication: Let f ( m ) = H ( 1 m , . . . , 1 ) m � �� � m ◮ f ( 3 2 ) = 2 f ( 3 ) = 2 H ( 1 3 , 1 3 , 1 3 ) ⇒ f ( 3 n ) = nf ( 3 ) . = Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  35. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Follow by the extended group axiom and the symmetry of H Implication: Let f ( m ) = H ( 1 m , . . . , 1 ) m � �� � m ◮ f ( 3 2 ) = 2 f ( 3 ) = 2 H ( 1 3 , 1 3 , 1 3 ) ⇒ f ( 3 n ) = nf ( 3 ) . = ◮ f ( mn ) = f ( m ) + f ( n ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  36. Further generalization of the grouping axiom Let 1 = k 1 < k 2 < . . . < k q < m and let C t = � k t + 1 − 1 p i (letting k q + 1 = m + 1). i = k t Claim 2 (Generalized ++ grouping axiom) H ( p 1 , p 2 , . . . , p m ) = p k 2 − 1 p kq + 1 H ( C 1 , . . . , C q ) + C 1 · H ( p 1 C q , . . . , p m C 1 , . . . , C 1 ) + . . . + C q · H ( C q ) Proof : Follow by the extended group axiom and the symmetry of H Implication: Let f ( m ) = H ( 1 m , . . . , 1 ) m � �� � m ◮ f ( 3 2 ) = 2 f ( 3 ) = 2 H ( 1 3 , 1 3 , 1 3 ) ⇒ f ( 3 n ) = nf ( 3 ) . = ◮ f ( mn ) = f ( m ) + f ( n ) ⇒ f ( m k ) = kf ( m ) = Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 6 / 12

  37. f ( m ) = log m Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  38. f ( m ) = log m We give a proof under the additional axiom A4 f ( m ) < f ( m + 1 ) (you can Google for a proof using only A 1– A 3) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  39. f ( m ) = log m We give a proof under the additional axiom A4 f ( m ) < f ( m + 1 ) (you can Google for a proof using only A 1– A 3) ◮ For n ∈ N let k = ⌊ n log 3 ⌋ . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  40. f ( m ) = log m We give a proof under the additional axiom A4 f ( m ) < f ( m + 1 ) (you can Google for a proof using only A 1– A 3) ◮ For n ∈ N let k = ⌊ n log 3 ⌋ . ◮ By A 4, f ( 2 k ) < f ( 3 n ) < f ( 2 k + 1 ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  41. f ( m ) = log m We give a proof under the additional axiom A4 f ( m ) < f ( m + 1 ) (you can Google for a proof using only A 1– A 3) ◮ For n ∈ N let k = ⌊ n log 3 ⌋ . ◮ By A 4, f ( 2 k ) < f ( 3 n ) < f ( 2 k + 1 ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  42. f ( m ) = log m We give a proof under the additional axiom A4 f ( m ) < f ( m + 1 ) (you can Google for a proof using only A 1– A 3) ◮ For n ∈ N let k = ⌊ n log 3 ⌋ . ◮ By A 4, f ( 2 k ) < f ( 3 n ) < f ( 2 k + 1 ) . ◮ By grouping axiom, k < nf ( 3 ) < k + 1. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  43. f ( m ) = log m We give a proof under the additional axiom A4 f ( m ) < f ( m + 1 ) (you can Google for a proof using only A 1– A 3) ◮ For n ∈ N let k = ⌊ n log 3 ⌋ . ◮ By A 4, f ( 2 k ) < f ( 3 n ) < f ( 2 k + 1 ) . ◮ By grouping axiom, k < nf ( 3 ) < k + 1. ⌊ n log 3 ⌋ < f ( 3 ) < ⌊ n log 3 ⌋ + 1 = ⇒ for any n ∈ N n n Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  44. f ( m ) = log m We give a proof under the additional axiom A4 f ( m ) < f ( m + 1 ) (you can Google for a proof using only A 1– A 3) ◮ For n ∈ N let k = ⌊ n log 3 ⌋ . ◮ By A 4, f ( 2 k ) < f ( 3 n ) < f ( 2 k + 1 ) . ◮ By grouping axiom, k < nf ( 3 ) < k + 1. ⌊ n log 3 ⌋ < f ( 3 ) < ⌊ n log 3 ⌋ + 1 = ⇒ for any n ∈ N n n = ⇒ f ( 3 ) = log 3. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  45. f ( m ) = log m We give a proof under the additional axiom A4 f ( m ) < f ( m + 1 ) (you can Google for a proof using only A 1– A 3) ◮ For n ∈ N let k = ⌊ n log 3 ⌋ . ◮ By A 4, f ( 2 k ) < f ( 3 n ) < f ( 2 k + 1 ) . ◮ By grouping axiom, k < nf ( 3 ) < k + 1. ⌊ n log 3 ⌋ < f ( 3 ) < ⌊ n log 3 ⌋ + 1 = ⇒ for any n ∈ N n n = ⇒ f ( 3 ) = log 3. ◮ Proof extends to any integer (not only 3) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 7 / 12

  46. H ( p , q ) = − p log p − q log q Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  47. H ( p , q ) = − p log p − q log q ◮ For rational p , q , let p = k m and q = m − k m , where m is the smallest common multiplier. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  48. H ( p , q ) = − p log p − q log q ◮ For rational p , q , let p = k m and q = m − k m , where m is the smallest common multiplier. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  49. H ( p , q ) = − p log p − q log q ◮ For rational p , q , let p = k m and q = m − k m , where m is the smallest common multiplier. ◮ By grouping axiom, f ( m ) = H ( p , q ) + p · f ( k ) + q · f ( m − k ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  50. H ( p , q ) = − p log p − q log q ◮ For rational p , q , let p = k m and q = m − k m , where m is the smallest common multiplier. ◮ By grouping axiom, f ( m ) = H ( p , q ) + p · f ( k ) + q · f ( m − k ) . ◮ Hence, H ( p , q ) = log m − p log k − q log ( m − k ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  51. H ( p , q ) = − p log p − q log q ◮ For rational p , q , let p = k m and q = m − k m , where m is the smallest common multiplier. ◮ By grouping axiom, f ( m ) = H ( p , q ) + p · f ( k ) + q · f ( m − k ) . ◮ Hence, H ( p , q ) = log m − p log k − q log ( m − k ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  52. H ( p , q ) = − p log p − q log q ◮ For rational p , q , let p = k m and q = m − k m , where m is the smallest common multiplier. ◮ By grouping axiom, f ( m ) = H ( p , q ) + p · f ( k ) + q · f ( m − k ) . ◮ Hence, H ( p , q ) = log m − p log k − q log ( m − k ) = p ( logm − logk ) + q ( log m − log ( m − k )) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  53. H ( p , q ) = − p log p − q log q ◮ For rational p , q , let p = k m and q = m − k m , where m is the smallest common multiplier. ◮ By grouping axiom, f ( m ) = H ( p , q ) + p · f ( k ) + q · f ( m − k ) . ◮ Hence, H ( p , q ) = log m − p log k − q log ( m − k ) = p ( logm − logk ) + q ( log m − log ( m − k )) − p log m k − q log m − k = = − p log p − q log q m Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  54. H ( p , q ) = − p log p − q log q ◮ For rational p , q , let p = k m and q = m − k m , where m is the smallest common multiplier. ◮ By grouping axiom, f ( m ) = H ( p , q ) + p · f ( k ) + q · f ( m − k ) . ◮ Hence, H ( p , q ) = log m − p log k − q log ( m − k ) = p ( logm − logk ) + q ( log m − log ( m − k )) − p log m k − q log m − k = = − p log p − q log q m ◮ By continuity axiom, holds for every p , q . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 8 / 12

  55. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  56. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  57. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. ◮ For rational p 1 , p 2 , p 3 , let p 1 = k 1 m , q = k 2 m and p 3 = k 3 m , where m = k 1 + k 2 + k 3 is the smallest common multiplier. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  58. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. ◮ For rational p 1 , p 2 , p 3 , let p 1 = k 1 m , q = k 2 m and p 3 = k 3 m , where m = k 1 + k 2 + k 3 is the smallest common multiplier. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  59. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. ◮ For rational p 1 , p 2 , p 3 , let p 1 = k 1 m , q = k 2 m and p 3 = k 3 m , where m = k 1 + k 2 + k 3 is the smallest common multiplier. ◮ f ( m ) = H ( p 1 , p 2 , p 3 ) + p 1 f ( k 1 ) + p 2 f ( k 2 ) + p 3 f ( k 3 ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  60. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. ◮ For rational p 1 , p 2 , p 3 , let p 1 = k 1 m , q = k 2 m and p 3 = k 3 m , where m = k 1 + k 2 + k 3 is the smallest common multiplier. ◮ f ( m ) = H ( p 1 , p 2 , p 3 ) + p 1 f ( k 1 ) + p 2 f ( k 2 ) + p 3 f ( k 3 ) ◮ Hence, H ( p 1 , p 2 , p 3 ) = log m − p 1 log k 1 − p 2 log k 2 − p 3 log k 3 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  61. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. ◮ For rational p 1 , p 2 , p 3 , let p 1 = k 1 m , q = k 2 m and p 3 = k 3 m , where m = k 1 + k 2 + k 3 is the smallest common multiplier. ◮ f ( m ) = H ( p 1 , p 2 , p 3 ) + p 1 f ( k 1 ) + p 2 f ( k 2 ) + p 3 f ( k 3 ) ◮ Hence, H ( p 1 , p 2 , p 3 ) = log m − p 1 log k 1 − p 2 log k 2 − p 3 log k 3 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  62. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. ◮ For rational p 1 , p 2 , p 3 , let p 1 = k 1 m , q = k 2 m and p 3 = k 3 m , where m = k 1 + k 2 + k 3 is the smallest common multiplier. ◮ f ( m ) = H ( p 1 , p 2 , p 3 ) + p 1 f ( k 1 ) + p 2 f ( k 2 ) + p 3 f ( k 3 ) ◮ Hence, H ( p 1 , p 2 , p 3 ) = log m − p 1 log k 1 − p 2 log k 2 − p 3 log k 3 − p 1 log k 1 m − p 2 log k 2 k 3 = m − p 3 m Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  63. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. ◮ For rational p 1 , p 2 , p 3 , let p 1 = k 1 m , q = k 2 m and p 3 = k 3 m , where m = k 1 + k 2 + k 3 is the smallest common multiplier. ◮ f ( m ) = H ( p 1 , p 2 , p 3 ) + p 1 f ( k 1 ) + p 2 f ( k 2 ) + p 3 f ( k 3 ) ◮ Hence, H ( p 1 , p 2 , p 3 ) = log m − p 1 log k 1 − p 2 log k 2 − p 3 log k 3 − p 1 log k 1 m − p 2 log k 2 k 3 = m − p 3 m = − p 1 log p 1 − p 2 log p 2 − p 3 log p 3 Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  64. H ( p 1 , p 2 , . . . , p m ) = − � m i − p i log p i We prove for m = 3. Proof for arbitrary m follows the same lines. ◮ For rational p 1 , p 2 , p 3 , let p 1 = k 1 m , q = k 2 m and p 3 = k 3 m , where m = k 1 + k 2 + k 3 is the smallest common multiplier. ◮ f ( m ) = H ( p 1 , p 2 , p 3 ) + p 1 f ( k 1 ) + p 2 f ( k 2 ) + p 3 f ( k 3 ) ◮ Hence, H ( p 1 , p 2 , p 3 ) = log m − p 1 log k 1 − p 2 log k 2 − p 3 log k 3 − p 1 log k 1 m − p 2 log k 2 k 3 = m − p 3 m = − p 1 log p 1 − p 2 log p 2 − p 3 log p 3 ◮ By continuity axiom, holds for every p 1 , p 2 , p 3 . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 9 / 12

  65. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  66. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  67. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  68. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . ◮ H ( p 1 , . . . , p m ) = log m for ( p 1 , . . . , p m ) = ( 1 m , . . . , 1 m ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  69. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . ◮ H ( p 1 , . . . , p m ) = log m for ( p 1 , . . . , p m ) = ( 1 m , . . . , 1 m ) . ◮ Non negativity is clear. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  70. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . ◮ H ( p 1 , . . . , p m ) = log m for ( p 1 , . . . , p m ) = ( 1 m , . . . , 1 m ) . ◮ Non negativity is clear. ◮ A function f is concave if ∀ t 1 , t 2 , λ ∈ [ 0 , 1 ] ≤ 1 λ f ( t 1 ) + ( 1 − λ ) f ( t 2 ) ≤ f ( λ t 1 + ( 1 − λ ) t 2 ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  71. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . ◮ H ( p 1 , . . . , p m ) = log m for ( p 1 , . . . , p m ) = ( 1 m , . . . , 1 m ) . ◮ Non negativity is clear. ◮ A function f is concave if ∀ t 1 , t 2 , λ ∈ [ 0 , 1 ] ≤ 1 λ f ( t 1 ) + ( 1 − λ ) f ( t 2 ) ≤ f ( λ t 1 + ( 1 − λ ) t 2 ) (by induction) ∀ t 1 , . . . , t k , λ 1 , . . . , λ k ∈ [ 0 , 1 ] with � = ⇒ i λ i = 1 � i λ i f ( λ i t i ) ≤ f ( � i λ i t i ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  72. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . ◮ H ( p 1 , . . . , p m ) = log m for ( p 1 , . . . , p m ) = ( 1 m , . . . , 1 m ) . ◮ Non negativity is clear. ◮ A function f is concave if ∀ t 1 , t 2 , λ ∈ [ 0 , 1 ] ≤ 1 λ f ( t 1 ) + ( 1 − λ ) f ( t 2 ) ≤ f ( λ t 1 + ( 1 − λ ) t 2 ) (by induction) ∀ t 1 , . . . , t k , λ 1 , . . . , λ k ∈ [ 0 , 1 ] with � = ⇒ i λ i = 1 � i λ i f ( λ i t i ) ≤ f ( � i λ i t i ) = ⇒ (Jensen inequality): E f ( X ) ≤ f ( E X ) for any random variable X . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  73. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . ◮ H ( p 1 , . . . , p m ) = log m for ( p 1 , . . . , p m ) = ( 1 m , . . . , 1 m ) . ◮ Non negativity is clear. ◮ A function f is concave if ∀ t 1 , t 2 , λ ∈ [ 0 , 1 ] ≤ 1 λ f ( t 1 ) + ( 1 − λ ) f ( t 2 ) ≤ f ( λ t 1 + ( 1 − λ ) t 2 ) (by induction) ∀ t 1 , . . . , t k , λ 1 , . . . , λ k ∈ [ 0 , 1 ] with � = ⇒ i λ i = 1 � i λ i f ( λ i t i ) ≤ f ( � i λ i t i ) = ⇒ (Jensen inequality): E f ( X ) ≤ f ( E X ) for any random variable X . ◮ log ( x ) is (strictly) concave for x > 0, since its second derivative ( − 1 x 2 ) is always negative. Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  74. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . ◮ H ( p 1 , . . . , p m ) = log m for ( p 1 , . . . , p m ) = ( 1 m , . . . , 1 m ) . ◮ Non negativity is clear. ◮ A function f is concave if ∀ t 1 , t 2 , λ ∈ [ 0 , 1 ] ≤ 1 λ f ( t 1 ) + ( 1 − λ ) f ( t 2 ) ≤ f ( λ t 1 + ( 1 − λ ) t 2 ) (by induction) ∀ t 1 , . . . , t k , λ 1 , . . . , λ k ∈ [ 0 , 1 ] with � = ⇒ i λ i = 1 � i λ i f ( λ i t i ) ≤ f ( � i λ i t i ) = ⇒ (Jensen inequality): E f ( X ) ≤ f ( E X ) for any random variable X . ◮ log ( x ) is (strictly) concave for x > 0, since its second derivative ( − 1 x 2 ) is always negative. ◮ Hence, H ( p 1 , . . . , p m ) = � p i ≤ log � i p i log 1 i p i 1 p i = log m Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  75. 0 ≤ H ( p 1 , . . . , p m ) ≤ log m ◮ Tight bounds ◮ H ( p 1 , . . . , p m ) = 0 for ( p 1 , . . . , p m ) = ( 1 , 0 , . . . , 0 ) . ◮ H ( p 1 , . . . , p m ) = log m for ( p 1 , . . . , p m ) = ( 1 m , . . . , 1 m ) . ◮ Non negativity is clear. ◮ A function f is concave if ∀ t 1 , t 2 , λ ∈ [ 0 , 1 ] ≤ 1 λ f ( t 1 ) + ( 1 − λ ) f ( t 2 ) ≤ f ( λ t 1 + ( 1 − λ ) t 2 ) (by induction) ∀ t 1 , . . . , t k , λ 1 , . . . , λ k ∈ [ 0 , 1 ] with � = ⇒ i λ i = 1 � i λ i f ( λ i t i ) ≤ f ( � i λ i t i ) = ⇒ (Jensen inequality): E f ( X ) ≤ f ( E X ) for any random variable X . ◮ log ( x ) is (strictly) concave for x > 0, since its second derivative ( − 1 x 2 ) is always negative. ◮ Hence, H ( p 1 , . . . , p m ) = � p i ≤ log � i p i log 1 i p i 1 p i = log m ◮ Alternatively, for X over { 1 , . . . , m } , 1 1 H ( X ) = E X log P X ( X ) ≤ log E X P X ( X ) = log m Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 10 / 12

  76. H ( g ( X )) ≤ H ( X ) Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 11 / 12

  77. H ( g ( X )) ≤ H ( X ) Let X be a random variable, and let g be over Supp ( X ) := { x : P X ( x ) > 0 } . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 11 / 12

  78. H ( g ( X )) ≤ H ( X ) Let X be a random variable, and let g be over Supp ( X ) := { x : P X ( x ) > 0 } . ◮ H ( Y = g ( X )) ≤ H ( X ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 11 / 12

  79. H ( g ( X )) ≤ H ( X ) Let X be a random variable, and let g be over Supp ( X ) := { x : P X ( x ) > 0 } . ◮ H ( Y = g ( X )) ≤ H ( X ) . Proof : Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 11 / 12

  80. H ( g ( X )) ≤ H ( X ) Let X be a random variable, and let g be over Supp ( X ) := { x : P X ( x ) > 0 } . ◮ H ( Y = g ( X )) ≤ H ( X ) . Proof : Iftach Haitner (TAU) Application of Information Theory, Lecture 1 October 28, 2014 11 / 12

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

1 Basic Definitions Below are some basic definitions and terminology that will be used throughout

1 Basic Definitions Below are some basic definitions and terminology that will be used throughout

1 Basic Definitions Below are some basic definitions and terminology that will be used throughout the course. Additional terms and definitions will be introduced as we come across them. A graph G is a tuple consisting of a set V ( G ) of elements

450 views • 3 slides
Plan 2 Outline: Basic definitions and facts Relation between L p , interpolation of norms

Plan 2 Outline: Basic definitions and facts Relation between L p , interpolation of norms

Function spaces L p Updated April 22, 2020 Plan 2 Outline: Basic definitions and facts Relation between L p , interpolation of norms Uniform convexity of L p with 1 p 8 Projections on closed convex sets Peculiarities of L p -spaces with

724 views • 28 slides
Street foods basic definitions and facts L. Filippo DAntuono Department of Agri-food science

Street foods basic definitions and facts L. Filippo DAntuono Department of Agri-food science

Street foods basic definitions and facts L. Filippo DAntuono Department of Agri-food science and technology Campus of Food Science, Cesena Alma Mater Studiorum, University of Bologna, Italy Traditional Food International - TFI-2012 Street

280 views • 11 slides
Basic Definitions Basic Definitions Swelling Pressure Due to volumetric Expansion of a

Basic Definitions Basic Definitions Swelling Pressure Due to volumetric Expansion of a

Basic Definitions Basic Definitions Swelling Pressure Due to volumetric Expansion of a rock mass having swelling minerals when comes in contact with water Primitive stress Stress in the state of equilibrium Primitive stress Stress

703 views • 69 slides
network Complex Networks Complex Networks experience for professional or social purposes : a

network Complex Networks Complex Networks experience for professional or social purposes : a

Overview of Overview of Outline Complex Networks Complex Networks Complex Networks Principles of Complex Systems Basic definitions Basic definitions Basic definitions Examples of Examples of Course CSYS/MATH 300, Fall, 2009 Complex

754 views • 30 slides
Convex hull: basic facts Convex hull: basic facts CG Lecture 1 CG Lecture 1 Problem : give a set

Convex hull: basic facts Convex hull: basic facts CG Lecture 1 CG Lecture 1 Problem : give a set

Convex hull: basic facts Convex hull: basic facts CG Lecture 1 CG Lecture 1 Problem : give a set of n points P in the plane, compute Convex Hull Algorithms its convex hull CH ( P ). Basic facts : 2D CH ( P ) is a convex polygon with

297 views • 7 slides
Review of basic probability and statistics Probability: basic definitions A random variable is

Review of basic probability and statistics Probability: basic definitions A random variable is

Review of basic probability and statistics Probability: basic definitions A random variable is the outcome of a natural process that can not be predicted with certainty. Examples: the maximum temperature next Tuesday in Chicago, the

1.85k views • 153 slides
Basics of Traditional Reliability Where we are going N Basic Definitions N Life and times of a

Basics of Traditional Reliability Where we are going N Basic Definitions N Life and times of a

Basics of Traditional Reliability Where we are going N Basic Definitions N Life and times of a Fault N Reliability Models N N-Modular redundant systems Definitions N RELIABILITY: SURVIVAL PROBABILITY When repair is costly or function is

411 views • 15 slides
CMU-Q 15-381 Lecture 1: Introduction AI, basic definitions, problems, road map Teacher:

CMU-Q 15-381 Lecture 1: Introduction AI, basic definitions, problems, road map Teacher:

CMU-Q 15-381 Lecture 1: Introduction AI, basic definitions, problems, road map Teacher: Gianni A. Di Caro O UTLINE AI? Representation and Problem Solving? What this course is about Fundamental definitions and notions Road map

458 views • 43 slides
1 2 Outlines Probability Basic definitions: Randomization experiment Sample

1 2 Outlines Probability Basic definitions: Randomization experiment Sample

1 2 Outlines Probability Basic definitions: Randomization experiment Sample spaces Elementary outcomes Basic operationsconditional probability Bayes Theorem Randomness 3 Things may happen randomly, for examples

839 views • 48 slides
Rings Definitions and Basic Properties Abhijit Das Department of Computer Science and

Rings Definitions and Basic Properties Abhijit Das Department of Computer Science and

Rings Definitions and Basic Properties Abhijit Das Department of Computer Science and Engineering Indian Institute of Technology Kharagpur October 5, 2020 Discrete Structures, Autumn 2020 Abhijit Das Definitions A set R with two binary

589 views • 30 slides
Outline Overview of Complex Networks Complex Networks Complex Networks Basic definitions

Outline Overview of Complex Networks Complex Networks Complex Networks Basic definitions

Overview of Outline Overview of Complex Networks Complex Networks Complex Networks Basic definitions Basic definitions Basic definitions Principles of Complex Systems Books Books Course 300, Fall, 2008 Examples of Examples of Books

588 views • 23 slides
Review Probability Basic definitions: Randomization experiment Sample spaces

Review Probability Basic definitions: Randomization experiment Sample spaces

Review Probability Basic definitions: Randomization experiment Sample spaces Elementary outcomes Event Basic operationsconditional probability Bayes Theorem Objectives Random Variable Discrete random

712 views • 49 slides
network Complex Networks Complex Networks Prof. Peter Dodds Nutshell Nutshell noun Basic

network Complex Networks Complex Networks Prof. Peter Dodds Nutshell Nutshell noun Basic

Overview of Thesaurus deliciousness: Overview of Complex Networks Complex Networks Complex Networks Principles of Complex Systems Basic definitions Basic definitions Examples of Examples of CSYS/MATH 300, Fall, 2011 Complex Networks

563 views • 21 slides
Geek Speak Can you keep up with internet terminology? April 24, 2018 Agenda 1. Fun Facts 2.

Geek Speak Can you keep up with internet terminology? April 24, 2018 Agenda 1. Fun Facts 2.

Geek Speak Can you keep up with internet terminology? April 24, 2018 Agenda 1. Fun Facts 2. Definitions 3. Planning for your event 4. Questions to ask your venue 5. Calculating your networking needs Fun Facts Fun Facts the iPad

522 views • 35 slides
Artificial Intelligence Photo CC BY-SA 3.0 History Definitions Basic Concepts

Artificial Intelligence Photo CC BY-SA 3.0 History Definitions Basic Concepts

Self Service is Artificial Intelligence Photo CC BY-SA 3.0 History Definitions Basic Concepts Challenges Contents Getting Started Applications Evaluation Process Technology Partners History From game playing

598 views • 13 slides
Todays exercises 5.17: Football Pools 5.18: Cells of Line and Hyperplane Arrangements

Todays exercises 5.17: Football Pools 5.18: Cells of Line and Hyperplane Arrangements

Exercise Session 9 03.05.2016 slide 1 Todays exercises 5.17: Football Pools 5.18: Cells of Line and Hyperplane Arrangements Inclass: PPZ on the formula F 6.1: Harmonic vs. Geometric Mean 6.2: Many j -isolated Satisfying

385 views • 13 slides
Probability, Entropy, and Inference Ensemble X is a triple ( x, A X , P X ) , where Based on

Probability, Entropy, and Inference Ensemble X is a triple ( x, A X , P X ) , where Based on

2 Ensembles and probabilities Probability, Entropy, and Inference Ensemble X is a triple ( x, A X , P X ) , where Based on David J.C. MacKay: x is the outcome of random variable Information Theory, Inference and Learning Algorithms, 2003

207 views • 4 slides
H Filtering of Uncertain LPV Systems with Time-Delay C.Briat, O.Sename and JF.Lafay August

H Filtering of Uncertain LPV Systems with Time-Delay C.Briat, O.Sename and JF.Lafay August

H Filtering of Uncertain LPV Systems with Time-Delay C.Briat, O.Sename and JF.Lafay August 2009 ECC09 - Budapest, Hungary C.Briat, O.Sename and JF.Lafay corentin.briat@briat.info 1/21 Outline Introduction Stability of Uncertain LPV

439 views • 21 slides
Robust Hidden Markov Models Inference in the Presence of Label Noise Benot Frnay February 7,

Robust Hidden Markov Models Inference in the Presence of Label Noise Benot Frnay February 7,

Robust Hidden Markov Models Inference in the Presence of Label Noise Benot Frnay February 7, 2014 What is Machine Learning ? What is Machine Learning ? Machine learning is about learning from data . A model is inferred from a training set

986 views • 73 slides
Low Rank Approximation Lecture 5 Daniel Kressner Chair for Numerical Algorithms and HPC

Low Rank Approximation Lecture 5 Daniel Kressner Chair for Numerical Algorithms and HPC

Low Rank Approximation Lecture 5 Daniel Kressner Chair for Numerical Algorithms and HPC Institute of Mathematics, EPFL daniel.kressner@epfl.ch 1 Randomized column/row sampling Aim: Obtain rank- r approximation from randomly selected rows and

289 views • 14 slides
The Total Curvature and Betti Numbers of Complex Projective Manifolds Convex, Discrete and

The Total Curvature and Betti Numbers of Complex Projective Manifolds Convex, Discrete and

The Total Curvature and Betti Numbers of Complex Projective Manifolds Convex, Discrete and Integral Geometry Friedrich-Schiller-Universit at Jena Joseph Ansel Hoisington University of Georgia 19th September, 2019 Hoisington Total

673 views • 42 slides
Data Mining in Aeronautics, Science, and Exploration Systems 2007 Conference June 26-27, 2007

Data Mining in Aeronautics, Science, and Exploration Systems 2007 Conference June 26-27, 2007

Data Mining in Aeronautics, Science, and Exploration Systems 2007 Conference June 26-27, 2007 Computer History Museum Mountain View, California, USA Sponsored by NASA Engineering and Safety Center Science Mission Directorate Aeronautics

651 views • 44 slides
EXO-200 Mike Jewell Stanford University NorCal HEP-EXchange December 2 nd , 2017 Neutrinoless

EXO-200 Mike Jewell Stanford University NorCal HEP-EXchange December 2 nd , 2017 Neutrinoless

Latest Results from EXO-200 Mike Jewell Stanford University NorCal HEP-EXchange December 2 nd , 2017 Neutrinoless Double Beta Decay (0 ) 136 I 53 I 136 Pr 59 Pr A = 136 9 Massive Majorana Neutrinos 7 Lepton

451 views • 27 slides

More recommend