channel capacity and isoperimetric inequality
play

Channel Capacity and Isoperimetric Inequality Iftach Haitner Tel - PowerPoint PPT Presentation

Dec 25, 2023 •890 likes •2.76k views

Application of Information Theory, Lecture 5 Channel Capacity and Isoperimetric Inequality Iftach Haitner Tel Aviv University. November 25, 2014 Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 1 / 21 Part

  1. Shannon’s result ◮ Shannon showed that you can reduce the error rate towards 0, without reducing the transmission rate towards 0 ◮ For any c < C p , exists a code with transmission rate c that is correct w.p. ◮ Example: for p = . 1, C p > 1 2 . Hence, for sending x = ( x 1 , . . . , x m ) , one can send 2 m bits, such that x is recovered w.p. close to 1 ◮ More generally, ∀ p ∈ [ 0 , 1 ] ∃ C p such that for sending x ∈ { 0 , 1 } m , one can send ≈ m C p bits, and x is recovered w.p. close to 1 ◮ C p might be 0 (i.e., for p = 1 2 ) ◮ A revolution in EE and the whole world Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 5 / 21

  2. Error correction code Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21

  3. Error correction code ◮ Message to send x = ( x 1 , . . . , x m ) ∈ { 0 , 1 } m Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21

  4. Error correction code ◮ Message to send x = ( x 1 , . . . , x m ) ∈ { 0 , 1 } m ◮ Encoding scheme: f : { 0 , 1 } m �→ { 0 , 1 } n ( n > m ) Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21

  5. Error correction code ◮ Message to send x = ( x 1 , . . . , x m ) ∈ { 0 , 1 } m ◮ Encoding scheme: f : { 0 , 1 } m �→ { 0 , 1 } n ( n > m ) ◮ Decoding scheme: g : { 0 , 1 } n �→ { 0 , 1 } m Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21

  6. Error correction code ◮ Message to send x = ( x 1 , . . . , x m ) ∈ { 0 , 1 } m ◮ Encoding scheme: f : { 0 , 1 } m �→ { 0 , 1 } n ( n > m ) ◮ Decoding scheme: g : { 0 , 1 } n �→ { 0 , 1 } m m n — transmission rate ◮ Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21

  7. Error correction code ◮ Message to send x = ( x 1 , . . . , x m ) ∈ { 0 , 1 } m ◮ Encoding scheme: f : { 0 , 1 } m �→ { 0 , 1 } n ( n > m ) ◮ Decoding scheme: g : { 0 , 1 } n �→ { 0 , 1 } m m n — transmission rate ◮ ◮ Sender sends f ( x ) rather than x Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21

  8. Error correction code ◮ Message to send x = ( x 1 , . . . , x m ) ∈ { 0 , 1 } m ◮ Encoding scheme: f : { 0 , 1 } m �→ { 0 , 1 } n ( n > m ) ◮ Decoding scheme: g : { 0 , 1 } n �→ { 0 , 1 } m m n — transmission rate ◮ ◮ Sender sends f ( x ) rather than x ◮ Receiver decodes the message by applying g Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21

  9. Error correction code ◮ Message to send x = ( x 1 , . . . , x m ) ∈ { 0 , 1 } m ◮ Encoding scheme: f : { 0 , 1 } m �→ { 0 , 1 } n ( n > m ) ◮ Decoding scheme: g : { 0 , 1 } n �→ { 0 , 1 } m m n — transmission rate ◮ ◮ Sender sends f ( x ) rather than x ◮ Receiver decodes the message by applying g encoding decoding channel x − → f ( x ) − → f ( x ) ⊕ Z − → g ( f ( x ) ⊕ Z ) ◮ ���� ���� � �� � m bits n bits bitwise XOR Z = ( Z 1 , . . . , Z n ) where Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) (i.e., over { 0 , 1 } with Pr [ Z i = 1 ] = p ) Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21

  10. Error correction code ◮ Message to send x = ( x 1 , . . . , x m ) ∈ { 0 , 1 } m ◮ Encoding scheme: f : { 0 , 1 } m �→ { 0 , 1 } n ( n > m ) ◮ Decoding scheme: g : { 0 , 1 } n �→ { 0 , 1 } m m n — transmission rate ◮ ◮ Sender sends f ( x ) rather than x ◮ Receiver decodes the message by applying g encoding decoding channel x − → f ( x ) − → f ( x ) ⊕ Z − → g ( f ( x ) ⊕ Z ) ◮ ���� ���� � �� � m bits n bits bitwise XOR Z = ( Z 1 , . . . , Z n ) where Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) (i.e., over { 0 , 1 } with Pr [ Z i = 1 ] = p ) ◮ We hope g ( f ( x ) ⊕ Z ) = x Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21

  11. Error correction code ◮ Message to send x = ( x 1 , . . . , x m ) ∈ { 0 , 1 } m ◮ Encoding scheme: f : { 0 , 1 } m �→ { 0 , 1 } n ( n > m ) ◮ Decoding scheme: g : { 0 , 1 } n �→ { 0 , 1 } m m n — transmission rate ◮ ◮ Sender sends f ( x ) rather than x ◮ Receiver decodes the message by applying g encoding decoding channel x − → f ( x ) − → f ( x ) ⊕ Z − → g ( f ( x ) ⊕ Z ) ◮ ���� ���� � �� � m bits n bits bitwise XOR Z = ( Z 1 , . . . , Z n ) where Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) (i.e., over { 0 , 1 } with Pr [ Z i = 1 ] = p ) ◮ We hope g ( f ( x ) ⊕ Z ) = x ◮ ECCs are everywhere Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21

  12. Error correction code ◮ Message to send x = ( x 1 , . . . , x m ) ∈ { 0 , 1 } m ◮ Encoding scheme: f : { 0 , 1 } m �→ { 0 , 1 } n ( n > m ) ◮ Decoding scheme: g : { 0 , 1 } n �→ { 0 , 1 } m m n — transmission rate ◮ ◮ Sender sends f ( x ) rather than x ◮ Receiver decodes the message by applying g encoding decoding channel x − → f ( x ) − → f ( x ) ⊕ Z − → g ( f ( x ) ⊕ Z ) ◮ ���� ���� � �� � m bits n bits bitwise XOR Z = ( Z 1 , . . . , Z n ) where Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) (i.e., over { 0 , 1 } with Pr [ Z i = 1 ] = p ) ◮ We hope g ( f ( x ) ⊕ Z ) = x ◮ ECCs are everywhere ◮ ECC Vs compression Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21

  13. Shannon’s theorem Theorem 1 ∃ m ε , s.t. ∀ m > m ε and n > m ( 1 ∀ p ∃ C p , s.t. ∀ ε > 0 C p + ε ) , ∃ f : { 0 , 1 } m �→ { 0 , 1 } n and g : { 0 , 1 } n �→ { 0 , 1 } m , s.t. ∀ x ∈ { 0 , 1 } m : z ← Z =( Z 1 ,..., Z n ) [ g ( f ( x ) ⊕ z ) � = x ] ≤ ε Pr for Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) . Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 7 / 21

  14. Shannon’s theorem Theorem 1 ∃ m ε , s.t. ∀ m > m ε and n > m ( 1 ∀ p ∃ C p , s.t. ∀ ε > 0 C p + ε ) , ∃ f : { 0 , 1 } m �→ { 0 , 1 } n and g : { 0 , 1 } n �→ { 0 , 1 } m , s.t. ∀ x ∈ { 0 , 1 } m : z ← Z =( Z 1 ,..., Z n ) [ g ( f ( x ) ⊕ z ) � = x ] ≤ ε Pr for Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) . ◮ C p = 1 − h ( p ) — the channel capacity Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 7 / 21

  15. Shannon’s theorem Theorem 1 ∃ m ε , s.t. ∀ m > m ε and n > m ( 1 ∀ p ∃ C p , s.t. ∀ ε > 0 C p + ε ) , ∃ f : { 0 , 1 } m �→ { 0 , 1 } n and g : { 0 , 1 } n �→ { 0 , 1 } m , s.t. ∀ x ∈ { 0 , 1 } m : z ← Z =( Z 1 ,..., Z n ) [ g ( f ( x ) ⊕ z ) � = x ] ≤ ε Pr for Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) . ◮ C p = 1 − h ( p ) — the channel capacity ⇒ C p = 0 . 5310 > 1 p = . 1 = 2 Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 7 / 21

  16. Shannon’s theorem Theorem 1 ∃ m ε , s.t. ∀ m > m ε and n > m ( 1 ∀ p ∃ C p , s.t. ∀ ε > 0 C p + ε ) , ∃ f : { 0 , 1 } m �→ { 0 , 1 } n and g : { 0 , 1 } n �→ { 0 , 1 } m , s.t. ∀ x ∈ { 0 , 1 } m : z ← Z =( Z 1 ,..., Z n ) [ g ( f ( x ) ⊕ z ) � = x ] ≤ ε Pr for Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) . ◮ C p = 1 − h ( p ) — the channel capacity ⇒ C p = 0 . 5310 > 1 p = . 1 = 2 ⇒ C p ≈ 1 p = . 25 = 5 Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 7 / 21

  17. Shannon’s theorem Theorem 1 ∃ m ε , s.t. ∀ m > m ε and n > m ( 1 ∀ p ∃ C p , s.t. ∀ ε > 0 C p + ε ) , ∃ f : { 0 , 1 } m �→ { 0 , 1 } n and g : { 0 , 1 } n �→ { 0 , 1 } m , s.t. ∀ x ∈ { 0 , 1 } m : z ← Z =( Z 1 ,..., Z n ) [ g ( f ( x ) ⊕ z ) � = x ] ≤ ε Pr for Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) . ◮ C p = 1 − h ( p ) — the channel capacity ⇒ C p = 0 . 5310 > 1 p = . 1 = 2 ⇒ C p ≈ 1 p = . 25 = 5 ◮ Tight theorem Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 7 / 21

  18. Shannon’s theorem Theorem 1 ∃ m ε , s.t. ∀ m > m ε and n > m ( 1 ∀ p ∃ C p , s.t. ∀ ε > 0 C p + ε ) , ∃ f : { 0 , 1 } m �→ { 0 , 1 } n and g : { 0 , 1 } n �→ { 0 , 1 } m , s.t. ∀ x ∈ { 0 , 1 } m : z ← Z =( Z 1 ,..., Z n ) [ g ( f ( x ) ⊕ z ) � = x ] ≤ ε Pr for Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) . ◮ C p = 1 − h ( p ) — the channel capacity ⇒ C p = 0 . 5310 > 1 p = . 1 = 2 ⇒ C p ≈ 1 p = . 25 = 5 ◮ Tight theorem ◮ We prove a weaker variant that holds w.h.p. over x ← { 0 , 1 } m Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 7 / 21

  19. Hamming distance Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 8 / 21

  20. Hamming distance ◮ For y = ( y 1 , . . . , y n ) ∈ { 0 , 1 } n , let | y | = � i y i — Hamming weight of y Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 8 / 21

  21. Hamming distance ◮ For y = ( y 1 , . . . , y n ) ∈ { 0 , 1 } n , let | y | = � i y i — Hamming weight of y ◮ | y − y ′ | = | y ⊕ y ′ | — Hamming distance of y from y ′ ; # of places differ. Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 8 / 21

  22. Proving the theorem Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21

  23. Proving the theorem ◮ Fix p ∈ [ 0 , 1 2 ) and ε > 0, and let m > m ε and n ≥ m ( 1 C p + ε ) , for m ε to be determined by the analysis. Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21

  24. Proving the theorem ◮ Fix p ∈ [ 0 , 1 2 ) and ε > 0, and let m > m ε and n ≥ m ( 1 C p + ε ) , for m ε to be determined by the analysis. Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21

  25. Proving the theorem ◮ Fix p ∈ [ 0 , 1 2 ) and ε > 0, and let m > m ε and n ≥ m ( 1 C p + ε ) , for m ε to be determined by the analysis. (Recall C p = 1 − h ( p ) ). Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21

  26. Proving the theorem ◮ Fix p ∈ [ 0 , 1 2 ) and ε > 0, and let m > m ε and n ≥ m ( 1 C p + ε ) , for m ε to be determined by the analysis. (Recall C p = 1 − h ( p ) ). ◮ We show ∃ f : { 0 , 1 } m �→ { 0 , 1 } n and g : { 0 , 1 } n �→ { 0 , 1 } m , s.t. Pr x ←{ 0 , 1 } m [ g ( f ( x ) ⊕ Z ) � = x ] ≤ ε Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21

  27. Proving the theorem ◮ Fix p ∈ [ 0 , 1 2 ) and ε > 0, and let m > m ε and n ≥ m ( 1 C p + ε ) , for m ε to be determined by the analysis. (Recall C p = 1 − h ( p ) ). ◮ We show ∃ f : { 0 , 1 } m �→ { 0 , 1 } n and g : { 0 , 1 } n �→ { 0 , 1 } m , s.t. Pr x ←{ 0 , 1 } m [ g ( f ( x ) ⊕ Z ) � = x ] ≤ ε ◮ g ( y ) returns argmin x ′ ∈{ 0 , 1 } m | y − f ( x ′ ) | Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21

  28. Proving the theorem ◮ Fix p ∈ [ 0 , 1 2 ) and ε > 0, and let m > m ε and n ≥ m ( 1 C p + ε ) , for m ε to be determined by the analysis. (Recall C p = 1 − h ( p ) ). ◮ We show ∃ f : { 0 , 1 } m �→ { 0 , 1 } n and g : { 0 , 1 } n �→ { 0 , 1 } m , s.t. Pr x ←{ 0 , 1 } m [ g ( f ( x ) ⊕ Z ) � = x ] ≤ ε ◮ g ( y ) returns argmin x ′ ∈{ 0 , 1 } m | y − f ( x ′ ) | ◮ So it all boils down to finding f s.t. � � | f ( x ) − y | < min x ′ ∈{ 0 , 1 } m \{ x } | f ( x ′ ) − y | Pr x ←{ 0 , 1 } m ; y = f ( x ) ⊕ Z ≥ 1 − ε Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21

  29. Proving the theorem ◮ Fix p ∈ [ 0 , 1 2 ) and ε > 0, and let m > m ε and n ≥ m ( 1 C p + ε ) , for m ε to be determined by the analysis. (Recall C p = 1 − h ( p ) ). ◮ We show ∃ f : { 0 , 1 } m �→ { 0 , 1 } n and g : { 0 , 1 } n �→ { 0 , 1 } m , s.t. Pr x ←{ 0 , 1 } m [ g ( f ( x ) ⊕ Z ) � = x ] ≤ ε ◮ g ( y ) returns argmin x ′ ∈{ 0 , 1 } m | y − f ( x ′ ) | ◮ So it all boils down to finding f s.t. � � | f ( x ) − y | < min x ′ ∈{ 0 , 1 } m \{ x } | f ( x ′ ) − y | Pr x ←{ 0 , 1 } m ; y = f ( x ) ⊕ Z ≥ 1 − ε ◮ Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21

  30. Proving the theorem ◮ Fix p ∈ [ 0 , 1 2 ) and ε > 0, and let m > m ε and n ≥ m ( 1 C p + ε ) , for m ε to be determined by the analysis. (Recall C p = 1 − h ( p ) ). ◮ We show ∃ f : { 0 , 1 } m �→ { 0 , 1 } n and g : { 0 , 1 } n �→ { 0 , 1 } m , s.t. Pr x ←{ 0 , 1 } m [ g ( f ( x ) ⊕ Z ) � = x ] ≤ ε ◮ g ( y ) returns argmin x ′ ∈{ 0 , 1 } m | y − f ( x ′ ) | ◮ So it all boils down to finding f s.t. � � | f ( x ) − y | < min x ′ ∈{ 0 , 1 } m \{ x } | f ( x ′ ) − y | Pr x ←{ 0 , 1 } m ; y = f ( x ) ⊕ Z ≥ 1 − ε ◮ Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21

  31. Proving the theorem ◮ Fix p ∈ [ 0 , 1 2 ) and ε > 0, and let m > m ε and n ≥ m ( 1 C p + ε ) , for m ε to be determined by the analysis. (Recall C p = 1 − h ( p ) ). ◮ We show ∃ f : { 0 , 1 } m �→ { 0 , 1 } n and g : { 0 , 1 } n �→ { 0 , 1 } m , s.t. Pr x ←{ 0 , 1 } m [ g ( f ( x ) ⊕ Z ) � = x ] ≤ ε ◮ g ( y ) returns argmin x ′ ∈{ 0 , 1 } m | y − f ( x ′ ) | ◮ So it all boils down to finding f s.t. � � | f ( x ) − y | < min x ′ ∈{ 0 , 1 } m \{ x } | f ( x ′ ) − y | Pr x ←{ 0 , 1 } m ; y = f ( x ) ⊕ Z ≥ 1 − ε ◮ Idea: for p ′ > p to be determined later, find f s.t. w.h.p. over x and Z : (1) | f ( x ) ⊕ Z , f ( x ) | ≤ p ′ n (2) | f ( x ) ⊕ Z , f ( x ′ ) | > p ′ n for all x ′ � = x Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21

  32. Proving the theorem ◮ Fix p ∈ [ 0 , 1 2 ) and ε > 0, and let m > m ε and n ≥ m ( 1 C p + ε ) , for m ε to be determined by the analysis. (Recall C p = 1 − h ( p ) ). ◮ We show ∃ f : { 0 , 1 } m �→ { 0 , 1 } n and g : { 0 , 1 } n �→ { 0 , 1 } m , s.t. Pr x ←{ 0 , 1 } m [ g ( f ( x ) ⊕ Z ) � = x ] ≤ ε ◮ g ( y ) returns argmin x ′ ∈{ 0 , 1 } m | y − f ( x ′ ) | ◮ So it all boils down to finding f s.t. � � | f ( x ) − y | < min x ′ ∈{ 0 , 1 } m \{ x } | f ( x ′ ) − y | Pr x ←{ 0 , 1 } m ; y = f ( x ) ⊕ Z ≥ 1 − ε ◮ Idea: for p ′ > p to be determined later, find f s.t. w.h.p. over x and Z : (1) | f ( x ) ⊕ Z , f ( x ) | ≤ p ′ n (2) | f ( x ) ⊕ Z , f ( x ′ ) | > p ′ n for all x ′ � = x Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21

  33. Proving the theorem ◮ Fix p ∈ [ 0 , 1 2 ) and ε > 0, and let m > m ε and n ≥ m ( 1 C p + ε ) , for m ε to be determined by the analysis. (Recall C p = 1 − h ( p ) ). ◮ We show ∃ f : { 0 , 1 } m �→ { 0 , 1 } n and g : { 0 , 1 } n �→ { 0 , 1 } m , s.t. Pr x ←{ 0 , 1 } m [ g ( f ( x ) ⊕ Z ) � = x ] ≤ ε ◮ g ( y ) returns argmin x ′ ∈{ 0 , 1 } m | y − f ( x ′ ) | ◮ So it all boils down to finding f s.t. � � | f ( x ) − y | < min x ′ ∈{ 0 , 1 } m \{ x } | f ( x ′ ) − y | Pr x ←{ 0 , 1 } m ; y = f ( x ) ⊕ Z ≥ 1 − ε ◮ Idea: for p ′ > p to be determined later, find f s.t. w.h.p. over x and Z : (1) | f ( x ) ⊕ Z , f ( x ) | ≤ p ′ n (2) | f ( x ) ⊕ Z , f ( x ′ ) | > p ′ n for all x ′ � = x ◮ We choose f uniformly at random (what does it mean?) Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21

  34. Proving the theorem ◮ Fix p ∈ [ 0 , 1 2 ) and ε > 0, and let m > m ε and n ≥ m ( 1 C p + ε ) , for m ε to be determined by the analysis. (Recall C p = 1 − h ( p ) ). ◮ We show ∃ f : { 0 , 1 } m �→ { 0 , 1 } n and g : { 0 , 1 } n �→ { 0 , 1 } m , s.t. Pr x ←{ 0 , 1 } m [ g ( f ( x ) ⊕ Z ) � = x ] ≤ ε ◮ g ( y ) returns argmin x ′ ∈{ 0 , 1 } m | y − f ( x ′ ) | ◮ So it all boils down to finding f s.t. � � | f ( x ) − y | < min x ′ ∈{ 0 , 1 } m \{ x } | f ( x ′ ) − y | Pr x ←{ 0 , 1 } m ; y = f ( x ) ⊕ Z ≥ 1 − ε ◮ Idea: for p ′ > p to be determined later, find f s.t. w.h.p. over x and Z : (1) | f ( x ) ⊕ Z , f ( x ) | ≤ p ′ n (2) | f ( x ) ⊕ Z , f ( x ′ ) | > p ′ n for all x ′ � = x ◮ We choose f uniformly at random (what does it mean?) ◮ Non-constructive proof Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21

  35. Proving the theorem ◮ Fix p ∈ [ 0 , 1 2 ) and ε > 0, and let m > m ε and n ≥ m ( 1 C p + ε ) , for m ε to be determined by the analysis. (Recall C p = 1 − h ( p ) ). ◮ We show ∃ f : { 0 , 1 } m �→ { 0 , 1 } n and g : { 0 , 1 } n �→ { 0 , 1 } m , s.t. Pr x ←{ 0 , 1 } m [ g ( f ( x ) ⊕ Z ) � = x ] ≤ ε ◮ g ( y ) returns argmin x ′ ∈{ 0 , 1 } m | y − f ( x ′ ) | ◮ So it all boils down to finding f s.t. � � | f ( x ) − y | < min x ′ ∈{ 0 , 1 } m \{ x } | f ( x ′ ) − y | Pr x ←{ 0 , 1 } m ; y = f ( x ) ⊕ Z ≥ 1 − ε ◮ Idea: for p ′ > p to be determined later, find f s.t. w.h.p. over x and Z : (1) | f ( x ) ⊕ Z , f ( x ) | ≤ p ′ n (2) | f ( x ) ⊕ Z , f ( x ′ ) | > p ′ n for all x ′ � = x ◮ We choose f uniformly at random (what does it mean?) ◮ Non-constructive proof ◮ Probabilistic method Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21

  36. Proving there exists good f Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21

  37. Proving there exists good f ◮ Fix p ′ > p such that 1 C p ≤ ε 1 C p ′ − 2 Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21

  38. Proving there exists good f ◮ Fix p ′ > p such that 1 C p ≤ ε 1 C p ′ − 2 ◮ For y ∈ { 0 , 1 } n , let B p ′ ( y ) = { y ∈ { 0 , 1 } n : | y ′ − y | ≤ p ′ n } Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21

  39. Proving there exists good f ◮ Fix p ′ > p such that 1 C p ≤ ε 1 C p ′ − 2 ◮ For y ∈ { 0 , 1 } n , let B p ′ ( y ) = { y ∈ { 0 , 1 } n : | y ′ − y | ≤ p ′ n } ( 1 ) By weak low of large numbers, ∃ n ′ ∈ N s.t. ∀ n ≥ n ′ and ∀ x ∈ { 0 , 1 } m : ∈ B p ′ ( f ( x ))] ≤ ε α n := Pr z ← Z [( f ( x ) ⊕ z ) / (for any fixed f ) 2 Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21

  40. Proving there exists good f ◮ Fix p ′ > p such that 1 C p ≤ ε 1 C p ′ − 2 ◮ For y ∈ { 0 , 1 } n , let B p ′ ( y ) = { y ∈ { 0 , 1 } n : | y ′ − y | ≤ p ′ n } ( 1 ) By weak low of large numbers, ∃ n ′ ∈ N s.t. ∀ n ≥ n ′ and ∀ x ∈ { 0 , 1 } m : ∈ B p ′ ( f ( x ))] ≤ ε α n := Pr z ← Z [( f ( x ) ⊕ z ) / (for any fixed f ) 2 ◮ Fact (proved later): b ( p ′ ) = | B p ′ ( y ) | ≤ 2 n · h ( p ′ ) Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21

  41. Proving there exists good f ◮ Fix p ′ > p such that 1 C p ≤ ε 1 C p ′ − 2 ◮ For y ∈ { 0 , 1 } n , let B p ′ ( y ) = { y ∈ { 0 , 1 } n : | y ′ − y | ≤ p ′ n } ( 1 ) By weak low of large numbers, ∃ n ′ ∈ N s.t. ∀ n ≥ n ′ and ∀ x ∈ { 0 , 1 } m : ∈ B p ′ ( f ( x ))] ≤ ε α n := Pr z ← Z [( f ( x ) ⊕ z ) / (for any fixed f ) 2 ◮ Fact (proved later): b ( p ′ ) = | B p ′ ( y ) | ≤ 2 n · h ( p ′ ) ∀ x � = x ′ ∈ { 0 , 1 } m : Pr f , Z [ f ( x ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] = b ( p ′ ) = ⇒ 2 n Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21

  42. Proving there exists good f ◮ Fix p ′ > p such that 1 C p ≤ ε 1 C p ′ − 2 ◮ For y ∈ { 0 , 1 } n , let B p ′ ( y ) = { y ∈ { 0 , 1 } n : | y ′ − y | ≤ p ′ n } ( 1 ) By weak low of large numbers, ∃ n ′ ∈ N s.t. ∀ n ≥ n ′ and ∀ x ∈ { 0 , 1 } m : ∈ B p ′ ( f ( x ))] ≤ ε α n := Pr z ← Z [( f ( x ) ⊕ z ) / (for any fixed f ) 2 ◮ Fact (proved later): b ( p ′ ) = | B p ′ ( y ) | ≤ 2 n · h ( p ′ ) ∀ x � = x ′ ∈ { 0 , 1 } m : Pr f , Z [ f ( x ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] = b ( p ′ ) = ⇒ 2 n Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21

  43. Proving there exists good f ◮ Fix p ′ > p such that 1 C p ≤ ε 1 C p ′ − 2 ◮ For y ∈ { 0 , 1 } n , let B p ′ ( y ) = { y ∈ { 0 , 1 } n : | y ′ − y | ≤ p ′ n } ( 1 ) By weak low of large numbers, ∃ n ′ ∈ N s.t. ∀ n ≥ n ′ and ∀ x ∈ { 0 , 1 } m : ∈ B p ′ ( f ( x ))] ≤ ε α n := Pr z ← Z [( f ( x ) ⊕ z ) / (for any fixed f ) 2 ◮ Fact (proved later): b ( p ′ ) = | B p ′ ( y ) | ≤ 2 n · h ( p ′ ) ∀ x � = x ′ ∈ { 0 , 1 } m : Pr f , Z [ f ( x ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] = b ( p ′ ) ≤ 2 n · h ( p ′ ) = 2 − nC p ′ = ⇒ 2 n 2 n Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21

  44. Proving there exists good f ◮ Fix p ′ > p such that 1 C p ≤ ε 1 C p ′ − 2 ◮ For y ∈ { 0 , 1 } n , let B p ′ ( y ) = { y ∈ { 0 , 1 } n : | y ′ − y | ≤ p ′ n } ( 1 ) By weak low of large numbers, ∃ n ′ ∈ N s.t. ∀ n ≥ n ′ and ∀ x ∈ { 0 , 1 } m : ∈ B p ′ ( f ( x ))] ≤ ε α n := Pr z ← Z [( f ( x ) ⊕ z ) / (for any fixed f ) 2 ◮ Fact (proved later): b ( p ′ ) = | B p ′ ( y ) | ≤ 2 n · h ( p ′ ) ∀ x � = x ′ ∈ { 0 , 1 } m : Pr f , Z [ f ( x ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] = b ( p ′ ) ≤ 2 n · h ( p ′ ) = 2 − nC p ′ = ⇒ 2 n 2 n ∀ x ∈ { 0 , 1 } m : Pr f , Z [ ∃ x ′ � = x ∈ { 0 , 1 } m : f ( x ′ ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] ≤ 2 m − nC p ′ = ⇒ Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21

  45. Proving there exists good f ◮ Fix p ′ > p such that 1 C p ≤ ε 1 C p ′ − 2 ◮ For y ∈ { 0 , 1 } n , let B p ′ ( y ) = { y ∈ { 0 , 1 } n : | y ′ − y | ≤ p ′ n } ( 1 ) By weak low of large numbers, ∃ n ′ ∈ N s.t. ∀ n ≥ n ′ and ∀ x ∈ { 0 , 1 } m : ∈ B p ′ ( f ( x ))] ≤ ε α n := Pr z ← Z [( f ( x ) ⊕ z ) / (for any fixed f ) 2 ◮ Fact (proved later): b ( p ′ ) = | B p ′ ( y ) | ≤ 2 n · h ( p ′ ) ∀ x � = x ′ ∈ { 0 , 1 } m : Pr f , Z [ f ( x ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] = b ( p ′ ) ≤ 2 n · h ( p ′ ) = 2 − nC p ′ = ⇒ 2 n 2 n ∀ x ∈ { 0 , 1 } m : Pr f , Z [ ∃ x ′ � = x ∈ { 0 , 1 } m : f ( x ′ ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] ≤ 2 m − nC p ′ = ⇒ = ⇒ ∃ f s.t. β m , n := Pr x ←{ 0 , 1 } m [ ∃ x ′ � = x ∈ { 0 , 1 } m : f ( x ′ ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] ≤ 2 m − nC p ′ Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21

  46. Proving there exists good f ◮ Fix p ′ > p such that 1 C p ≤ ε 1 C p ′ − 2 ◮ For y ∈ { 0 , 1 } n , let B p ′ ( y ) = { y ∈ { 0 , 1 } n : | y ′ − y | ≤ p ′ n } ( 1 ) By weak low of large numbers, ∃ n ′ ∈ N s.t. ∀ n ≥ n ′ and ∀ x ∈ { 0 , 1 } m : ∈ B p ′ ( f ( x ))] ≤ ε α n := Pr z ← Z [( f ( x ) ⊕ z ) / (for any fixed f ) 2 ◮ Fact (proved later): b ( p ′ ) = | B p ′ ( y ) | ≤ 2 n · h ( p ′ ) ∀ x � = x ′ ∈ { 0 , 1 } m : Pr f , Z [ f ( x ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] = b ( p ′ ) ≤ 2 n · h ( p ′ ) = 2 − nC p ′ = ⇒ 2 n 2 n ∀ x ∈ { 0 , 1 } m : Pr f , Z [ ∃ x ′ � = x ∈ { 0 , 1 } m : f ( x ′ ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] ≤ 2 m − nC p ′ = ⇒ = ⇒ ∃ f s.t. β m , n := Pr x ←{ 0 , 1 } m [ ∃ x ′ � = x ∈ { 0 , 1 } m : f ( x ′ ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] ≤ 2 m − nC p ′ C p ′ + 1 − log ε β m , n ≤ ε C p ′ − log ε + 1 = m ( 1 m = ⇒ 2 , for n ≥ ) m Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21

  47. Proving there exists good f ◮ Fix p ′ > p such that 1 C p ≤ ε 1 C p ′ − 2 ◮ For y ∈ { 0 , 1 } n , let B p ′ ( y ) = { y ∈ { 0 , 1 } n : | y ′ − y | ≤ p ′ n } ( 1 ) By weak low of large numbers, ∃ n ′ ∈ N s.t. ∀ n ≥ n ′ and ∀ x ∈ { 0 , 1 } m : ∈ B p ′ ( f ( x ))] ≤ ε α n := Pr z ← Z [( f ( x ) ⊕ z ) / (for any fixed f ) 2 ◮ Fact (proved later): b ( p ′ ) = | B p ′ ( y ) | ≤ 2 n · h ( p ′ ) ∀ x � = x ′ ∈ { 0 , 1 } m : Pr f , Z [ f ( x ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] = b ( p ′ ) ≤ 2 n · h ( p ′ ) = 2 − nC p ′ = ⇒ 2 n 2 n ∀ x ∈ { 0 , 1 } m : Pr f , Z [ ∃ x ′ � = x ∈ { 0 , 1 } m : f ( x ′ ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] ≤ 2 m − nC p ′ = ⇒ = ⇒ ∃ f s.t. β m , n := Pr x ←{ 0 , 1 } m [ ∃ x ′ � = x ∈ { 0 , 1 } m : f ( x ′ ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] ≤ 2 m − nC p ′ C p ′ + 1 − log ε β m , n ≤ ε C p ′ − log ε + 1 = m ( 1 m = ⇒ 2 , for n ≥ ) m 2 , for m ≥ m ′ = 2 ( 1 − log ε ) ( 2 ) β m , n ≤ ε and n ≥ m ( 1 C p + ε 2 + 1 − log ε ) = m ( 1 C p + ε ) ε m Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21

  48. Proving there exists good f ◮ Fix p ′ > p such that 1 C p ≤ ε 1 C p ′ − 2 ◮ For y ∈ { 0 , 1 } n , let B p ′ ( y ) = { y ∈ { 0 , 1 } n : | y ′ − y | ≤ p ′ n } ( 1 ) By weak low of large numbers, ∃ n ′ ∈ N s.t. ∀ n ≥ n ′ and ∀ x ∈ { 0 , 1 } m : ∈ B p ′ ( f ( x ))] ≤ ε α n := Pr z ← Z [( f ( x ) ⊕ z ) / (for any fixed f ) 2 ◮ Fact (proved later): b ( p ′ ) = | B p ′ ( y ) | ≤ 2 n · h ( p ′ ) ∀ x � = x ′ ∈ { 0 , 1 } m : Pr f , Z [ f ( x ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] = b ( p ′ ) ≤ 2 n · h ( p ′ ) = 2 − nC p ′ = ⇒ 2 n 2 n ∀ x ∈ { 0 , 1 } m : Pr f , Z [ ∃ x ′ � = x ∈ { 0 , 1 } m : f ( x ′ ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] ≤ 2 m − nC p ′ = ⇒ = ⇒ ∃ f s.t. β m , n := Pr x ←{ 0 , 1 } m [ ∃ x ′ � = x ∈ { 0 , 1 } m : f ( x ′ ) ⊕ Z ∈ B p ′ ( f ( x ′ ))] ≤ 2 m − nC p ′ C p ′ + 1 − log ε β m , n ≤ ε C p ′ − log ε + 1 = m ( 1 m = ⇒ 2 , for n ≥ ) m 2 , for m ≥ m ′ = 2 ( 1 − log ε ) ( 2 ) β m , n ≤ ε and n ≥ m ( 1 C p + ε 2 + 1 − log ε ) = m ( 1 C p + ε ) ε m ◮ Hence, for m > m ε = max { m ′ , n ′ } and n > m ( 1 C p + ε ) , it holds that Pr x ←{ 0 , 1 } m [ g ( f ( x ) ⊕ Z ) � = x ] ≤ α n + β m , n ≤ ε . Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21

  49. Why C p = 1 − h ( p ) ?

  50. Why C p = 1 − h ( p ) ? ◮ Let X ← { 0 , 1 } , Z ∼ ( 1 − p , p ) and Y = X ⊕ Z

  51. Why C p = 1 − h ( p ) ? ◮ Let X ← { 0 , 1 } , Z ∼ ( 1 − p , p ) and Y = X ⊕ Z X Y 1 − p 0 0 p p 1 1 1 − p Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 11 / 21

  52. Why C p = 1 − h ( p ) ? ◮ Let X ← { 0 , 1 } , Z ∼ ( 1 − p , p ) and Y = X ⊕ Z X Y 1 − p 0 0 p p 1 1 1 − p ◮ I ( X ; Y ) = H ( Y ) − H ( Y | X ) = H ( Y ) − H ( Z ) = 1 − h ( p ) = C p Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 11 / 21

  53. Why C p = 1 − h ( p ) ? ◮ Let X ← { 0 , 1 } , Z ∼ ( 1 − p , p ) and Y = X ⊕ Z X Y 1 − p 0 0 p p 1 1 1 − p ◮ I ( X ; Y ) = H ( Y ) − H ( Y | X ) = H ( Y ) − H ( Z ) = 1 − h ( p ) = C p ◮ Received bit “gives" C p information about transmitted bit Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 11 / 21

  54. Why C p = 1 − h ( p ) ? ◮ Let X ← { 0 , 1 } , Z ∼ ( 1 − p , p ) and Y = X ⊕ Z X Y 1 − p 0 0 p p 1 1 1 − p ◮ I ( X ; Y ) = H ( Y ) − H ( Y | X ) = H ( Y ) − H ( Z ) = 1 − h ( p ) = C p ◮ Received bit “gives" C p information about transmitted bit ◮ Hence, to recover m bits, we need to send at least m · 1 C p bits Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 11 / 21

  55. Size of bounding ball Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 12 / 21

  56. Size of bounding ball Claim 2 � n � 2 ] and n ∈ N : it holds that � ⌊ pn ⌋ For p ∈ [ 0 , 1 ≤ 2 n · h ( p ) k = 0 k Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 12 / 21

  57. Size of bounding ball Claim 2 � n � 2 ] and n ∈ N : it holds that � ⌊ pn ⌋ For p ∈ [ 0 , 1 ≤ 2 n · h ( p ) k = 0 k � n � ≈ 2 n · h ( p ) ) Proof in a few slides (we already saw that pn Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 12 / 21

  58. Size of bounding ball Claim 2 � n � 2 ] and n ∈ N : it holds that � ⌊ pn ⌋ For p ∈ [ 0 , 1 ≤ 2 n · h ( p ) k = 0 k � n � ≈ 2 n · h ( p ) ) Proof in a few slides (we already saw that pn Corollary 3 For y ∈ { 0 , 1 } n and p ∈ [ 0 , 1 2 ] , let B p ( y ) = { y ∈ { 0 , 1 } n : | y ′ − y | ≤ pn } . Then | B p ( y ) | = � ⌊ pn ⌋ � n � ≤ 2 n · h ( p ) k = 0 k Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 12 / 21

  59. Size of bounding ball Claim 2 � n � 2 ] and n ∈ N : it holds that � ⌊ pn ⌋ For p ∈ [ 0 , 1 ≤ 2 n · h ( p ) k = 0 k � n � ≈ 2 n · h ( p ) ) Proof in a few slides (we already saw that pn Corollary 3 For y ∈ { 0 , 1 } n and p ∈ [ 0 , 1 2 ] , let B p ( y ) = { y ∈ { 0 , 1 } n : | y ′ − y | ≤ pn } . Then | B p ( y ) | = � ⌊ pn ⌋ � n � ≤ 2 n · h ( p ) k = 0 k Very useful estimation. Weaker variants follows by AEP or Stirling, Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 12 / 21

  60. Tightness Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21

  61. Tightness ◮ X ← { 0 , 1 } m , Z = ( Z 1 , . . . , Z n ) where Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21

  62. Tightness ◮ X ← { 0 , 1 } m , Z = ( Z 1 , . . . , Z n ) where Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) X − → f ( X ) − → f ( X ) ⊕ Z − → g ( f ( X ) ⊕ Z ) ◮ ���� ���� � �� � � �� � m bits n bits Y g ( Y ) Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21

  63. Tightness ◮ X ← { 0 , 1 } m , Z = ( Z 1 , . . . , Z n ) where Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) X − → f ( X ) − → f ( X ) ⊕ Z − → g ( f ( X ) ⊕ Z ) ◮ ���� ���� � �� � � �� � m bits n bits Y g ( Y ) ◮ Assuming Pr [ g ( Y ) = X ] ≥ 1 − ε , we show nC p ≥ m ( 1 − ε ) − 1 Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21

  64. Tightness ◮ X ← { 0 , 1 } m , Z = ( Z 1 , . . . , Z n ) where Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) X − → f ( X ) − → f ( X ) ⊕ Z − → g ( f ( X ) ⊕ Z ) ◮ ���� ���� � �� � � �� � m bits n bits Y g ( Y ) ◮ Assuming Pr [ g ( Y ) = X ] ≥ 1 − ε , we show nC p ≥ m ( 1 − ε ) − 1 ◮ Compare to nC p > m ( 1 + ε C p ) in Thm 1 Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21

  65. Tightness ◮ X ← { 0 , 1 } m , Z = ( Z 1 , . . . , Z n ) where Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) X − → f ( X ) − → f ( X ) ⊕ Z − → g ( f ( X ) ⊕ Z ) ◮ ���� ���� � �� � � �� � m bits n bits Y g ( Y ) ◮ Assuming Pr [ g ( Y ) = X ] ≥ 1 − ε , we show nC p ≥ m ( 1 − ε ) − 1 ◮ Compare to nC p > m ( 1 + ε C p ) in Thm 1 ◮ Hence, lim ε → 0 m n = C p Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21

  66. Tightness ◮ X ← { 0 , 1 } m , Z = ( Z 1 , . . . , Z n ) where Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) X − → f ( X ) − → f ( X ) ⊕ Z − → g ( f ( X ) ⊕ Z ) ◮ ���� ���� � �� � � �� � m bits n bits Y g ( Y ) ◮ Assuming Pr [ g ( Y ) = X ] ≥ 1 − ε , we show nC p ≥ m ( 1 − ε ) − 1 ◮ Compare to nC p > m ( 1 + ε C p ) in Thm 1 ◮ Hence, lim ε → 0 m n = C p ◮ By Fano, H ( X | Y ) ≤ h ( ε ) + ε log ( 2 m − 1 ) ≤ 1 + ε m Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21

  67. Tightness ◮ X ← { 0 , 1 } m , Z = ( Z 1 , . . . , Z n ) where Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) X − → f ( X ) − → f ( X ) ⊕ Z − → g ( f ( X ) ⊕ Z ) ◮ ���� ���� � �� � � �� � m bits n bits Y g ( Y ) ◮ Assuming Pr [ g ( Y ) = X ] ≥ 1 − ε , we show nC p ≥ m ( 1 − ε ) − 1 ◮ Compare to nC p > m ( 1 + ε C p ) in Thm 1 ◮ Hence, lim ε → 0 m n = C p ◮ By Fano, H ( X | Y ) ≤ h ( ε ) + ε log ( 2 m − 1 ) ≤ 1 + ε m ◮ I ( X ; Y ) = H ( X ) − H ( X | Y ) ≥ m − ε m − 1 = m ( 1 − ε ) − 1 Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21

  68. Tightness ◮ X ← { 0 , 1 } m , Z = ( Z 1 , . . . , Z n ) where Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) X − → f ( X ) − → f ( X ) ⊕ Z − → g ( f ( X ) ⊕ Z ) ◮ ���� ���� � �� � � �� � m bits n bits Y g ( Y ) ◮ Assuming Pr [ g ( Y ) = X ] ≥ 1 − ε , we show nC p ≥ m ( 1 − ε ) − 1 ◮ Compare to nC p > m ( 1 + ε C p ) in Thm 1 ◮ Hence, lim ε → 0 m n = C p ◮ By Fano, H ( X | Y ) ≤ h ( ε ) + ε log ( 2 m − 1 ) ≤ 1 + ε m ◮ I ( X ; Y ) = H ( X ) − H ( X | Y ) ≥ m − ε m − 1 = m ( 1 − ε ) − 1 ◮ H ( Y | X ) = H ( X , Y ) − H ( X ) = H ( X , Z ) − H ( X ) = H ( Z ) = nh ( p ) Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21

  69. Tightness ◮ X ← { 0 , 1 } m , Z = ( Z 1 , . . . , Z n ) where Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) X − → f ( X ) − → f ( X ) ⊕ Z − → g ( f ( X ) ⊕ Z ) ◮ ���� ���� � �� � � �� � m bits n bits Y g ( Y ) ◮ Assuming Pr [ g ( Y ) = X ] ≥ 1 − ε , we show nC p ≥ m ( 1 − ε ) − 1 ◮ Compare to nC p > m ( 1 + ε C p ) in Thm 1 ◮ Hence, lim ε → 0 m n = C p ◮ By Fano, H ( X | Y ) ≤ h ( ε ) + ε log ( 2 m − 1 ) ≤ 1 + ε m ◮ I ( X ; Y ) = H ( X ) − H ( X | Y ) ≥ m − ε m − 1 = m ( 1 − ε ) − 1 ◮ H ( Y | X ) = H ( X , Y ) − H ( X ) = H ( X , Z ) − H ( X ) = H ( Z ) = nh ( p ) ◮ I ( X ; Y ) = H ( Y ) − H ( Y | X ) = n − nh ( p ) Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21

  70. Tightness ◮ X ← { 0 , 1 } m , Z = ( Z 1 , . . . , Z n ) where Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) X − → f ( X ) − → f ( X ) ⊕ Z − → g ( f ( X ) ⊕ Z ) ◮ ���� ���� � �� � � �� � m bits n bits Y g ( Y ) ◮ Assuming Pr [ g ( Y ) = X ] ≥ 1 − ε , we show nC p ≥ m ( 1 − ε ) − 1 ◮ Compare to nC p > m ( 1 + ε C p ) in Thm 1 ◮ Hence, lim ε → 0 m n = C p ◮ By Fano, H ( X | Y ) ≤ h ( ε ) + ε log ( 2 m − 1 ) ≤ 1 + ε m ◮ I ( X ; Y ) = H ( X ) − H ( X | Y ) ≥ m − ε m − 1 = m ( 1 − ε ) − 1 ◮ H ( Y | X ) = H ( X , Y ) − H ( X ) = H ( X , Z ) − H ( X ) = H ( Z ) = nh ( p ) ◮ I ( X ; Y ) = H ( Y ) − H ( Y | X ) = n − nh ( p ) ◮ Hence, m ( 1 − ε ) ≤ I ( X ; Y ) + 1 = n ( 1 − h ( p )) + 1 = nC p + 1 Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21

  71. Tightness ◮ X ← { 0 , 1 } m , Z = ( Z 1 , . . . , Z n ) where Z 1 , . . . , Z n iid ∼ ( 1 − p , p ) X − → f ( X ) − → f ( X ) ⊕ Z − → g ( f ( X ) ⊕ Z ) ◮ ���� ���� � �� � � �� � m bits n bits Y g ( Y ) ◮ Assuming Pr [ g ( Y ) = X ] ≥ 1 − ε , we show nC p ≥ m ( 1 − ε ) − 1 ◮ Compare to nC p > m ( 1 + ε C p ) in Thm 1 ◮ Hence, lim ε → 0 m n = C p ◮ By Fano, H ( X | Y ) ≤ h ( ε ) + ε log ( 2 m − 1 ) ≤ 1 + ε m ◮ I ( X ; Y ) = H ( X ) − H ( X | Y ) ≥ m − ε m − 1 = m ( 1 − ε ) − 1 ◮ H ( Y | X ) = H ( X , Y ) − H ( X ) = H ( X , Z ) − H ( X ) = H ( Z ) = nh ( p ) ◮ I ( X ; Y ) = H ( Y ) − H ( Y | X ) = n − nh ( p ) ◮ Hence, m ( 1 − ε ) ≤ I ( X ; Y ) + 1 = n ( 1 − h ( p )) + 1 = nC p + 1 ◮ . . . Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21

  72. General communication channel

  73. General communication channel Y = Q ( X ) X Q : [ k ] �→ [ k ] that channel (a probabilistic p 1 , 1 function) 1 1 p 1 , 4 2 2 p i , j = Pr [ Q ( i ) = j ] . p 2 , 4 . . . . . p k , 2 k k Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 14 / 21

  74. General communication channel Y = Q ( X ) X Q : [ k ] �→ [ k ] that channel (a probabilistic p 1 , 1 function) 1 1 p 1 , 4 2 2 p i , j = Pr [ Q ( i ) = j ] . p 2 , 4 . . . . . ◮ x = ( x 1 , . . . , x m ) ∈ { 0 , 1 } m p k , 2 k k Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 14 / 21

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

An isoperimetric inequality for a nonlinear eigenvalue problem Gisella Croce joint work with A.

An isoperimetric inequality for a nonlinear eigenvalue problem Gisella Croce joint work with A.

An isoperimetric inequality for a nonlinear eigenvalue problem Gisella Croce joint work with A. Henrot and G. Pisante PICOF 12 - Ecole Polytechnique, Palaiseau Let be an open bounded subset of R N . v L p ()

665 views • 28 slides
Lecture 7. Outline. 1. Isoperimetric inequality for hypercube. 2. Modular Arithmetic. Clock

Lecture 7. Outline. 1. Isoperimetric inequality for hypercube. 2. Modular Arithmetic. Clock

Lecture 7. Outline. 1. Isoperimetric inequality for hypercube. 2. Modular Arithmetic. Clock Math!!! 3. Inverses for Modular Arithmetic: Greatest Common Divisor. Division!!! 4. Euclids GCD Algorithm. A little tricky here! 1 / 36

556 views • 30 slides
Part V. AWGN Channel Capacity AWGN Capacity Formula; Sphere Packing; Resources in AWGN Channel

Part V. AWGN Channel Capacity AWGN Capacity Formula; Sphere Packing; Resources in AWGN Channel

Part V. AWGN Channel Capacity AWGN Capacity Formula; Sphere Packing; Resources in AWGN Channel 61 Channel coding theorem For every memoryless channel, there is a definite number C (computable) such that: If the data rate R &lt; C ,

910 views • 12 slides
Chapter 7 Channel Capacity Peng-Hua Wang Graduate Inst. of Comm. Engineering National Taipei

Chapter 7 Channel Capacity Peng-Hua Wang Graduate Inst. of Comm. Engineering National Taipei

Chapter 7 Channel Capacity Peng-Hua Wang Graduate Inst. of Comm. Engineering National Taipei University Chapter Outline Chap. 7 Channel Capacity 7.1 Examples of Channel Capacity 7.2 Symmetric Channels 7.3 Properties of Channel Capacity 7.4

1.2k views • 62 slides
Achieving channel capacity ... Shannon says this is possible ... how ? Make use of the gap

Achieving channel capacity ... Shannon says this is possible ... how ? Make use of the gap

Introduction Turbo Principle SISO (Soft Input Soft Output) Example of a product code Achieving channel capacity ... Shannon says this is possible ... how ? Make use of the gap between the source rate and the channel capacity: coding scheme

729 views • 14 slides
Further Properties of Wireless Channel Capacity Fengyou Sun and Yuming Jiang Norwegian University

Further Properties of Wireless Channel Capacity Fengyou Sun and Yuming Jiang Norwegian University

Wireless Channel Capacity Analysis of Cumulative Capacity Analysis of Maximum and Minimum Cumulative Capacity Further Properties of Wireless Channel Capacity Fengyou Sun and Yuming Jiang Norwegian University of Science and Technology (NTNU)

260 views • 22 slides
Compound channel with feedback: Opportunistic capacity and error exponents Aditya Mahajan and

Compound channel with feedback: Opportunistic capacity and error exponents Aditya Mahajan and

Compound channel with feedback: Opportunistic capacity and error exponents Aditya Mahajan and Sekhar Tatikonda Yale University March 17, 2010 CISS Compound channel Channel model | , =

791 views • 57 slides
Lecture 1 Capacity of the Gaussian Channel Basic concepts in information theory: Appendix B

Lecture 1 Capacity of the Gaussian Channel Basic concepts in information theory: Appendix B

Lecture 1 Capacity of the Gaussian Channel Basic concepts in information theory: Appendix B Capacity of the Gaussian channel: Appendix B, Ch. 5.13 Mikael Skoglund, Theoretical Foundations of Wireless 1/23 Entropy and Mutual

307 views • 12 slides
Information Theory Lecture 10 Network Information Theory (CT15); a focus on channel capacity

Information Theory Lecture 10 Network Information Theory (CT15); a focus on channel capacity

Information Theory Lecture 10 Network Information Theory (CT15); a focus on channel capacity results The (two-user) multiple access channel (15.3) The (two-user) broadcast channel (15.6) The relay channel (15.7) Some remarks on

598 views • 13 slides
Previously... Joint typical sequences Covering and Packing Lemmas Channel Coding Theorem

Previously... Joint typical sequences Covering and Packing Lemmas Channel Coding Theorem

Lecture 12 Review Previously... Joint typical sequences Covering and Packing Lemmas Channel Coding Theorem Capacity of Gaussian channel Capacity of additive white Gaussian channel Forward proof of Channel Coding Theorem S. Cheng (OU-Tulsa)

2.44k views • 134 slides
Information Capacity of the BSC Permutation Channel Anuran Makur EECS Department, Massachusetts

Information Capacity of the BSC Permutation Channel Anuran Makur EECS Department, Massachusetts

Information Capacity of the BSC Permutation Channel Anuran Makur EECS Department, Massachusetts Institute of Technology Allerton Conference 2018 A. Makur (MIT) Capacity of BSC Permutation Channel 5 October 2018 1 / 21 Outline Introduction

1.14k views • 100 slides
A General Formula for Channel Capacity 1 Definitions Information variable { 1 , . . .

A General Formula for Channel Capacity 1 Definitions Information variable { 1 , . . .

A General Formula for Channel Capacity 1 Definitions Information variable { 1 , . . . , M } , p ( i ) = Pr( = i ) Channel input X X and output Y Y , finite alphabets Codewords { x N 1 ( i ) : i = 1 , . . . , M } , x n

198 views • 6 slides
Lecture 2: Wireless Physical Lecture 2: Wireless Physical Layer, Channel, and Capacity Layer,

Lecture 2: Wireless Physical Lecture 2: Wireless Physical Layer, Channel, and Capacity Layer,

Lecture 2: Wireless Physical Lecture 2: Wireless Physical Layer, Channel, and Capacity Layer, Channel, and Capacity Mythili Vutukuru CS 653 Spring 2014 Jan 9, Thursday Radio waves and spectrum Physical layer of mobile systems wireless

327 views • 20 slides
Capacity of a Broadcast Channel with Luo, Gohary, Yanikomeroglu Gaussian Jamming and a Friendly

Capacity of a Broadcast Channel with Luo, Gohary, Yanikomeroglu Gaussian Jamming and a Friendly

Capacity of a Broadcast Channel with Gaussian Jamming and a Friendly Eavesdropper 1 Capacity of a Broadcast Channel with Luo, Gohary, Yanikomeroglu Gaussian Jamming and a Friendly Channel Eavesdropper Model Relaying Schemes Kevin

507 views • 18 slides
ECS 289M Lecture 23 May 24, 2006 Measuring Capacity Intuitively, difference between

ECS 289M Lecture 23 May 24, 2006 Measuring Capacity Intuitively, difference between

ECS 289M Lecture 23 May 24, 2006 Measuring Capacity Intuitively, difference between unmodulated, modulated channel Normal uncertainty in channel is 8 bits Attacker modulates channel to send information, reducing uncertainty to 5

611 views • 20 slides
The Approximate Sum Capacity of the Symmetric Gaussian K -User Interference Channel Or Ordentlich

The Approximate Sum Capacity of the Symmetric Gaussian K -User Interference Channel Or Ordentlich

The Approximate Sum Capacity of the Symmetric Gaussian K -User Interference Channel Or Ordentlich Joint work with Uri Erez and Bobak Nazer July 5th, ISIT 2012 MIT, Cambridge, Massachusetts Ordentlich, Erez, Nazer Approx. Sum Capacity of the

746 views • 59 slides
Effect of Administration of Aqueous Extract of Securidaca Longepedunculata Stem Bark on Enzymes of

Effect of Administration of Aqueous Extract of Securidaca Longepedunculata Stem Bark on Enzymes of

Effect of Administration of Aqueous Extract of Securidaca Longepedunculata Stem Bark on Enzymes of the Small Intestine Allozan Induced Diabetic Rat Oluwaferanmi Adefolaju 1, 2, *, Julius Arannilewa 2 , and Ayokanmi Olowo 2 1. Department of

771 views • 8 slides
hEPAtic Study Rilpivirine-TDF-FTC in HIV-HCV Coinfected Patients hEPAtic: Design Study Design:

hEPAtic Study Rilpivirine-TDF-FTC in HIV-HCV Coinfected Patients hEPAtic: Design Study Design:

Rilpivirine-TDF-FTC in HIV-HCV Coinfected Patients hEPAtic Study Rilpivirine-TDF-FTC in HIV-HCV Coinfected Patients hEPAtic: Design Study Design: hEPAtic STUDY Background : Retrospective, case-control study to evaluate the hepatic safety

235 views • 9 slides
Spinning black hole binaries for ET: SNR estimates and parameter estimation calculations Eliu

Spinning black hole binaries for ET: SNR estimates and parameter estimation calculations Eliu

Spinning black hole binaries for ET: SNR estimates and parameter estimation calculations Eliu Huerta, IoA, Cambridge Jonathan Gair, IoA, Cambridge Nikhef, Amsterdam, February 2010 1 Outline Motivation to study spinning black hole binaries

578 views • 20 slides
Homework Assignment: 5 11-721: Grammars and Lexicons 11-721: Grammars and Lexicons Fall 2007

Homework Assignment: 5 11-721: Grammars and Lexicons 11-721: Grammars and Lexicons Fall 2007

Homework Assignment: 5 11-721: Grammars and Lexicons 11-721: Grammars and Lexicons Fall 2007 Task 1 Task 1 S N V P N Pronoun V -Trans N I love DET N -comp S the N -comp C S fact that N V P N BE

399 views • 12 slides
Monolingual Transduction Graham Neubig Site https://phontron.com/class/mtandseq2seq2019/

Monolingual Transduction Graham Neubig Site https://phontron.com/class/mtandseq2seq2019/

CS11-731 Machine Translation and Sequence-to-Sequence Models Monolingual Transduction Graham Neubig Site https://phontron.com/class/mtandseq2seq2019/ (Slides by Graham Neubig and Antonis Anastasopoulos) Examples Summarization

595 views • 15 slides
Agent architectures and hierarchical control Overview: Agents and Robots Agent systems and

Agent architectures and hierarchical control Overview: Agents and Robots Agent systems and

Agent architectures and hierarchical control Overview: Agents and Robots Agent systems and architectures Agent controllers Hierarchical controllers D. Poole and A. Mackworth 2019 c Artificial Intelligence, Lecture 2.1 1 / 12 Example:

607 views • 21 slides
Agent-Based Systems Michael Rovatsos mrovatso@inf.ed.ac.uk Lecture 3 Deductive Reasoning

Agent-Based Systems Michael Rovatsos mrovatso@inf.ed.ac.uk Lecture 3 Deductive Reasoning

Agent-Based Systems Agent-Based Systems Michael Rovatsos mrovatso@inf.ed.ac.uk Lecture 3 Deductive Reasoning Agents 1 / 16 Agent-Based Systems Where are we? Last time . . . Talked about abstract agent architectures Agents with and

339 views • 18 slides
Transfer Theorems Igor Walukiewicz Bordeaux University 1 / 59 Recursion stacks 2 / 59

Transfer Theorems Igor Walukiewicz Bordeaux University 1 / 59 Recursion stacks 2 / 59

Transfer Theorems Igor Walukiewicz Bordeaux University 1 / 59 Recursion stacks 2 / 59 Recursion stacks F x . if x = 0 then 1 else F ( x 1) x . 3 / 59 Recursion stacks F x . if x = 0 then 1 else F ( x 1) x

1.29k views • 59 slides

More recommend