Channel Capacity and Isoperimetric Inequality Iftach Haitner Tel - - PowerPoint PPT Presentation
Application of Information Theory, Lecture 5 Channel Capacity and Isoperimetric Inequality Iftach Haitner Tel Aviv University. November 25, 2014 Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 1 / 21 Part
Tel Aviv University.
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 1 / 21
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 2 / 21
◮ We want to send a message x = (x1, . . . , xm) ∈ {0, 1}m, but the
◮ We want to send a message x = (x1, . . . , xm) ∈ {0, 1}m, but the
◮ Each bit is (independently) flipped w.p. p (e.g., 0.1)
◮ We want to send a message x = (x1, . . . , xm) ∈ {0, 1}m, but the
◮ Each bit is (independently) flipped w.p. p (e.g., 0.1) 1 1 1 − p p p 1 − p
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 3 / 21
◮ We want to send a message x = (x1, . . . , xm) ∈ {0, 1}m, but the
◮ Each bit is (independently) flipped w.p. p (e.g., 0.1) 1 1 1 − p p p 1 − p ◮ (expected) Error rate is p
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 3 / 21
◮ We want to send a message x = (x1, . . . , xm) ∈ {0, 1}m, but the
◮ Each bit is (independently) flipped w.p. p (e.g., 0.1) 1 1 1 − p p p 1 − p ◮ (expected) Error rate is p ◮ Such “channel” is called Binary Symmetric Channel (BSC)
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 3 / 21
◮ We want to send a message x = (x1, . . . , xm) ∈ {0, 1}m, but the
◮ Each bit is (independently) flipped w.p. p (e.g., 0.1) 1 1 1 − p p p 1 − p ◮ (expected) Error rate is p ◮ Such “channel” is called Binary Symmetric Channel (BSC) ◮ When sending m bits, we have ≈ pm errors
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 3 / 21
◮ We want to send a message x = (x1, . . . , xm) ∈ {0, 1}m, but the
◮ Each bit is (independently) flipped w.p. p (e.g., 0.1) 1 1 1 − p p p 1 − p ◮ (expected) Error rate is p ◮ Such “channel” is called Binary Symmetric Channel (BSC) ◮ When sending m bits, we have ≈ pm errors ◮ Can we send bits with smaller error?
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 3 / 21
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 4 / 21
◮ Obvious solution is “error correction codes (ECC)”
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 4 / 21
◮ Obvious solution is “error correction codes (ECC)” ◮ Most simple example: send each bit three times, and take majority
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 4 / 21
◮ Obvious solution is “error correction codes (ECC)” ◮ Most simple example: send each bit three times, and take majority ◮ Error happens if the channel errs at least twice
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 4 / 21
◮ Obvious solution is “error correction codes (ECC)” ◮ Most simple example: send each bit three times, and take majority ◮ Error happens if the channel errs at least twice ◮ For p = 0.1: happens w.p.
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 4 / 21
◮ Obvious solution is “error correction codes (ECC)” ◮ Most simple example: send each bit three times, and take majority ◮ Error happens if the channel errs at least twice ◮ For p = 0.1: happens w.p.
◮ Error rate: .028
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 4 / 21
◮ Obvious solution is “error correction codes (ECC)” ◮ Most simple example: send each bit three times, and take majority ◮ Error happens if the channel errs at least twice ◮ For p = 0.1: happens w.p.
◮ Error rate: .028 ◮ Transmission rate: 1/3 (i.e., # of bits recovered / #of bits transmitted)
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 4 / 21
◮ Obvious solution is “error correction codes (ECC)” ◮ Most simple example: send each bit three times, and take majority ◮ Error happens if the channel errs at least twice ◮ For p = 0.1: happens w.p.
◮ Error rate: .028 ◮ Transmission rate: 1/3 (i.e., # of bits recovered / #of bits transmitted) ◮ We reduced the error rate by reducing the transmission rate.
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 4 / 21
◮ Obvious solution is “error correction codes (ECC)” ◮ Most simple example: send each bit three times, and take majority ◮ Error happens if the channel errs at least twice ◮ For p = 0.1: happens w.p.
◮ Error rate: .028 ◮ Transmission rate: 1/3 (i.e., # of bits recovered / #of bits transmitted) ◮ We reduced the error rate by reducing the transmission rate. ◮ Can we reduce the error rate, without reducing the transmitting rate too
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 4 / 21
◮ Obvious solution is “error correction codes (ECC)” ◮ Most simple example: send each bit three times, and take majority ◮ Error happens if the channel errs at least twice ◮ For p = 0.1: happens w.p.
◮ Error rate: .028 ◮ Transmission rate: 1/3 (i.e., # of bits recovered / #of bits transmitted) ◮ We reduced the error rate by reducing the transmission rate. ◮ Can we reduce the error rate, without reducing the transmitting rate too
◮ Before Shannon it was believed that very small error rate requires very
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 4 / 21
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 5 / 21
◮ Shannon showed that you can reduce the error rate towards 0, without
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 5 / 21
◮ Shannon showed that you can reduce the error rate towards 0, without
◮ For any c < Cp, exists a code with transmission rate c that is correct w.p.
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 5 / 21
◮ Shannon showed that you can reduce the error rate towards 0, without
◮ For any c < Cp, exists a code with transmission rate c that is correct w.p. ◮ Example: for p = .1, Cp > 1
2.
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 5 / 21
◮ Shannon showed that you can reduce the error rate towards 0, without
◮ For any c < Cp, exists a code with transmission rate c that is correct w.p. ◮ Example: for p = .1, Cp > 1
2.
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 5 / 21
◮ Shannon showed that you can reduce the error rate towards 0, without
◮ For any c < Cp, exists a code with transmission rate c that is correct w.p. ◮ Example: for p = .1, Cp > 1
2.
◮ More generally, ∀p ∈ [0, 1]
Cp bits, and x is recovered w.p. close to 1
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 5 / 21
◮ Shannon showed that you can reduce the error rate towards 0, without
◮ For any c < Cp, exists a code with transmission rate c that is correct w.p. ◮ Example: for p = .1, Cp > 1
2.
◮ More generally, ∀p ∈ [0, 1]
Cp bits, and x is recovered w.p. close to 1
◮ Cp might be 0 (i.e., for p = 1
2)
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 5 / 21
◮ Shannon showed that you can reduce the error rate towards 0, without
◮ For any c < Cp, exists a code with transmission rate c that is correct w.p. ◮ Example: for p = .1, Cp > 1
2.
◮ More generally, ∀p ∈ [0, 1]
Cp bits, and x is recovered w.p. close to 1
◮ Cp might be 0 (i.e., for p = 1
2)
◮ A revolution in EE and the whole world
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 5 / 21
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21
◮ Message to send x = (x1, . . . , xm) ∈ {0, 1}m
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21
◮ Message to send x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding scheme: f : {0, 1}m → {0, 1}n
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21
◮ Message to send x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding scheme: f : {0, 1}m → {0, 1}n
◮ Decoding scheme: g : {0, 1}n → {0, 1}m
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21
◮ Message to send x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding scheme: f : {0, 1}m → {0, 1}n
◮ Decoding scheme: g : {0, 1}n → {0, 1}m ◮
m n — transmission rate
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21
◮ Message to send x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding scheme: f : {0, 1}m → {0, 1}n
◮ Decoding scheme: g : {0, 1}n → {0, 1}m ◮
m n — transmission rate
◮ Sender sends f(x) rather than x
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21
◮ Message to send x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding scheme: f : {0, 1}m → {0, 1}n
◮ Decoding scheme: g : {0, 1}n → {0, 1}m ◮
m n — transmission rate
◮ Sender sends f(x) rather than x ◮ Receiver decodes the message by applying g
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21
◮ Message to send x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding scheme: f : {0, 1}m → {0, 1}n
◮ Decoding scheme: g : {0, 1}n → {0, 1}m ◮
m n — transmission rate
◮ Sender sends f(x) rather than x ◮ Receiver decodes the message by applying g ◮
encoding
channel
decoding
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21
◮ Message to send x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding scheme: f : {0, 1}m → {0, 1}n
◮ Decoding scheme: g : {0, 1}n → {0, 1}m ◮
m n — transmission rate
◮ Sender sends f(x) rather than x ◮ Receiver decodes the message by applying g ◮
encoding
channel
decoding
◮ We hope g(f(x) ⊕ Z) = x
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21
◮ Message to send x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding scheme: f : {0, 1}m → {0, 1}n
◮ Decoding scheme: g : {0, 1}n → {0, 1}m ◮
m n — transmission rate
◮ Sender sends f(x) rather than x ◮ Receiver decodes the message by applying g ◮
encoding
channel
decoding
◮ We hope g(f(x) ⊕ Z) = x ◮ ECCs are everywhere
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21
◮ Message to send x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding scheme: f : {0, 1}m → {0, 1}n
◮ Decoding scheme: g : {0, 1}n → {0, 1}m ◮
m n — transmission rate
◮ Sender sends f(x) rather than x ◮ Receiver decodes the message by applying g ◮
encoding
channel
decoding
◮ We hope g(f(x) ⊕ Z) = x ◮ ECCs are everywhere ◮ ECC Vs compression
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 6 / 21
Cp + ε),
z←Z=(Z1,...,Zn) [g(f(x) ⊕ z) = x] ≤ ε
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 7 / 21
Cp + ε),
z←Z=(Z1,...,Zn) [g(f(x) ⊕ z) = x] ≤ ε
◮ Cp = 1 − h(p) — the channel capacity
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 7 / 21
Cp + ε),
z←Z=(Z1,...,Zn) [g(f(x) ⊕ z) = x] ≤ ε
◮ Cp = 1 − h(p) — the channel capacity
2
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 7 / 21
Cp + ε),
z←Z=(Z1,...,Zn) [g(f(x) ⊕ z) = x] ≤ ε
◮ Cp = 1 − h(p) — the channel capacity
2
5
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 7 / 21
Cp + ε),
z←Z=(Z1,...,Zn) [g(f(x) ⊕ z) = x] ≤ ε
◮ Cp = 1 − h(p) — the channel capacity
2
5
◮ Tight theorem
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 7 / 21
Cp + ε),
z←Z=(Z1,...,Zn) [g(f(x) ⊕ z) = x] ≤ ε
◮ Cp = 1 − h(p) — the channel capacity
2
5
◮ Tight theorem ◮ We prove a weaker variant that holds w.h.p. over x ← {0, 1}m
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 7 / 21
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 8 / 21
◮ For y = (y1, . . . , yn) ∈ {0, 1}n, let |y| =
i yi — Hamming weight of y
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 8 / 21
◮ For y = (y1, . . . , yn) ∈ {0, 1}n, let |y| =
i yi — Hamming weight of y
◮ |y − y′| = |y ⊕ y′| — Hamming distance of y from y′; # of places differ.
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 8 / 21
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21
◮ Fix p ∈ [0, 1
2) and ε > 0, and let m > mε and n ≥ m( 1 Cp + ε), for mε to be
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21
◮ Fix p ∈ [0, 1
2) and ε > 0, and let m > mε and n ≥ m( 1 Cp + ε), for mε to be
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21
◮ Fix p ∈ [0, 1
2) and ε > 0, and let m > mε and n ≥ m( 1 Cp + ε), for mε to be
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21
◮ Fix p ∈ [0, 1
2) and ε > 0, and let m > mε and n ≥ m( 1 Cp + ε), for mε to be
◮ We show ∃ f : {0, 1}m → {0, 1}n and g : {0, 1}n → {0, 1}m, s.t.
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21
◮ Fix p ∈ [0, 1
2) and ε > 0, and let m > mε and n ≥ m( 1 Cp + ε), for mε to be
◮ We show ∃ f : {0, 1}m → {0, 1}n and g : {0, 1}n → {0, 1}m, s.t.
◮ g(y) returns argminx′∈{0,1}m |y − f(x′)|
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21
◮ Fix p ∈ [0, 1
2) and ε > 0, and let m > mε and n ≥ m( 1 Cp + ε), for mε to be
◮ We show ∃ f : {0, 1}m → {0, 1}n and g : {0, 1}n → {0, 1}m, s.t.
◮ g(y) returns argminx′∈{0,1}m |y − f(x′)| ◮ So it all boils down to finding f s.t.
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21
◮ Fix p ∈ [0, 1
2) and ε > 0, and let m > mε and n ≥ m( 1 Cp + ε), for mε to be
◮ We show ∃ f : {0, 1}m → {0, 1}n and g : {0, 1}n → {0, 1}m, s.t.
◮ g(y) returns argminx′∈{0,1}m |y − f(x′)| ◮ So it all boils down to finding f s.t.
◮
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21
◮ Fix p ∈ [0, 1
2) and ε > 0, and let m > mε and n ≥ m( 1 Cp + ε), for mε to be
◮ We show ∃ f : {0, 1}m → {0, 1}n and g : {0, 1}n → {0, 1}m, s.t.
◮ g(y) returns argminx′∈{0,1}m |y − f(x′)| ◮ So it all boils down to finding f s.t.
◮
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21
◮ Fix p ∈ [0, 1
2) and ε > 0, and let m > mε and n ≥ m( 1 Cp + ε), for mε to be
◮ We show ∃ f : {0, 1}m → {0, 1}n and g : {0, 1}n → {0, 1}m, s.t.
◮ g(y) returns argminx′∈{0,1}m |y − f(x′)| ◮ So it all boils down to finding f s.t.
◮ Idea: for p′ > p to be determined later, find f
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21
◮ Fix p ∈ [0, 1
2) and ε > 0, and let m > mε and n ≥ m( 1 Cp + ε), for mε to be
◮ We show ∃ f : {0, 1}m → {0, 1}n and g : {0, 1}n → {0, 1}m, s.t.
◮ g(y) returns argminx′∈{0,1}m |y − f(x′)| ◮ So it all boils down to finding f s.t.
◮ Idea: for p′ > p to be determined later, find f
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21
◮ Fix p ∈ [0, 1
2) and ε > 0, and let m > mε and n ≥ m( 1 Cp + ε), for mε to be
◮ We show ∃ f : {0, 1}m → {0, 1}n and g : {0, 1}n → {0, 1}m, s.t.
◮ g(y) returns argminx′∈{0,1}m |y − f(x′)| ◮ So it all boils down to finding f s.t.
◮ Idea: for p′ > p to be determined later, find f
◮ We choose f uniformly at random (what does it mean?)
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21
◮ Fix p ∈ [0, 1
2) and ε > 0, and let m > mε and n ≥ m( 1 Cp + ε), for mε to be
◮ We show ∃ f : {0, 1}m → {0, 1}n and g : {0, 1}n → {0, 1}m, s.t.
◮ g(y) returns argminx′∈{0,1}m |y − f(x′)| ◮ So it all boils down to finding f s.t.
◮ Idea: for p′ > p to be determined later, find f
◮ We choose f uniformly at random (what does it mean?) ◮ Non-constructive proof
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21
◮ Fix p ∈ [0, 1
2) and ε > 0, and let m > mε and n ≥ m( 1 Cp + ε), for mε to be
◮ We show ∃ f : {0, 1}m → {0, 1}n and g : {0, 1}n → {0, 1}m, s.t.
◮ g(y) returns argminx′∈{0,1}m |y − f(x′)| ◮ So it all boils down to finding f s.t.
◮ Idea: for p′ > p to be determined later, find f
◮ We choose f uniformly at random (what does it mean?) ◮ Non-constructive proof ◮ Probabilistic method
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 9 / 21
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21
◮ Fix p′ > p such that
1 Cp′ − 1 Cp ≤ ε 2
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21
◮ Fix p′ > p such that
1 Cp′ − 1 Cp ≤ ε 2
◮ For y ∈ {0, 1}n, let Bp′(y) = {y ∈ {0, 1}n : |y′ − y| ≤ p′n}
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21
◮ Fix p′ > p such that
1 Cp′ − 1 Cp ≤ ε 2
◮ For y ∈ {0, 1}n, let Bp′(y) = {y ∈ {0, 1}n : |y′ − y| ≤ p′n}
2
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21
◮ Fix p′ > p such that
1 Cp′ − 1 Cp ≤ ε 2
◮ For y ∈ {0, 1}n, let Bp′(y) = {y ∈ {0, 1}n : |y′ − y| ≤ p′n}
2
◮ Fact (proved later): b(p′) = |Bp′(y)| ≤ 2n·h(p′)
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21
◮ Fix p′ > p such that
1 Cp′ − 1 Cp ≤ ε 2
◮ For y ∈ {0, 1}n, let Bp′(y) = {y ∈ {0, 1}n : |y′ − y| ≤ p′n}
2
◮ Fact (proved later): b(p′) = |Bp′(y)| ≤ 2n·h(p′)
2n
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21
◮ Fix p′ > p such that
1 Cp′ − 1 Cp ≤ ε 2
◮ For y ∈ {0, 1}n, let Bp′(y) = {y ∈ {0, 1}n : |y′ − y| ≤ p′n}
2
◮ Fact (proved later): b(p′) = |Bp′(y)| ≤ 2n·h(p′)
2n
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21
◮ Fix p′ > p such that
1 Cp′ − 1 Cp ≤ ε 2
◮ For y ∈ {0, 1}n, let Bp′(y) = {y ∈ {0, 1}n : |y′ − y| ≤ p′n}
2
◮ Fact (proved later): b(p′) = |Bp′(y)| ≤ 2n·h(p′)
2n
2n
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21
◮ Fix p′ > p such that
1 Cp′ − 1 Cp ≤ ε 2
◮ For y ∈ {0, 1}n, let Bp′(y) = {y ∈ {0, 1}n : |y′ − y| ≤ p′n}
2
◮ Fact (proved later): b(p′) = |Bp′(y)| ≤ 2n·h(p′)
2n
2n
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21
◮ Fix p′ > p such that
1 Cp′ − 1 Cp ≤ ε 2
◮ For y ∈ {0, 1}n, let Bp′(y) = {y ∈ {0, 1}n : |y′ − y| ≤ p′n}
2
◮ Fact (proved later): b(p′) = |Bp′(y)| ≤ 2n·h(p′)
2n
2n
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21
◮ Fix p′ > p such that
1 Cp′ − 1 Cp ≤ ε 2
◮ For y ∈ {0, 1}n, let Bp′(y) = {y ∈ {0, 1}n : |y′ − y| ≤ p′n}
2
◮ Fact (proved later): b(p′) = |Bp′(y)| ≤ 2n·h(p′)
2n
2n
2, for n ≥ m Cp′ − log ε + 1 = m( 1 Cp′ + 1−log ε m
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21
◮ Fix p′ > p such that
1 Cp′ − 1 Cp ≤ ε 2
◮ For y ∈ {0, 1}n, let Bp′(y) = {y ∈ {0, 1}n : |y′ − y| ≤ p′n}
2
◮ Fact (proved later): b(p′) = |Bp′(y)| ≤ 2n·h(p′)
2n
2n
2, for n ≥ m Cp′ − log ε + 1 = m( 1 Cp′ + 1−log ε m
2, for m ≥ m′ = 2(1−log ε) ε
Cp + ε 2 + 1−log ε m
Cp + ε)
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21
◮ Fix p′ > p such that
1 Cp′ − 1 Cp ≤ ε 2
◮ For y ∈ {0, 1}n, let Bp′(y) = {y ∈ {0, 1}n : |y′ − y| ≤ p′n}
2
◮ Fact (proved later): b(p′) = |Bp′(y)| ≤ 2n·h(p′)
2n
2n
2, for n ≥ m Cp′ − log ε + 1 = m( 1 Cp′ + 1−log ε m
2, for m ≥ m′ = 2(1−log ε) ε
Cp + ε 2 + 1−log ε m
Cp + ε)
◮ Hence, for m > mε = max{m′, n′} and n > m( 1
Cp + ε), it holds that
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 10 / 21
◮ Let X ← {0, 1} , Z ∼ (1 − p, p) and Y = X ⊕ Z
◮ Let X ← {0, 1} , Z ∼ (1 − p, p) and Y = X ⊕ Z
1 1 1 − p p p 1 − p
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 11 / 21
◮ Let X ← {0, 1} , Z ∼ (1 − p, p) and Y = X ⊕ Z
1 1 1 − p p p 1 − p ◮ I(X; Y) = H(Y) − H(Y|X) = H(Y) − H(Z) = 1 − h(p) = Cp
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 11 / 21
◮ Let X ← {0, 1} , Z ∼ (1 − p, p) and Y = X ⊕ Z
1 1 1 − p p p 1 − p ◮ I(X; Y) = H(Y) − H(Y|X) = H(Y) − H(Z) = 1 − h(p) = Cp ◮ Received bit “gives" Cp information about transmitted bit
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 11 / 21
◮ Let X ← {0, 1} , Z ∼ (1 − p, p) and Y = X ⊕ Z
1 1 1 − p p p 1 − p ◮ I(X; Y) = H(Y) − H(Y|X) = H(Y) − H(Z) = 1 − h(p) = Cp ◮ Received bit “gives" Cp information about transmitted bit ◮ Hence, to recover m bits, we need to send at least m ·
1 Cp bits
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 11 / 21
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 12 / 21
2] and n ∈ N: it holds that ⌊pn⌋ k=0
k
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 12 / 21
2] and n ∈ N: it holds that ⌊pn⌋ k=0
k
pn
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 12 / 21
2] and n ∈ N: it holds that ⌊pn⌋ k=0
k
pn
2], let Bp(y) = {y ∈ {0, 1}n : |y′ − y| ≤ pn}. Then
k=0
k
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 12 / 21
2] and n ∈ N: it holds that ⌊pn⌋ k=0
k
pn
2], let Bp(y) = {y ∈ {0, 1}n : |y′ − y| ≤ pn}. Then
k=0
k
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 12 / 21
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21
◮ X ← {0, 1}m, Z = (Z1, . . . , Zn) where Z1, . . . , Zn iid ∼ (1 − p, p)
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21
◮ X ← {0, 1}m, Z = (Z1, . . . , Zn) where Z1, . . . , Zn iid ∼ (1 − p, p) ◮
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21
◮ X ← {0, 1}m, Z = (Z1, . . . , Zn) where Z1, . . . , Zn iid ∼ (1 − p, p) ◮
◮ Assuming Pr [g(Y) = X] ≥ 1 − ε, we show nCp ≥ m(1 − ε) − 1
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21
◮ X ← {0, 1}m, Z = (Z1, . . . , Zn) where Z1, . . . , Zn iid ∼ (1 − p, p) ◮
◮ Assuming Pr [g(Y) = X] ≥ 1 − ε, we show nCp ≥ m(1 − ε) − 1 ◮ Compare to nCp > m(1 + εCp) in Thm 1
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21
◮ X ← {0, 1}m, Z = (Z1, . . . , Zn) where Z1, . . . , Zn iid ∼ (1 − p, p) ◮
◮ Assuming Pr [g(Y) = X] ≥ 1 − ε, we show nCp ≥ m(1 − ε) − 1 ◮ Compare to nCp > m(1 + εCp) in Thm 1 ◮ Hence, limε→0 m
n = Cp
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21
◮ X ← {0, 1}m, Z = (Z1, . . . , Zn) where Z1, . . . , Zn iid ∼ (1 − p, p) ◮
◮ Assuming Pr [g(Y) = X] ≥ 1 − ε, we show nCp ≥ m(1 − ε) − 1 ◮ Compare to nCp > m(1 + εCp) in Thm 1 ◮ Hence, limε→0 m
n = Cp
◮ By Fano, H(X|Y) ≤ h(ε) + ε log(2m − 1) ≤ 1 + εm
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21
◮ X ← {0, 1}m, Z = (Z1, . . . , Zn) where Z1, . . . , Zn iid ∼ (1 − p, p) ◮
◮ Assuming Pr [g(Y) = X] ≥ 1 − ε, we show nCp ≥ m(1 − ε) − 1 ◮ Compare to nCp > m(1 + εCp) in Thm 1 ◮ Hence, limε→0 m
n = Cp
◮ By Fano, H(X|Y) ≤ h(ε) + ε log(2m − 1) ≤ 1 + εm ◮ I(X; Y) = H(X) − H(X|Y) ≥ m − εm − 1 = m(1 − ε) − 1
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21
◮ X ← {0, 1}m, Z = (Z1, . . . , Zn) where Z1, . . . , Zn iid ∼ (1 − p, p) ◮
◮ Assuming Pr [g(Y) = X] ≥ 1 − ε, we show nCp ≥ m(1 − ε) − 1 ◮ Compare to nCp > m(1 + εCp) in Thm 1 ◮ Hence, limε→0 m
n = Cp
◮ By Fano, H(X|Y) ≤ h(ε) + ε log(2m − 1) ≤ 1 + εm ◮ I(X; Y) = H(X) − H(X|Y) ≥ m − εm − 1 = m(1 − ε) − 1 ◮ H(Y|X) = H(X, Y) − H(X) = H(X, Z) − H(X) = H(Z) = nh(p)
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21
◮ X ← {0, 1}m, Z = (Z1, . . . , Zn) where Z1, . . . , Zn iid ∼ (1 − p, p) ◮
◮ Assuming Pr [g(Y) = X] ≥ 1 − ε, we show nCp ≥ m(1 − ε) − 1 ◮ Compare to nCp > m(1 + εCp) in Thm 1 ◮ Hence, limε→0 m
n = Cp
◮ By Fano, H(X|Y) ≤ h(ε) + ε log(2m − 1) ≤ 1 + εm ◮ I(X; Y) = H(X) − H(X|Y) ≥ m − εm − 1 = m(1 − ε) − 1 ◮ H(Y|X) = H(X, Y) − H(X) = H(X, Z) − H(X) = H(Z) = nh(p) ◮ I(X; Y) = H(Y) − H(Y|X) = n − nh(p)
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21
◮ X ← {0, 1}m, Z = (Z1, . . . , Zn) where Z1, . . . , Zn iid ∼ (1 − p, p) ◮
◮ Assuming Pr [g(Y) = X] ≥ 1 − ε, we show nCp ≥ m(1 − ε) − 1 ◮ Compare to nCp > m(1 + εCp) in Thm 1 ◮ Hence, limε→0 m
n = Cp
◮ By Fano, H(X|Y) ≤ h(ε) + ε log(2m − 1) ≤ 1 + εm ◮ I(X; Y) = H(X) − H(X|Y) ≥ m − εm − 1 = m(1 − ε) − 1 ◮ H(Y|X) = H(X, Y) − H(X) = H(X, Z) − H(X) = H(Z) = nh(p) ◮ I(X; Y) = H(Y) − H(Y|X) = n − nh(p) ◮ Hence, m(1 − ε) ≤ I(X; Y) + 1 = n(1 − h(p)) + 1 = nCp + 1
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21
◮ X ← {0, 1}m, Z = (Z1, . . . , Zn) where Z1, . . . , Zn iid ∼ (1 − p, p) ◮
◮ Assuming Pr [g(Y) = X] ≥ 1 − ε, we show nCp ≥ m(1 − ε) − 1 ◮ Compare to nCp > m(1 + εCp) in Thm 1 ◮ Hence, limε→0 m
n = Cp
◮ By Fano, H(X|Y) ≤ h(ε) + ε log(2m − 1) ≤ 1 + εm ◮ I(X; Y) = H(X) − H(X|Y) ≥ m − εm − 1 = m(1 − ε) − 1 ◮ H(Y|X) = H(X, Y) − H(X) = H(X, Z) − H(X) = H(Z) = nh(p) ◮ I(X; Y) = H(Y) − H(Y|X) = n − nh(p) ◮ Hence, m(1 − ε) ≤ I(X; Y) + 1 = n(1 − h(p)) + 1 = nCp + 1 ◮ . . .
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 13 / 21
1 2 . . . k 1 2 . . . k p1,1 p1,4 p2,4 pk,2
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 14 / 21
1 2 . . . k 1 2 . . . k p1,1 p1,4 p2,4 pk,2 ◮ x = (x1, . . . , xm) ∈ {0, 1}m
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 14 / 21
1 2 . . . k 1 2 . . . k p1,1 p1,4 p2,4 pk,2 ◮ x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding function f : {0, 1}m → {1, . . . , k}n
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 14 / 21
1 2 . . . k 1 2 . . . k p1,1 p1,4 p2,4 pk,2 ◮ x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding function f : {0, 1}m → {1, . . . , k}n ◮ Decoding function g{1, . . . , k}n → {0, 1}m
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 14 / 21
1 2 . . . k 1 2 . . . k p1,1 p1,4 p2,4 pk,2 ◮ x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding function f : {0, 1}m → {1, . . . , k}n ◮ Decoding function g{1, . . . , k}n → {0, 1}m ◮ x
encoding
channel
decoding
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 14 / 21
1 2 . . . k 1 2 . . . k p1,1 p1,4 p2,4 pk,2 ◮ x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding function f : {0, 1}m → {1, . . . , k}n ◮ Decoding function g{1, . . . , k}n → {0, 1}m ◮ x
encoding
channel
decoding
◮ We hope for g(Q(f(x))) = x
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 14 / 21
1 2 . . . k 1 2 . . . k p1,1 p1,4 p2,4 pk,2 ◮ x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding function f : {0, 1}m → {1, . . . , k}n ◮ Decoding function g{1, . . . , k}n → {0, 1}m ◮ x
encoding
channel
decoding
◮ We hope for g(Q(f(x))) = x ◮ Channel capacity CQ = maxX I(X; Y)
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 14 / 21
1 2 . . . k 1 2 . . . k p1,1 p1,4 p2,4 pk,2 ◮ x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding function f : {0, 1}m → {1, . . . , k}n ◮ Decoding function g{1, . . . , k}n → {0, 1}m ◮ x
encoding
channel
decoding
◮ We hope for g(Q(f(x))) = x ◮ Channel capacity CQ = maxX I(X; Y) ◮ The maximal information Y gives on X
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 14 / 21
1 2 . . . k 1 2 . . . k p1,1 p1,4 p2,4 pk,2 ◮ x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding function f : {0, 1}m → {1, . . . , k}n ◮ Decoding function g{1, . . . , k}n → {0, 1}m ◮ x
encoding
channel
decoding
◮ We hope for g(Q(f(x))) = x ◮ Channel capacity CQ = maxX I(X; Y) ◮ The maximal information Y gives on X ◮ Shannon theorem: ∀Q and ∀ε > 0, ∃mε: ∀m > mε and ∀n > m( 1
CQ + ε) :
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 14 / 21
1 2 . . . k 1 2 . . . k p1,1 p1,4 p2,4 pk,2 ◮ x = (x1, . . . , xm) ∈ {0, 1}m ◮ Encoding function f : {0, 1}m → {1, . . . , k}n ◮ Decoding function g{1, . . . , k}n → {0, 1}m ◮ x
encoding
channel
decoding
◮ We hope for g(Q(f(x))) = x ◮ Channel capacity CQ = maxX I(X; Y) ◮ The maximal information Y gives on X ◮ Shannon theorem: ∀Q and ∀ε > 0, ∃mε: ∀m > mε and ∀n > m( 1
CQ + ε) :
◮ Proof: similar lines to the binary case, but more subtle distribution for f
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 14 / 21
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 15 / 21
◮ Tight result
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 15 / 21
◮ Tight result ◮ Non-constructive
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 15 / 21
◮ Tight result ◮ Non-constructive ◮ Coding theory: design explicit (and efficient) code achieving the same
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 15 / 21
◮ Tight result ◮ Non-constructive ◮ Coding theory: design explicit (and efficient) code achieving the same
◮ Application: faulty communication, storage
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 15 / 21
◮ Tight result ◮ Non-constructive ◮ Coding theory: design explicit (and efficient) code achieving the same
◮ Application: faulty communication, storage ◮ Combination of data compression and ECC
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 15 / 21
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 16 / 21
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 17 / 21
◮ 2n people, m = 3n movies.
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 17 / 21
◮ 2n people, m = 3n movies. ◮ Every pair of movies was seen by at least 90% of the people
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 17 / 21
◮ 2n people, m = 3n movies. ◮ Every pair of movies was seen by at least 90% of the people ◮ Claim: there exist two people who saw exactly the same set of movies
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 17 / 21
◮ 2n people, m = 3n movies. ◮ Every pair of movies was seen by at least 90% of the people ◮ Claim: there exist two people who saw exactly the same set of movies ◮ X ← [2n] — a random person
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 17 / 21
◮ 2n people, m = 3n movies. ◮ Every pair of movies was seen by at least 90% of the people ◮ Claim: there exist two people who saw exactly the same set of movies ◮ X ← [2n] — a random person ◮ gi(x) =
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 17 / 21
◮ 2n people, m = 3n movies. ◮ Every pair of movies was seen by at least 90% of the people ◮ Claim: there exist two people who saw exactly the same set of movies ◮ X ← [2n] — a random person ◮ gi(x) =
◮ Yi = gi(X)
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 17 / 21
◮ 2n people, m = 3n movies. ◮ Every pair of movies was seen by at least 90% of the people ◮ Claim: there exist two people who saw exactly the same set of movies ◮ X ← [2n] — a random person ◮ gi(x) =
◮ Yi = gi(X) ◮ ∀i, j: H(Yi, Yj) ≤ H(0.9, 0.1
3 , 0.1 3 , 0.1 3 ) ≤ 2 3
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 17 / 21
◮ 2n people, m = 3n movies. ◮ Every pair of movies was seen by at least 90% of the people ◮ Claim: there exist two people who saw exactly the same set of movies ◮ X ← [2n] — a random person ◮ gi(x) =
◮ Yi = gi(X) ◮ ∀i, j: H(Yi, Yj) ≤ H(0.9, 0.1
3 , 0.1 3 , 0.1 3 ) ≤ 2 3
◮ H(Y = (Y1, . . . , Ym)) ≤ m/2
i
2 )
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 17 / 21
◮ 2n people, m = 3n movies. ◮ Every pair of movies was seen by at least 90% of the people ◮ Claim: there exist two people who saw exactly the same set of movies ◮ X ← [2n] — a random person ◮ gi(x) =
◮ Yi = gi(X) ◮ ∀i, j: H(Yi, Yj) ≤ H(0.9, 0.1
3 , 0.1 3 , 0.1 3 ) ≤ 2 3
◮ H(Y = (Y1, . . . , Ym)) ≤ m/2
i
2 )
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 17 / 21
◮ 2n people, m = 3n movies. ◮ Every pair of movies was seen by at least 90% of the people ◮ Claim: there exist two people who saw exactly the same set of movies ◮ X ← [2n] — a random person ◮ gi(x) =
◮ Yi = gi(X) ◮ ∀i, j: H(Yi, Yj) ≤ H(0.9, 0.1
3 , 0.1 3 , 0.1 3 ) ≤ 2 3
◮ H(Y = (Y1, . . . , Ym)) ≤ m/2
i
2 ) < 3n
2 · 2 3
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 17 / 21
◮ 2n people, m = 3n movies. ◮ Every pair of movies was seen by at least 90% of the people ◮ Claim: there exist two people who saw exactly the same set of movies ◮ X ← [2n] — a random person ◮ gi(x) =
◮ Yi = gi(X) ◮ ∀i, j: H(Yi, Yj) ≤ H(0.9, 0.1
3 , 0.1 3 , 0.1 3 ) ≤ 2 3
◮ H(Y = (Y1, . . . , Ym)) ≤ m/2
i
2 ) < 3n
2 · 2 3 = n
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 17 / 21
◮ 2n people, m = 3n movies. ◮ Every pair of movies was seen by at least 90% of the people ◮ Claim: there exist two people who saw exactly the same set of movies ◮ X ← [2n] — a random person ◮ gi(x) =
◮ Yi = gi(X) ◮ ∀i, j: H(Yi, Yj) ≤ H(0.9, 0.1
3 , 0.1 3 , 0.1 3 ) ≤ 2 3
◮ H(Y = (Y1, . . . , Ym)) ≤ m/2
i
2 ) < 3n
2 · 2 3 = n = H(X)
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 17 / 21
◮ 2n people, m = 3n movies. ◮ Every pair of movies was seen by at least 90% of the people ◮ Claim: there exist two people who saw exactly the same set of movies ◮ X ← [2n] — a random person ◮ gi(x) =
◮ Yi = gi(X) ◮ ∀i, j: H(Yi, Yj) ≤ H(0.9, 0.1
3 , 0.1 3 , 0.1 3 ) ≤ 2 3
◮ H(Y = (Y1, . . . , Ym)) ≤ m/2
i
2 ) < 3n
2 · 2 3 = n = H(X)
◮ Hence, X is not determined by Y
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 17 / 21
i H(Xi) so useful?
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 18 / 21
i H(Xi) so useful? ◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 18 / 21
i H(Xi) so useful? ◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S ◮ log |S| = H(X) ≤
i(Xi) implies |S| ≤ 2
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 18 / 21
i H(Xi) so useful? ◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S ◮ log |S| = H(X) ≤
i(Xi) implies |S| ≤ 2
◮ If
i H(Xi) is small, then S is small
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 18 / 21
i H(Xi) so useful? ◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S ◮ log |S| = H(X) ≤
i(Xi) implies |S| ≤ 2
◮ If
i H(Xi) is small, then S is small
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 18 / 21
i H(Xi) so useful? ◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S ◮ log |S| = H(X) ≤
i(Xi) implies |S| ≤ 2
◮ If
i H(Xi) is small, then S is small
◮ S is large implies
i H(Xi) is large, hence most Xi are almost balanced
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 18 / 21
i H(Xi) so useful? ◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S ◮ log |S| = H(X) ≤
i(Xi) implies |S| ≤ 2
◮ If
i H(Xi) is small, then S is small
◮ S is large implies
i H(Xi) is large, hence most Xi are almost balanced
◮ |S| ≥ 2n/2 implies Ei←[n] [H(Xi)] ≥ 1 − 1
n
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 18 / 21
i H(Xi) so useful? ◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S ◮ log |S| = H(X) ≤
i(Xi) implies |S| ≤ 2
◮ If
i H(Xi) is small, then S is small
◮ S is large implies
i H(Xi) is large, hence most Xi are almost balanced
◮ |S| ≥ 2n/2 implies Ei←[n] [H(Xi)] ≥ 1 − 1
n
◮ Most Xi are close to uniform
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 18 / 21
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 19 / 21
◮ p ≤ 1
2; S = {(a1, . . . , an) ∈ {0, 1}n : i ai ≤ pn}
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 19 / 21
◮ p ≤ 1
2; S = {(a1, . . . , an) ∈ {0, 1}n : i ai ≤ pn}
◮ |S| = ⌊pn⌋
k=0
k
Application of Information Theory, Lecture 5 November 25, 2014 19 / 21
◮ p ≤ 1
2; S = {(a1, . . . , an) ∈ {0, 1}n : i ai ≤ pn}
◮ |S| = ⌊pn⌋
k=0
k
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 19 / 21
◮ p ≤ 1
2; S = {(a1, . . . , an) ∈ {0, 1}n : i ai ≤ pn}
◮ |S| = ⌊pn⌋
k=0
k
◮
i Xi ≤ pn =
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 19 / 21
◮ p ≤ 1
2; S = {(a1, . . . , an) ∈ {0, 1}n : i ai ≤ pn}
◮ |S| = ⌊pn⌋
k=0
k
◮
i Xi ≤ pn =
◮ ∀i, j: Pr[Xi = 1] = Pr[Xj = 1] ≤ p
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 19 / 21
◮ p ≤ 1
2; S = {(a1, . . . , an) ∈ {0, 1}n : i ai ≤ pn}
◮ |S| = ⌊pn⌋
k=0
k
◮
i Xi ≤ pn =
◮ ∀i, j: Pr[Xi = 1] = Pr[Xj = 1] ≤ p
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 19 / 21
◮ p ≤ 1
2; S = {(a1, . . . , an) ∈ {0, 1}n : i ai ≤ pn}
◮ |S| = ⌊pn⌋
k=0
k
◮
i Xi ≤ pn =
◮ ∀i, j: Pr[Xi = 1] = Pr[Xj = 1] ≤ p
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 19 / 21
◮ p ≤ 1
2; S = {(a1, . . . , an) ∈ {0, 1}n : i ai ≤ pn}
◮ |S| = ⌊pn⌋
k=0
k
◮
i Xi ≤ pn =
◮ ∀i, j: Pr[Xi = 1] = Pr[Xj = 1] ≤ p
k=0
k
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 19 / 21
◮ p ≤ 1
2; S = {(a1, . . . , an) ∈ {0, 1}n : i ai ≤ pn}
◮ |S| = ⌊pn⌋
k=0
k
◮
i Xi ≤ pn =
◮ ∀i, j: Pr[Xi = 1] = Pr[Xj = 1] ≤ p
k=0
k
◮ . . .
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 19 / 21
◮ p ≤ 1
2; S = {(a1, . . . , an) ∈ {0, 1}n : i ai ≤ pn}
◮ |S| = ⌊pn⌋
k=0
k
◮
i Xi ≤ pn =
◮ ∀i, j: Pr[Xi = 1] = Pr[Xj = 1] ≤ p
k=0
k
◮ . . .
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 19 / 21
◮ p ≤ 1
2; S = {(a1, . . . , an) ∈ {0, 1}n : i ai ≤ pn}
◮ |S| = ⌊pn⌋
k=0
k
◮
i Xi ≤ pn =
◮ ∀i, j: Pr[Xi = 1] = Pr[Xj = 1] ≤ p
k=0
k
◮ . . .
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 19 / 21
◮ p ≤ 1
2; S = {(a1, . . . , an) ∈ {0, 1}n : i ai ≤ pn}
◮ |S| = ⌊pn⌋
k=0
k
◮
i Xi ≤ pn =
◮ ∀i, j: Pr[Xi = 1] = Pr[Xj = 1] ≤ p
k=0
k
◮ . . .
◮ X1, . . . , Xn iid uniform bits (i.e., ∼ ( 1
2, 1 2))
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 19 / 21
◮ p ≤ 1
2; S = {(a1, . . . , an) ∈ {0, 1}n : i ai ≤ pn}
◮ |S| = ⌊pn⌋
k=0
k
◮
i Xi ≤ pn =
◮ ∀i, j: Pr[Xi = 1] = Pr[Xj = 1] ≤ p
k=0
k
◮ . . .
◮ X1, . . . , Xn iid uniform bits (i.e., ∼ ( 1
2, 1 2))
◮ Pr [
i Xi ≤ pn] = Pr [(X1, . . . , Xn) ∈ S] ≤ 2nh(p) · 2−n = 2−n(1−h(p))
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 19 / 21
◮ p ≤ 1
2; S = {(a1, . . . , an) ∈ {0, 1}n : i ai ≤ pn}
◮ |S| = ⌊pn⌋
k=0
k
◮
i Xi ≤ pn =
◮ ∀i, j: Pr[Xi = 1] = Pr[Xj = 1] ≤ p
k=0
k
◮ . . .
◮ X1, . . . , Xn iid uniform bits (i.e., ∼ ( 1
2, 1 2))
◮ Pr [
i Xi ≤ pn] = Pr [(X1, . . . , Xn) ∈ S] ≤ 2nh(p) · 2−n = 2−n(1−h(p))
◮ Very useful inequality. No Chernoff, just IT
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 19 / 21
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
2 · |S| · log |S|
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
2 · |S| · log |S|
◮ Equality if S is “face" : S = {(x, y): y ∈ {0, 1}d} for some x ∈ {0, 1}n−d
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
2 · |S| · log |S|
◮ Equality if S is “face" : S = {(x, y): y ∈ {0, 1}d} for some x ∈ {0, 1}n−d ◮ Example: S is a face of the 3-dimensional cube
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
2 · |S| · log |S|
◮ Equality if S is “face" : S = {(x, y): y ∈ {0, 1}d} for some x ∈ {0, 1}n−d ◮ Example: S is a face of the 3-dimensional cube
2 · 4 · log 4 = 4
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
2 · |S| · log |S|
◮ Equality if S is “face" : S = {(x, y): y ∈ {0, 1}d} for some x ∈ {0, 1}n−d ◮ Example: S is a face of the 3-dimensional cube
2 · 4 · log 4 = 4
◮ Ei — edges of E in direction i
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
2 · |S| · log |S|
◮ Equality if S is “face" : S = {(x, y): y ∈ {0, 1}d} for some x ∈ {0, 1}n−d ◮ Example: S is a face of the 3-dimensional cube
2 · 4 · log 4 = 4
◮ Ei — edges of E in direction i
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
2 · |S| · log |S|
◮ Equality if S is “face" : S = {(x, y): y ∈ {0, 1}d} for some x ∈ {0, 1}n−d ◮ Example: S is a face of the 3-dimensional cube
2 · 4 · log 4 = 4
◮ Ei — edges of E in direction i
i∈[n] Ei)
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
2 · |S| · log |S|
◮ Equality if S is “face" : S = {(x, y): y ∈ {0, 1}d} for some x ∈ {0, 1}n−d ◮ Example: S is a face of the 3-dimensional cube
2 · 4 · log 4 = 4
◮ Ei — edges of E in direction i
i∈[n] Ei)
◮ X = (X1, . . . , Xn) ← S and X−i = (X1, X2, . . . , Xi−1, Xi+1, . . . , Xn)
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
2 · |S| · log |S|
◮ Equality if S is “face" : S = {(x, y): y ∈ {0, 1}d} for some x ∈ {0, 1}n−d ◮ Example: S is a face of the 3-dimensional cube
2 · 4 · log 4 = 4
◮ Ei — edges of E in direction i
i∈[n] Ei)
◮ X = (X1, . . . , Xn) ← S and X−i = (X1, X2, . . . , Xi−1, Xi+1, . . . , Xn)
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
2 · |S| · log |S|
◮ Equality if S is “face" : S = {(x, y): y ∈ {0, 1}d} for some x ∈ {0, 1}n−d ◮ Example: S is a face of the 3-dimensional cube
2 · 4 · log 4 = 4
◮ Ei — edges of E in direction i
i∈[n] Ei)
◮ X = (X1, . . . , Xn) ← S and X−i = (X1, X2, . . . , Xi−1, Xi+1, . . . , Xn)
|S|
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
2 · |S| · log |S|
◮ Equality if S is “face" : S = {(x, y): y ∈ {0, 1}d} for some x ∈ {0, 1}n−d ◮ Example: S is a face of the 3-dimensional cube
2 · 4 · log 4 = 4
◮ Ei — edges of E in direction i
i∈[n] Ei)
◮ X = (X1, . . . , Xn) ← S and X−i = (X1, X2, . . . , Xi−1, Xi+1, . . . , Xn)
|S|
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
2 · |S| · log |S|
◮ Equality if S is “face" : S = {(x, y): y ∈ {0, 1}d} for some x ∈ {0, 1}n−d ◮ Example: S is a face of the 3-dimensional cube
2 · 4 · log 4 = 4
◮ Ei — edges of E in direction i
i∈[n] Ei)
◮ X = (X1, . . . , Xn) ← S and X−i = (X1, X2, . . . , Xi−1, Xi+1, . . . , Xn)
|S|
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
2 · |S| · log |S|
◮ Equality if S is “face" : S = {(x, y): y ∈ {0, 1}d} for some x ∈ {0, 1}n−d ◮ Example: S is a face of the 3-dimensional cube
2 · 4 · log 4 = 4
◮ Ei — edges of E in direction i
i∈[n] Ei)
◮ X = (X1, . . . , Xn) ← S and X−i = (X1, X2, . . . , Xi−1, Xi+1, . . . , Xn)
|S|
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
2 · |S| · log |S|
◮ Equality if S is “face" : S = {(x, y): y ∈ {0, 1}d} for some x ∈ {0, 1}n−d ◮ Example: S is a face of the 3-dimensional cube
2 · 4 · log 4 = 4
◮ Ei — edges of E in direction i
i∈[n] Ei)
◮ X = (X1, . . . , Xn) ← S and X−i = (X1, X2, . . . , Xi−1, Xi+1, . . . , Xn)
|S|
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
2 · |S| · log |S|
◮ Equality if S is “face" : S = {(x, y): y ∈ {0, 1}d} for some x ∈ {0, 1}n−d ◮ Example: S is a face of the 3-dimensional cube
2 · 4 · log 4 = 4
◮ Ei — edges of E in direction i
i∈[n] Ei)
◮ X = (X1, . . . , Xn) ← S and X−i = (X1, X2, . . . , Xi−1, Xi+1, . . . , Xn)
|S|
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
◮ S ⊆ {0, 1}n ◮ Edges of S — E = {(u, v) ∈ S : |u − v| = 1}
2 · |S| · log |S|
◮ Equality if S is “face" : S = {(x, y): y ∈ {0, 1}d} for some x ∈ {0, 1}n−d ◮ Example: S is a face of the 3-dimensional cube
2 · 4 · log 4 = 4
◮ Ei — edges of E in direction i
i∈[n] Ei)
◮ X = (X1, . . . , Xn) ← S and X−i = (X1, X2, . . . , Xi−1, Xi+1, . . . , Xn)
|S|
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 20 / 21
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 21 / 21
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 21 / 21
◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 21 / 21
◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S ◮ E = {(u, v) ∈ S : |u − v| = 1} and E1 contains edges of E in direction 1
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 21 / 21
◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S ◮ E = {(u, v) ∈ S : |u − v| = 1} and E1 contains edges of E in direction 1 ◮ S−1 = {y ∈ {0, 1}n−1 : ∃x ∈ {0, 1} s.t. (x, y) ∈ S}.
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 21 / 21
◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S ◮ E = {(u, v) ∈ S : |u − v| = 1} and E1 contains edges of E in direction 1 ◮ S−1 = {y ∈ {0, 1}n−1 : ∃x ∈ {0, 1} s.t. (x, y) ∈ S}.
◮ Se
−1 = {y ∈ {0, 1}n−1 : (0, y), (1, y) ∈ S} and S¬e −1 = S−1 \ Se −1
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 21 / 21
◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S ◮ E = {(u, v) ∈ S : |u − v| = 1} and E1 contains edges of E in direction 1 ◮ S−1 = {y ∈ {0, 1}n−1 : ∃x ∈ {0, 1} s.t. (x, y) ∈ S}.
◮ Se
−1 = {y ∈ {0, 1}n−1 : (0, y), (1, y) ∈ S} and S¬e −1 = S−1 \ Se −1
◮ |S| = 2
−1
−1
Application of Information Theory, Lecture 5 November 25, 2014 21 / 21
◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S ◮ E = {(u, v) ∈ S : |u − v| = 1} and E1 contains edges of E in direction 1 ◮ S−1 = {y ∈ {0, 1}n−1 : ∃x ∈ {0, 1} s.t. (x, y) ∈ S}.
◮ Se
−1 = {y ∈ {0, 1}n−1 : (0, y), (1, y) ∈ S} and S¬e −1 = S−1 \ Se −1
◮ |S| = 2
−1
−1
−1
Application of Information Theory, Lecture 5 November 25, 2014 21 / 21
◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S ◮ E = {(u, v) ∈ S : |u − v| = 1} and E1 contains edges of E in direction 1 ◮ S−1 = {y ∈ {0, 1}n−1 : ∃x ∈ {0, 1} s.t. (x, y) ∈ S}.
◮ Se
−1 = {y ∈ {0, 1}n−1 : (0, y), (1, y) ∈ S} and S¬e −1 = S−1 \ Se −1
◮ |S| = 2
−1
−1
−1
−1
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 21 / 21
◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S ◮ E = {(u, v) ∈ S : |u − v| = 1} and E1 contains edges of E in direction 1 ◮ S−1 = {y ∈ {0, 1}n−1 : ∃x ∈ {0, 1} s.t. (x, y) ∈ S}.
◮ Se
−1 = {y ∈ {0, 1}n−1 : (0, y), (1, y) ∈ S} and S¬e −1 = S−1 \ Se −1
◮ |S| = 2
−1
−1
−1
−1
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 21 / 21
◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S ◮ E = {(u, v) ∈ S : |u − v| = 1} and E1 contains edges of E in direction 1 ◮ S−1 = {y ∈ {0, 1}n−1 : ∃x ∈ {0, 1} s.t. (x, y) ∈ S}.
◮ Se
−1 = {y ∈ {0, 1}n−1 : (0, y), (1, y) ∈ S} and S¬e −1 = S−1 \ Se −1
◮ |S| = 2
−1
−1
−1
−1
2|Se
−1|
2|Se
−1|+|S¬e −1|
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 21 / 21
◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S ◮ E = {(u, v) ∈ S : |u − v| = 1} and E1 contains edges of E in direction 1 ◮ S−1 = {y ∈ {0, 1}n−1 : ∃x ∈ {0, 1} s.t. (x, y) ∈ S}.
◮ Se
−1 = {y ∈ {0, 1}n−1 : (0, y), (1, y) ∈ S} and S¬e −1 = S−1 \ Se −1
◮ |S| = 2
−1
−1
−1
−1
2|Se
−1|
2|Se
−1|+|S¬e −1| = 2|E1|
|S|
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 21 / 21
◮ S ⊆ {0, 1}n; X = (X1, . . . , Xn) ← S ◮ E = {(u, v) ∈ S : |u − v| = 1} and E1 contains edges of E in direction 1 ◮ S−1 = {y ∈ {0, 1}n−1 : ∃x ∈ {0, 1} s.t. (x, y) ∈ S}.
◮ Se
−1 = {y ∈ {0, 1}n−1 : (0, y), (1, y) ∈ S} and S¬e −1 = S−1 \ Se −1
◮ |S| = 2
−1
−1
−1
−1
2|Se
−1|
2|Se
−1|+|S¬e −1| = 2|E1|
|S|
◮ . . .
Iftach Haitner (TAU) Application of Information Theory, Lecture 5 November 25, 2014 21 / 21