Bounding Techniques for the Intrinsic Uncertainty of Channels Or - - PowerPoint PPT Presentation

▶
bounding techniques for the intrinsic uncertainty of
SMART_READER_LITE
LIVE PREVIEW

Bounding Techniques for the Intrinsic Uncertainty of Channels Or - - PowerPoint PPT Presentation

Oct 23, 2022 213 likes •724 views

Bounding Techniques for the Intrinsic Uncertainty of Channels Or Ordentlich Joint work with Ofer Shayevitz July 4th, 2014 ISIT, Honolulu, HI, USA Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-1
SLIDE 1

Bounding Techniques for the Intrinsic Uncertainty

  • f Channels

Or Ordentlich Joint work with Ofer Shayevitz July 4th, 2014 ISIT, Honolulu, HI, USA

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-2
SLIDE 2

Motivation

DMC

For DMCs C = maxP(X) I(X; Y ) Calculating I(X; Y ) is easy

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-3
SLIDE 3

Motivation

DMC

For DMCs C = maxP(X) I(X; Y ) Calculating I(X; Y ) is easy

Channels with memory

Assuming information stability [Dobrushin 1973] C = lim

n→∞ max PX

1 nI(X; Y) Calculating I(X; Y) may be difficult

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-4
SLIDE 4

Motivation

DMC

For DMCs C = maxP(X) I(X; Y ) Calculating I(X; Y ) is easy

Channels with memory

Assuming information stability [Dobrushin 1973] C = lim

n→∞ max PX

1 nI(X; Y) Calculating I(X; Y) may be difficult For many interesting channels PXY has sparse support:

  • deletion, insertion, trapdoor,...

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-5
SLIDE 5

Motivation

DMC

For DMCs C = maxP(X) I(X; Y ) Calculating I(X; Y ) is easy

Channels with memory

Assuming information stability [Dobrushin 1973] C = lim

n→∞ max PX

1 nI(X; Y) Calculating I(X; Y) may be difficult For many interesting channels PXY has sparse support:

  • deletion, insertion, trapdoor,...

Want lower bounds on I(X; Y) that are useful for such channels

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-6
SLIDE 6

Motivation

X, Y are random vectors with joint distribution PXY ¯ X ∼ PX, ¯ Y ∼ PY, ¯ X | = ¯ Y

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-7
SLIDE 7

Motivation

X, Y are random vectors with joint distribution PXY ¯ X ∼ PX, ¯ Y ∼ PY, ¯ X | = ¯ Y AEP: I(X; Y) ≈ − log

  • Pr
  • (¯

X, ¯ Y) ∈ T

  • Or Ordentlich and Ofer Shayevitz

Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-8
SLIDE 8

Motivation

X, Y are random vectors with joint distribution PXY ¯ X ∼ PX, ¯ Y ∼ PY, ¯ X | = ¯ Y AEP: I(X; Y) ≈ − log

  • Pr
  • (¯

X, ¯ Y) ∈ T

  • Computing Pr
  • (¯

X, ¯ Y) ∈ T

  • may be difficult

Lower bound by replacing T with some S ⊇ T

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-9
SLIDE 9

Motivation

X, Y are random vectors with joint distribution PXY ¯ X ∼ PX, ¯ Y ∼ PY, ¯ X | = ¯ Y

✘✘ ✘

AEP: I(X; Y) ≥ − log

  • Pr
  • (¯

X, ¯ Y) ∈ S

  • Computing Pr
  • (¯

X, ¯ Y) ∈ T

  • may be difficult

Lower bound by replacing T with some S ⊇ T

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-10
SLIDE 10

Motivation

X, Y are random vectors with joint distribution PXY ¯ X ∼ PX, ¯ Y ∼ PY, ¯ X | = ¯ Y

✘✘ ✘

AEP: I(X; Y) ≥ − log

  • EXEY
1{(X,Y)∈S}
  • Computing Pr
  • (¯

X, ¯ Y) ∈ T

  • may be difficult

Lower bound by replacing T with some S ⊇ T

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-11
SLIDE 11

Motivation

X, Y are random vectors with joint distribution PXY ¯ X ∼ PX, ¯ Y ∼ PY, ¯ X | = ¯ Y

✘✘ ✘

AEP: I(X; Y) ≥ − log

  • EXEY
1{(X,Y)∈S}
  • Computing Pr
  • (¯

X, ¯ Y) ∈ T

  • may be difficult

Lower bound by replacing T with some S ⊇ T A simple choice is the support S {(x, y) : PXY(x, y) > 0}

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-12
SLIDE 12

Motivation

I(X; Y) ≥ − log

  • EXEY
1{(X,Y)∈S}
  • Bound relatively easy to compute: involves only marginals and support

Gives reasonable results for certain “sparse” distributions

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-13
SLIDE 13

Motivation

I(X; Y) ≥ − log

  • EXEY
1{(X,Y)∈S}
  • Bound relatively easy to compute: involves only marginals and support

Gives reasonable results for certain “sparse” distributions

Can we find better bounds that only involve marginals and support?

Yes - replace S with ¯ S {x ∈ TX, y ∈ TY : PXY(x, y) > 0} I(X; Y) ≥ − log

  • EX|TXEY|TY
1{(X,Y)∈S}
  • [Diggavi & Grossglauser ’01] [Drinea & Mitzenmacher ’07]. . .

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-14
SLIDE 14

Motivation

I(X; Y) ≥ − log

  • EXEY
1{(X,Y)∈S}
  • Bound relatively easy to compute: involves only marginals and support

Gives reasonable results for certain “sparse” distributions

Can we find better bounds that only involve marginals and support?

Yes - replace S with ¯ S {x ∈ TX, y ∈ TY : PXY(x, y) > 0} I(X; Y) ≥ − log

  • EX|TXEY|TY
1{(X,Y)∈S}
  • [Diggavi & Grossglauser ’01] [Drinea & Mitzenmacher ’07]. . .

Main Result

I(X; Y) ≥ −EY log EX

1{(X,Y)∈S} − EX log EY 1{(X,Y)∈S}

EX

1{(X,Y)∈S}

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-15
SLIDE 15

Examples

I(X; Y) ≥ −EY log EX

1{(X,Y)∈S} − EX log EY 1{(X,Y)∈S}

EX

1{(X,Y)∈S}

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-16
SLIDE 16

Examples

I(X; Y) ≥ −EY log EX

1{(X,Y)∈S} − EX log EY 1{(X,Y)∈S}

EX

1{(X,Y)∈S}

When is this bound useful?

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-17
SLIDE 17

Examples

I(X; Y) ≥ −EY log EX

1{(X,Y)∈S} − EX log EY 1{(X,Y)∈S}

EX

1{(X,Y)∈S}

When is this bound useful? Not for “fully-connected” channels: All pairs (x, y) ∈ S - the bound gives I(X; Y) ≥ 0 Can be pretty good for channels with “low-connectivity”

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-18
SLIDE 18

Example: Z-Channel

1 1/2 1 1 1/2

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-19
SLIDE 19

Example: Z-Channel

1 1/2 1 1 1/2

Bounds for IID Ber(p) Input

Mutual information: I(X; Y ) = H 1

2(1 + p)

  • − (1 − p)

Naive bound:

I(X, Y ) ≥ log

  • EX EY
1{(X,Y )∈S}
  • = −1

2 log

  • 1 − p

2 (1 − p)

  • Our bound:

I(X, Y ) ≥ −1 2(1−p) log(1−p)−p log 1 2(1 + p)

  • −(1−p) log

1 2(2 + p)

  • Or Ordentlich and Ofer Shayevitz

Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-20
SLIDE 20

Example: Z-Channel

0.2 0.4 0.6 0.8 1 0.05 0.1 0.15 0.2 0.25 0.3 0.35 p Bounds on I(X;Y) Naive Lower Bound Lower Bound − 1st term Lower Bound − both terms Mutual Information

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-21
SLIDE 21

Preliminaries

Channels via Conditional Probability

Channel ⇔ Conditional distribution PY|X Input alphabet X n Output alphabet Y∗

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-22
SLIDE 22

Preliminaries

Channels via Conditional Probability

Channel ⇔ Conditional distribution PY|X Input alphabet X n Output alphabet Y∗

Channels via Actions (Functional Representation Lemma)

PA - a distribution over mappings X n → Y∗ Channel ⇔ Action A ∼ PA, A | = X Y = A(X) The choice of PA is not unique

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-23
SLIDE 23

Preliminaries

The Intrinsic Uncertainty

Input distribution PX H(A|X, Y) is the intrinsic uncertainty

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-24
SLIDE 24

Preliminaries

The Intrinsic Uncertainty

Input distribution PX H(A|X, Y) is the intrinsic uncertainty

Capacity

I(X; Y) = H(Y) − H(Y|X) = H(Y) − (H(Y, A|X) − H(A|X, Y)) = H(Y) − H(A|X) − H(Y|A, X) + H(A|X, Y) = H(Y) − H(A) + H(A|X, Y) Lower bounding the intrinsic uncertainty = lower bounding MI

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-25
SLIDE 25

Examples for Action Sets and Intrinsic Uncertainty

The Binary Symmetric Channel

Action ⇔ IID Noise sequence W ∼ Ber(p) Y = A(X) = X ⊕ W H(A|X, Y) = 0

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-26
SLIDE 26

Examples for Action Sets and Intrinsic Uncertainty

The Binary Symmetric Channel

Action ⇔ IID Noise sequence W ∼ Ber(p) Y = A(X) = X ⊕ W H(A|X, Y) = 0

The Z-Channel

Action ⇔ IID Noise sequence W ∼ Ber(1

2)

Yi = {A(X)}i = Xi Xi = 0 Xi ⊕ Wi Xi = 1 Action masked when Xi = 0 ⇒ H(A|X, Y) > 0

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-27
SLIDE 27

Examples for Action Sets and Intrinsic Uncertainty

The Binary Deletion Channel

Deletes bits independently with probability d Action ⇔ IID deletion pattern W ∼ Ber(d) X → Y via many different actions ⇒ H(A|X, Y) > 0 For example: x = 01100 and y = 110

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-28
SLIDE 28

Examples for Action Sets and Intrinsic Uncertainty

The Binary Deletion Channel

Deletes bits independently with probability d Action ⇔ IID deletion pattern W ∼ Ber(d) X → Y via many different actions ⇒ H(A|X, Y) > 0 For example: x = ✁ 011✁ 00 and y = 110

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-29
SLIDE 29

Examples for Action Sets and Intrinsic Uncertainty

The Binary Deletion Channel

Deletes bits independently with probability d Action ⇔ IID deletion pattern W ∼ Ber(d) X → Y via many different actions ⇒ H(A|X, Y) > 0 For example: x = ✁ 0110✁ 0 and y = 110

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-30
SLIDE 30

Examples for Action Sets and Intrinsic Uncertainty

The Binary Deletion Channel

Deletes bits independently with probability d Action ⇔ IID deletion pattern W ∼ Ber(d) X → Y via many different actions ⇒ H(A|X, Y) > 0 For example: x = ✁ 0110✁ 0 and y = 110

Other Channels with memory and positive intrinsic uncertainty

Insertion channel Trapdoor channel Permutation channels ....

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-31
SLIDE 31

Main Tool

We would like to lower bound the intrinsic uncertainty H(A|X, Y) = E log

  • 1

P(A|X, Y)

  • Or Ordentlich and Ofer Shayevitz

Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-32
SLIDE 32

Main Tool

We would like to lower bound the intrinsic uncertainty H(A|X, Y) = E log

  • 1

P(A|X, Y)

  • Variational Principle [Dupuis & Ellis]

For any distribution P and function f (x) s.t. |EP log f (X)| < ∞, EP log f (X) = min

Q

(log EQf (X) + D(P||Q)) The minimum is uniquely attained by Q∗(x) = P(x)/f (x) EP(1/f (x))

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-33
SLIDE 33

Main Tool

We would like to lower bound the intrinsic uncertainty H(A|X, Y) = E log

  • 1

P(A|X, Y)

  • Variational Principle [Dupuis & Ellis]

For any distribution P and function f (x) s.t. |EP log f (X)| < ∞, EP log f (X) = min

Q

(log EQf (X) + D(P||Q)) The minimum is uniquely attained by Q∗(x) = P(x)/f (x) EP(1/f (x)) In our case f = 1/P(A|X, Y)

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-34
SLIDE 34

A General Bound

Using the variational principle + chain rule of relative entropy + convexity

  • f relative entropy

Theorem

The intrinsic uncertainty is lower bounded by H(A|X, Y) ≥ −H(Y) − EY log EX,AP(A|X, Y) − EX,A log EY P(A|X, Y) EX,AP(A|X, Y)

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-35
SLIDE 35

A General Bound

Using the variational principle + chain rule of relative entropy + convexity

  • f relative entropy

Theorem

The intrinsic uncertainty is lower bounded by I(X; Y) ≥ −H(A) − EY log EX,AP(A|X, Y) − EX,A log EY P(A|X, Y) EX,AP(A|X, Y)

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-36
SLIDE 36

A General Bound

Using the variational principle + chain rule of relative entropy + convexity

  • f relative entropy

Theorem

The intrinsic uncertainty is lower bounded by I(X; Y) ≥ −H(A) − EY log EX,AP(A|X, Y) − EX,A log EY P(A|X, Y) EX,AP(A|X, Y) Bound’s tightness depends on the choice of PA For BSC certain choices of PA yield tight bounds and other choices yield I(X; Y) ≥ 0

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-37
SLIDE 37

A General Bound

Using the variational principle + chain rule of relative entropy + convexity

  • f relative entropy

Theorem

The intrinsic uncertainty is lower bounded by I(X; Y) ≥ −H(A) − EY log EX,AP(A|X, Y) − EX,A log EY P(A|X, Y) EX,AP(A|X, Y) Bound’s tightness depends on the choice of PA For BSC certain choices of PA yield tight bounds and other choices yield I(X; Y) ≥ 0 For a certain choice of PA the bound becomes much simpler...

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-38
SLIDE 38

Uniform Action Set

Definition

A channel has a uniform action set if A ∼ Uniform(A)

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-39
SLIDE 39

Uniform Action Set

Definition

A channel has a uniform action set if A ∼ Uniform(A)

Theorem

For channels with uniform action set: I(X; Y) ≥ − EY log EX

1{(X,Y)∈S} − EX log EY 1{(X,Y)∈S}

EX

1{(X,Y)∈S}

where S {(x, y) : ∃a ∈ A s.t. a(x) = y} = {(x, y) : PXY(x, y) > 0}

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-40
SLIDE 40

Uniform Action Set

Proposition

For each channel PY|X there exist a uniform action set

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-41
SLIDE 41

Uniform Action Set

Proposition

For each channel PY|X there exist a uniform action set

Proof

Let A = {a1, . . . , a|A|} be some action set PA is a probability assignment on A consistent with PY|X Duplicate each action ai to Mi identical actions with equal probabilities PA(ai)

Mi

Choose the Mis such that all actions in the extended set are equiprobable

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-42
SLIDE 42

Uniform Action Set

Proposition

For each channel PY|X there exist a uniform action set

Corollary (Our Main Result)

For any joint distribution PXY I(X; Y) ≥ − EY log EX

1{(X,Y)∈S} − EX log EY 1{(X,Y)∈S}

EX

1{(X,Y)∈S}

where S {(x, y) : PXY(x, y) > 0}.

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-43
SLIDE 43

Example: Binary Deletion Channel

Capacity

Only bounds are known Best lower bounds use input with memory [Diggavi & Grossglauser ’01]

[Drinea & Mitzenmacher ’07] [Kirsch & Drinea ’10] . . .

Some implicitly analyze the first summand in our bound

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-44
SLIDE 44

Example: Binary Deletion Channel

Capacity

Only bounds are known Best lower bounds use input with memory [Diggavi & Grossglauser ’01]

[Drinea & Mitzenmacher ’07] [Kirsch & Drinea ’10] . . .

Some implicitly analyze the first summand in our bound

Rates for a memoryless input

Only bounds are known 1 − H2(d) achievable for d ∈ [0, 1

2) [Gallager ’61]

Recently improved for

◮ Small d [Rahmati & Duman ’13] ◮ d → 0 [Kanoria & Montanari ’13] [Drmota et al ’12]

And our bound?

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-45
SLIDE 45

Example: Binary Deletion Channel

New Bound (Memoryless Input) lim

n→∞

1 nI(X; Y) ≥ 1 − H2(d) + g(d)

where g(d) > 0 for all d ∈ (0, 1

2), and is given by g(d) = min

α∈[0,1] (D2(α||1 − d) − (1 − H2(α)) + Λ∗(α))

Λ∗(α) = max

t>0

 αt − 1 5

  • k1
  • k2

2−(k1+k2−1) log λZk1,k2 (t)   λk1,k2

Zi

(t) = 2k1(t−1) + 2t−1 1 − 2k1(t−1) 1 − 2t−1

  • 2t−1 1 − 2k2(t−1)

1 − 2t−1 + 2k2(t−1)−t

  • Or Ordentlich and Ofer Shayevitz

Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-46
SLIDE 46

Example: Binary Deletion Channel

% improvement over Gallager’s bound 1 − H2(d) (IID input):

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 5 10 15 20 25 30 d % improvement over 1 − H(d) New result Best previous result

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-47
SLIDE 47

Concluding Remarks

Summary

Novel lower bound on I(X; Y) that depends only on PX, PY and the support of PXY Bound is useful for channels with memory and low-connectivity Main tool: The Variational Principle For the deletion channel with IID input our bound improves best existing bounds (for some regime of d)

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-48
SLIDE 48

Concluding Remarks

Summary

Novel lower bound on I(X; Y) that depends only on PX, PY and the support of PXY Bound is useful for channels with memory and low-connectivity Main tool: The Variational Principle For the deletion channel with IID input our bound improves best existing bounds (for some regime of d)

Future Research

Evaluate bound for different inputs and different channels, e.g., deletion with Markov input, trapdoor channels, etc... Can improve the bound to better trade-off complexity and accuracy: Replace S with a subset of the support whose probability approaches 1

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels

slide-49
SLIDE 49

Thanks for your attention!

Or Ordentlich and Ofer Shayevitz Bounding Techniques for the Intrinsic Uncertainty of Channels