Permutation Channels Anuran Makur Department of Electrical - - PowerPoint PPT Presentation

Permutation Channels

Anuran Makur

Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology

10 July 2020

Anuran Makur (MIT) Permutation Channels 10 July 2020 1 / 40

Outline

1

Introduction Three Motivations Permutation Channel Model Information Capacity Example: Binary Symmetric Channel

2

Achievability and Converse for the BSC

3

General Achievability Bound

4

General Converse Bounds

5

Conclusion

Anuran Makur (MIT) Permutation Channels 10 July 2020 2 / 40

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . .

Anuran Makur (MIT) Permutation Channels 10 July 2020 3 / 40

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . . Random deletion channel: LDPC codes nearly achieve capacity for large alphabets Codes correct for transpositions of symbols

Anuran Makur (MIT) Permutation Channels 10 July 2020 3 / 40

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . . Random deletion channel: LDPC codes nearly achieve capacity for large alphabets Codes correct for transpositions of symbols Permutation channels with insertions, deletions, substitutions, or erasures Construction and analysis of multiset codes

Anuran Makur (MIT) Permutation Channels 10 July 2020 3 / 40

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . . Random deletion channel: LDPC codes nearly achieve capacity for large alphabets Codes correct for transpositions of symbols Permutation channels with insertions, deletions, substitutions, or erasures Construction and analysis of multiset codes Communication networks: [XZ02], [WWM09], [GG10], [KV13], . . . Mobile ad hoc networks, multipath routed networks, etc.

Anuran Makur (MIT) Permutation Channels 10 July 2020 3 / 40

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . . Random deletion channel: LDPC codes nearly achieve capacity for large alphabets Codes correct for transpositions of symbols Permutation channels with insertions, deletions, substitutions, or erasures Construction and analysis of multiset codes Communication networks: [XZ02], [WWM09], [GG10], [KV13], . . . Mobile ad hoc networks, multipath routed networks, etc. Out-of-order delivery of packets Correct for packet errors/losses when packets do not have sequence numbers

Anuran Makur (MIT) Permutation Channels 10 July 2020 3 / 40

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . . Random deletion channel: LDPC codes nearly achieve capacity for large alphabets Codes correct for transpositions of symbols Permutation channels with insertions, deletions, substitutions, or erasures Construction and analysis of multiset codes Communication networks: [XZ02], [WWM09], [GG10], [KV13], . . . Mobile ad hoc networks, multipath routed networks, etc. Out-of-order delivery of packets Correct for packet errors/losses when packets do not have sequence numbers Molecular/Biological Communications: [YKGR+15], [KPM16], [HSRT17], [SH19], . . .

Anuran Makur (MIT) Permutation Channels 10 July 2020 3 / 40

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . . Random deletion channel: LDPC codes nearly achieve capacity for large alphabets Codes correct for transpositions of symbols Permutation channels with insertions, deletions, substitutions, or erasures Construction and analysis of multiset codes Communication networks: [XZ02], [WWM09], [GG10], [KV13], . . . Mobile ad hoc networks, multipath routed networks, etc. Out-of-order delivery of packets Correct for packet errors/losses when packets do not have sequence numbers Molecular/Biological Communications: [YKGR+15], [KPM16], [HSRT17], [SH19], . . . DNA based storage systems Source data encoded into DNA molecules

Anuran Makur (MIT) Permutation Channels 10 July 2020 3 / 40

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . . Random deletion channel: LDPC codes nearly achieve capacity for large alphabets Codes correct for transpositions of symbols Permutation channels with insertions, deletions, substitutions, or erasures Construction and analysis of multiset codes Communication networks: [XZ02], [WWM09], [GG10], [KV13], . . . Mobile ad hoc networks, multipath routed networks, etc. Out-of-order delivery of packets Correct for packet errors/losses when packets do not have sequence numbers Molecular/Biological Communications: [YKGR+15], [KPM16], [HSRT17], [SH19], . . . DNA based storage systems Source data encoded into DNA molecules Fragments of DNA molecules cached Receiver reads encoded data by shotgun sequencing (i.e., random sampling)

Anuran Makur (MIT) Permutation Channels 10 July 2020 3 / 40

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . . Random deletion channel: LDPC codes nearly achieve capacity for large alphabets Codes correct for transpositions of symbols Permutation channels with insertions, deletions, substitutions, or erasures Construction and analysis of multiset codes Communication networks: [XZ02], [WWM09], [GG10], [KV13], . . . Mobile ad hoc networks, multipath routed networks, etc. Out-of-order delivery of packets Correct for packet errors/losses when packets do not have sequence numbers Molecular/Biological Communications: [YKGR+15], [KPM16], [HSRT17], [SH19], . . . DNA based storage systems Source data encoded into DNA molecules Fragments of DNA molecules cached Receiver reads encoded data by shotgun sequencing (i.e., random sampling)

Anuran Makur (MIT) Permutation Channels 10 July 2020 3 / 40

Motivation: Point-to-point Communication in Packet Networks

NETWORK SENDER RECEIVER

Anuran Makur (MIT) Permutation Channels 10 July 2020 4 / 40

Motivation: Point-to-point Communication in Packet Networks

NETWORK SENDER RECEIVER

Model communication network as a channel

Anuran Makur (MIT) Permutation Channels 10 July 2020 4 / 40

Motivation: Point-to-point Communication in Packet Networks

NETWORK SENDER RECEIVER

Model communication network as a channel: Alphabet symbols = all possible b-bit packets ⇒ 2b input symbols

Anuran Makur (MIT) Permutation Channels 10 July 2020 4 / 40

Motivation: Point-to-point Communication in Packet Networks

NETWORK SENDER RECEIVER

Model communication network as a channel: Alphabet symbols = all possible b-bit packets Multipath routed network or evolving network topology

Anuran Makur (MIT) Permutation Channels 10 July 2020 4 / 40

Motivation: Point-to-point Communication in Packet Networks

NETWORK SENDER RECEIVER

Model communication network as a channel: Alphabet symbols = all possible b-bit packets Multipath routed network ⇒ packets received with transpositions

Anuran Makur (MIT) Permutation Channels 10 July 2020 4 / 40

Motivation: Point-to-point Communication in Packet Networks

NETWORK SENDER RECEIVER

Model communication network as a channel: Alphabet symbols = all possible b-bit packets Multipath routed network ⇒ packets received with transpositions Packets are impaired (e.g., deletions, substitutions, etc.)

Anuran Makur (MIT) Permutation Channels 10 July 2020 4 / 40

Motivation: Point-to-point Communication in Packet Networks

NETWORK SENDER RECEIVER

Model communication network as a channel: Alphabet symbols = all possible b-bit packets Multipath routed network ⇒ packets received with transpositions Packets are impaired ⇒ model using channel probabilities

Anuran Makur (MIT) Permutation Channels 10 July 2020 4 / 40

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER

Anuran Makur (MIT) Permutation Channels 10 July 2020 5 / 40

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER RANDOM DELETION RANDOM PERMUTATION

Abstraction: n-length codeword = sequence of n packets

Anuran Makur (MIT) Permutation Channels 10 July 2020 5 / 40

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER RANDOM DELETION RANDOM PERMUTATION

Abstraction: n-length codeword = sequence of n packets Random deletion channel: Delete each symbol/packet independently with prob p ∈ (0, 1)

Anuran Makur (MIT) Permutation Channels 10 July 2020 5 / 40

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER RANDOM DELETION RANDOM PERMUTATION

Abstraction: n-length codeword = sequence of n packets Random deletion channel: Delete each symbol/packet independently with prob p ∈ (0, 1)

Anuran Makur (MIT) Permutation Channels 10 July 2020 5 / 40

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER RANDOM DELETION RANDOM PERMUTATION

Abstraction: n-length codeword = sequence of n packets Random deletion channel: Delete each symbol/packet independently with prob p ∈ (0, 1) Random permutation block: Randomly permute packets of codeword

Anuran Makur (MIT) Permutation Channels 10 July 2020 5 / 40

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER RANDOM DELETION RANDOM PERMUTATION

Abstraction: n-length codeword = sequence of n packets Random deletion channel: Delete each symbol/packet independently with prob p ∈ (0, 1) Random permutation block: Randomly permute packets of codeword

Anuran Makur (MIT) Permutation Channels 10 July 2020 5 / 40

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER ERASURE CHANNEL RANDOM PERMUTATION

? ?

Abstraction: n-length codeword = sequence of n packets Equivalent Erasure channel: Erase each symbol/packet independently with prob p ∈ (0, 1) Random permutation block: Randomly permute packets of codeword

Anuran Makur (MIT) Permutation Channels 10 July 2020 5 / 40

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER ERASURE CHANNEL RANDOM PERMUTATION

? ? 1 2 3 3 3 1 1

Abstraction: n-length codeword = sequence of n packets Erasure channel: Erase each symbol/packet independently with prob p ∈ (0, 1) Random permutation block: Randomly permute packets of codeword Coding: Add sequence numbers (packet size = b + log(n) bits, alphabet size = n 2b)

Anuran Makur (MIT) Permutation Channels 10 July 2020 5 / 40

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER ERASURE CHANNEL RANDOM PERMUTATION

? ? 1 2 3 3 3 1 1

Abstraction: n-length codeword = sequence of n packets Erasure channel: Erase each symbol/packet independently with prob p ∈ (0, 1) Random permutation block: Randomly permute packets of codeword Coding: Add sequence numbers and use standard coding techniques

Anuran Makur (MIT) Permutation Channels 10 July 2020 5 / 40

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER ERASURE CHANNEL RANDOM PERMUTATION

? ? 1 2 3 3 3 1 1

Abstraction: n-length codeword = sequence of n packets Erasure channel: Erase each symbol/packet independently with prob p ∈ (0, 1) Random permutation block: Randomly permute packets of codeword Coding: Add sequence numbers and use standard coding techniques More refined coding techniques simulate sequence numbers [Mit06], [Met09]

Anuran Makur (MIT) Permutation Channels 10 July 2020 5 / 40

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER ERASURE CHANNEL RANDOM PERMUTATION

? ?

Abstraction: n-length codeword = sequence of n packets Erasure channel: Erase each symbol/packet independently with prob p ∈ (0, 1) Random permutation block: Randomly permute packets of codeword How do you code in such channels without increasing alphabet size?

Anuran Makur (MIT) Permutation Channels 10 July 2020 5 / 40

Permutation Channel Model

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Sender sends message M ∼ Uniform(M)

n = blocklength

Anuran Makur (MIT) Permutation Channels 10 July 2020 6 / 40

Permutation Channel Model

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Sender sends message M ∼ Uniform(M)

n = blocklength Randomized encoder fn : M → X n produces codeword X n

1 = (X1, . . . , Xn) = fn(M)

Anuran Makur (MIT) Permutation Channels 10 July 2020 6 / 40

Permutation Channel Model

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Sender sends message M ∼ Uniform(M)

n = blocklength Randomized encoder fn : M → X n produces codeword X n

1 = (X1, . . . , Xn) = fn(M)

Discrete memoryless channel PZ|X with input & output alphabets X & Y produces Z n

1 :

PZ n

1 |X n 1 (zn

1 |xn 1 ) = n

• i=1

PZ|X(zi|xi)

Anuran Makur (MIT) Permutation Channels 10 July 2020 6 / 40

Permutation Channel Model

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Sender sends message M ∼ Uniform(M)

n = blocklength Randomized encoder fn : M → X n produces codeword X n

1 = (X1, . . . , Xn) = fn(M)

Discrete memoryless channel PZ|X with input & output alphabets X & Y produces Z n

1 :

PZ n

1 |X n 1 (zn

1 |xn 1 ) = n

• i=1

PZ|X(zi|xi) Random permutation π generates Y n

1 from Z n 1 : Yπ(i) = Zi for i ∈ {1, . . . , n}

Anuran Makur (MIT) Permutation Channels 10 July 2020 6 / 40

Permutation Channel Model

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Sender sends message M ∼ Uniform(M)

n = blocklength Randomized encoder fn : M → X n produces codeword X n

1 = (X1, . . . , Xn) = fn(M)

Discrete memoryless channel PZ|X with input & output alphabets X & Y produces Z n

1 :

PZ n

1 |X n 1 (zn

1 |xn 1 ) = n

• i=1

PZ|X(zi|xi) Random permutation π generates Y n

1 from Z n 1 : Yπ(i) = Zi for i ∈ {1, . . . , n}

Randomized decoder gn : Yn → M ∪ {error} produces estimate ˆ M = gn(Y n

1 ) at receiver

Anuran Makur (MIT) Permutation Channels 10 July 2020 6 / 40

Permutation Channel Model

What if we analyze the “swapped” model?

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑊
• 𝑋
• 𝑁
• ENCODER

CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Anuran Makur (MIT)

Permutation Channels 10 July 2020 7 / 40

Permutation Channel Model

What if we analyze the “swapped” model?

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑊
• 𝑋
• 𝑁
• Proposition (Equivalent Models)

If channel PW |V is equal to channel PZ|X, then channel PW n

1 |X n 1 is equal to channel PY n 1 |X n 1 .

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Anuran Makur (MIT)

Permutation Channels 10 July 2020 7 / 40

Permutation Channel Model

What if we analyze the “swapped” model?

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑊
• 𝑋
• 𝑁
• Proposition (Equivalent Models)

If channel PW |V is equal to channel PZ|X, then channel PW n

1 |X n 1 is equal to channel PY n 1 |X n 1 .

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Remarks:

Proof follows from direct calculation.

Anuran Makur (MIT) Permutation Channels 10 July 2020 7 / 40

Permutation Channel Model

What if we analyze the “swapped” model?

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑊
• 𝑋
• 𝑁
• Proposition (Equivalent Models)

If channel PW |V is equal to channel PZ|X, then channel PW n

1 |X n 1 is equal to channel PY n 1 |X n 1 .

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Remarks:

Proof follows from direct calculation. Can analyze either model!

Anuran Makur (MIT) Permutation Channels 10 July 2020 7 / 40

Coding for the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• General Principle:

“Encode the information in an object that is invariant under the [permutation] transformation.” [KV13]

Anuran Makur (MIT) Permutation Channels 10 July 2020 8 / 40

Coding for the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• General Principle:

“Encode the information in an object that is invariant under the [permutation] transformation.” [KV13] Multiset codes are studied in [KV13], [KV15], and [KT18].

Anuran Makur (MIT) Permutation Channels 10 July 2020 8 / 40

Coding for the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• General Principle:

“Encode the information in an object that is invariant under the [permutation] transformation.” [KV13] Multiset codes are studied in [KV13], [KV15], and [KT18]. What are the fundamental information theoretic limits of this model?

Anuran Makur (MIT) Permutation Channels 10 July 2020 8 / 40

Information Capacity of the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Average probability of error Pn

error P(M = ˆ

M)

Anuran Makur (MIT) Permutation Channels 10 July 2020 9 / 40

Information Capacity of the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Average probability of error Pn

error P(M = ˆ

M) “Rate” of coding scheme (fn, gn) is R log(|M|) log(n)

Anuran Makur (MIT) Permutation Channels 10 July 2020 9 / 40

Information Capacity of the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Average probability of error Pn

error P(M = ˆ

M) “Rate” of coding scheme (fn, gn) is R log(|M|) log(n) |M| = nR

Anuran Makur (MIT) Permutation Channels 10 July 2020 9 / 40

Information Capacity of the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Average probability of error Pn

error P(M = ˆ

M) “Rate” of coding scheme (fn, gn) is R log(|M|) log(n) |M| = nR because number of empirical distributions of Y n

1 is poly(n)

Anuran Makur (MIT) Permutation Channels 10 July 2020 9 / 40

Information Capacity of the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Average probability of error Pn

error P(M = ˆ

M) “Rate” of coding scheme (fn, gn) is R log(|M|) log(n) |M| = nR Rate R ≥ 0 is achievable ⇔ ∃ {(fn, gn)}n∈N such that lim

n→∞ Pn error = 0

Anuran Makur (MIT) Permutation Channels 10 July 2020 9 / 40

Information Capacity of the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Average probability of error Pn

error P(M = ˆ

M) “Rate” of coding scheme (fn, gn) is R log(|M|) log(n) |M| = nR Rate R ≥ 0 is achievable ⇔ ∃ {(fn, gn)}n∈N such that lim

n→∞ Pn error = 0

Definition (Permutation Channel Capacity)

Cperm(PZ|X) sup{R ≥ 0 : R is achievable}

Anuran Makur (MIT) Permutation Channels 10 July 2020 9 / 40

Information Capacity of the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Average probability of error Pn

error P(M = ˆ

M) “Rate” of coding scheme (fn, gn) is R log(|M|) log(n) |M| = nR Rate R ≥ 0 is achievable ⇔ ∃ {(fn, gn)}n∈N such that lim

n→∞ Pn error = 0

Definition (Permutation Channel Capacity)

Cperm(PZ|X) sup{R ≥ 0 : R is achievable}

Main Question

What is the permutation channel capacity of a general PZ|X?

Anuran Makur (MIT) Permutation Channels 10 July 2020 9 / 40

Example: Binary Symmetric Channel

ENCODER BSC 𝒒 RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Channel is binary symmetric channel, denoted BSC(p):

∀z, x ∈ {0, 1}, PZ|X(z|x) =

• 1 − p,

for z = x p, for z = x

Anuran Makur (MIT) Permutation Channels 10 July 2020 10 / 40

Example: Binary Symmetric Channel

ENCODER BSC 𝒒 RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Channel is binary symmetric channel, denoted BSC(p):

∀z, x ∈ {0, 1}, PZ|X(z|x) =

• 1 − p,

for z = x p, for z = x Alphabets are X = Y = {0, 1}

Anuran Makur (MIT) Permutation Channels 10 July 2020 10 / 40

Example: Binary Symmetric Channel

ENCODER BSC 𝒒 RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Channel is binary symmetric channel, denoted BSC(p):

∀z, x ∈ {0, 1}, PZ|X(z|x) =

• 1 − p,

for z = x p, for z = x Alphabets are X = Y = {0, 1} Assume crossover probability p ∈ (0, 1) and p = 1

2

Anuran Makur (MIT) Permutation Channels 10 July 2020 10 / 40

Example: Binary Symmetric Channel

ENCODER BSC 𝒒 RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Channel is binary symmetric channel, denoted BSC(p):

∀z, x ∈ {0, 1}, PZ|X(z|x) =

• 1 − p,

for z = x p, for z = x Alphabets are X = Y = {0, 1} Assume crossover probability p ∈ (0, 1) and p = 1

2

Question: What is the permutation channel capacity of the BSC?

Anuran Makur (MIT) Permutation Channels 10 July 2020 10 / 40

Outline

1

Introduction

2

Achievability and Converse for the BSC Encoder and Decoder Testing between Converging Hypotheses Second Moment Method for TV Distance Fano’s Inequality and CLT Approximation

3

General Achievability Bound

4

General Converse Bounds

5

Conclusion

Anuran Makur (MIT) Permutation Channels 10 July 2020 11 / 40

Warm-up: Sending Two Messages

ENCODER BSC 𝒒 RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Fix a message m ∈ {0, 1}

𝑟 1 3 𝑟 2 3 1

✶

Anuran Makur (MIT) Permutation Channels 10 July 2020 12 / 40

Warm-up: Sending Two Messages

ENCODER BSC 𝒒 RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Fix a message m ∈ {0, 1}, and encode m as fn(m) = X n

1 i.i.d.

∼ Ber(qm)

𝑟 1 3 𝑟 2 3 1

✶

Anuran Makur (MIT) Permutation Channels 10 July 2020 12 / 40

Warm-up: Sending Two Messages

ENCODER BSC 𝒒 RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Fix a message m ∈ {0, 1}, and encode m as fn(m) = X n

1 i.i.d.

∼ Ber(qm)

𝑟 1 3 𝑟 2 3 1

Memoryless BSC(p) outputs Z n

1 i.i.d.

∼ Ber(p ∗ qm), where p ∗ qm p(1 − qm) + qm(1 − p) is the convolution of p and qm ✶

Anuran Makur (MIT) Permutation Channels 10 July 2020 12 / 40

Warm-up: Sending Two Messages

ENCODER BSC 𝒒 RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Fix a message m ∈ {0, 1}, and encode m as fn(m) = X n

1 i.i.d.

∼ Ber(qm)

𝑟 1 3 𝑟 2 3 1

Memoryless BSC(p) outputs Z n

1 i.i.d.

∼ Ber(p ∗ qm), where p ∗ qm p(1 − qm) + qm(1 − p) is the convolution of p and qm Random permutation generates Y n

1 i.i.d.

∼ Ber(p ∗ qm) ✶

Anuran Makur (MIT) Permutation Channels 10 July 2020 12 / 40

Warm-up: Sending Two Messages

ENCODER BSC 𝒒 RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Fix a message m ∈ {0, 1}, and encode m as fn(m) = X n

1 i.i.d.

∼ Ber(qm)

𝑟 1 3 𝑟 2 3 1

Memoryless BSC(p) outputs Z n

1 i.i.d.

∼ Ber(p ∗ qm), where p ∗ qm p(1 − qm) + qm(1 − p) is the convolution of p and qm Random permutation generates Y n

1 i.i.d.

∼ Ber(p ∗ qm) Maximum Likelihood (ML) decoder: ˆ M = ✶ 1

n

n

i=1 Yi ≥ 1 2

• (for p < 1

2)

Anuran Makur (MIT) Permutation Channels 10 July 2020 12 / 40

Warm-up: Sending Two Messages

ENCODER BSC 𝒒 RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Fix a message m ∈ {0, 1}, and encode m as fn(m) = X n

1 i.i.d.

∼ Ber(qm)

𝑟 1 3 𝑟 2 3 1

Memoryless BSC(p) outputs Z n

1 i.i.d.

∼ Ber(p ∗ qm), where p ∗ qm p(1 − qm) + qm(1 − p) is the convolution of p and qm Random permutation generates Y n

1 i.i.d.

∼ Ber(p ∗ qm) Maximum Likelihood (ML) decoder: ˆ M = ✶ 1

n

n

i=1 Yi ≥ 1 2

• (for p < 1

2) 1 n

n

i=1 Yi → p ∗ qm in probability as n → ∞

⇒ lim

n→∞ Pn error = 0 as p ∗ q0 = p ∗ q1

Anuran Makur (MIT) Permutation Channels 10 July 2020 12 / 40

Encoder and Decoder

Suppose M = {1, . . . , nR} for some R > 0

1 𝑜

Anuran Makur (MIT) Permutation Channels 10 July 2020 13 / 40

Encoder and Decoder

Suppose M = {1, . . . , nR} for some R > 0 Randomized encoder: Given m ∈ M, fn(m) = X n

1 i.i.d.

∼ Ber m nR

• 1

𝑜

Anuran Makur (MIT) Permutation Channels 10 July 2020 13 / 40

Encoder and Decoder

Suppose M = {1, . . . , nR} for some R > 0 Randomized encoder: Given m ∈ M, fn(m) = X n

1 i.i.d.

∼ Ber m nR

• 1

𝑜

Given m ∈ M, Y n

1 i.i.d.

∼ Ber

• p ∗ m

nR

• (as before)

Anuran Makur (MIT) Permutation Channels 10 July 2020 13 / 40

Encoder and Decoder

Suppose M = {1, . . . , nR} for some R > 0 Randomized encoder: Given m ∈ M, fn(m) = X n

1 i.i.d.

∼ Ber m nR

• 1

𝑜

Given m ∈ M, Y n

1 i.i.d.

∼ Ber

• p ∗ m

nR

• ML decoder: For yn

1 ∈ {0, 1}n, gn(yn 1 ) = arg max m∈M

PY n

1 |M(yn

1 |m)

Anuran Makur (MIT) Permutation Channels 10 July 2020 13 / 40

Encoder and Decoder

Suppose M = {1, . . . , nR} for some R > 0 Randomized encoder: Given m ∈ M, fn(m) = X n

1 i.i.d.

∼ Ber m nR

• 1

𝑜

Given m ∈ M, Y n

1 i.i.d.

∼ Ber

• p ∗ m

nR

• ML decoder: For yn

1 ∈ {0, 1}n, gn(yn 1 ) = arg max m∈M

PY n

1 |M(yn

1 |m)

Challenge: Although 1

n

n

i=1 Yi → p ∗ m nR in probability as n → ∞, consecutive messages

become indistinguishable, i.e.

m nR − m+1 nR

→ 0

Anuran Makur (MIT) Permutation Channels 10 July 2020 13 / 40

Encoder and Decoder

Suppose M = {1, . . . , nR} for some R > 0 Randomized encoder: Given m ∈ M, fn(m) = X n

1 i.i.d.

∼ Ber m nR

• 1

𝑜

Given m ∈ M, Y n

1 i.i.d.

∼ Ber

• p ∗ m

nR

• ML decoder: For yn

1 ∈ {0, 1}n, gn(yn 1 ) = arg max m∈M

PY n

1 |M(yn

1 |m)

Challenge: Although 1

n

n

i=1 Yi → p ∗ m nR in probability as n → ∞, consecutive messages

become indistinguishable, i.e.

m nR − m+1 nR

→ 0 Fact: Consecutive messages distinguishable ⇒ lim

n→∞ Pn error = 0

Anuran Makur (MIT) Permutation Channels 10 July 2020 13 / 40

Encoder and Decoder

Suppose M = {1, . . . , nR} for some R > 0 Randomized encoder: Given m ∈ M, fn(m) = X n

1 i.i.d.

∼ Ber m nR

• 1

𝑜

Given m ∈ M, Y n

1 i.i.d.

∼ Ber

• p ∗ m

nR

• ML decoder: For yn

1 ∈ {0, 1}n, gn(yn 1 ) = arg max m∈M

PY n

1 |M(yn

1 |m)

Challenge: Although 1

n

n

i=1 Yi → p ∗ m nR in probability as n → ∞, consecutive messages

become indistinguishable, i.e.

m nR − m+1 nR

→ 0 Fact: Consecutive messages distinguishable ⇒ lim

n→∞ Pn error = 0

What is the largest R such that two consecutive messages can be distinguished?

Anuran Makur (MIT) Permutation Channels 10 July 2020 13 / 40

Testing between Converging Hypotheses

Binary Hypothesis Testing: Consider hypothesis H ∼ Ber 1

2

• with uniform prior

Anuran Makur (MIT) Permutation Channels 10 July 2020 14 / 40

Testing between Converging Hypotheses

Binary Hypothesis Testing: Consider hypothesis H ∼ Ber 1

2

• with uniform prior

For any n ∈ N, q ∈ (0, 1), and R > 0, consider likelihoods: Given H = 0 : X n

1 i.i.d.

∼ PX|H=0 = Ber(q) Given H = 1 : X n

1 i.i.d.

∼ PX|H=1 = Ber

• q + 1

nR

• Anuran Makur (MIT)

Permutation Channels 10 July 2020 14 / 40

Testing between Converging Hypotheses

Binary Hypothesis Testing: Consider hypothesis H ∼ Ber 1

2

• with uniform prior

For any n ∈ N, q ∈ (0, 1), and R > 0, consider likelihoods: Given H = 0 : X n

1 i.i.d.

∼ PX|H=0 = Ber(q) Given H = 1 : X n

1 i.i.d.

∼ PX|H=1 = Ber

• q + 1

nR

• Define the zero-mean sufficient statistic of X n

1 for H:

Tn 1 n

n

• i=1

Xi − q − 1 2nR

Anuran Makur (MIT) Permutation Channels 10 July 2020 14 / 40

Testing between Converging Hypotheses

Binary Hypothesis Testing: Consider hypothesis H ∼ Ber 1

2

• with uniform prior

For any n ∈ N, q ∈ (0, 1), and R > 0, consider likelihoods: Given H = 0 : X n

1 i.i.d.

∼ PX|H=0 = Ber(q) Given H = 1 : X n

1 i.i.d.

∼ PX|H=1 = Ber

• q + 1

nR

• Define the zero-mean sufficient statistic of X n

1 for H:

Tn 1 n

n

• i=1

Xi − q − 1 2nR Let ˆ Hn

ML(Tn) denote the ML decoder for H based on Tn with minimum probability of

error Pn

ML P( ˆ

Hn

ML(Tn) = H)

Anuran Makur (MIT) Permutation Channels 10 July 2020 14 / 40

Testing between Converging Hypotheses

Binary Hypothesis Testing: Consider hypothesis H ∼ Ber 1

2

• with uniform prior

For any n ∈ N, q ∈ (0, 1), and R > 0, consider likelihoods: Given H = 0 : X n

1 i.i.d.

∼ PX|H=0 = Ber(q) Given H = 1 : X n

1 i.i.d.

∼ PX|H=1 = Ber

• q + 1

nR

• Define the zero-mean sufficient statistic of X n

1 for H:

Tn 1 n

n

• i=1

Xi − q − 1 2nR Let ˆ Hn

ML(Tn) denote the ML decoder for H based on Tn with minimum probability of

error Pn

ML P( ˆ

Hn

ML(Tn) = H)

Want: Largest R > 0 such that lim

n→∞ Pn ML = 0?

Anuran Makur (MIT) Permutation Channels 10 July 2020 14 / 40

Intuition via Central Limit Theorem

For large n, PTn|H(·|0) and PTn|H(·|1) are Gaussian distributions Figure: 𝑢 𝑄

| 𝑢|0

𝑄

| 𝑢|1

1 2𝑜 1 2𝑜

Anuran Makur (MIT) Permutation Channels 10 July 2020 15 / 40

Intuition via Central Limit Theorem

For large n, PTn|H(·|0) and PTn|H(·|1) are Gaussian distributions |E[Tn|H = 0] − E[Tn|H = 1]| = 1/nR Figure: 𝑢 𝑄

| 𝑢|0

𝑄

| 𝑢|1

𝛴 1 𝑜 1 𝑜 𝛴 1 𝑜

Anuran Makur (MIT) Permutation Channels 10 July 2020 15 / 40

Intuition via Central Limit Theorem

For large n, PTn|H(·|0) and PTn|H(·|1) are Gaussian distributions |E[Tn|H = 0] − E[Tn|H = 1]| = 1/nR Standard deviations are Θ

• 1/

√n

• Figure:

𝑢 𝑄

| 𝑢|0

𝑄

| 𝑢|1

𝛴 1 𝑜 1 𝑜 𝛴 1 𝑜

Anuran Makur (MIT) Permutation Channels 10 July 2020 15 / 40

Intuition via Central Limit Theorem

For large n, PTn|H(·|0) and PTn|H(·|1) are Gaussian distributions |E[Tn|H = 0] − E[Tn|H = 1]| = 1/nR Standard deviations are Θ

• 1/

√n

• Case R < 1

2:

𝑢 𝑄

| 𝑢|0

𝑄

| 𝑢|1

𝛴 1 𝑜 1 𝑜 𝛴 1 𝑜

Anuran Makur (MIT) Permutation Channels 10 July 2020 15 / 40

Intuition via Central Limit Theorem

For large n, PTn|H(·|0) and PTn|H(·|1) are Gaussian distributions |E[Tn|H = 0] − E[Tn|H = 1]| = 1/nR Standard deviations are Θ

• 1/

√n

• Case R < 1

2: Decoding is possible

𝑢 𝑄

| 𝑢|0

𝑄

| 𝑢|1

𝛴 1 𝑜 1 𝑜 𝛴 1 𝑜

Anuran Makur (MIT) Permutation Channels 10 July 2020 15 / 40

Intuition via Central Limit Theorem

For large n, PTn|H(·|0) and PTn|H(·|1) are Gaussian distributions |E[Tn|H = 0] − E[Tn|H = 1]| = 1/nR Standard deviations are Θ

• 1/

√n

• Case R > 1

2:

𝑢 𝑄

| 𝑢|0

𝑄

| 𝑢|1

𝛴 1 𝑜 1 𝑜 𝛴 1 𝑜

Anuran Makur (MIT) Permutation Channels 10 July 2020 15 / 40

Intuition via Central Limit Theorem

For large n, PTn|H(·|0) and PTn|H(·|1) are Gaussian distributions |E[Tn|H = 0] − E[Tn|H = 1]| = 1/nR Standard deviations are Θ

• 1/

√n

• Case R > 1

2: Decoding is impossible

𝑢 𝑄

| 𝑢|0

𝑄

| 𝑢|1

𝛴 1 𝑜 1 𝑜 𝛴 1 𝑜

Anuran Makur (MIT) Permutation Channels 10 July 2020 15 / 40

Second Moment Method for TV Distance

Lemma (2nd Moment Method [EKPS00])

• PTn|H=1 − PTn|H=0
• TV ≥ (E[Tn|H = 1] − E[Tn|H = 0])2

4 VAR(Tn) where P − QTV = 1

2 P − Q1 denotes the total variation (TV) distance between the

distributions P and Q.

Anuran Makur (MIT) Permutation Channels 10 July 2020 16 / 40

Second Moment Method for TV Distance

Lemma (2nd Moment Method [EKPS00])

• PTn|H=1 − PTn|H=0
• TV ≥ (E[Tn|H = 1] − E[Tn|H = 0])2

4 VAR(Tn) where P − QTV = 1

2 P − Q1 denotes the total variation (TV) distance between the

distributions P and Q. Proof: Let T +

n ∼ PTn|H=1 and T − n ∼ PTn|H=0

• E
• T +

n

• − E
• T −

n

2 =

• t

t

• PTn|H(t|1) − PTn|H(t|0)
• 2

Anuran Makur (MIT) Permutation Channels 10 July 2020 16 / 40

Second Moment Method for TV Distance

Lemma (2nd Moment Method [EKPS00])

• PTn|H=1 − PTn|H=0
• TV ≥ (E[Tn|H = 1] − E[Tn|H = 0])2

4 VAR(Tn) where P − QTV = 1

2 P − Q1 denotes the total variation (TV) distance between the

distributions P and Q. Proof: Let T +

n ∼ PTn|H=1 and T − n ∼ PTn|H=0

• E
• T +

n

• − E
• T −

n

2 =

• t

t

• PTn(t)
• PTn|H(t|1) − PTn|H(t|0)
• PTn(t)
• 2

Anuran Makur (MIT) Permutation Channels 10 July 2020 16 / 40

Second Moment Method for TV Distance

Lemma (2nd Moment Method [EKPS00])

• PTn|H=1 − PTn|H=0
• TV ≥ (E[Tn|H = 1] − E[Tn|H = 0])2

4 VAR(Tn) where P − QTV = 1

2 P − Q1 denotes the total variation (TV) distance between the

distributions P and Q. Proof: Cauchy-Schwarz inequality

• E
• T +

n

• − E
• T −

n

2 =

• t

t

• PTn(t)
• PTn|H(t|1) − PTn|H(t|0)
• PTn(t)
• 2

≤

• t

t2PTn(t)

• t
• PTn|H(t|1) − PTn|H(t|0)

2 PTn(t)

• Anuran Makur (MIT)

Permutation Channels 10 July 2020 16 / 40

Second Moment Method for TV Distance

Lemma (2nd Moment Method [EKPS00])

• PTn|H=1 − PTn|H=0
• TV ≥ (E[Tn|H = 1] − E[Tn|H = 0])2

4 VAR(Tn) where P − QTV = 1

2 P − Q1 denotes the total variation (TV) distance between the

distributions P and Q. Proof: Recall that Tn is zero-mean

• E
• T +

n

• − E
• T −

n

2 =

• t

t

• PTn(t)
• PTn|H(t|1) − PTn|H(t|0)
• PTn(t)
• 2

≤ VAR(Tn)

• t
• PTn|H(t|1) − PTn|H(t|0)

2 PTn(t)

• Anuran Makur (MIT)

Permutation Channels 10 July 2020 16 / 40

Second Moment Method for TV Distance

Lemma (2nd Moment Method [EKPS00])

• PTn|H=1 − PTn|H=0
• TV ≥ (E[Tn|H = 1] − E[Tn|H = 0])2

4 VAR(Tn) where P − QTV = 1

2 P − Q1 denotes the total variation (TV) distance between the

distributions P and Q. Proof: Hammersley-Chapman-Robbins bound

• E
• T +

n

• − E
• T −

n

2 =

• t

t

• PTn(t)
• PTn|H(t|1) − PTn|H(t|0)
• PTn(t)
• 2

≤ 4 VAR(Tn)

• 1

4

• t
• PTn|H(t|1) − PTn|H(t|0)

2 PTn(t)

• Vincze-Le Cam distance

Anuran Makur (MIT) Permutation Channels 10 July 2020 16 / 40

Second Moment Method for TV Distance

Lemma (2nd Moment Method [EKPS00])

• PTn|H=1 − PTn|H=0
• TV ≥ (E[Tn|H = 1] − E[Tn|H = 0])2

4 VAR(Tn) where P − QTV = 1

2 P − Q1 denotes the total variation (TV) distance between the

distributions P and Q. Proof:

• E
• T +

n

• − E
• T −

n

2 =

• t

t

• PTn(t)
• PTn|H(t|1) − PTn|H(t|0)
• PTn(t)
• 2

≤ 4 VAR(Tn)

• 1

4

• t
• PTn|H(t|1) − PTn|H(t|0)

2 PTn(t)

• ≤ 4 VAR(Tn)
• PTn|H=1 − PTn|H=0
• TV

Anuran Makur (MIT) Permutation Channels 10 July 2020 16 / 40

BSC Achievability Proof

Proposition (BSC Achievability)

For any 0 < R < 1/2, consider the binary hypothesis testing problem with H ∼ Ber 1

2

• , and

X n

1 i.i.d.

∼ Ber

• q + h

nR

• given H = h ∈ {0, 1}.

Proof: Start with Le Cam’s relation Pn

ML = 1

2

• 1 −
• PTn|H=1 − PTn|H=0
• TV
• Anuran Makur (MIT)

Permutation Channels 10 July 2020 17 / 40

BSC Achievability Proof

Proposition (BSC Achievability)

For any 0 < R < 1/2, consider the binary hypothesis testing problem with H ∼ Ber 1

2

• , and

X n

1 i.i.d.

∼ Ber

• q + h

nR

• given H = h ∈ {0, 1}.

Proof: Apply second moment method lemma Pn

ML = 1

2

• 1 −
• PTn|H=1 − PTn|H=0
• TV
• ≤ 1

2

• 1 − (E[Tn|H = 1] − E[Tn|H = 0])2

4 VAR(Tn)

• Anuran Makur (MIT)

Permutation Channels 10 July 2020 17 / 40

BSC Achievability Proof

Proposition (BSC Achievability)

For any 0 < R < 1/2, consider the binary hypothesis testing problem with H ∼ Ber 1

2

• , and

X n

1 i.i.d.

∼ Ber

• q + h

nR

• given H = h ∈ {0, 1}.

Proof: After explicit computation and simplification... Pn

ML = 1

2

• 1 −
• PTn|H=1 − PTn|H=0
• TV
• ≤ 1

2

• 1 − (E[Tn|H = 1] − E[Tn|H = 0])2

4 VAR(Tn)

• Anuran Makur (MIT)

Permutation Channels 10 July 2020 17 / 40

BSC Achievability Proof

Proposition (BSC Achievability)

For any 0 < R < 1/2, consider the binary hypothesis testing problem with H ∼ Ber 1

2

• , and

X n

1 i.i.d.

∼ Ber

• q + h

nR

• given H = h ∈ {0, 1}.

Proof: For any 0 < R < 1

2,

Pn

ML = 1

2

• 1 −
• PTn|H=1 − PTn|H=0
• TV
• ≤ 1

2

• 1 − (E[Tn|H = 1] − E[Tn|H = 0])2

4 VAR(Tn)

• ≤

3 2n1−2R

Anuran Makur (MIT) Permutation Channels 10 July 2020 17 / 40

BSC Achievability Proof

Proposition (BSC Achievability)

For any 0 < R < 1/2, consider the binary hypothesis testing problem with H ∼ Ber 1

2

• , and

X n

1 i.i.d.

∼ Ber

• q + h

nR

• given H = h ∈ {0, 1}.

Then, lim

n→∞ Pn ML = 0.

Proof: For any 0 < R < 1

2,

Pn

ML = 1

2

• 1 −
• PTn|H=1 − PTn|H=0
• TV
• ≤ 1

2

• 1 − (E[Tn|H = 1] − E[Tn|H = 0])2

4 VAR(Tn)

• ≤

3 2n1−2R → 0 as n → ∞

Anuran Makur (MIT) Permutation Channels 10 July 2020 17 / 40

BSC Achievability Proof

Proposition (BSC Achievability)

For any 0 < R < 1/2, consider the binary hypothesis testing problem with H ∼ Ber 1

2

• , and

X n

1 i.i.d.

∼ Ber

• q + h

nR

• given H = h ∈ {0, 1}.

Then, lim

n→∞ Pn ML = 0. This implies that:

Cperm(BSC(p)) ≥ 1 2 . Proof: For any 0 < R < 1

2,

Pn

ML = 1

2

• 1 −
• PTn|H=1 − PTn|H=0
• TV
• ≤ 1

2

• 1 − (E[Tn|H = 1] − E[Tn|H = 0])2

4 VAR(Tn)

• ≤

3 2n1−2R → 0 as n → ∞

Anuran Makur (MIT) Permutation Channels 10 July 2020 17 / 40

Outline

1

Introduction

2

Achievability and Converse for the BSC Encoder and Decoder Testing between Converging Hypotheses Second Moment Method for TV Distance Fano’s Inequality and CLT Approximation

3

General Achievability Bound

4

General Converse Bounds

5

Conclusion

Anuran Makur (MIT) Permutation Channels 10 July 2020 18 / 40

Recall: Basic Definitions of Information Measures

Consider discrete random variables X ∈ X and Y ∈ Y with joint distribution PX,Y .

Anuran Makur (MIT) Permutation Channels 10 July 2020 19 / 40

Recall: Basic Definitions of Information Measures

Consider discrete random variables X ∈ X and Y ∈ Y with joint distribution PX,Y . Shannon Entropy: H(X) −

• x∈X

PX(x) log(PX(x))

Anuran Makur (MIT) Permutation Channels 10 July 2020 19 / 40

Recall: Basic Definitions of Information Measures

Consider discrete random variables X ∈ X and Y ∈ Y with joint distribution PX,Y . Shannon Entropy: H(X) −

• x∈X

PX(x) log(PX(x)) Conditional Shannon Entropy: H(X|Y ) −

• x∈X
• y∈Y

PX,Y (x, y) log

• PX|Y (x|y)
• Anuran Makur (MIT)

Permutation Channels 10 July 2020 19 / 40

Recall: Basic Definitions of Information Measures

Consider discrete random variables X ∈ X and Y ∈ Y with joint distribution PX,Y . Shannon Entropy: H(X) −

• x∈X

PX(x) log(PX(x)) Conditional Shannon Entropy: H(X|Y ) −

• x∈X
• y∈Y

PX,Y (x, y) log

• PX|Y (x|y)
• Mutual Information:

I(X; Y )

• x∈X
• y∈Y

PX,Y (x, y) log PX,Y (x, y) PX(y)PY (y)

• Anuran Makur (MIT)

Permutation Channels 10 July 2020 19 / 40

Recall: Basic Definitions of Information Measures

Consider discrete random variables X ∈ X and Y ∈ Y with joint distribution PX,Y . Shannon Entropy: H(X) −

• x∈X

PX(x) log(PX(x)) Conditional Shannon Entropy: H(X|Y ) −

• x∈X
• y∈Y

PX,Y (x, y) log

• PX|Y (x|y)
• Mutual Information:

I(X; Y )

• x∈X
• y∈Y

PX,Y (x, y) log PX,Y (x, y) PX(y)PY (y)

• = H(X) − H(X|Y )

Anuran Makur (MIT) Permutation Channels 10 July 2020 19 / 40

Recall: Two Information Inequalities

Consider discrete random variables X, Y , Z that form a Markov chain X → Y → Z.

Anuran Makur (MIT) Permutation Channels 10 July 2020 20 / 40

Recall: Two Information Inequalities

Consider discrete random variables X, Y , Z that form a Markov chain X → Y → Z.

Lemma (Data Processing Inequality [CT06])

I(X; Z) ≤ I(X; Y ) with equality if and only if Z is a sufficient statistic of Y for X, i.e., X → Z → Y also forms a Markov chain.

Anuran Makur (MIT) Permutation Channels 10 July 2020 20 / 40

Recall: Two Information Inequalities

Consider discrete random variables X, Y , Z that form a Markov chain X → Y → Z.

Lemma (Data Processing Inequality [CT06])

I(X; Z) ≤ I(X; Y ) with equality if and only if Z is a sufficient statistic of Y for X, i.e., X → Z → Y also forms a Markov chain.

Lemma (Fano’s Inequality [CT06])

If X takes values in the finite alphabet X, then H(X|Z) ≤ 1 + P(X = Z) log(|X|) where we perceive Z as an estimator for X based on Y .

Anuran Makur (MIT) Permutation Channels 10 July 2020 20 / 40

BSC Converse Proof: Fano’s Inequality Argument

Consider the Markov chain M → X n

1 → Z n 1 → Y n 1 → Sn n i=1 Yi → ˆ

M, and a sequence of encoder-decoder pairs {(fn, gn)}n∈N such that |M| = nR and lim

n→∞ Pn error = 0

Anuran Makur (MIT) Permutation Channels 10 July 2020 21 / 40

BSC Converse Proof: Fano’s Inequality Argument

Consider the Markov chain M → X n

1 → Z n 1 → Y n 1 → Sn n i=1 Yi → ˆ

M, and a sequence of encoder-decoder pairs {(fn, gn)}n∈N such that |M| = nR and lim

n→∞ Pn error = 0

Standard argument [CT06]: M is uniform R log(n) = H(M)

Anuran Makur (MIT) Permutation Channels 10 July 2020 21 / 40

BSC Converse Proof: Fano’s Inequality Argument

Consider the Markov chain M → X n

1 → Z n 1 → Y n 1 → Sn n i=1 Yi → ˆ

M, and a sequence of encoder-decoder pairs {(fn, gn)}n∈N such that |M| = nR and lim

n→∞ Pn error = 0

Standard argument [CT06]: Fano’s inequality, data processing inequality R log(n) = H(M| ˆ M) + I(M; ˆ M) ≤ 1 + Pn

errorR log(n) + I(M; Y n 1 )

Anuran Makur (MIT) Permutation Channels 10 July 2020 21 / 40

BSC Converse Proof: Fano’s Inequality Argument

Consider the Markov chain M → X n

1 → Z n 1 → Y n 1 → Sn n i=1 Yi → ˆ

M, and a sequence of encoder-decoder pairs {(fn, gn)}n∈N such that |M| = nR and lim

n→∞ Pn error = 0

Standard argument [CT06]: sufficiency R log(n) = H(M| ˆ M) + I(M; ˆ M) ≤ 1 + Pn

errorR log(n) + I(M; Y n 1 )

= 1 + Pn

errorR log(n) + I(M; Sn)

Anuran Makur (MIT) Permutation Channels 10 July 2020 21 / 40

BSC Converse Proof: Fano’s Inequality Argument

Consider the Markov chain M → X n

1 → Z n 1 → Y n 1 → Sn n i=1 Yi → ˆ

M, and a sequence of encoder-decoder pairs {(fn, gn)}n∈N such that |M| = nR and lim

n→∞ Pn error = 0

Standard argument [CT06]: data processing inequality R log(n) = H(M| ˆ M) + I(M; ˆ M) ≤ 1 + Pn

errorR log(n) + I(M; Y n 1 )

= 1 + Pn

errorR log(n) + I(M; Sn)

≤ 1 + Pn

errorR log(n) + I(X n 1 ; Sn)

Anuran Makur (MIT) Permutation Channels 10 July 2020 21 / 40

BSC Converse Proof: Fano’s Inequality Argument

Consider the Markov chain M → X n

1 → Z n 1 → Y n 1 → Sn n i=1 Yi → ˆ

M, and a sequence of encoder-decoder pairs {(fn, gn)}n∈N such that |M| = nR and lim

n→∞ Pn error = 0

Standard argument [CT06]: R log(n) = H(M| ˆ M) + I(M; ˆ M) ≤ 1 + Pn

errorR log(n) + I(M; Y n 1 )

= 1 + Pn

errorR log(n) + I(M; Sn)

≤ 1 + Pn

errorR log(n) + I(X n 1 ; Sn)

Divide by log(n) R ≤ 1 log(n) + Pn

errorR + I(X n 1 ; Sn)

log(n)

Anuran Makur (MIT) Permutation Channels 10 July 2020 21 / 40

BSC Converse Proof: Fano’s Inequality Argument

Consider the Markov chain M → X n

1 → Z n 1 → Y n 1 → Sn n i=1 Yi → ˆ

M, and a sequence of encoder-decoder pairs {(fn, gn)}n∈N such that |M| = nR and lim

n→∞ Pn error = 0

Standard argument [CT06]: R log(n) = H(M| ˆ M) + I(M; ˆ M) ≤ 1 + Pn

errorR log(n) + I(M; Y n 1 )

= 1 + Pn

errorR log(n) + I(M; Sn)

≤ 1 + Pn

errorR log(n) + I(X n 1 ; Sn)

Divide by log(n) and let n → ∞: R ≤ lim

n→∞

I(X n

1 ; Sn)

log(n)

Anuran Makur (MIT) Permutation Channels 10 July 2020 21 / 40

BSC Converse Proof: CLT Approximation

Upper bound on I(X n

1 ; Sn):

I(X n

1 ; Sn) = H(Sn) − H(Sn|X n 1 )

Anuran Makur (MIT) Permutation Channels 10 July 2020 22 / 40

BSC Converse Proof: CLT Approximation

Since Sn ∈ {0, . . . , n}, I(X n

1 ; Sn) = H(Sn) − H(Sn|X n 1 )

≤ log(n + 1) −

• xn

1 ∈{0,1}n

PX n

1 (xn

1 ) H(Sn|X n 1 = xn 1 )

Anuran Makur (MIT) Permutation Channels 10 July 2020 22 / 40

BSC Converse Proof: CLT Approximation

Given X n

1 = xn 1 with n i=1 xi = k, Sn = bin(k, 1 − p) + bin(n − k, p):

I(X n

1 ; Sn) = H(Sn) − H(Sn|X n 1 )

≤ log(n + 1) −

• xn

1 ∈{0,1}n

PX n

1 (xn

1 ) H(bin(k, 1 − p) + bin(n − k, p))

Anuran Makur (MIT) Permutation Channels 10 July 2020 22 / 40

BSC Converse Proof: CLT Approximation

Using [CT06, Problem 2.14], i.e., max{H(X), H(Y )} ≤ H(X + Y ) for X ⊥ ⊥ Y , I(X n

1 ; Sn) = H(Sn) − H(Sn|X n 1 )

≤ log(n + 1) −

• xn

1 ∈{0,1}n

PX n

1 (xn

1 ) H(bin(k, 1 − p) + bin(n − k, p))

≤ log(n + 1) −

• xn

1 ∈{0,1}n

PX n

1 (xn

1 ) H

• bin

n 2, p

• Anuran Makur (MIT)

Permutation Channels 10 July 2020 22 / 40

BSC Converse Proof: CLT Approximation

Approximate binomial entropy using CLT [ALY10]: I(X n

1 ; Sn) = H(Sn) − H(Sn|X n 1 )

≤ log(n + 1) −

• xn

1 ∈{0,1}n

PX n

1 (xn

1 ) H(bin(k, 1 − p) + bin(n − k, p))

≤ log(n + 1) −

• xn

1 ∈{0,1}n

PX n

1 (xn

1 ) H

• bin

n 2, p

• = log(n + 1) −
• xn

1 ∈{0,1}n

PX n

1 (xn

1 )

1 2 log(πep(1 − p)n) + O 1 n

• Anuran Makur (MIT)

Permutation Channels 10 July 2020 22 / 40

BSC Converse Proof: CLT Approximation

Upper bound on I(X n

1 ; Sn):

I(X n

1 ; Sn) = H(Sn) − H(Sn|X n 1 )

≤ log(n + 1) −

• xn

1 ∈{0,1}n

PX n

1 (xn

1 ) H(bin(k, 1 − p) + bin(n − k, p))

≤ log(n + 1) −

• xn

1 ∈{0,1}n

PX n

1 (xn

1 ) H

• bin

n 2, p

• = log(n + 1) − 1

2 log(πep(1 − p)n) + O 1 n

• Anuran Makur (MIT)

Permutation Channels 10 July 2020 22 / 40

BSC Converse Proof: CLT Approximation

Upper bound on I(X n

1 ; Sn):

I(X n

1 ; Sn) = H(Sn) − H(Sn|X n 1 )

≤ log(n + 1) −

• xn

1 ∈{0,1}n

PX n

1 (xn

1 ) H(bin(k, 1 − p) + bin(n − k, p))

≤ log(n + 1) −

• xn

1 ∈{0,1}n

PX n

1 (xn

1 ) H

• bin

n 2, p

• = log(n + 1) − 1

2 log(πep(1 − p)n) + O 1 n

• Hence, we have R ≤ lim

n→∞ I(X n 1 ; Sn)/log(n) = 1 2.

Anuran Makur (MIT) Permutation Channels 10 July 2020 22 / 40

BSC Converse Proof: CLT Approximation

Upper bound on I(X n

1 ; Sn):

I(X n

1 ; Sn) = H(Sn) − H(Sn|X n 1 )

≤ log(n + 1) −

• xn

1 ∈{0,1}n

PX n

1 (xn

1 ) H(bin(k, 1 − p) + bin(n − k, p))

≤ log(n + 1) −

• xn

1 ∈{0,1}n

PX n

1 (xn

1 ) H

• bin

n 2, p

• = log(n + 1) − 1

2 log(πep(1 − p)n) + O 1 n

• Hence, we have R ≤ lim

n→∞ I(X n 1 ; Sn)/log(n) = 1 2.

Proposition (BSC Converse)

Cperm(BSC(p)) ≤ 1 2

Anuran Makur (MIT) Permutation Channels 10 July 2020 22 / 40

Information Capacity of the BSC Permutation Channel

Proposition (Pemutation Channel Capacity of BSC)

Cperm(BSC(p)) =      1, for p = 0, 1

1 2,

for p ∈

• 0, 1

2

• ∪

1

2, 1

• 0,

for p = 1

2

𝑞 𝐷perm BSC 𝑞 1 1

1 2 1 2

Anuran Makur (MIT) Permutation Channels 10 July 2020 23 / 40

Information Capacity of the BSC Permutation Channel

Proposition (Pemutation Channel Capacity of BSC)

Cperm(BSC(p)) =      1, for p = 0, 1

1 2,

for p ∈

• 0, 1

2

• ∪

1

2, 1

• 0,

for p = 1

2

𝑞 𝐷perm BSC 𝑞 1 1

1 2 1 2

Remarks: Cperm(·) is discontinuous and non-convex

Anuran Makur (MIT) Permutation Channels 10 July 2020 23 / 40

Information Capacity of the BSC Permutation Channel

Proposition (Pemutation Channel Capacity of BSC)

Cperm(BSC(p)) =      1, for p = 0, 1

1 2,

for p ∈

• 0, 1

2

• ∪

1

2, 1

• 0,

for p = 1

2

𝑞 𝐷perm BSC 𝑞 1 1

1 2 1 2

Remarks: Cperm(·) is discontinuous and non-convex Cperm(·) is generally agnostic to parameters of channel

Anuran Makur (MIT) Permutation Channels 10 July 2020 23 / 40

Information Capacity of the BSC Permutation Channel

Proposition (Pemutation Channel Capacity of BSC)

Cperm(BSC(p)) =      1, for p = 0, 1

1 2,

for p ∈

• 0, 1

2

• ∪

1

2, 1

• 0,

for p = 1

2

𝑞 𝐷perm BSC 𝑞 1 1

1 2 1 2

Remarks: Cperm(·) is discontinuous and non-convex Cperm(·) is generally agnostic to parameters of channel Computationally tractable coding scheme in achievability proof

Anuran Makur (MIT) Permutation Channels 10 July 2020 23 / 40

Outline

1

Introduction

2

Achievability and Converse for the BSC

3

General Achievability Bound Coding Scheme Rank Bound

4

General Converse Bounds

5

Conclusion

Anuran Makur (MIT) Permutation Channels 10 July 2020 24 / 40

Recall General Problem

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Average probability of error Pn

error P(M = ˆ

M) “Rate” of coding scheme (fn, gn) is R log(|M|) log(n) Rate R ≥ 0 is achievable ⇔ ∃ {(fn, gn)}n∈N such that lim

n→∞ Pn error = 0

Definition (Permutation Channel Capacity)

Cperm(PZ|X) sup{R ≥ 0 : R is achievable}

Main Question

What is the permutation channel capacity of a general PZ|X?

Anuran Makur (MIT) Permutation Channels 10 July 2020 25 / 40

Achievability: Coding Scheme

Let r = rank(PZ|X) and k = √n

• Anuran Makur (MIT)

Permutation Channels 10 July 2020 26 / 40

Achievability: Coding Scheme

Let r = rank(PZ|X) and k = √n

• Consider X ′ ⊆ X with |X ′| = r such that {PZ|X(·|x) : x ∈ X ′} are linearly independent

Anuran Makur (MIT) Permutation Channels 10 July 2020 26 / 40

Achievability: Coding Scheme

Let r = rank(PZ|X) and k = √n

• Consider X ′ ⊆ X with |X ′| = r such that {PZ|X(·|x) : x ∈ X ′} are linearly independent

Message set: M

• p = (p(x) : x ∈ X ′) ∈ (Z+)X ′ :
• x∈X ′

p(x) = k

• Anuran Makur (MIT)

Permutation Channels 10 July 2020 26 / 40

Achievability: Coding Scheme

Let r = rank(PZ|X) and k = √n

• Consider X ′ ⊆ X with |X ′| = r such that {PZ|X(·|x) : x ∈ X ′} are linearly independent

Message set: M

• p = (p(x) : x ∈ X ′) ∈ (Z+)X ′ :
• x∈X ′

p(x) = k

• where |M| =

k+r−1

r−1

• = Θ
• n

r−1 2 Anuran Makur (MIT) Permutation Channels 10 July 2020 26 / 40

Achievability: Coding Scheme

Let r = rank(PZ|X) and k = √n

• Consider X ′ ⊆ X with |X ′| = r such that {PZ|X(·|x) : x ∈ X ′} are linearly independent

Message set: M

• p = (p(x) : x ∈ X ′) ∈ (Z+)X ′ :
• x∈X ′

p(x) = k

• where |M| =

k+r−1

r−1

• = Θ
• n

r−1 2

Randomized Encoder: ∀p ∈ M, fn(p) = X n

1 i.i.d.

∼ PX where PX(x) = p(x)

k ,

for x ∈ X ′ 0, for x ∈ X\X ′

Anuran Makur (MIT) Permutation Channels 10 July 2020 26 / 40

Achievability: Coding Scheme

Let stochastic matrix ˜ PZ|X ∈ Rr×|Y| have rows {PZ|X(·|x) : x ∈ X ′} Let ˜ P†

Z|X denote its Moore-Penrose pseudoinverse

✶

Anuran Makur (MIT) Permutation Channels 10 July 2020 27 / 40

Achievability: Coding Scheme

Let stochastic matrix ˜ PZ|X ∈ Rr×|Y| have rows {PZ|X(·|x) : x ∈ X ′} Let ˜ P†

Z|X denote its Moore-Penrose pseudoinverse

(Sub-optimal) Thresholding Decoder: For any yn

1 ∈ Yn,

Step 1: Construct its type/empirical distribution/histogram ∀y ∈ Y, ˆ Pyn

1 (y) = 1

n

n

• i=1

✶{yi = y}

Anuran Makur (MIT) Permutation Channels 10 July 2020 27 / 40

Achievability: Coding Scheme

Let stochastic matrix ˜ PZ|X ∈ Rr×|Y| have rows {PZ|X(·|x) : x ∈ X ′} Let ˜ P†

Z|X denote its Moore-Penrose pseudoinverse

(Sub-optimal) Thresholding Decoder: For any yn

1 ∈ Yn,

Step 1: Construct its type/empirical distribution/histogram ∀y ∈ Y, ˆ Pyn

1 (y) = 1

n

n

• i=1

✶{yi = y} Step 2: Generate estimate ˆ p ∈ (Z+)X ′ with components ∀x ∈ X ′, ˆ p(x) = arg min

j∈{0,...,k}

• y∈Y

ˆ Pyn

1 (y)

• ˜

P†

Z|X

• y,x − j

k

• Anuran Makur (MIT)

Permutation Channels 10 July 2020 27 / 40

Achievability: Coding Scheme

Let stochastic matrix ˜ PZ|X ∈ Rr×|Y| have rows {PZ|X(·|x) : x ∈ X ′} Let ˜ P†

Z|X denote its Moore-Penrose pseudoinverse

(Sub-optimal) Thresholding Decoder: For any yn

1 ∈ Yn,

Step 1: Construct its type/empirical distribution/histogram ∀y ∈ Y, ˆ Pyn

1 (y) = 1

n

n

• i=1

✶{yi = y} Step 2: Generate estimate ˆ p ∈ (Z+)X ′ with components ∀x ∈ X ′, ˆ p(x) = arg min

j∈{0,...,k}

• y∈Y

ˆ Pyn

1 (y)

• ˜

P†

Z|X

• y,x − j

k

• Step 3: Output decoded message

gn(yn

1 ) =

• ˆ

p, if ˆ p ∈ M error,

• therwise

Anuran Makur (MIT) Permutation Channels 10 July 2020 27 / 40

Achievability: Rank Bound

Theorem (Rank Bound)

For any channel PZ|X: Cperm(PZ|X) ≥ rank(PZ|X) − 1 2 . Remarks about Coding Scheme: Showing limn→∞ Pn

error = 0 proves theorem.

Anuran Makur (MIT) Permutation Channels 10 July 2020 28 / 40

Achievability: Rank Bound

Theorem (Rank Bound)

For any channel PZ|X: Cperm(PZ|X) ≥ rank(PZ|X) − 1 2 . Remarks about Coding Scheme: Showing limn→∞ Pn

error = 0 proves theorem.

Intuition: Conditioned on M = p, ˆ PY n

1 ≈ PZ with high probability as n → ∞. Anuran Makur (MIT) Permutation Channels 10 July 2020 28 / 40

Achievability: Rank Bound

Theorem (Rank Bound)

For any channel PZ|X: Cperm(PZ|X) ≥ rank(PZ|X) − 1 2 . Remarks about Coding Scheme: Showing limn→∞ Pn

error = 0 proves theorem.

Intuition: Conditioned on M = p, ˆ PY n

1 ≈ PZ with high probability as n → ∞.

Hence,

y∈Y ˆ

PY n

1 (y)

˜ P†

Z|X

• y,x ≈ PX(x) for all x ∈ X ′ with high probability.

Anuran Makur (MIT) Permutation Channels 10 July 2020 28 / 40

Achievability: Rank Bound

Theorem (Rank Bound)

For any channel PZ|X: Cperm(PZ|X) ≥ rank(PZ|X) − 1 2 . Remarks about Coding Scheme: Showing limn→∞ Pn

error = 0 proves theorem.

Intuition: Conditioned on M = p, ˆ PY n

1 ≈ PZ with high probability as n → ∞.

Hence,

y∈Y ˆ

PY n

1 (y)

˜ P†

Z|X

• y,x ≈ PX(x) for all x ∈ X ′ with high probability.

Computational complexity: Decoder has O(n) running time.

Anuran Makur (MIT) Permutation Channels 10 July 2020 28 / 40

Achievability: Rank Bound

Theorem (Rank Bound)

For any channel PZ|X: Cperm(PZ|X) ≥ rank(PZ|X) − 1 2 . Remarks about Coding Scheme: Showing limn→∞ Pn

error = 0 proves theorem.

Intuition: Conditioned on M = p, ˆ PY n

1 ≈ PZ with high probability as n → ∞.

Hence,

y∈Y ˆ

PY n

1 (y)

˜ P†

Z|X

• y,x ≈ PX(x) for all x ∈ X ′ with high probability.

Computational complexity: Decoder has O(n) running time. Probabilistic method: Good deterministic codes exist.

Anuran Makur (MIT) Permutation Channels 10 July 2020 28 / 40

Achievability: Rank Bound

Theorem (Rank Bound)

For any channel PZ|X: Cperm(PZ|X) ≥ rank(PZ|X) − 1 2 . Remarks about Coding Scheme: Showing limn→∞ Pn

error = 0 proves theorem.

Intuition: Conditioned on M = p, ˆ PY n

1 ≈ PZ with high probability as n → ∞.

Hence,

y∈Y ˆ

PY n

1 (y)

˜ P†

Z|X

• y,x ≈ PX(x) for all x ∈ X ′ with high probability.

Computational complexity: Decoder has O(n) running time. Probabilistic method: Good deterministic codes exist. Expurgation: Achievability bound holds under maximal probability of error criterion.

Anuran Makur (MIT) Permutation Channels 10 July 2020 28 / 40

Outline

1

Introduction

2

Achievability and Converse for the BSC

3

General Achievability Bound

4

General Converse Bounds Output Alphabet Bound Effective Input Alphabet Bound Degradation by Symmetric Channels

5

Conclusion

Anuran Makur (MIT) Permutation Channels 10 July 2020 29 / 40

Converse: Output Alphabet Bound

Theorem (Output Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ |Y| − 1 2 .

Anuran Makur (MIT) Permutation Channels 10 July 2020 30 / 40

Converse: Output Alphabet Bound

Theorem (Output Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ |Y| − 1 2 . Remarks: Proof hinges on Fano’s inequality and CLT approximation of binomial entropy.

Anuran Makur (MIT) Permutation Channels 10 July 2020 30 / 40

Converse: Output Alphabet Bound

Theorem (Output Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ |Y| − 1 2 . Remarks: Proof hinges on Fano’s inequality and CLT approximation of binomial entropy. What if |X| is much smaller than |Y|?

Anuran Makur (MIT) Permutation Channels 10 July 2020 30 / 40

Converse: Output Alphabet Bound

Theorem (Output Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ |Y| − 1 2 . Remarks: Proof hinges on Fano’s inequality and CLT approximation of binomial entropy. What if |X| is much smaller than |Y|? Want: Converse bound in terms of input alphabet size.

Anuran Makur (MIT) Permutation Channels 10 July 2020 30 / 40

Converse: Effective Input Alphabet Bound

Theorem (Effective Input Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ ext(PZ|X) − 1 2 where ext(PZ|X) denotes the number of extreme points of conv

• PZ|X(·|x) : x ∈ X
• .

Anuran Makur (MIT) Permutation Channels 10 July 2020 31 / 40

Converse: Effective Input Alphabet Bound

Theorem (Effective Input Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ ext(PZ|X) − 1 2 where ext(PZ|X) denotes the number of extreme points of conv

• PZ|X(·|x) : x ∈ X
• .

Remarks: Effective input alphabet size: rank(PZ|X) ≤ ext(PZ|X) ≤ |X|.

Anuran Makur (MIT) Permutation Channels 10 July 2020 31 / 40

Converse: Effective Input Alphabet Bound

Theorem (Effective Input Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ ext(PZ|X) − 1 2 where ext(PZ|X) denotes the number of extreme points of conv

• PZ|X(·|x) : x ∈ X
• .

Remarks: Effective input alphabet size: rank(PZ|X) ≤ ext(PZ|X) ≤ |X|. For any channel PZ|X > 0, Cperm(PZ|X) ≤

• min{ext(PZ|X), |Y|} − 1
• /2.

Anuran Makur (MIT) Permutation Channels 10 July 2020 31 / 40

Converse: Effective Input Alphabet Bound

Theorem (Effective Input Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ ext(PZ|X) − 1 2 where ext(PZ|X) denotes the number of extreme points of conv

• PZ|X(·|x) : x ∈ X
• .

Remarks: Effective input alphabet size: rank(PZ|X) ≤ ext(PZ|X) ≤ |X|. For any channel PZ|X > 0, Cperm(PZ|X) ≤

• min{ext(PZ|X), |Y|} − 1
• /2.

For any general channel PZ|X, Cperm(PZ|X) ≤ min{ext(PZ|X), |Y|} − 1.

Anuran Makur (MIT) Permutation Channels 10 July 2020 31 / 40

Converse: Effective Input Alphabet Bound

Theorem (Effective Input Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ ext(PZ|X) − 1 2 where ext(PZ|X) denotes the number of extreme points of conv

• PZ|X(·|x) : x ∈ X
• .

Remarks: Effective input alphabet size: rank(PZ|X) ≤ ext(PZ|X) ≤ |X|. For any channel PZ|X > 0, Cperm(PZ|X) ≤

• min{ext(PZ|X), |Y|} − 1
• /2.

For any general channel PZ|X, Cperm(PZ|X) ≤ min{ext(PZ|X), |Y|} − 1. How do we prove above theorem?

Anuran Makur (MIT) Permutation Channels 10 July 2020 31 / 40

Brief Digression: Degradation

Definition (Degradation/Blackwell Order [Bla51], [She51], [Ste51], [Cov72], [Ber73])

Given channels PZ1|X and PZ2|X with common input alphabet X, PZ2|X is a degraded version

• f PZ1|X if PZ2|X = PZ1|XPZ2|Z1 for some channel PZ2|Z1.

Anuran Makur (MIT) Permutation Channels 10 July 2020 32 / 40

Brief Digression: Degradation

Definition (Degradation/Blackwell Order [Bla51], [She51], [Ste51], [Cov72], [Ber73])

Given channels PZ1|X and PZ2|X with common input alphabet X, PZ2|X is a degraded version

• f PZ1|X if PZ2|X = PZ1|XPZ2|Z1 for some channel PZ2|Z1.

Theorem (Blackwell-Sherman-Stein [Bla51], [She51], [Ste51])

The observation model PZ2|X is a degraded version of PZ1|X if and only if for every prior distribution PX, and every loss function L : X × X → R, the Bayes risks satisfy: min

f (·) E [L(X, f (Z1))] ≤ min g(·) E [L(X, g(Z2))]

where the minima are over all randomized estimators of X.

Anuran Makur (MIT) Permutation Channels 10 July 2020 32 / 40

Brief Digression: Symmetric Channels

Definition (q-ary Symmetric Channel)

A q-ary symmetric channel, denoted q-SC(δ), with total crossover probability δ ∈ [0, 1] and alphabet X where |X| = q, is given by the doubly stochastic matrix: Wδ       1 − δ

δ q−1

· · ·

δ q−1 δ q−1

1 − δ · · ·

δ q−1

. . . . . . ... . . .

δ q−1 δ q−1

· · · 1 − δ       .

Anuran Makur (MIT) Permutation Channels 10 July 2020 33 / 40

Brief Digression: Symmetric Channels

Definition (q-ary Symmetric Channel)

A q-ary symmetric channel, denoted q-SC(δ), with total crossover probability δ ∈ [0, 1] and alphabet X where |X| = q, is given by the doubly stochastic matrix: Wδ       1 − δ

δ q−1

· · ·

δ q−1 δ q−1

1 − δ · · ·

δ q−1

. . . . . . ... . . .

δ q−1 δ q−1

· · · 1 − δ       .

Proposition (Degradation by Symmetric Channels)

Given channel PZ|X with ν = min

x∈X, y∈Y PZ|X(y|x),

if 0 ≤ δ ≤ ν 1 − ν +

ν q−1

, then PZ|X is a degraded version of q-SC(δ).

Anuran Makur (MIT) Permutation Channels 10 July 2020 33 / 40

Brief Digression: Symmetric Channels

Proposition (Degradation by Symmetric Channels)

Given channel PZ|X with ν = min

x∈X, y∈Y PZ|X(y|x),

if 0 ≤ δ ≤ ν 1 − ν +

ν q−1

, then PZ|X is a degraded version of q-SC(δ). Remarks: Prop follows from computing extremal δ such that W −1

δ

PZ|X is row stochastic.

Anuran Makur (MIT) Permutation Channels 10 July 2020 34 / 40

Brief Digression: Symmetric Channels

Proposition (Degradation by Symmetric Channels)

Given channel PZ|X with ν = min

x∈X, y∈Y PZ|X(y|x),

if 0 ≤ δ ≤ ν 1 − ν +

ν q−1

, then PZ|X is a degraded version of q-SC(δ). Remarks: Prop follows from computing extremal δ such that W −1

δ

PZ|X is row stochastic. Bound on δ can be improved when more is known about PZ|X:

Anuran Makur (MIT) Permutation Channels 10 July 2020 34 / 40

Brief Digression: Symmetric Channels

Proposition (Degradation by Symmetric Channels)

Given channel PZ|X with ν = min

x∈X, y∈Y PZ|X(y|x),

if 0 ≤ δ ≤ ν 1 − ν +

ν q−1

, then PZ|X is a degraded version of q-SC(δ). Remarks: Prop follows from computing extremal δ such that W −1

δ

PZ|X is row stochastic. Bound on δ can be improved when more is known about PZ|X: Markov chain [MP18]: δ ≤ ν/

• 1 − (q − 1)ν +

ν q−1

• .

Anuran Makur (MIT) Permutation Channels 10 July 2020 34 / 40

Brief Digression: Symmetric Channels

Proposition (Degradation by Symmetric Channels)

Given channel PZ|X with ν = min

x∈X, y∈Y PZ|X(y|x),

if 0 ≤ δ ≤ ν 1 − ν +

ν q−1

, then PZ|X is a degraded version of q-SC(δ). Remarks: Prop follows from computing extremal δ such that W −1

δ

PZ|X is row stochastic. Bound on δ can be improved when more is known about PZ|X: Markov chain [MP18]: δ ≤ ν/

• 1 − (q − 1)ν +

ν q−1

• .

Additive noise channel on Abelian group X [MP18]: δ ≤ (q − 1)ν.

Anuran Makur (MIT) Permutation Channels 10 July 2020 34 / 40

Brief Digression: Symmetric Channels

Proposition (Degradation by Symmetric Channels)

Given channel PZ|X with ν = min

x∈X, y∈Y PZ|X(y|x),

if 0 ≤ δ ≤ ν 1 − ν +

ν q−1

, then PZ|X is a degraded version of q-SC(δ). Remarks: Prop follows from computing extremal δ such that W −1

δ

PZ|X is row stochastic. Bound on δ can be improved when more is known about PZ|X: Markov chain [MP18]: δ ≤ ν/

• 1 − (q − 1)ν +

ν q−1

• .

Additive noise channel on Abelian group X [MP18]: δ ≤ (q − 1)ν. Alternative bounds for Markov chains [MOS13].

Anuran Makur (MIT) Permutation Channels 10 July 2020 34 / 40

Brief Digression: Symmetric Channels

Proposition (Degradation by Symmetric Channels)

Given channel PZ|X with ν = min

x∈X, y∈Y PZ|X(y|x),

if 0 ≤ δ ≤ ν 1 − ν +

ν q−1

, then PZ|X is a degraded version of q-SC(δ). Remarks: Prop follows from computing extremal δ such that W −1

δ

PZ|X is row stochastic. Bound on δ can be improved when more is known about PZ|X: Markov chain [MP18]: δ ≤ ν/

• 1 − (q − 1)ν +

ν q−1

• .

Additive noise channel on Abelian group X [MP18]: δ ≤ (q − 1)ν. Alternative bounds for Markov chains [MOS13]. Many applications in information theory, statistics, and probability [MP18], [MOS13].

Anuran Makur (MIT) Permutation Channels 10 July 2020 34 / 40

Proof Idea: Degradation by Symmetric Channels

Theorem (Effective Input Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ ext(PZ|X) − 1 2 . Proof Sketch: Degradation by symmetric channels + tensorization of degradation + data processing ⇒ I(X n

1 ; Y n 1 ) ≤ I(X n 1 ; ˜

Y n

1 )

where Y n

1 and ˜

Y n

1 are outputs of permutation channels with PZ|X and q-SC(δ), resp.

Anuran Makur (MIT) Permutation Channels 10 July 2020 35 / 40

Proof Idea: Degradation by Symmetric Channels

Theorem (Effective Input Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ ext(PZ|X) − 1 2 . Proof Sketch: Degradation by symmetric channels + tensorization of degradation + data processing ⇒ I(X n

1 ; Y n 1 ) ≤ I(X n 1 ; ˜

Y n

1 )

where Y n

1 and ˜

Y n

1 are outputs of permutation channels with PZ|X and q-SC(δ), resp.

Convexity of KL divergence ⇒ Reduce |X| to ext(PZ|X).

Anuran Makur (MIT) Permutation Channels 10 July 2020 35 / 40

Proof Idea: Degradation by Symmetric Channels

Theorem (Effective Input Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ ext(PZ|X) − 1 2 . Proof Sketch: Degradation by symmetric channels + tensorization of degradation + data processing ⇒ I(X n

1 ; Y n 1 ) ≤ I(X n 1 ; ˜

Y n

1 )

where Y n

1 and ˜

Y n

1 are outputs of permutation channels with PZ|X and q-SC(δ), resp.

Convexity of KL divergence ⇒ Reduce |X| to ext(PZ|X). Fano argument of output alphabet bound ⇒ effective input alphabet bound.

Anuran Makur (MIT) Permutation Channels 10 July 2020 35 / 40

Outline

1

Introduction

2

Achievability and Converse for the BSC

3

General Achievability Bound

4

General Converse Bounds

5

Conclusion Strictly Positive and “Full Rank” Channels

Anuran Makur (MIT) Permutation Channels 10 July 2020 36 / 40

Strictly Positive and “Full Rank” Channels

Achievability and converse bounds yield:

Theorem (Strictly Positive and “Full Rank” Channels)

For any entry-wise strictly positive channel PZ|X > 0 that is “full rank” in the sense that r rank(PZ|X) = min{ext(PZ|X), |Y|}: Cperm(PZ|X) = r − 1 2 .

Anuran Makur (MIT) Permutation Channels 10 July 2020 37 / 40

Strictly Positive and “Full Rank” Channels

Achievability and converse bounds yield:

Theorem (Strictly Positive and “Full Rank” Channels)

For any entry-wise strictly positive channel PZ|X > 0 that is “full rank” in the sense that r rank(PZ|X) = min{ext(PZ|X), |Y|}: Cperm(PZ|X) = r − 1 2 . Recall Example: Cperm of non-trivial binary symmetric channel is 1

2.

Anuran Makur (MIT) Permutation Channels 10 July 2020 37 / 40

Conclusion

Main Result: For any entry-wise strictly positive channel PZ|X > 0: rank(PZ|X) − 1 2 ≤ Cperm(PZ|X) ≤ min{ext(PZ|X), |Y|} − 1 2 .

Anuran Makur (MIT) Permutation Channels 10 July 2020 38 / 40

Conclusion

Main Result: For any entry-wise strictly positive channel PZ|X > 0: rank(PZ|X) − 1 2 ≤ Cperm(PZ|X) ≤ min{ext(PZ|X), |Y|} − 1 2 . Future Directions: Characterize Cperm of all (entry-wise strictly positive) channels.

Anuran Makur (MIT) Permutation Channels 10 July 2020 38 / 40

Conclusion

Main Result: For any entry-wise strictly positive channel PZ|X > 0: rank(PZ|X) − 1 2 ≤ Cperm(PZ|X) ≤ min{ext(PZ|X), |Y|} − 1 2 . Future Directions: Characterize Cperm of all (entry-wise strictly positive) channels. Perform error exponent analysis (i.e., tight bounds on Pn

error).

Anuran Makur (MIT) Permutation Channels 10 July 2020 38 / 40

Conclusion

Main Result: For any entry-wise strictly positive channel PZ|X > 0: rank(PZ|X) − 1 2 ≤ Cperm(PZ|X) ≤ min{ext(PZ|X), |Y|} − 1 2 . Future Directions: Characterize Cperm of all (entry-wise strictly positive) channels. Perform error exponent analysis (i.e., tight bounds on Pn

error).

Prove strong converse results (i.e., phase transition for Pn

error).

Anuran Makur (MIT) Permutation Channels 10 July 2020 38 / 40

Conclusion

Main Result: For any entry-wise strictly positive channel PZ|X > 0: rank(PZ|X) − 1 2 ≤ Cperm(PZ|X) ≤ min{ext(PZ|X), |Y|} − 1 2 . Future Directions: Characterize Cperm of all (entry-wise strictly positive) channels. Perform error exponent analysis (i.e., tight bounds on Pn

error).

Prove strong converse results (i.e., phase transition for Pn

error).

Perform finite blocklength analysis (i.e., exact asymptotics for maximum achievable |M|).

Anuran Makur (MIT) Permutation Channels 10 July 2020 38 / 40

Conclusion

Main Result: For any entry-wise strictly positive channel PZ|X > 0: rank(PZ|X) − 1 2 ≤ Cperm(PZ|X) ≤ min{ext(PZ|X), |Y|} − 1 2 . Future Directions: Characterize Cperm of all (entry-wise strictly positive) channels. Perform error exponent analysis (i.e., tight bounds on Pn

error).

Prove strong converse results (i.e., phase transition for Pn

error).

Perform finite blocklength analysis (i.e., exact asymptotics for maximum achievable |M|). Analyze permutation channels with more complex probability models in the random permutation block.

Anuran Makur (MIT) Permutation Channels 10 July 2020 38 / 40

References

This talk was based on:

• A. Makur, “Information capacity of BSC and BEC permutation channels,” in

Proceedings of the 56th Annual Allerton Conference on Communication, Control, and Computing, Monticello, IL, USA, October 2-5 2018, pp. 1112–1119.

• A. Makur, “Bounds on permutation channel capacity,” in Proceedings of the IEEE

International Symposium on Information Theory (ISIT), Los Angeles, CA, USA, June 21-26 2020.

• A. Makur, “Coding theorems for noisy permutation channels,” accepted to IEEE

Transactions on Information Theory, July 2020.

Anuran Makur (MIT) Permutation Channels 10 July 2020 39 / 40

Thank You!

Anuran Makur (MIT) Permutation Channels 10 July 2020 40 / 40