Bounds on Permutation Channel Capacity Anuran Makur Department of - - PowerPoint PPT Presentation

Bounds on Permutation Channel Capacity

Anuran Makur

Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology

IEEE International Symposium on Information Theory 2020

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 1 / 22

Outline

1

Introduction Three Motivations Permutation Channel Model Information Capacity

2

Achievability Bound

3

Converse Bounds

4

Conclusion

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 2 / 22

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . .

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 3 / 22

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . . Random deletion channel: LDPC codes nearly achieve capacity for large alphabets Codes correct for transpositions of symbols

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 3 / 22

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . . Random deletion channel: LDPC codes nearly achieve capacity for large alphabets Codes correct for transpositions of symbols Permutation channels with insertions, deletions, substitutions, or erasures Construction and analysis of multiset codes

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 3 / 22

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . . Random deletion channel: LDPC codes nearly achieve capacity for large alphabets Codes correct for transpositions of symbols Permutation channels with insertions, deletions, substitutions, or erasures Construction and analysis of multiset codes Communication networks: [XZ02], [WWM09], [GG10], [KV13], . . . Mobile ad hoc networks, multipath routed networks, etc.

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 3 / 22

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . . Random deletion channel: LDPC codes nearly achieve capacity for large alphabets Codes correct for transpositions of symbols Permutation channels with insertions, deletions, substitutions, or erasures Construction and analysis of multiset codes Communication networks: [XZ02], [WWM09], [GG10], [KV13], . . . Mobile ad hoc networks, multipath routed networks, etc. Out-of-order delivery of packets Correct for packet errors/losses when packets do not have sequence numbers

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 3 / 22

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . . Random deletion channel: LDPC codes nearly achieve capacity for large alphabets Codes correct for transpositions of symbols Permutation channels with insertions, deletions, substitutions, or erasures Construction and analysis of multiset codes Communication networks: [XZ02], [WWM09], [GG10], [KV13], . . . Mobile ad hoc networks, multipath routed networks, etc. Out-of-order delivery of packets Correct for packet errors/losses when packets do not have sequence numbers Molecular/Biological Communications: [YKGR+15], [KPM16], [HSRT17], [SH19], . . .

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 3 / 22

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . . Random deletion channel: LDPC codes nearly achieve capacity for large alphabets Codes correct for transpositions of symbols Permutation channels with insertions, deletions, substitutions, or erasures Construction and analysis of multiset codes Communication networks: [XZ02], [WWM09], [GG10], [KV13], . . . Mobile ad hoc networks, multipath routed networks, etc. Out-of-order delivery of packets Correct for packet errors/losses when packets do not have sequence numbers Molecular/Biological Communications: [YKGR+15], [KPM16], [HSRT17], [SH19], . . . DNA based storage systems Source data encoded into DNA molecules

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 3 / 22

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . . Random deletion channel: LDPC codes nearly achieve capacity for large alphabets Codes correct for transpositions of symbols Permutation channels with insertions, deletions, substitutions, or erasures Construction and analysis of multiset codes Communication networks: [XZ02], [WWM09], [GG10], [KV13], . . . Mobile ad hoc networks, multipath routed networks, etc. Out-of-order delivery of packets Correct for packet errors/losses when packets do not have sequence numbers Molecular/Biological Communications: [YKGR+15], [KPM16], [HSRT17], [SH19], . . . DNA based storage systems Source data encoded into DNA molecules Fragments of DNA molecules cached Receiver reads encoded data by shotgun sequencing (i.e., random sampling)

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 3 / 22

Three Motivations

Coding theory: [DG01], [Mit06], [Met09], [KV15], [KT18], . . . Random deletion channel: LDPC codes nearly achieve capacity for large alphabets Codes correct for transpositions of symbols Permutation channels with insertions, deletions, substitutions, or erasures Construction and analysis of multiset codes Communication networks: [XZ02], [WWM09], [GG10], [KV13], . . . Mobile ad hoc networks, multipath routed networks, etc. Out-of-order delivery of packets Correct for packet errors/losses when packets do not have sequence numbers Molecular/Biological Communications: [YKGR+15], [KPM16], [HSRT17], [SH19], . . . DNA based storage systems Source data encoded into DNA molecules Fragments of DNA molecules cached Receiver reads encoded data by shotgun sequencing (i.e., random sampling)

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 3 / 22

Motivation: Point-to-point Communication in Packet Networks

NETWORK SENDER RECEIVER

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 4 / 22

Motivation: Point-to-point Communication in Packet Networks

NETWORK SENDER RECEIVER

Model communication network as a channel

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 4 / 22

Motivation: Point-to-point Communication in Packet Networks

NETWORK SENDER RECEIVER

Model communication network as a channel: Alphabet symbols = all possible b-bit packets ⇒ 2b input symbols

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 4 / 22

Motivation: Point-to-point Communication in Packet Networks

NETWORK SENDER RECEIVER

Model communication network as a channel: Alphabet symbols = all possible b-bit packets Multipath routed network or evolving network topology

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 4 / 22

Motivation: Point-to-point Communication in Packet Networks

NETWORK SENDER RECEIVER

Model communication network as a channel: Alphabet symbols = all possible b-bit packets Multipath routed network ⇒ packets received with transpositions

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 4 / 22

Motivation: Point-to-point Communication in Packet Networks

NETWORK SENDER RECEIVER

Model communication network as a channel: Alphabet symbols = all possible b-bit packets Multipath routed network ⇒ packets received with transpositions Packets are impaired (e.g., deletions, substitutions, etc.)

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 4 / 22

Motivation: Point-to-point Communication in Packet Networks

NETWORK SENDER RECEIVER

Model communication network as a channel: Alphabet symbols = all possible b-bit packets Multipath routed network ⇒ packets received with transpositions Packets are impaired ⇒ model using channel probabilities

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 4 / 22

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 5 / 22

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER RANDOM DELETION RANDOM PERMUTATION

Abstraction: n-length codeword = sequence of n packets

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 5 / 22

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER RANDOM DELETION RANDOM PERMUTATION

Abstraction: n-length codeword = sequence of n packets Random deletion channel: Delete each symbol/packet independently with prob p ∈ (0, 1)

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 5 / 22

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER RANDOM DELETION RANDOM PERMUTATION

Abstraction: n-length codeword = sequence of n packets Random deletion channel: Delete each symbol/packet independently with prob p ∈ (0, 1)

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 5 / 22

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER RANDOM DELETION RANDOM PERMUTATION

Abstraction: n-length codeword = sequence of n packets Random deletion channel: Delete each symbol/packet independently with prob p ∈ (0, 1) Random permutation block: Randomly permute packets of codeword

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 5 / 22

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER RANDOM DELETION RANDOM PERMUTATION

Abstraction: n-length codeword = sequence of n packets Random deletion channel: Delete each symbol/packet independently with prob p ∈ (0, 1) Random permutation block: Randomly permute packets of codeword

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 5 / 22

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER ERASURE CHANNEL RANDOM PERMUTATION

? ?

Abstraction: n-length codeword = sequence of n packets Equivalent Erasure channel: Erase each symbol/packet independently with prob p ∈ (0, 1) Random permutation block: Randomly permute packets of codeword

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 5 / 22

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER ERASURE CHANNEL RANDOM PERMUTATION

? ? 1 2 3 3 3 1 1

Abstraction: n-length codeword = sequence of n packets Erasure channel: Erase each symbol/packet independently with prob p ∈ (0, 1) Random permutation block: Randomly permute packets of codeword Coding: Add sequence numbers (packet size = b + log(n) bits, alphabet size = n 2b)

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 5 / 22

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER ERASURE CHANNEL RANDOM PERMUTATION

? ? 1 2 3 3 3 1 1

Abstraction: n-length codeword = sequence of n packets Erasure channel: Erase each symbol/packet independently with prob p ∈ (0, 1) Random permutation block: Randomly permute packets of codeword Coding: Add sequence numbers and use standard coding techniques

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 5 / 22

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER ERASURE CHANNEL RANDOM PERMUTATION

? ? 1 2 3 3 3 1 1

Abstraction: n-length codeword = sequence of n packets Erasure channel: Erase each symbol/packet independently with prob p ∈ (0, 1) Random permutation block: Randomly permute packets of codeword Coding: Add sequence numbers and use standard coding techniques More refined coding techniques simulate sequence numbers [Mit06], [Met09]

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 5 / 22

Example: Coding for Random Deletion Network

Consider a communication network where packets can be dropped:

NETWORK SENDER RECEIVER ERASURE CHANNEL RANDOM PERMUTATION

? ?

Abstraction: n-length codeword = sequence of n packets Erasure channel: Erase each symbol/packet independently with prob p ∈ (0, 1) Random permutation block: Randomly permute packets of codeword How do you code in such channels without increasing alphabet size?

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 5 / 22

Permutation Channel Model

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Sender sends message M ∼ Uniform(M)

n = blocklength

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 6 / 22

Permutation Channel Model

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Sender sends message M ∼ Uniform(M)

n = blocklength Randomized encoder fn : M → X n produces codeword X n

1 = (X1, . . . , Xn) = fn(M)

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 6 / 22

Permutation Channel Model

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Sender sends message M ∼ Uniform(M)

n = blocklength Randomized encoder fn : M → X n produces codeword X n

1 = (X1, . . . , Xn) = fn(M)

Discrete memoryless channel PZ|X with input & output alphabets X & Y produces Z n

1 :

PZ n

1 |X n 1 (zn

1 |xn 1 ) = n

• i=1

PZ|X(zi|xi)

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 6 / 22

Permutation Channel Model

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Sender sends message M ∼ Uniform(M)

n = blocklength Randomized encoder fn : M → X n produces codeword X n

1 = (X1, . . . , Xn) = fn(M)

Discrete memoryless channel PZ|X with input & output alphabets X & Y produces Z n

1 :

PZ n

1 |X n 1 (zn

1 |xn 1 ) = n

• i=1

PZ|X(zi|xi) Random permutation π generates Y n

1 from Z n 1 : Yπ(i) = Zi for i ∈ {1, . . . , n}

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 6 / 22

Permutation Channel Model

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Sender sends message M ∼ Uniform(M)

n = blocklength Randomized encoder fn : M → X n produces codeword X n

1 = (X1, . . . , Xn) = fn(M)

Discrete memoryless channel PZ|X with input & output alphabets X & Y produces Z n

1 :

PZ n

1 |X n 1 (zn

1 |xn 1 ) = n

• i=1

PZ|X(zi|xi) Random permutation π generates Y n

1 from Z n 1 : Yπ(i) = Zi for i ∈ {1, . . . , n}

Randomized decoder gn : Yn → M ∪ {error} produces estimate ˆ M = gn(Y n

1 ) at receiver

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 6 / 22

Permutation Channel Model

What if we analyze the “swapped” model?

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑊
• 𝑋
• 𝑁
• ENCODER

CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Anuran Makur (MIT)

Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 7 / 22

Permutation Channel Model

What if we analyze the “swapped” model?

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑊
• 𝑋
• 𝑁
• Proposition (Equivalent Models)

If channel PW |V is equal to channel PZ|X, then channel PW n

1 |X n 1 is equal to channel PY n 1 |X n 1 .

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Anuran Makur (MIT)

Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 7 / 22

Permutation Channel Model

What if we analyze the “swapped” model?

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑊
• 𝑋
• 𝑁
• Proposition (Equivalent Models)

If channel PW |V is equal to channel PZ|X, then channel PW n

1 |X n 1 is equal to channel PY n 1 |X n 1 .

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Remarks:

Proof follows from direct calculation.

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 7 / 22

Permutation Channel Model

What if we analyze the “swapped” model?

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑊
• 𝑋
• 𝑁
• Proposition (Equivalent Models)

If channel PW |V is equal to channel PZ|X, then channel PW n

1 |X n 1 is equal to channel PY n 1 |X n 1 .

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Remarks:

Proof follows from direct calculation. Can analyze either model!

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 7 / 22

Coding for the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• General Principle:

“Encode the information in an object that is invariant under the [permutation] transformation.” [KV13]

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 8 / 22

Coding for the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• General Principle:

“Encode the information in an object that is invariant under the [permutation] transformation.” [KV13] Multiset codes are studied in [KV13], [KV15], and [KT18].

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 8 / 22

Coding for the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• General Principle:

“Encode the information in an object that is invariant under the [permutation] transformation.” [KV13] Multiset codes are studied in [KV13], [KV15], and [KT18]. In contrast, in [Mak18], we asked: What are the fundamental information theoretic limits?

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 8 / 22

Information Capacity of the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Average probability of error Pn

error P(M = ˆ

M)

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 9 / 22

Information Capacity of the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Average probability of error Pn

error P(M = ˆ

M) “Rate” of coding scheme (fn, gn) is R log(|M|) log(n)

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 9 / 22

Information Capacity of the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Average probability of error Pn

error P(M = ˆ

M) “Rate” of coding scheme (fn, gn) is R log(|M|) log(n) |M| = nR

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 9 / 22

Information Capacity of the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Average probability of error Pn

error P(M = ˆ

M) “Rate” of coding scheme (fn, gn) is R log(|M|) log(n) |M| = nR because number of empirical distributions of Y n

1 is poly(n)

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 9 / 22

Information Capacity of the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Average probability of error Pn

error P(M = ˆ

M) “Rate” of coding scheme (fn, gn) is R log(|M|) log(n) |M| = nR Rate R ≥ 0 is achievable ⇔ ∃ {(fn, gn)}n∈N such that lim

n→∞ Pn error = 0

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 9 / 22

Information Capacity of the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Average probability of error Pn

error P(M = ˆ

M) “Rate” of coding scheme (fn, gn) is R log(|M|) log(n) |M| = nR Rate R ≥ 0 is achievable ⇔ ∃ {(fn, gn)}n∈N such that lim

n→∞ Pn error = 0

Definition (Permutation Channel Capacity [Mak18])

Cperm(PZ|X) sup{R ≥ 0 : R is achievable}

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 9 / 22

Information Capacity of the Permutation Channel

ENCODER CHANNEL RANDOM PERMUTATION DECODER 𝑁 𝑌

• 𝑎
• 𝑍
• 𝑁
• Average probability of error Pn

error P(M = ˆ

M) “Rate” of coding scheme (fn, gn) is R log(|M|) log(n) |M| = nR Rate R ≥ 0 is achievable ⇔ ∃ {(fn, gn)}n∈N such that lim

n→∞ Pn error = 0

Definition (Permutation Channel Capacity [Mak18])

Cperm(PZ|X) sup{R ≥ 0 : R is achievable}

Main Question

What is the permutation channel capacity of a general PZ|X?

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 9 / 22

Outline

1

Introduction

2

Achievability Bound Coding Scheme Rank Bound

3

Converse Bounds

4

Conclusion

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 10 / 22

Achievability: Coding Scheme

Let r = rank(PZ|X) and k = √n

• Anuran Makur (MIT)

Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 11 / 22

Achievability: Coding Scheme

Let r = rank(PZ|X) and k = √n

• Consider X ′ ⊆ X with |X ′| = r such that {PZ|X(·|x) : x ∈ X ′} are linearly independent

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 11 / 22

Achievability: Coding Scheme

Let r = rank(PZ|X) and k = √n

• Consider X ′ ⊆ X with |X ′| = r such that {PZ|X(·|x) : x ∈ X ′} are linearly independent

Message set: M

• p = (p(x) : x ∈ X ′) ∈ (Z+)X ′ :
• x∈X ′

p(x) = k

• Anuran Makur (MIT)

Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 11 / 22

Achievability: Coding Scheme

Let r = rank(PZ|X) and k = √n

• Consider X ′ ⊆ X with |X ′| = r such that {PZ|X(·|x) : x ∈ X ′} are linearly independent

Message set: M

• p = (p(x) : x ∈ X ′) ∈ (Z+)X ′ :
• x∈X ′

p(x) = k

• where |M| =

k+r−1

r−1

• = Θ
• n

r−1 2 Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 11 / 22

Achievability: Coding Scheme

Let r = rank(PZ|X) and k = √n

• Consider X ′ ⊆ X with |X ′| = r such that {PZ|X(·|x) : x ∈ X ′} are linearly independent

Message set: M

• p = (p(x) : x ∈ X ′) ∈ (Z+)X ′ :
• x∈X ′

p(x) = k

• where |M| =

k+r−1

r−1

• = Θ
• n

r−1 2

Randomized Encoder: ∀p ∈ M, fn(p) = X n

1 i.i.d.

∼ PX where PX(x) = p(x)

k ,

for x ∈ X ′ 0, for x ∈ X\X ′

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 11 / 22

Achievability: Coding Scheme

Let stochastic matrix ˜ PZ|X ∈ Rr×|Y| have rows {PZ|X(·|x) : x ∈ X ′} Let ˜ P†

Z|X denote its Moore-Penrose pseudoinverse

✶

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 12 / 22

Achievability: Coding Scheme

Let stochastic matrix ˜ PZ|X ∈ Rr×|Y| have rows {PZ|X(·|x) : x ∈ X ′} Let ˜ P†

Z|X denote its Moore-Penrose pseudoinverse

(Sub-optimal) Thresholding Decoder: For any yn

1 ∈ Yn,

Step 1: Construct its type/empirical distribution/histogram ∀y ∈ Y, ˆ Pyn

1 (y) = 1

n

n

• i=1

✶{yi = y}

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 12 / 22

Achievability: Coding Scheme

Let stochastic matrix ˜ PZ|X ∈ Rr×|Y| have rows {PZ|X(·|x) : x ∈ X ′} Let ˜ P†

Z|X denote its Moore-Penrose pseudoinverse

(Sub-optimal) Thresholding Decoder: For any yn

1 ∈ Yn,

Step 1: Construct its type/empirical distribution/histogram ∀y ∈ Y, ˆ Pyn

1 (y) = 1

n

n

• i=1

✶{yi = y} Step 2: Generate estimate ˆ p ∈ (Z+)X ′ with components ∀x ∈ X ′, ˆ p(x) = arg min

j∈{0,...,k}

• y∈Y

ˆ Pyn

1 (y)

• ˜

P†

Z|X

• y,x − j

k

• Anuran Makur (MIT)

Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 12 / 22

Achievability: Coding Scheme

Let stochastic matrix ˜ PZ|X ∈ Rr×|Y| have rows {PZ|X(·|x) : x ∈ X ′} Let ˜ P†

Z|X denote its Moore-Penrose pseudoinverse

(Sub-optimal) Thresholding Decoder: For any yn

1 ∈ Yn,

Step 1: Construct its type/empirical distribution/histogram ∀y ∈ Y, ˆ Pyn

1 (y) = 1

n

n

• i=1

✶{yi = y} Step 2: Generate estimate ˆ p ∈ (Z+)X ′ with components ∀x ∈ X ′, ˆ p(x) = arg min

j∈{0,...,k}

• y∈Y

ˆ Pyn

1 (y)

• ˜

P†

Z|X

• y,x − j

k

• Step 3: Output decoded message

gn(yn

1 ) =

• ˆ

p, if ˆ p ∈ M error,

• therwise

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 12 / 22

Achievability: Rank Bound

Theorem (Rank Bound)

For any channel PZ|X: Cperm(PZ|X) ≥ rank(PZ|X) − 1 2 . Remarks about Coding Scheme: Showing limn→∞ Pn

error = 0 proves theorem.

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 13 / 22

Achievability: Rank Bound

Theorem (Rank Bound)

For any channel PZ|X: Cperm(PZ|X) ≥ rank(PZ|X) − 1 2 . Remarks about Coding Scheme: Showing limn→∞ Pn

error = 0 proves theorem.

Intuition: Conditioned on M = p, ˆ PY n

1 ≈ PZ with high probability as n → ∞. Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 13 / 22

Achievability: Rank Bound

Theorem (Rank Bound)

For any channel PZ|X: Cperm(PZ|X) ≥ rank(PZ|X) − 1 2 . Remarks about Coding Scheme: Showing limn→∞ Pn

error = 0 proves theorem.

Intuition: Conditioned on M = p, ˆ PY n

1 ≈ PZ with high probability as n → ∞.

Hence,

y∈Y ˆ

PY n

1 (y)

˜ P†

Z|X

• y,x ≈ PX(x) for all x ∈ X ′ with high probability.

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 13 / 22

Achievability: Rank Bound

Theorem (Rank Bound)

For any channel PZ|X: Cperm(PZ|X) ≥ rank(PZ|X) − 1 2 . Remarks about Coding Scheme: Showing limn→∞ Pn

error = 0 proves theorem.

Intuition: Conditioned on M = p, ˆ PY n

1 ≈ PZ with high probability as n → ∞.

Hence,

y∈Y ˆ

PY n

1 (y)

˜ P†

Z|X

• y,x ≈ PX(x) for all x ∈ X ′ with high probability.

Computational complexity: Decoder has O(n) running time.

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 13 / 22

Achievability: Rank Bound

Theorem (Rank Bound)

For any channel PZ|X: Cperm(PZ|X) ≥ rank(PZ|X) − 1 2 . Remarks about Coding Scheme: Showing limn→∞ Pn

error = 0 proves theorem.

Intuition: Conditioned on M = p, ˆ PY n

1 ≈ PZ with high probability as n → ∞.

Hence,

y∈Y ˆ

PY n

1 (y)

˜ P†

Z|X

• y,x ≈ PX(x) for all x ∈ X ′ with high probability.

Computational complexity: Decoder has O(n) running time. Probabilistic method: Good deterministic codes exist.

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 13 / 22

Achievability: Rank Bound

Theorem (Rank Bound)

For any channel PZ|X: Cperm(PZ|X) ≥ rank(PZ|X) − 1 2 . Remarks about Coding Scheme: Showing limn→∞ Pn

error = 0 proves theorem.

Intuition: Conditioned on M = p, ˆ PY n

1 ≈ PZ with high probability as n → ∞.

Hence,

y∈Y ˆ

PY n

1 (y)

˜ P†

Z|X

• y,x ≈ PX(x) for all x ∈ X ′ with high probability.

Computational complexity: Decoder has O(n) running time. Probabilistic method: Good deterministic codes exist. Expurgation: Achievability bound holds under maximal probability of error criterion.

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 13 / 22

Outline

1

Introduction

2

Achievability Bound

3

Converse Bounds Output Alphabet Bound Effective Input Alphabet Bound Degradation by Symmetric Channels

4

Conclusion

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 14 / 22

Converse: Output Alphabet Bound

Theorem (Output Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ |Y| − 1 2 .

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 15 / 22

Converse: Output Alphabet Bound

Theorem (Output Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ |Y| − 1 2 . Remarks: Proof hinges on Fano’s inequality and CLT-based approximation of binomial entropy.

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 15 / 22

Converse: Output Alphabet Bound

Theorem (Output Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ |Y| − 1 2 . Remarks: Proof hinges on Fano’s inequality and CLT-based approximation of binomial entropy. What if |X| is much smaller than |Y|?

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 15 / 22

Converse: Output Alphabet Bound

Theorem (Output Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ |Y| − 1 2 . Remarks: Proof hinges on Fano’s inequality and CLT-based approximation of binomial entropy. What if |X| is much smaller than |Y|? Want: Converse bound in terms of input alphabet size.

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 15 / 22

Converse: Effective Input Alphabet Bound

Theorem (Effective Input Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ ext(PZ|X) − 1 2 where ext(PZ|X) denotes the number of extreme points of conv

• PZ|X(·|x) : x ∈ X
• .

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 16 / 22

Converse: Effective Input Alphabet Bound

Theorem (Effective Input Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ ext(PZ|X) − 1 2 where ext(PZ|X) denotes the number of extreme points of conv

• PZ|X(·|x) : x ∈ X
• .

Remarks: Effective input alphabet size: rank(PZ|X) ≤ ext(PZ|X) ≤ |X|.

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 16 / 22

Converse: Effective Input Alphabet Bound

Theorem (Effective Input Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ ext(PZ|X) − 1 2 where ext(PZ|X) denotes the number of extreme points of conv

• PZ|X(·|x) : x ∈ X
• .

Remarks: Effective input alphabet size: rank(PZ|X) ≤ ext(PZ|X) ≤ |X|. For any channel PZ|X > 0, Cperm(PZ|X) ≤

• min{ext(PZ|X), |Y|} − 1
• /2.

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 16 / 22

Converse: Effective Input Alphabet Bound

Theorem (Effective Input Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ ext(PZ|X) − 1 2 where ext(PZ|X) denotes the number of extreme points of conv

• PZ|X(·|x) : x ∈ X
• .

Remarks: Effective input alphabet size: rank(PZ|X) ≤ ext(PZ|X) ≤ |X|. For any channel PZ|X > 0, Cperm(PZ|X) ≤

• min{ext(PZ|X), |Y|} − 1
• /2.

For any general channel PZ|X, Cperm(PZ|X) ≤ min{ext(PZ|X), |Y|} − 1.

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 16 / 22

Converse: Effective Input Alphabet Bound

Theorem (Effective Input Alphabet Bound)

For any entry-wise strictly positive channel PZ|X > 0: Cperm(PZ|X) ≤ ext(PZ|X) − 1 2 where ext(PZ|X) denotes the number of extreme points of conv

• PZ|X(·|x) : x ∈ X
• .

Remarks: Effective input alphabet size: rank(PZ|X) ≤ ext(PZ|X) ≤ |X|. For any channel PZ|X > 0, Cperm(PZ|X) ≤

• min{ext(PZ|X), |Y|} − 1
• /2.

For any general channel PZ|X, Cperm(PZ|X) ≤ min{ext(PZ|X), |Y|} − 1. How do we prove above theorem?

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 16 / 22

Proof Idea: Degradation by Symmetric Channels

Definition (Degradation/Blackwell Order [Bla51], [She51], [Ste51], [Cov72], [Ber73])

Given channels PZ1|X and PZ2|X with common input alphabet X, PZ2|X is a degraded version

• f PZ1|X if PZ2|X = PZ1|XPZ2|Z1 for some channel PZ2|Z1.

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 17 / 22

Proof Idea: Degradation by Symmetric Channels

Definition (Degradation/Blackwell Order [Bla51], [She51], [Ste51], [Cov72], [Ber73])

Given channels PZ1|X and PZ2|X with common input alphabet X, PZ2|X is a degraded version

• f PZ1|X if PZ2|X = PZ1|XPZ2|Z1 for some channel PZ2|Z1.

Definition (q-ary Symmetric Channel)

A q-ary symmetric channel, denoted q-SC(δ), with total crossover probability δ ∈ [0, 1] and alphabet X where |X| = q, is given by the doubly stochastic matrix: Wδ       1 − δ

δ q−1

· · ·

δ q−1 δ q−1

1 − δ · · ·

δ q−1

. . . . . . ... . . .

δ q−1 δ q−1

· · · 1 − δ       .

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 17 / 22

Proof Idea: Degradation by Symmetric Channels

Proposition (Degradation by Symmetric Channels)

Given channel PZ|X with ν = min

x∈X, y∈Y PZ|X(y|x), if we have:

0 ≤ δ ≤ ν 1 − ν +

ν q−1

, then PZ|X is a degraded version of q-SC(δ).

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 18 / 22

Proof Idea: Degradation by Symmetric Channels

Proposition (Degradation by Symmetric Channels)

Given channel PZ|X with ν = min

x∈X, y∈Y PZ|X(y|x), if we have:

0 ≤ δ ≤ ν 1 − ν +

ν q−1

, then PZ|X is a degraded version of q-SC(δ). Prop follows from computing extremal δ such that W −1

δ

PZ|X is row stochastic. Many other applications in information theory and statistics [MP18], [MOS13].

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 18 / 22

Proof Idea: Degradation by Symmetric Channels

Proposition (Degradation by Symmetric Channels)

Given channel PZ|X with ν = min

x∈X, y∈Y PZ|X(y|x), if we have:

0 ≤ δ ≤ ν 1 − ν +

ν q−1

, then PZ|X is a degraded version of q-SC(δ). Prop follows from computing extremal δ such that W −1

δ

PZ|X is row stochastic. Many other applications in information theory and statistics [MP18], [MOS13]. Prop + “swapped” model + tensorization of degradation ⇒ I(X n

1 ; Y n 1 ) ≤ I(X n 1 ; ˜

Y n

1 ),

where Y n

1 and ˜

Y n

1 are outputs of permutation channels with PZ|X and q-SC(δ).

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 18 / 22

Proof Idea: Degradation by Symmetric Channels

Proposition (Degradation by Symmetric Channels)

Given channel PZ|X with ν = min

x∈X, y∈Y PZ|X(y|x), if we have:

0 ≤ δ ≤ ν 1 − ν +

ν q−1

, then PZ|X is a degraded version of q-SC(δ). Prop follows from computing extremal δ such that W −1

δ

PZ|X is row stochastic. Many other applications in information theory and statistics [MP18], [MOS13]. Prop + “swapped” model + tensorization of degradation ⇒ I(X n

1 ; Y n 1 ) ≤ I(X n 1 ; ˜

Y n

1 ),

where Y n

1 and ˜

Y n

1 are outputs of permutation channels with PZ|X and q-SC(δ).

Convexity of KL divergence ⇒ Reduce |X| to ext(PZ|X).

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 18 / 22

Proof Idea: Degradation by Symmetric Channels

Proposition (Degradation by Symmetric Channels)

Given channel PZ|X with ν = min

x∈X, y∈Y PZ|X(y|x), if we have:

0 ≤ δ ≤ ν 1 − ν +

ν q−1

, then PZ|X is a degraded version of q-SC(δ). Prop follows from computing extremal δ such that W −1

δ

PZ|X is row stochastic. Many other applications in information theory and statistics [MP18], [MOS13]. Prop + “swapped” model + tensorization of degradation ⇒ I(X n

1 ; Y n 1 ) ≤ I(X n 1 ; ˜

Y n

1 ),

where Y n

1 and ˜

Y n

1 are outputs of permutation channels with PZ|X and q-SC(δ).

Convexity of KL divergence ⇒ Reduce |X| to ext(PZ|X). Fano argument of output alphabet bound ⇒ effective input alphabet bound.

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 18 / 22

Outline

1

Introduction

2

Achievability Bound

3

Converse Bounds

4

Conclusion Strictly Positive and “Full Rank” Channels

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 19 / 22

Strictly Positive and “Full Rank” Channels

Achievability and converse bounds yield:

Theorem (Strictly Positive and “Full Rank” Channels)

For any entry-wise strictly positive channel PZ|X > 0 that is “full rank” in the sense that r rank(PZ|X) = min{ext(PZ|X), |Y|}: Cperm(PZ|X) = r − 1 2 .

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 20 / 22

Strictly Positive and “Full Rank” Channels

Achievability and converse bounds yield:

Theorem (Strictly Positive and “Full Rank” Channels)

For any entry-wise strictly positive channel PZ|X > 0 that is “full rank” in the sense that r rank(PZ|X) = min{ext(PZ|X), |Y|}: Cperm(PZ|X) = r − 1 2 . Example [Mak18]: Cperm of non-trivial binary symmetric channel is 1

2.

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 20 / 22

Conclusion

Main Result: For any entry-wise strictly positive channel PZ|X > 0: rank(PZ|X) − 1 2 ≤ Cperm(PZ|X) ≤ min{ext(PZ|X), |Y|} − 1 2 .

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 21 / 22

Conclusion

Main Result: For any entry-wise strictly positive channel PZ|X > 0: rank(PZ|X) − 1 2 ≤ Cperm(PZ|X) ≤ min{ext(PZ|X), |Y|} − 1 2 . Future Direction: Characterize Cperm of all entry-wise strictly positive channels, and more generally, all channels.

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 21 / 22

Thank You!

Anuran Makur (MIT) Bounds on Permutation Channel Capacity ISIT 21-26 June 2020 22 / 22