Hypergraph-based Coding Schemes for Two Source Coding Problems - - PowerPoint PPT Presentation

Hypergraph-based Coding Schemes for Two Source Coding Problems under Maximal Distortion

Sourya Basu: CSL, ECE Dept., University of Illinois at Urbana-Champaign Daewon Seo: ECE Dept., University of Wisconsin-Madison Lav Varshney: CSL, ECE Dept., University of Illinois at Urbana-

Champaign; Salesforce Research

This work was funded in part by the IBM-Illinois Center for Cognitive Computing Systems Research (C3SR), a research collaboration as part of the IBM AI Horizons Network.

1

Introduction

X1 X2 M1 M2 ̂ f(X1, X2) || f(X1, X2) − ̂ f(X, Y)|| ≤ ϵ

• Distributed coding for computing under maximal distortion

E1 E2 D

2

R1 > I(X1; U1|U2, Q) R2 > I(X2; U2|U1, Q) R1 + R2 > I(X1, X2; U1, U2|Q)

Let if and only if:

(R1, R2) ∈ ℛi,ϵ

for some joint pmf with , , and some function such that , where .

p(q)p(u1|x1, q)p(u2|x2, q) |𝒭| ≤ 4 |𝒱j| ≤ |𝒴j| + 4 j = 1,2 ̂ f(u1, u2) 𝔽 [dϵ(X1, X2, ̂ f(U1, U2))] = 0 dϵ(x1, x2, z) = 1{||z−f(x1,x2)||>ϵ}

• Distributed coding for computing: Berger-Tung achievable

region

Introduction

3

• Successive refinement coding under maximal distortion

X E1 E2 D1 D2 ̂ X1 ̂ X2 ||X − ̂ X1|| ≤ ϵ1 ||X − ̂ X2|| ≤ ϵ2

Introduction

M1 M2

4

Let if and only if for some conditional pmf , for .

(R1, R2) ∈ ℛ(ϵ1,ϵ2) R1 > I(X; ̂ X1) R2 > I(X; ̂ X1, ̂ X2), p( ̂ x1, ̂ x2|x) 𝔽 [1{|| ̂

Xi−X||>ϵi}] = 0

i ∈ {1,2}

• Successive refinement coding: known rate region

Introduction

5

• Not much is known about the auxiliary random variables involved,

which makes it difficult to actually attain these regions in practice.

• Hence, this work is concerned with hypergraph-based auxiliary

variables under maximal distortion, which makes it easier to attain the rate regions.

Motivation

• W. Gu, “On achievable rate regions for source coding over networks,” Ph.D. dissertation, California Institute of Technology, 2009.

6

• Distributed coding for computing: we provide a hypergraph-

based coding scheme that outperforms existing hypergraph- based coding schemes.

• Successive refinement coding: we attain the entire rate region

using a hypergraph-based coding scheme.

Our contributions

7

X Y

M

f(X, Y) || ̂ f(X, Y) − f(X, Y)|| ≤ ϵ

where is the set of all hyperedges of a -characteristic hypergraph which can be constructed based on f, X, Y as a part

• f coding scheme.

Rmin = min

W − X − Y X ∈ W ∈ Γ(Gϵ)

I(W; X|Y)

Γ(Gϵ)

ϵ

, where is the set of all p(w|x), such that there exists a satisfying

.

Rmin = min

W − X − Y p ∈ 𝒬(0)

I(W; X|Y)

𝒬(0) g : 𝒳 × 𝒵 ↦ 𝒶 E[1||f(X,Y)−g(W,Y)||>D] ≤ 0

• S. Basu, D. Seo, and L. Varshney, “Functional Epsilon Entropy”, in Proceedings of the IEEE Data Compression Conference, Snowbird, Utah,

24-27 March 2020.

Optimal practical codes

Previous work

E D

8

Overview

• Problem settings
• Definitions
• Our rates regions
• Some special cases

9

Problem setting: distributed coding for computing

• are N iid random variables, where

and are finite sets.

• Reconstruct

as such that as .

{X1,i, X2,i}N

i=1

Xj,i ∈ 𝒴j for j ∈ {1,2} and i ∈ {1,…, N} 𝒴j {f(X1,i, X2,i)}N

i=1

{ ̂ Zi}N

i=1

Pavg

ϵ

( ̂ ZN, XN

1 , XN 2 ) → 0

N → ∞

Pavg

ϵ

( ̂ ZN, XN

1 , XN 2 ) = 1

N

N

∑

i=1

Pr [|| ̂ Zi − f(X1,i, X2,i)|| > ϵ]

E1 E2 D

X1 X2 M1 M2 ̂ f(X1, X2) || f(X1, X2) − ̂ f(X1, X2)|| ≤ ϵ

10

X E1 E2 D1 D2 ̂ X1 ̂ X2 ||X − ̂ X1|| ≤ ϵ1 ||X − ̂ X2|| ≤ ϵ2

• are N iid random variables, where

and is a finite set.

• Decoder

reconstructs as for such that as .

{Xi}N

i=1

Xi ∈ 𝒴 for i ∈ {1,…, N} 𝒴 Dj {Xi}N

i=1

{ ̂ Xj,i}N

i=1

j ∈ {1,2} Pavg

ϵi ( ̂

XN

i , XN) → 0

N → ∞

Pavg

ϵi ( ̂

XN

j , XN) = 1

N

N

∑

i=1

Pr [|| ̂ Xj,i − Xi|| > ϵj]

Problem setting: successive refinement coding

M1 M2

11

Smallest enclosing circles

Smallest enclosing circles:

• For a set of points , the circle with smallest radius covering all the points in

is called the smallest enclosing circle of

S S S

Images constructed using https://www.nayuki.io/page/smallest-enclosing-circle 12

Distributed source coding: some definitions

Let for . A pair of hypergraphs is called an

• achievable hypergraph pair with respect to a function

, if, for any and the radius of the smallest enclosing circle of the set of points is less than or equal to .

Gϵ

i = (𝒴i, Ei)

i ∈ {1,2} (Gϵ

1, Gϵ 2)

ϵ f : 𝒴1 × 𝒴2 ↦ 𝒶 w1 ∈ E1 w2 ∈ E2 {f(x1, x2) : x1 ∈ w1, x2 ∈ w2, and p(x1, x2) > 0} ϵ

A -characteristic hypergraph pair set, , consists of all -achievable hypergraph pairs with respect to a function , i.e.

ϵ 𝒣ϵ ϵ f 𝒣ϵ = {(G1, G2) : (G1, G2) is an ϵ-achievable hypergraph pair}

13

Distributed coding for computing: hypergraph-based coding rate region

Let if and only if

(R1, R2) ∈ ℛ𝒣,ϵ R1 > I(X1; W1|W2, Q) R2 > I(X2; W2|W1, Q) R1 + R2 > I(X1, X2; W1, W2|Q)

for some joint pmf such that form a Markov chain and and for some .

p(q)p(w1, w2|x1, x2, q) W1 − X1 − X2 − W2 X1 ∈ W1 ∈ Γ(Gϵ

1)

X2 ∈ W2 ∈ Γ(Gϵ

2)

(Gϵ

1, Gϵ 2) ∈ 𝒣ϵ

For , induces a pmf over the vertices of the graph and is

• btained by defining the pmfs
• ver all hyperedges

that contain , i.e. for all and .

i ∈ {1,2} Xi Gϵ

i

Wi p(wi|xi) wi ∈ Γ(Gϵ

i )

xi p(wi|xi) ≥ 0 xi ∈ wi ∈ Γ(Gϵ

i )

∑

wi∋xi

p(wi|xi) = 1

14

Distributed source coding: main result

Theorem 1: The region is achievable.

ℛ𝒣,ϵ

Theorem 2: The region and match on the sum-rate bound, i.e. .

ℛsb

𝒣,ϵ

ℛsb

i,ϵ

ℛsb

𝒣,ϵ = ℛsb i,ϵ

15

Distributed source coding: special cases

Theorem 3: When is independent of , .

X1 X2 ℛind

i,ϵ = ℛind 𝒣,ϵ

such that for some distribution and time-sharing random variable , such that for some function .

ℛind

i,ϵ = (R1, R2)

R1 ≥ I(X1; U1|Q) R2 ≥ I(X2; U2|Q), p(u1|x1), p(u2|x2) Q 𝔽[dϵ( ̂ f(U1, U2), f(X1, X2))] = 0 ̂ f

such that for some distributions and time-sharing random variable , such that for for some

ℛind

𝒣,ϵ = (R1, R2)

R1 ≥ I(X1; W1|Q) R2 ≥ I(X2; W2|Q), p(w1|x1), p(w2|x2) Q Xi ∈ Wi ∈ Γ(Gϵ

i )

i ∈ {1,2} (Gϵ

1, Gϵ 2) ∈ 𝒣ϵ

16

Distributed source coding: improvement in rate region

Example: Let be i.i.d. Bern(1/2) random variables. Encoder observes for and the decoder wants to compute the identity function with .

(X1, X2) i Xi i ∈ {1,2} f(x1, x2) = (x1, x2) ϵ = 0.5

17

• The convex region COD correspond to the hypergraph-based region in previous work.
• The convex region YABX correspond to the hypergraph-based region in our work.
• S. Feizi and M. Médard, “On network functional compression,” IEEE Trans. Inf. Theory, vol. 60, no. 9, pp. 5387–5401, Sep. 2014.

Distributed source coding: improvement in rate region

18

Example: Consider the point to point source coding problem with no side information, uniformly distributed over and take the function as the identity function.

X {0,1,2} f

Distributed source coding: improvement in rate region

19

• Proof: Will use the Berger-Tung inner bound. Basically, we use

as the center of the smallest enclosing circle containing all the points .

• Further, by definition of the -achievable hypergraph pair set, we have

.

• This completes the proof to achievability.

W1 = U1, W2 = U2 and ̂ f(W1, W2) {f(x1, x2) : x1 ∈ W1, x2 ∈ W2} ϵ 𝔽 [dϵ(X1, X2, ̂ f(W1, W2))] = 0

Theorem 1: The region is achievable.

ℛ𝒣,ϵ

Distributed source coding: outline of proofs

20

• Proof:

is trivial. We need to show that .

ℛsb

𝒣,ϵ ≥ ℛsb i,ϵ

ℛsb

𝒣,ϵ ≤ ℛsb i,ϵ

Theorem 2: The region and match on the sum-rate bound, i.e. .

ℛsb

𝒣,ϵ

ℛsb

i,ϵ

ℛsb

𝒣,ϵ = ℛsb i,ϵ

Distributed source coding: outline of proofs

21

• Proof:

is trivial. We need to show that .

• Idea: For every

satisfying conditions in , we need to find corresponding that satisfy conditions in .

ℛsb

𝒣,ϵ ≥ ℛsb i,ϵ

ℛsb

𝒣,ϵ ≤ ℛsb i,ϵ

U1, U2, ̂ f ℛsb

i,ϵ

W1, W2 ℛsb

𝒣,ϵ

Theorem 2: The region and match on the sum-rate bound, i.e. .

ℛsb

𝒣,ϵ

ℛsb

i,ϵ

ℛsb

𝒣,ϵ = ℛsb i,ϵ

Distributed source coding: outline of proofs

22

• Proof:

is trivial. We need to show that .

• Idea: For every

satisfying conditions in , we need to find corresponding that satisfy conditions in .

• Define

.

ℛsb

𝒣,ϵ ≥ ℛsb i,ϵ

ℛsb

𝒣,ϵ ≤ ℛsb i,ϵ

U1, U2, ̂ f ℛsb

i,ϵ

W1, W2 ℛsb

𝒣,ϵ

̂ wi(ui) = {xi : p(ui, xi) > 0}

Theorem 2: The region and match on the sum-rate bound, i.e. .

ℛsb

𝒣,ϵ

ℛsb

i,ϵ

ℛsb

𝒣,ϵ = ℛsb i,ϵ

Distributed source coding: outline of proofs

23

• Proof:

is trivial. We need to show that .

• Idea: For every

satisfying conditions in , we need to find corresponding that satisfy conditions in .

• Define

.

• Define

as

ℛsb

𝒣,ϵ ≥ ℛsb i,ϵ

ℛsb

𝒣,ϵ ≤ ℛsb i,ϵ

U1, U2, ̂ f ℛsb

i,ϵ

W1, W2 ℛsb

𝒣,ϵ

̂ wi(ui) = {xi : p(ui, xi) > 0} W1, W2 p(w1, w2|u1, u2, x1, x2) = { 1, if w1 = ̂ w1(u1), w2 = ̂ w2(u2) 0, otherwise.

Theorem 2: The region and match on the sum-rate bound, i.e. .

ℛsb

𝒣,ϵ

ℛsb

i,ϵ

ℛsb

𝒣,ϵ = ℛsb i,ϵ

Distributed source coding: outline of proofs

24

• Proof:

is trivial. We need to show that .

• Idea: For every

satisfying conditions in , we need to find corresponding that satisfy conditions in .

• Define

.

• Define

as

• Rest of the proof is showing that the above definitions of

satisfy the required conditions.

ℛsb

𝒣,ϵ ≥ ℛsb i,ϵ

ℛsb

𝒣,ϵ ≤ ℛsb i,ϵ

U1, U2, ̂ f ℛsb

i,ϵ

W1, W2 ℛsb

𝒣,ϵ

̂ wi(ui) = {xi : p(ui, xi) > 0} W1, W2 p(w1, w2|u1, u2, x1, x2) = { 1, ifw1 = ̂ w1(u1), w2 = ̂ w2(u2) 0, otherwise. W1, W2

Theorem 2: The region and match on the sum-rate bound, i.e. .

ℛsb

𝒣,ϵ

ℛsb

i,ϵ

ℛsb

𝒣,ϵ = ℛsb i,ϵ

Distributed source coding: outline of proofs

25

Successive refinement coding: some definitions

A hypergraph is an -achievable hypergraph with respect to , if, for any the radius of the smallest enclosing circle of the set of points is less than or equal to .

Gϵ = (𝒴, E) ϵ f : 𝒴 ↦ 𝒶 w ∈ E {f(x) : x ∈ w and p(x) > 0} ϵ

26

Successive refinement coding: practical codes rate region

Let if and only if for some conditional pmf such that and where is an -achievable hypergraph.

(R1, R2) ∈ ℛG,(ϵ1,ϵ2) R1 > I(X; W1) R2 > I(X; W1, W2), p(w1, w2|x) X ∈ W1 ∈ Γ(Gϵ1) X ∈ W2 ∈ Γ(Gϵ2) Gϵ ϵ

27

Successive refinement coding: main result

Theorem 4: Hypergraph-based codes for this problem matches the optimal codes, i.e. .

ℛ(ϵ1,ϵ2) = ℛG,(ϵ1,ϵ2)

28

Future work

• Hypergraph-based coding schemes for other source

coding problems.

• Practical implementation of the proposed coding

schemes.

29