Structure of Optimal Quantizer for Binary-Input Continuous-Output - - PowerPoint PPT Presentation

▶
structure of optimal quantizer for binary input
SMART_READER_LITE
LIVE PREVIEW

Structure of Optimal Quantizer for Binary-Input Continuous-Output - - PowerPoint PPT Presentation

Sep 08, 2022 112 likes •356 views

Structure of Optimal Quantizer for Binary-Input Continuous-Output Channels with Output Constraints Thuan Nguyen and Thinh Nguyen Oregon State University nguyeth9@oregonstate.edu, thinhq@oregonstate.edu June 6, 2020 Thuan Nguyen and Thinh

slide-1
SLIDE 1

Structure of Optimal Quantizer for Binary-Input Continuous-Output Channels with Output Constraints

Thuan Nguyen and Thinh Nguyen

Oregon State University nguyeth9@oregonstate.edu, thinhq@oregonstate.edu

June 6, 2020

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 1 / 23

slide-2
SLIDE 2

Outline

1

Introduction

2

Problem Formulation

3

Structure of optimal quantizer

4

Applications

5

Conclusion

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 2 / 23

slide-3
SLIDE 3
  • 1. Introduction

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 3 / 23

slide-4
SLIDE 4

Quantization maximizing mutual information

  • Designing quantizer maximizing mutual information receives

tremendous attention [1,2,3,4,5,6,7,8,9,10,11].

  • Finding the optimal quantizer under some output constraints has a

long history: entropy-constrained scalar quantization [12], [13], [14], entropy-constrained vector quantization [15], [16], [17]. → We investigate the problem of designing optimal quantizers that maximize the mutual information between input and the quantized

  • utput under quantized output constraints.

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 4 / 23

slide-5
SLIDE 5

Related work

  • Strouse et al. proposed an iterative algorithm to find a local optimal

quantizer that maximizes the mutual information under the entropy constraint of output [18].

  • Gyorgy and Linder proved that convex cell quantizers are optimal for

entropy-constrained scalar quantization [14].

  • Kurkoski and Yagi showed that if the input is binary then the
  • ptimal quantizer maximizing mutual information must belong to class
  • f convex cell quantizers [1].

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 5 / 23

slide-6
SLIDE 6

Our Results

  • The optimal quantizer maximizing mutual information under any
  • utput constraint must belong to the class of convex cell quantizers.
  • A fast algorithm for finding a globally optimal quantizer if the

channel input is binary.

  • A sufficient condition for which a single threshold quantizer is
  • ptimal.

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 6 / 23

slide-7
SLIDE 7
  • 2. Problem formulation

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 7 / 23

slide-8
SLIDE 8

Problem formulation

  • Binary random input X = {x1, x2}, pX = [px1, px2] = [p1, p2].
  • Continuous output Y is specified by two given conditional densities

py|x1 = φ1(y) and py|x2 = φ2(y).

  • Quantizer Q is used to quantize Y to a discrete output set

Z = {z1, z2, . . . , zN}. Our objective is to find the solution to the following optimization problem: max

Q βI(X; Z) − C(pZ),

(1)

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 8 / 23

slide-9
SLIDE 9

Notations

  • ry = px1|y =

p1φ1(y) p1φ1(y) + p2φ2(y).

  • vy = px|y = [px1|y, px2|y] = [ry, 1 − ry].
  • µ(y) denotes the density function of Y , µ(y) = p1φ1(y) + p2φ2(y).
  • Zi denotes the set of y that is quantized to the ith output zi.

Zi = {y : Q(y) = zi}.

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 9 / 23

slide-10
SLIDE 10

Definitions

Definition

(Kullback-Leibler (KL) divergence) KL divergence of two probability vectors a = (a1, a2, . . . , aJ) and b = (b1, b2, . . . , bJ) is defined by D(a||b) =

J

  • i=1

ai log(ai bi ). (2)

Definition

(Centroid) Centroid of subset Zi ⊂ R is a two dimensional vector ci = [ci, 1 − ci] that globally minimizes the total KL divergence vy to ci from all y ∈ Zi: ci = arg min

c

  • y∈Zi

D(vy||c)µ(y)dy. (3)

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 10 / 23

slide-11
SLIDE 11

Definitions cont.

Definition

(Vector order) For two given conditional probability vectors vy1 and vy2, we define vy1 ≤ vy2 if and only if px1|y1 ≤ px1|y2 or ry1 ≤ ry2. Similarly, for two centroid vectors ci = [ci, 1 − ci] and cj = [cj, 1 − cj], ci ≤ cj if and

  • nly if ci ≤ cj.

Definition

(Set order) Given two arbitrary sets A ⊂ R and B ⊂ R , we define A ≤ B if and only if for all ya ∈ A and any yb ∈ B, we have vya ≤ vyb. We define A ≡ B if and only if A ⊂ B and B ⊂ A.

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 11 / 23

slide-12
SLIDE 12

Definitions cont.

Definition

(Convex cell quantizer) A quantizer is a convex cell quantizer in ry if Q∗(ry) = zi, if a∗

i−1 ≤ ry < a∗ i ,

(4) for some optimal thresholds a∗

0 = 0 < a∗ 1 < · · · < a∗ N−1 < a∗ N = 1.

In other words, Q is convex cell quantizer if Z1 ≤ Z2 ≤ · · · ≤ ZN.

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 12 / 23

slide-13
SLIDE 13
  • 3. Structure of optimal quantizer

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 13 / 23

slide-14
SLIDE 14

Optimal quantizer and KL-divergence

  • A quantizer Q maximizing I(X; Z) is equivalent to minimizing the

average distortion of KL divergence EY [DKL(vy||ci)] = D(Q) [2].

  • If D(Q) ≤ D(Q′) then I(X; Z)Q ≥ I(X; Z)Q′.

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 14 / 23

slide-15
SLIDE 15

Structure of optimal quantizer for binary partition (N = 2)

Theorem

Let Q be an arbitrary quantizer that produces two disjoint output sets {Z1, Z2} corresponding to two centroid vectors c1, c2 such that c1 ≤ c2, there exists a convex cell quantizer ¯ Q with two output sets { ¯ Z1, ¯ Z2} and the corresponding centroids {¯ c1, ¯ c2} such that ¯ Z1 ≤ ¯ Z2, pZi = p ¯

Zi for

i = 1, 2 and D( ¯ Q) ≤ D(Q).

Proof.

For any given quantizer Q(y), we show that existing a convex cell quantizer ¯ Q(ry) such that:

  • pZ = p ¯

Z.

  • D( ¯

Q) ≤ D(Q). → The optimal quantizer must belong to the class of convex cell quantizers.

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 15 / 23

slide-16
SLIDE 16

Optimal quantizer structure for N > 2

Theorem

Let Q be a quantizer with arbitrary disjoint quantized-output sets {Z1, Z2, . . . , ZN} corresponding to N centroids c1, c2, . . . , cN such that ci ≤ ci+1 ∀ i, there exists an other convex cell quantizer ¯ Q with the output sets { ¯ Z1, ¯ Z2, . . . , ¯ ZN} and the corresponding centroids {¯ c1, ¯ c2, . . . , ¯ cN} such that ¯ Zi ≤ ¯ Zi+1, pZi = p ¯

Zi ∀ i and D( ¯

Q) ≤ D(Q).

Proof.

Our proof is based on the induction method using the base case in Theorem 1.

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 16 / 23

slide-17
SLIDE 17
  • 4. Applications

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 17 / 23

slide-18
SLIDE 18

Finding optimal binary partition N = 2

Based on Theorem 1, Q∗(ry) =

  • z1 if ry ≤ a∗

1,

z2 if ry > a∗

1,

for an optimal a∗

1 ∈ (0, 1).

  • The optimal quantizer can be found by an exhaustive searching over

a∗

1 ∈ (0, 1).

  • If φ2(y)

φ1(y) is a strictly increasing/decreasing function, the classical

  • ne threshold quantizer is optimal.

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 18 / 23

slide-19
SLIDE 19

Finding optimal quantizer for N > 2

From Theorem 2, finding the optimal quantizer is equivalent to finding N + 1 scalar thresholds a0 = 0 < a1 < · · · < aN−1 < aN = 1 such that Q(y) = zi, if ai−1 ≤ ry = px1|y < ai. → The problem of finding globally optimal quantizer can be cast as a 1-dimensional scalar quantization problem that can be solved efficiently using the well-known dynamic programming [1], [11], [19].

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 19 / 23

slide-20
SLIDE 20
  • 5. Conclusion

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 20 / 23

slide-21
SLIDE 21

In this paper,

  • We show that the optimal quantizer that maximizes the mutual

information between input and output under an output constraint separates ry into convex cells.

  • We provide a sufficient condition for which a single threshold

quantizer is optimal.

  • Some fast algorithms are proposed for determining the optimal

quantizers.

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 21 / 23

slide-22
SLIDE 22

Thank you for listening !!!

Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 22 / 23

slide-23
SLIDE 23

References

Brian M Kurkoski and Hideki Yagi. Quantization of binary-input discrete memoryless channels. IEEE Transactions on Information Theory, 60(8):4544–4552, 2014. Jiuyang Alan Zhang and Brian M Kurkoski. Low-complexity quantization of discrete memoryless channels. In 2016 International Symposium on Information Theory and Its Applications (ISITA), pages 448–452. IEEE, 2016. Ken-ichi Iwata and Shin-ya Ozawa. Quantizer design for outputs of binary-input discrete memoryless channels using smawk algorithm. In Information Theory (ISIT), 2014 IEEE International Symposium on, pages 191–195. IEEE, 2014. Rudolf Mathar and Meik Dörpinghaus. Threshold optimization for capacity-achieving discrete input one-bit output quantization. In 2013 IEEE International Symposium on Information Theory, pages 1999–2003. IEEE, 2013. Yuta Sakai and Ken-ichi Iwata. Suboptimal quantizer design for outputs of discrete memoryless channels with a finite-input alphabet. In Information Theory and its Applications (ISITA), 2014 International Symposium on, pages 120–124. IEEE, 2014. Harish Vangala, Emanuele Viterbo, and Yi Hong. Quantization of binary input dmc at optimal mutual information using constrained shortest path problem. In 2015 22nd International Conference on Telecommunications (ICT), pages 151–155. IEEE, 2015. Tobias Koch and Amos Lapidoth. At low snr, asymmetric quantizers are better. IEEE Trans. Information Theory, 59(9):5421–5445, 2013. Brian M Kurkoski and Hideki Yagi. Single-bit quantization of binary-input, continuous-output channels. Thuan Nguyen and Thinh Nguyen (OSU) June 6, 2020 23 / 23