From Feedback Capacity to Tight Achievable Rates without Feedback for - - PowerPoint PPT Presentation

▶
from feedback capacity to tight achievable rates without
SMART_READER_LITE
LIVE PREVIEW

From Feedback Capacity to Tight Achievable Rates without Feedback for - - PowerPoint PPT Presentation

Jun 22, 2023 482 likes •770 views

From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise Christos Kourtellaris, Charalambos D. Charalambous and Sergey Loyka ISIT 2020 1 Introduction: Objectives &

slide-1
SLIDE 1

From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

Christos Kourtellaris, Charalambos D. Charalambous and Sergey Loyka ISIT 2020

slide-2
SLIDE 2

1 Introduction: Objectives & Methodology 2 Problem Formulation 3 Achievable rates without feedback 4 Application Examples 5 Summary

slide-3
SLIDE 3

1 Introduction: Objectives & Methodology 2 Problem Formulation 3 Achievable rates without feedback 4 Application Examples 5 Summary

slide-4
SLIDE 4

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Objectives

Additive Gaussian Noise (AGN) channels for stable and unstable autoregressive (AR) noise. Objectives: Achievable rates which emerge directly from the time-domain analysis of feedback capacity, and

  • ptimal channel input processes.

Expressions of tight lower bounds on nofeedback capacity, which are computed based on finite memory channel input processes, that induce asymptotic stationarity:

Markov channel input process. Independent and Identically Distributed (IID) channel input processes.

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

4/27

slide-5
SLIDE 5

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Methodology

Methodology1: Time domain analysis and methods. Define feedback capacity and optimal channel input process. Employ the results of feedback capacity to derive achievable rates without feedback. The achievable rates without feedback provide lower bounds on no-feedback capacity. Application examples where the achievable rates are computed explicitly.

  • 1C. K. Kourtellaris and C. D. Charalambous, ”Information Structures for Feedback Capacity of Channels With Memory and

Transmission Cost: Stochastic Optimal Control and Variational Equalities,” in IEEE Transactions on Information Theory, vol. 64, no. 7, pp. 4962-4992, July 2018.

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

5/27

slide-6
SLIDE 6

1 Introduction: Objectives & Methodology 2 Problem Formulation 3 Achievable rates without feedback 4 Application Examples 5 Summary

slide-7
SLIDE 7

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Problem formulation I: Channel Model

Channel model: Yt = Xt +Vt, t = 1,...,n, 1

nEs0

  • ∑n

t=1(Xt)2

≤ κ X n {X1,X2,...,Xn} is the sequence of channel input random variables (RVs). Y n {Y1,Y2,...,Yn} is the sequence of channel output RVs. V n {V1,V2,...,Vn} is the sequence of nonstationary, jointly Gaussian distributed RVs, for fixed initial state, with distribution PV n|S0(dvn|s0). The autoregressive (AR) noise with memory M, is defined by Vt =

M

∑

j=1

ct,jVt−j +Wt = CtSt−1 +Wt, St (Vt Vt−1 ... Vt−M+1)T , t = 1,...,n, (2.1) where S0 is Gaussian independent of Wt ∈ N(0,KWt ),KWt > 0,t = 1,...,n, Ct is nonrandom.

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

7/27

slide-8
SLIDE 8

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Problem formulation II: State Space Representation

A state space representation of Vt is 2 St =AtSt−1 +BtWt, S0 = s0, t = 1,...,n, (2.2) Vt =DtSt, Yt = Xt +Vt = Xt +HtSt−1 +NtWt, (2.3) Ht

△

=DtAt, Nt

△

= DtBt (2.4) where (At,Bt,Dt) are easily determined.

  • 2T. Kailauth, A. Sayed, and B. Hassibi, Linear Estimation, Prentice Hall, 2000.
  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

8/27

slide-9
SLIDE 9

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Problem formulation III: Codes with & without Feedback

Let W : Ω → M (n) △ = {1,2,...,⌈Mn⌉} be the set of uniformly distributed messages, which are independent of (V n,S0). A time-varying feedback code for the AGN channel is denoted by C fb

Z+, and consists of codewords of

block length n, defined by E[0,n](κ)

  • X1 = e1(W ,S0),...,Xn = en(W ,S0,X n−1,Y n−1) : 1

n Ee

s0

n

∑

i=1

(Xt)2 ≤ κ

  • .

(2.5) A time-varying code without feedback for the AGN Channel, denoted by C nfb

Z+ , is the restriction of the

feedback code C fb

Z+ to the subset E nfb [0,n](κ) ⊂ E[0,n](κ), defined by

E nfb

[0,n](κ)

  • X1 = enfb

1

(W ,S0),...,Xn = enfb

n

(W ,S0,X n−1) : 1 n Eenfb

s0

n

∑

i=1

(Xt)2 ≤ κ

  • .

(2.6)

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

9/27

slide-10
SLIDE 10

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Problem formulation IV: An Important Observation

For the AGN channel with AR noise and feedback code, we consider S0 = s0 is known to encoder. Hence, V t−1 uniquely defines St−1 and vice-versa, for t = 1,...,n. Consequently, PXt|X t−1,Y t−1,S0 =PXt|V t−1,Y t−1,S0, by Yt = Xt +Vt =PXt|St−1,Y t−1,S0, t = 1,...,n where the last is due to V t−1 uniquely defines St−1 (since it is assumed that S0 = s0 is known to the encoder).

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

10/27

slide-11
SLIDE 11

1 Introduction: Objectives & Methodology 2 Problem Formulation 3 Achievable rates without feedback 4 Application Examples 5 Summary

slide-12
SLIDE 12

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Results on Feedback Capacity I

Define the error covariance by Kt

△

=Es0

  • St −

St

  • St −

St T ,

  • St

△

= Es0

  • St
  • Y t

= Es0

  • Xt −

Xt

  • Xt −

Xt T ,

  • Xt

△

=Es0

  • Xt
  • Y t

, Xt

△

=

  • Xt Xt−1

... Xt−M+1 T , t = 1,...,n. (a) Any achievable rate R(s0) for feedback code C fb

Z+ satisfies

R(s0) ≤ C(κ,s0)

△

= lim

n− →∞

1 n Cn(κ,s0) (3.7) where Cn(κ,s0) is the FTFI capacity given by Cn(κ,s0)

△

= sup

P[0,n](κ)

H(Y n|s0)−H(V n|s0) (3.8) H(·|s0) denotes differential entropy,

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

12/27

slide-13
SLIDE 13

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Results on Feedback Capacity II

The channel input and output (X n,Y n) is jointly Gaussian represented by Xt = Λt

  • St−1 −

St−1

  • +Zt,

S0 = s0, t = 1,...,n, Zt ∈ N(0,KZt )

  • indep. of

(S0,X t−1,V t−1,Y t−1), Yt = Λt

  • St−1 −

St−1

  • +Zt +Vt,

It

△

= Yt −Es0

  • Yt
  • Y t−1

=

  • Λt +Ht
  • St−1 −

St−1

  • +NtWt +Zt,

KYt|Y t−1,s0 = Es0

  • ItI T

t

  • =
  • Λt +Ht
  • Kt−1
  • Λt +Ht

T +KZt +KWt KY1|s0 = Es0

  • I1I T

1

  • I n is an orthogonal innovations process, Λt ∈ R1×M is nonrandom, and KZt ≥ 0. Moreover, Cn(κ,s0) is

given by Cn(s0,κ) = sup

P[0,n]

1 2

n

∑

t=1

log KYt|Y t−1,s0 KWt (3.9)

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

13/27

slide-14
SLIDE 14

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Results on Feedback Capacity III

  • St and Kt are solutions of the generalized Kalman-filter and generalized difference Riccati equation

(DRE):

  • St = At

St−1 + BtKWt NT

t +AtKt−1

  • Λt +Ht

T NtKWt NT

t +KZt +

  • Λt +Ht
  • Kt−1
  • Λt +Ht

T It, t = 1,...,n, Kt =AtKt−1AT

t +BtKWt BT t −

BtKWt NT

t +AtKt−1

  • Λt +Ht

T NtKWt NT

t +KZt +

  • Λt +Ht
  • Kt−1
  • Λt +Ht

T .

  • BtKWt NT

t +AtKt−1

  • Λt +Ht

T , t = 1,...,n (3.10) with initial conditions S0 = s0 and K0 = 0.

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

14/27

slide-15
SLIDE 15

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Achievable Rates Without Feedback

If the nofeedback code is used, then any achievable rate without feedback Rnfb(s0), satisfies Rnfb(s0) ≤C nfb(κ,s0)

△

= lim

n− →∞

1 n C nfb

n

(κ,s0), (3.11) C nfb

n

(κ,s0)

△

= sup

P[0,n](κ)

H(Y n|s0)−H(V n|s0) (3.12) where (X n,Y n) is jointly Gaussian, the distribution of X n is PXt|X t−1,S0,t = 1,...,n. A realization of the optimal input process is Xt = Γ0S0 +

t−1

∑

j=1

Γ1

t,jXj +Zt,

S0 = s0, Zt ∈ N(0,KZt )

  • indep. of

(S0,X t−1,V t−1,Y t−1), Yt = Γ0S0 +

t−1

∑

j=1

Γ1

t,jXj +Zt +Vt,

Pnfb

[0,n](κ) : 1

n Es0

n

∑

t=1

  • Xt

2 ≤ κ

  • .
  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

15/27

slide-16
SLIDE 16

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Achievable Rates Without Feedback II

From feedback capacity to achievable rates without feedback. Consider the feedback process X n: Xt = Λt

  • St−1 −

St−1

  • +Zt.

By Vt = Yt −Xt, we have Xt = −Λt

  • Xt−1 −Es0
  • Xt−1
  • Y t−1

+Zt. By restriction to nofeedback input X n (relabeled as the initial sequence) we deduce Xt = −ΛtXt−1 +Zt = ΛtXt−1 +Zt where Λt = −Λt.

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

16/27

slide-17
SLIDE 17

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Achievable Rates Without Feedback III

Theorem 1: Lower bounds on C nfb

n

(κ,s0) A lower bound on the FTwFI capacity C nfb

n

(κ,s0) is C nfb

n,LB(κ,s0) △

= sup

PLB

[0,n](κ)

H(Y n|s0)−H(V n|s0) (3.13) where (X n,Y n) is represented for t = 1,...,n, by Xt = ΛtXt−1 +Zt, Zt ∈ N(0,KZt )

  • indep. of

(S0,X t−1,V t−1,Y t−1), Yt = ΛtXt−1 +Zt +Vt, PLB

[0,n](κ) △

=

  • Λt,KZt
  • ,t = 1,...,n : 1

n Es0 n

∑

t=1

  • Xt

2 ≤ κ

  • ,

Moreover, C nfb

n,LB(κ,s0) is given by C nfb n,LB(s0,κ) = supPLB

[0,n]

1 2 ∑n t=1 log KYt |Y t−1,s0 KWt

where KYt|Y t−1,s0 is a function of the generalized difference Riccati equation of estimating Xt−1 from (Y t−1,S0 = s0).

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

17/27

slide-18
SLIDE 18

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Achievable Rates Without Feedback IV

From the above theorem we can identify conditions for existence of limits and asymptotic stationarity, irrespectively, if the noise is stable or unstable. From the above Theorem we obtain simpler lower bounds by further simplifying the input to (a) Xt = Zt,t = 1,...,n an independent process, (b) Xt = ΛtXt−1 +Zt, Λt ∈ R, a Markov process X n. For (a) and (b) we compute in the next section achievable lower bounds on C nfb

n

(κ,s0), and we show they induce output processes Y ∞ (for the case of stable noise) or innovation processes (for the case of unstable noise) which are asymptotically ergodic.

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

18/27

slide-19
SLIDE 19

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Achievable Rates Without Feedback V

Interchange of limit and supremum Define the variant of FTFI capacity Cn(κ,s0) with the limit and supremum operations interchanged. C −(κ,s0)

△

= sup

limn−

→∞ 1 n Es0

  • ∑n

t=1

  • Xt

2

≤κ

lim

n− →∞

1 n

  • H(Y n|s0)−H(V n|s0)
  • ≤ C fb(κ,s0)

△

= lim

n− →∞

1 n C fb

n (κ,s0).

The supremum is taken over all jointly Gaussian channel input processes X n,n = 1,2,... with feedback,

  • r distributions with feedback PXt|X t−1,Y t−1 = PXt|V t−1,Y t−1,t = 1,2,..., such that (X n,Y n),n = 1,2,...,

is jointly stationary or asymptotically stationary Gaussian. We also have similar expressions for nofeedback, denoted by C nfb

LB (κ,s0), C nfb,−(κ,s0), C nfb(κ,s0).

For any of the two limiting problems, we need to identify necessary and/or sufficient conditions for the per unit time limits C −(κ,s0), C(κ,s0) to exist, and to be independent on the initial state s0.

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

19/27

slide-20
SLIDE 20

1 Introduction: Objectives & Methodology 2 Problem Formulation 3 Achievable rates without feedback 4 Application Examples 5 Summary

slide-21
SLIDE 21

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Achievable rates without feedback for the unit memory AR(c) noise channel I

Consider the AGN channel driven by a time-varying AR noise with unit memory, AR(ct) Vt = ctVt−1 +Wt, t = 1,...,n, V0 = S0 = v0, ct ∈ (−∞,∞) We utilise Theorem 1, to provide a lower bound on C nfb

n

(κ,s0), by considering a Markov process without feedback, Xt = ΛtXt−1 +Zt, t = 1,...,n, 1 n Ev0 n

∑

t=1

  • Xt

2 = 1 n

n

∑

t=1

  • Λt

2KXt−1 +KZt

  • ≤ κ

where KXt = Ev0{X 2

t } satisfies the time-varying difference Lyapunov equation KXt =

  • Λt

2KXt−1 +KZt . The estimation error Kt Es0

  • Xt −

Xt 2 , satisfies the generalized time-varying DRE, Kt = Λ2

t Kt−1 +KZt −

  • KZt +ΛtKt−1
  • Λt −ct

2

  • KZt +KWt +
  • Λt −ct

2 Kt−1 , Kt ≥ 0, K0 = 0, t = 1,...,n,

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

21/27

slide-22
SLIDE 22

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Achievable rates without feedback for the unit memory AR(c) noise channel II

Then, the lower bound on no feedback capacity is given by C nfb

n,LB(κ,s0)

sup

  • Λt,KZt
  • ,t=1,...,n:

1 n ∑n t=1

  • Λt

2

KXt−1+KZt

  • ≤κ

1 2

n

∑

t=1

log

  • Λt −ct

2Kt−1 +KZt +KWt KWt

  • For time-invariant noise AR(c), c ∈ (−∞,∞), and by restricting, Λt = Λ∞ and KZt = K ∞

Z , then

C nfb,TI

n,LB (κ,s0) C nfb n,LB(κ,s0)

  • Λt=Λ

∞,KZt =K ∞ Z

To address the asymptotic limit C nfb,TI

LB

(κ,s0) limn−

→∞ 1 nC nfb,TI n,LB (κ,s0) we need to ensure

limn−

→∞ Kn = K ∞, where

K ∞ =

  • Λ

2K ∞ +K ∞

Z −

  • K ∞

Z +Λ∞K ∞

Λ∞ −c 2

  • K ∞

Z +KW +

  • Λ∞ −c

2K ∞ , K ∞ ≥ 0, is unique and stabilizable.

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

22/27

slide-23
SLIDE 23

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Achievable rates without feedback for the unit memory AR(c) noise channel III

Theorem 2: Achievable rates for Markov channel input process Consider the AGN with noise AR(c),c ∈ (−∞,∞), and nofeedback code. Define the set P∞,nfb

  • (Λ∞,K ∞

Z ) ∈ (−1,1)×[0,∞)

  • : detectability and stabilizability holds. Then, the lower bound
  • n nofeedback capacity is

C nfb,TI

LB

(κ,s0) = C ∞

LB(κ)

max

  • Λ

∞,K ∞ Z

  • ∈P∞,nfb(κ)

1 2 log

  • Λ∞ −c

2K ∞ +K ∞

Z +KW

KW

  • ,

(4.14) P∞,nfb(κ)

  • (Λ∞,K ∞

Z ) ∈ P∞,nfb :

K ∞

Z

1−

  • Λ∞2 ≤ κ
  • (4.15)

Moreover, there exist maximum element (Λ∞,K ∞

Z ) ∈ P∞,nfb(κ), such that the channel input process

induces asymptotic stationarity of the joint input and innovation process, and the output process is ergodic, for all κ ∈ [0,∞).

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

23/27

slide-24
SLIDE 24

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Achievable rates without feedback for the unit memory AR(c) noise channel IV

Theorem 3: Achievable rates for IID channel input process Consider the AGN with with noise AR(c),c ∈ (−∞,∞), KW = 1, and nofeedback code. The lower bound on nofeedback capacity with an IID channel input process is given by C IID

LB (κ) = 1

2 log

  • c2K ∞,∗ +κ +1
  • , κ ∈ K ∞,nfb(c,Kw), K ∞,nfb(c,KW )
  • κ ∈ [0,∞) : K ∞ ≥ 0
  • where K ∞,∗ ≥ 0 and K ∞,∗

Z

, are given as follows (i) For c = 0, κ ∈ K ∞,nfb(c,KW ), then K ∞,∗

Z

= κ and K ∞,∗ = −κ

  • 1−c2

−1+

  • κ
  • 1−c2

+1 2 +4c2κ 2c2 . (ii) For c = 0,κ ∈ [0,∞), then K ∞,∗

Z

= κ and K ∞,∗ =

κ κ+1.

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

24/27

slide-25
SLIDE 25

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Achievable rates without feedback for the unit memory AR(c) noise channel V

2 4 6 8 10 12 14 16 18 0.5 1 1.5 2 2.5

Rate (bits/channel use)

Feedforward Capacity (using water-filling) Achievable rate (IID channel input) Achievable rate (unit memory channel input) 4 4.05 4.1 4.15 4.2 4.25 4.3 1.2 1.21 1.22 1.23 1.24

Figure 1: Comparison of nofeedback capacity C nfb

WF (κ) based

  • n water-filling formula, lower

bound C IID

LB (κ) of Theorem 3 of

transmitting an IID input Zt ∈ N(0,κ), and a Markov input based on Theorem 2, for an AR(c), noise, c = 0.75 and KW = 1.

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

25/27

slide-26
SLIDE 26

1 Introduction: Objectives & Methodology 2 Problem Formulation 3 Achievable rates without feedback 4 Application Examples 5 Summary

slide-27
SLIDE 27

Introduction: Objectives & Methodology Problem Formulation Achievable rates without feedback Application Examples Summary

Conclusions

Time-domain analysis and tools of feedback capacity. Achievable rates of nofeedback capacity, and optimal channel input processes. Expressions of tight lower bounds on nofeedback capacity, which are computed based on finite memory channel input processes, that induce asymptotic stationarity:

Markov channel input process. Independent and Identically Distributed (IID) channel input processes.

  • C. Kourtellaris, C. D. Charalambous and S. Loyka — From Feedback Capacity to Tight Achievable Rates without Feedback for AGN Channels with Stable and Unstable Autoregressive Noise

27/27