FUNDAMENTAL PERFORMANCE LIMITS OF ANALOG-TO-DIGITAL COMPRESSION - - PowerPoint PPT Presentation

FUNDAMENTAL PERFORMANCE LIMITS OF ANALOG-TO-DIGITAL COMPRESSION

April 2016

Andrea Goldsmith

Stanford University

Yonina Eldar

Technion

1

Tsachy Weissman

Stanford University

Alon Kipnis

Stanford University

OUTLINE

2

Optimal sampling structure

Optimal sampling frequency under bitrate constraint

Combined sampling and source coding

Below the Nyquist rate

Motivation: fundamental performance limits

• f analog-to-digital compression

3

MOTIVATION:

Analog to digital (A/D) conversion:

010010011001 001000010000 1000100111…

Main goal: measuring and minimizing distortion in A/D

error bitrate(memory) [bit/sec]

D R

Encoder

4

COMBINED SAMPLING AND SOURCE CODING

Distortion is due to:

Finite bit representation (lossy compression) Time discretization / sampling (next slide) MOTIVATION

01001001 00101001

?

System model: combined sampling and source coding Decoder

ˆ X(·) R

Encoder

X(·)

fs

Y [·]

Analog source coding:

Encoder Decoder

X(0 : T)

ˆ X(0 : T)

W ∈ {0, 1}bRT c

fs

Quantizer

X(0 : T) {0, 1}bRT c

5

WHY SUB-SAMPLING?

Why sub-sampling?

MOTIVATION

High sampling rate bloats memory

Sampling rate is limited by technology

Decoder

ˆ X(·) R

Encoder

X(·)

fs Y [·]

10 10

2

10

4

10

6

10

8

10

10

2 4 6 8 10 12 14 16 18 20 22 24 Fsample (HZ) SNRbits (effective number of bits)

2005 1999 P=9.13x1011 P=4.1x1011

Data from 1978 to 1999 Data from 2000 to 2005

Analog Devices: 24 bit 2.5MS/s 16 bit 100 MS/s

6

WHY SUB-SAMPLING?

MOTIVATION

“we are not interested in exact transmission when we have a continuous source, but only in transmission to within a given tolerance”

Shannon [1948]:

R

D(R)

Can we achieve D(R) by sampling below fNyq?

t

Sampling theorem:

fs > fNyq , 2fB

X(t) = X

n∈Z

X ✓ n fs ◆ sin (fst − n) fst − n

D(R) is the minimal distortion possible using bitrate R

7

INFORMATION THEORETIC REPRESENTATION

ˆ X(·) X(·)

fs

Y [·] I(Y ; ˆ X) ≤ R PY | ˆ

X

Decoder

X(0 : T)

Y [0 : bTfsc]

n 1, . . . , 2bT Rco

ˆ X(0 : T)

Encoder

fs

PROBLEM FORMULATION

Remote (indirect) source coding [Dobrushin [Wolf & Ziv ’70], [Berger ’71]

[Witsenhausen ’80 CEO…

I(Y ; ˆ X) ≤ R

D(fs, R) = inf

T

inf

Y → ˆ X

1 2T Z T

−T

Ed (x(t), ˆ x(t)) dt d (x(t), ˆ x(t))

8

REMOTE SOURCE CODING

BACKGROUND

Dec Enc PY|X

Xn Y m(n)

• 1, . . . , 2nR

ˆ Xn

d ⇣ Xn, ˆ Xn ⌘

Dec Enc

Y m(n)

• 1, . . . , 2nR

ˆ Y m ˆ d (ym

0 , ˆ

ym

0 ) , E

h d(Xn

0 , ˆ

Xn

0 |Y m

= ym

0 , ˆ

Y m = ym i

Reduced distortion measure [Witsenhausen ’80]

Edn ⇣ Xn

0 , ˆ

Xn ⌘ = E ˆ dm ⇣ Y m

0 , ˆ

Y m ⌘

I(Y, ˆ Y ) ≤ R

DX|Y (R) = inf

n

inf

Y → ˆ Y

Edn ⇣ Xn

0 , ˆ

Xn

0 ( ˆ

Y m

0 )

⌘

Corollary: source coding theorem:

9

ASSUMPTIONS

PROBLEM FORMULATION

ˆ X(·) X(·)

fs

Y [·] I(Y ; ˆ X) ≤ R PY | ˆ

X

d (x(t), ˆ x(t)) = (x(t) − ˆ x(t))2 quadratic distortion X(·) is stationary Gaussian with PSD SX(f) SX(f) is unimodal

SX(f)

f

pointwise uniform sampler Y [n] = X (n/fs)

D(fs, R) = mmseX|Y (fs) + inf

Y → ˆ X

lim

T →∞

1 2T Z T

−T

⇣ E [X(t)|Y ] − ˆ X(t) ⌘2 dt

I(Y ; ˆ X) ≤ R

⇒

[Wolf & Ziv ‘70]

R

D(R)

10

SPECIAL CASE: QUADRATIC GAUSSIAN DISTORTION-RATE

[Pinsker1954]

R(θ) = 1 2 Z ∞

−∞

log+ [SX(f)/θ] d f D(θ) = Z ∞

−∞

min {SX(f), θ} d f

SX(f)

f θ

⇥

f SX(f)

θ

Decoder

ˆ X(·) R

Encoder

fs Y [·]

X(·) D(fs, R) = D(R) fs > fNyq

RESULT I: SAMPLING-RATE-DISTORTION FUNCTION

11

Decoder

ˆ X(·) R

Encoder

X(·)

fs Y [·]

RESULT I

mmseX|Y (fs) = σ2

X −

Z

fs 2

− fs

2

e SX|Y (f)d f

e SX|Y (f) = P

k∈Z S2 X(f − fsk)

P

k∈Z SX(f − fsk)

Theorem [K., Goldsmith, Weissman, Eldar 2014]

R(fs, θ) = 1 2 Z

fs 2

− fs

2

log+ h e SX|Y (f)/θ i d f D(fs, θ) = mmseX|Y (fs) + Z

fs 2

− fs

2

min{e SX|Y (f), θ}d f

WATERFILLING INTERPRETATION

12

R(fs, θ) = 1 2 Z

fs 2

− fs

2

log+ h e SX|Y (f)/θ i d f

D(fs, θ) = mmseX|Y (fs) + Z

fs 2

− fs

2

min{e SX|Y (f), θ}d f

D(fs, R) = mmse + waterfilling

X

k∈Z

SX(f − fsk)

e SX|Y (f)

f

fs

RESULT I

θ

mmseX|Y (fs) = σ2

X −

Z

fs 2

− fs

2

e SX|Y (f)d f

σ2

X =

Z

fs 2

− fs

2

X

k∈Z

SX(f − fsk)d f Distortion due to sampling Distortion due to rate constraint

EXAMPLE

13

RESULT I D(fs, R) = ( 1 −

fs 2W + fs 2W 2− 2R

fs

fs < 2W 2−R/W fs ≥ 2W

fs

D ( R , fs )

Distortion

fNyq

D(R)

f

SX(f) W

1

1 4

D(fs, R) vs fs (R = 1)

EXTENDED MODEL: PRE-SAMPLING FILTER

14

Decoder

ˆ X(·) R

Encoder

X(·)

fs Y [·]

H(f) What is D?(R, fs) , inf

H D(R, fs, H) ?

˜ SX|Y (f) = P

k∈Z S2 X(f − fsk) |H(f − fsk)|2

P

k∈Z SX(f − fsk) |H(f − fsk)|2

Theorem [K. Goldsmith, Weissman, Eldar 2014]

R(fs, θ) = 1 2 Z

fs 2

− fs

2

log+ h e SX|Y (f)/θ i d f D(fs, θ, H) = mmseX|Y (fs) + Z

fs 2

− fs

2

min n e SX|Y (f), θ

• d

f

RESULT II

RESULT II: OPTIMAL PRE-SAMPLING FILTER

15 15

SX(f)

θ

−fs 2 fs 2

fs H?(f)

Suppresses lower energy bands

RESULT II

When PSD is non-unimodal filter-bank sampling must be used

Optimal pre-sampling filter maximizes passband energy under an aliasing-free constraint. For unimodal PSD: Theorem [K. Goldsmith, Eldar, Weissman 2014]

R(θ) = 1 2 Z

fs 2

− fs

2

log+ [SX(f)/θ] d f

D?(fs, θ) = mmse?

X|Y (fs) +

Z

fs 2

− fs

2

min {SX(f), θ} d f

WHY ANTI-ALIASING IS OPTIMAL?

X(·)

fs Y [·]

H(f)

X1 ∼ N(0, σ2

1)

X2 ∼ N(0, σ2

2)

mmse(X1,X2)|Y = ⇣ X1 − f X1 ⌘2 + ⇣ X2 − f X2 ⌘2

inf

h1,h2 mmse(X1,X2)|Y = min

• σ2

1, σ2 2

( h1 = 0, h2 = 1, σ2

1 < σ2 2

h1 = 1, h2 = 0, σ2

1 > σ2 2

Y = h1X1 + h2X2

= h2

1σ4 1 + h2 2σ4 2

h2

1σ2 1 + h2 2σ2 2

h2 h1 +

16

e X1 , E [X1|Y ]

e X2 , E [X2|Y ] mmse

fs

SX(f)

H?(f)

RESULT II

OPTIMAL PRE-SAMPLING FILTER - EXAMPLE

Decoder

ˆ X(·) R

Encoder

X(·)

fs Y [·]

H(f)

fs

D? ( R , fs )

D(R, fs)

D?(R, fs) and D(R, fs) vs fs

0.5 1 1.5 2 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Distortion

f

fs

θ D(R, fs)

RESULT II

θ

−fs 2

fs 2

fs H?(f)

D?(R, fs)

fs

0.5 1 1.5 2

D?(R, fs) vs fs

D?(fs, R)

D(R)

OPTIMAL SAMPLING RATE

18

fNy

COROLLARY:

SX(f)

θ

fs

SX(f)

θ

fs

SX(f)

θ

fs

D?(fs, R) = D(R) for fs ≥ fDR(R) (!)

fDR(R)

RESULT III: OPTIMAL SAMPLING RATE IN ANALOG-TO-DIGITAL COMPRESSION

RESULT III:

Extends Shannon-Nyquist-Kotelinkov- Whittaker sampling theorem:

Valid when X() is not bandlimited Incorporates lossy compression

Theorem [K. Goldsmith, Eldar 2015]

D?(fs, R) = D(R)

fs ≥ fDR(R)

A new sampling theorem for Gaussian stationary processes:

SX(f)

θ

H?(f)

fDR(R)

Interpretation: lossy compression reduces degrees of freedom

Holds under non-uniform sampling

OPTIMAL SAMPLING RATE - EXAMPLE

20

RESULT III

fs fNyq

1 σ2

X

R = 0.5 R = 1

R = 2

R = 20

fDR(R)

D?(fs, R) [dB]

SX(f)

D?(fs, R) as a function of

fs fNyq

OPTIMAL SAMPLING RATE - EXAMPLE

21

RESULT III

SAMPLING NON-BANDLIMITED SIGNALS

22 22

SX(f)

θ

−fs 2 fs 2

fs

H?(f)

f

MSE [dB]

fs fNyq = ∞ fDR(R) < ∞

fDR(R)

fDR(R) → ∞ Sampling a Brownian motion ? ISIT 2016 ….

SUMMARY

23

Constraining sampling frequency in the analog source coding problem leads to D(fs,R) = mmse + `water-filling’ Optimal pre-sampling filter eliminates aliasing New critical sampling frequency f performance D(R) under bitrate constraints — typically below the Nyquist rate

THE END!

24

fs

θ

D(R, fs)

f

OPEN QUESTIONS AND FUTURE WORK

25

Optimal pre-sampling filter with different loss criterion Unknown source statistic Effect of lossy compression on DoF in other signal model

(e.g. sparse signals)

Optimal sampling rate - bit allocation strategy in existing A/D schemes

Linear pre-processing reduces sampling distortion. Can non-linear (time-preserving)

REFERENCES

26

• A. Kipnis, A. J. Goldsmith, T. Weissman and Y. C. Eldar, “Rate distortion
• f Gaussian stationary processes”

2015, available online

• A. Kipnis, A. J. Goldsmith and Y. C. Eldar, “Sub-Nyquist sampling

achieves optimal rate-distortion”

• A. Kipnis, A. J. Goldsmith and Y. C. Eldar, “Distrotion-rate function

under sub-Nyquist nonuniform samplig”

• A. Kipnis, A. J. Goldsmith and Y. C. Eldar, “Rate-Distortion Function of

Cyclostationary Gaussian Processes”

• A. Kipnis, A. J. Goldsmith and Y. C. Eldar, “Optimal tradeoff between

sampling rate and quantizer resolution in A/D conversion”

27

EXTENSION TO GENERAL PSD

ˆ X(·) X(·)

fs

Y [·] I(Y ; ˆ X) ≤ R PY | ˆ

X

DP (fs, R) D?(fs, R) , lim

P →∞

inf

H1,...,HP DP (fs, R)

H1(f) HP(f)

X(·)

fs/P fs/P

Y [·]

sampler

GENERAL (NON-UNIMODAL) PSD

ˆ X(·) X(·)

fs

Y [·] I(Y ; ˆ X) ≤ R PY | ˆ

X

D?(fs, R) =? Theorem [K. Goldsmith, Weissman, Eldar 2014]

SX(f)

f

θ

D?(fs, θ) = Z ∞

−∞

SX(f)d f − Z

A? [SX(f) − θ]+ d

f R = 1 2 Z

A? log+ [SX(f)/θ] d

f Z

A? SX(f)d

f = inf

λ(A)≤fs SX(f)d

f

29

NONUNIFORM SAMPLING

EXTENSION I

t →

t1t2 t3

t4

Λ , {t1, t2, . . .} d−(Λ) = lim

T →∞

n−(Λ) T

minimal no. of elements of in an interval of length T

Λ n−(Λ) ,

Decoder

ˆ X(·) R

Encoder

X(·)

Y [n]

tn ∈ Λ

Theorem [K. Goldsmith, Eldar 2014]

DΛ(R) ≥ D(d−(Λ), R) DΛ(R) , minimal expected distortion

using rate R codes on Y[]

May still reduce pre-processing complexity Nonuniform sampling does not provide theoretical gain

X(·)

fs Y [·]

H(f)

SY (f) = X

k

H∗(f − fsk)SX(f − fsk) Analogy: aliasing in sub-Nyquist sampling

f

fs 2fs −fs

PASS ONLY THE HIGHEST FREQUENCIES, SUPPRESS THE REST

fs 2

−fs 2

SX(f)

SY (f)

fNyq 2

WHY ANTI-ALIASING IS OPTIMAL?

( h1 = 0, h2 = 1, σ2

1 < σ2 2

h1 = 1, h2 = 0, σ2

1 > σ2 2

30