Analog-to-Digital Compression Oral PhD Exam Alon Kipnis Advisor: - - PowerPoint PPT Presentation

/32

Analog-to-Digital Compression

Oral PhD Exam Alon Kipnis

Fundamental performance limits of

Advisor: Andrea Goldsmith

1

/32

Outline

Motivation — Factors affecting analog-to-digital conversion

Main problem — Combined problem sampling and lossy compression

Corollary — Optimal sampling under compression constraints Summary — Toward a unified spectral theory of analog signal processing and lossy compression

2

sampling

analog

quantization (lossy compression)

010010011001001000 0100101010010001…

digital

/32

Motivation

010010011001 001000010000 1000100111…

information loss

A/D conversion

Challenges: 1) measure 2) minimize

The analog-to-digital (A/D) conversion problem:

3

/32

Motivation: measuring information loss

distortion

A/D parameters

Minimal distortion in A/D:

4

analog

010010011001001 000010010101001

digital

reconstruction

analog

quantization (lossy compression)

sampling

/32

Background: Lossy Compression

5

sampling

quantization (lossy compression)

digital analog

. . .

0 . . . 00 0 . . . 01

1 . . . 11

RT

. . .

T X(t)

R

bitrate:

[bits/sec]

The Source Coding Theorem [Shannon ‘48]:

D(R)

Shannon’s distortion-rate function

=

Theoretic lower bound for distortion in A/D Ignores effect of sampling

=

• ptimization over

probability distributions

reconstruction

analog

Enc

/32

fs

The Sampling Theorem [Whittaker, Kotelinkov, Shannon]:

distortion

sampling rate fs

Background: The Sampling Theorem

t

X(t)

fs > fNyq , 2fB

t

sinc(t)

∗

=

6

fNyq = 2fB

Y [n] = X(t/fs)

Ignores effect of quantization

sampling

quantization (lossy compression)

digital analog

Shannon’s distortion-rate function

D(R)

/32

Combined sampling and lossy compression

7

sampling

• ptimal lossy

compression

digital

analog

D(fs, R)

?

=

Minimal distortion under sampling and lossy compression

distortion

sampling rate fs

u n l i m i t e d b i t r a t e

Shannon’s distortion-rate function

D(R)

unlimited sampling rate

fNyq

/32

Sampling under Bitrate Constraints

Can we attain D(R) by sampling below Nyquist ?

The Sampling Theorem [Whittaker, Kotelinkov, Shannon]

t X(t)

Y [n] = X(t/fs)

fs > fNyq , 2fB t

sinc(t)

∗

=

8

“we are not interested in exact transmission when we have a continuous [amplitude] source, but only in transmission to within a given tolerance” [Shannon ’48]

D(fs, R)

?

=

distortion

sampling rate fs

D(R)

fNyq = 2fB

/32

Motivation — Summary

2) Can we attain D(R) by sampling below Nyquist ?

9

distortion

sampling rate fs

fNyq = 2fB

D(R)

D(fs, R)

?

=

?

1) What is the minimal distortion in sampling and lossy compression?

u n l i m i t e d b i t r a t e

unlimited sampling rate

/32

Combined Sampling and Source Coding

, inf

enc−dec,T

1 T Z T E ⇣ X(t) − b X(t) ⌘2 dt D(fs, R)

Assumptions:

is zero mean Gaussian stationary with PSD

X(t) SX(f)

SX(f)

f

is unimodal

SX(f)

Pointwise uniform sampling

Y [n] = X(n/fs)

10

sampling lossy compression

reconstruction

Enc Dec

fs Y [·]

X(t) b X(t) R

/32

Special case I: Gaussian Distortion Rate Function

Enc Dec

Y [·]

fs > fNyq

fs

⇒ = D(R) D(fs, R)

[Pinsker ’54]

Dθ(R) = Z ∞

−∞

min {SX(f), θ} d f Rθ = 1 2 Z ∞

−∞

log+ [SX(f)/θ] d f

θ

SX(f) f

θ

SX(f) f

R

D(R)

, WF(SX)

(

(water-filling) X(t) b X(t)

11

R

/32

Special case II: MMSE in sub-Nyquist Sampling

Y [·]

Enc

R

fs

R → ∞ ⇒ mmse(X|Y ) = D(fs, R) mmse(fs) =

MMSE in sub-Nyquist sampling [Chan & Donaldson ‘71, Matthews ’00]

X

k∈Z

SX(f − fsk)

e SX|Y (f) = P

k S2 X(f − fsk)

P

k SX(f − fsk)

fs

f

SX(f)

SX(f − fs)

S

X

( f + f

s

)

b X(t)

Dec

X(t)

12

/32

Combined Sampling and Source Coding

e SX|Y (f)

f

fs

θ

Distortion due to sampling Distortion due to bitrate constraint

Theorem*[K., Goldsmith, Eldar, Weissman ‘13]

D(fs, R) mmse(fs) = + WF ⇣ e SX|Y ⌘

(*) A. Kipnis, A. J. Goldsmith, T. Weissman and

• Y. C. Eldar, ‘Rate-distortion function of sub-

Nyquist sampled Gaussian sources corrupted by noise’, Allerton 2013

13

Enc Dec

fs Y [·]

R X(t) b X(t)

/32

Example: Uniform PSD

D(fs, R)

f

SX(f)

fB

distortion

fs

D(R)

D(fs, R) vs fs (R = 1)

mmse(fs)

fNyq = 2fB

14

/32

Achievability Scheme

Enc Dec

fs Y [·]

D(fs, R) = + Y [·]

estimator

E [X(t)|Y [·]]

e X(·)

Enc Enc Enc

mmse(fs) WF ⇣ e SX|Y ⌘

• rthogonalizing

transformation

e X∆2[·]

e X∆k[·]

e X∆1[·]

. . .

*

15

(*) A. Kipnis, A. J. Goldsmith and

• Y. C. Eldar, ‘The distortion rate function of cyclostationary

Gaussian processes’, (under review) 2016

R X(t) b X(t)

X

i

Ri ≤ R R1

R2

Rk

/32

Pre-Sampling Operation

Enc Dec

fs

H(f)

e SX|Y (f)

fs

θ

without pre-sampling filter

Linear pre-processing can reduce distortion

fs

θ

with pre-sampling filter

e SX|Y (f)

fs D(R) distortion

16

b X(t) X(t) R

H(f) ≡ 1

H(f)

/32

Optimal pre-Sampling Filter

Theorem* [K., Goldsmith, Eldar, Weissman ’14] The optimal pre-sampling filter is (i) anti-aliasing (ii) maximizes passband energy

(*) A. Kipnis, A. J. Goldsmith,

• Y. C. Eldar and T. Weissman, ‘Distortion-Rate function of sub-Nyquist

sampled Gaussian sources’, IEEE Trans. on Information Theory, January, 2016 SX(f)

fs

θ

H?(f)

SX(f)

fs

θ

H?(f)

no aliasing

D?(fs, R) = mmse?(fs) + WF ⇣ |H?|2 SX ⌘

17

/32

Optimal pre-Sampling Filter

(*) A. Kipnis, A. J. Goldsmith,

• Y. C. Eldar and T. Weissman, ‘Distortion-Rate function of sub-Nyquist

sampled Gaussian sources’, IEEE Trans. on Information Theory, January, 2016

Theorem* [K., Goldsmith, Eldar, Weissman ’14] The optimal pre-sampling filter is (i) anti-aliasing (ii) maximizes passband energy fs

f

low-pass is optimal

fs fs fs

f

maximal aliasing-free set is optimal

18

/32

Why anti-aliasing is optimal ?

X1 ∼ N

• 0, σ2

1

• X2 ∼ N
• 0, σ2

2

• X1

X2 + = Y h1 h2

fs fs fs f

1(σ1 > σ2) 1(σ1 < σ2)

h1

h2

= = ∗ ∗ Answer:

19

h1 h2

argmin ? = Question: {mmse(X1|Y ) + mmse(X2|Y )} mmse(Xi|Y ) = E (Xi − E[Xi|Y ])2

fs

/32

Critical Sub-Nyquist Sampling Rate

D?(R, fs) vs fs

D(R)

fNyq

fR

mmse(fs)

θ

fs

θ

fs

θ

fs

fs

distortion

(R is fixed) Sub-Nyquist sampling achieves optimal distortion-rate performance

D?(fs, R) = D(R) fs ≥ fR

SX(f)

20

/32

Critical Sub-Nyquist Sampling Rate

Theorem* [K., Goldsmith, Eldar ’15]

D?(fs, R) = D(R) fs ≥ fR

θ

fR

(+) A. Kipnis, A. J. Goldsmith and

• Y. C. Eldar, ‘Gaussian distortion-rate function under sub-

Nyquist nonuniform sampling’, Allerton 2014

Extends Kotelnikov-Whittaker-Shannon sampling theorem:

Incorporates lossy compression Valid when input signal is not band limited

Alignment of degrees of freedom

Holds under non-uniform sampling

+

(*) A. Kipnis, A. J. Goldsmith and

• Y. C. Eldar, ‘Sub-Nyquist sampling achieves optimal rate-distortion’,

Information Theory Workshop (ITW), 2015

21

/32

R 1 fR fNyq

Critical Sub-Nyquist Sampling Rate

critical sub-sampling ratio vs R

*

22

/32

Summary

Transforming analog signals to bits involves sampling and lossy compression

Parts of the signal removed due to lossy compression can be removed at the sampling stage

23

Closed-form expression for the minimal distortion as a function of the sampling rate and bitrate

• Sub-Nyquist sampling is optimal under bitrate constraint

/32

Future Work I

Degrees of freedom alignment in other sampling models ?

24

Enc Dec

{0, 1}nR b X

Example: compressed sensing

X Y sampler ∈ Rn

/32

Noisy Input Signal

(*) A. Kipnis, A. J. Goldsmith, T. Weissman and

• Y. C. Eldar, ‘Rate-distortion function of sub-

Nyquist sampled Gaussian sources corrupted by noise’, Allerton 2013

25

Enc Dec

fs

sampler

Y [·]

X(t) b X(t)

H(f)

+

η(t)

R

Theorem*[K. Goldsmith, Weissman, Eldar ’13]

∗

fs

θ e SX|Y (f)

P

k S2 X (f − fsk) |H(f − fsk)|2

P

k (SX(f − fsk) + Sη (f − fsk)) |H(f − fsk)|2

=

sampling quantization (lossy compression) digital

analog

noise

/32

Toward a Unified Spectral Theory

• f Processing Time Series

e SX|Y (f)

fs

θ

H(f)

+

η(t)

Enc Dec

26

Lossy compression Sampling Linear filtering

Does not incorporate time-flow

/32

Future Work II

Incorporating time-flow and lossy compression

[Kolmogorov ’56]: “Since a function with a bounded spectrum is always singular in the sense of my work and the

• bservation of such a function is

not related … to the stationary flow

• f new information, then the sense
• f this kind of argumentation does

not remain completely clear”

θ

SX(f)

f

27

Example: minimal distortion in causal estimation under bitrate constraint

X(t) t past future

/32

References

[3] A. Kipnis, A. J. Goldsmith and

• Y. C. Eldar, ‘The distortion-rate function of sampled Wiener

processes’, (under review) 2016 [2] A. Kipnis,

• Y. C. Eldar and A. J. Goldsmith, ‘Fundamental distortion limits of analog-to-digital

compression’, (under review) 2015 [1] A. Kipnis, A. J. Goldsmith,

• Y. C. Eldar and T. Weissman, ‘Distortion-rate function of sub-Nyquist

sampled Gaussian sources’, IEEE Trans. on Information Theory, January, 2016

• Conference version: A. Kipnis, A. J. Goldsmith and
• Y. C. Eldar, ‘Sub-Nyquist sampling achieves
• ptimal rate-distortion’, Information Theory Workshop (ITW), 2015
• Conference version: A. Kipnis, A. J. Goldsmith and
• Y. C. Eldar, ‘Gaussian distortion-rate

function under sub-Nyquist nonuniform sampling’, Allerton 2015

• Conference version: A. Kipnis, A. J. Goldsmith, T. Weissman and
• Y. C. Eldar, ‘Rate-distortion

function of sub-Nyquist sampled Gaussian sources corrupted by noise’, Allerton 2013

Analog-to-digital compression:

• Conference version: A. Kipnis, A. J. Goldsmith and
• Y. C. Eldar, ‘Information rates of

sampled Wiener processes’, ISIT 2016

• Conference version: A. Kipnis, A. J. Goldsmith and
• Y. C. Eldar, ‘Optimal trade-off between

sampling rate and quantization precision in Sigma-Delta A/D conversion’, SampTA 2015

• Conference version: A. Kipnis, A. J. Goldsmith and
• Y. C. Eldar, ‘Optimal Trade-off

Between Sampling Rate and Quantization Precision in A/D conversion’, Allerton 2015

28

/32

References (cont.)

Lossy source coding:

[4] A. Kipnis, A. J. Goldsmith and

• Y. C. Eldar, ‘The distortion rate function of cyclostationary

Gaussian processes’, (under review) 2016 [5] A. Kipnis, S. Rini and A. J. Goldsmith, ‘Multiterminal compress-and-estimate rate-distortion’, (in progress)

• Conference version: A. Kipnis, and A. J. Goldsmith, ‘Distortion rate function of cyclo-

stationary Gaussian processes’, ISIT 2014

• Conference version: A. Kipnis, S. Rini and A. J. Goldsmith, ‘The indirect rate-distortion

function of a binary i.i.d source’, ITW 2015

• Conference version: A. Kipnis, S. Rini and A. J. Goldsmith, ‘Multiterminal compress-and-

estimate source coding’, ISIT 2016

29

/32

Acknowledgments

Committee members: Emanuel Candes Abbas El-Gammal John Duchi

Yonina Eldar Tsachy Weissman

WSLers and ISLers Stefano Yuxin Milind Mainak Alexandros Nima Mahnoosh Yonathan Nariman Jiantao Kartik Idoia Miguel

30

/32

Acknowledgments

31

/32

The End!

e SX|Y (f)

fs

θ

32

D?(R, fs) vs fs

mmse(fs)

fs

distortion

D(R)

fNyq

fR

/32

Appendix I: Sampling in Source Coding

Sampling in practice:

[Berger ’68]: Joint typicality with respect to continuous-time waveform [Yaglom-Pinsker ‘57, Gallager ’68]: Karhunen–Loève transform [Shannon ’49]: Degrees of freedom = time X bandwidth [Berger ’71, Neuhoff & Pradhan 2013]: Analog distortion-rate function by discrete-time time approximations

X(t)

fs

Sampler

Y [n] continuous-time discrete-time

LTI

Constrained by hardware Constrained in bandwidth Modelling constraint

33

/32

digital

Relation to Remote Source Coding

Remote source coding [Dubroshin&Tsybakov ’62, Wolf&Ziv ’70]:

Enc Dec

X(0 : T)

information source reconstruction

b X(0 : T)

fs

Sampler Y [·]

M ∈ {0, 1}bT Rc

(*) A. Kipnis, A. J. Goldsmith and

• Y. C. Eldar, ‘The Distortion Rate Function of Cyclostationary

Gaussian Processes’, (under review) 2016

cyclo-stationary∗ e n c

• d

e c

D(fs, R)

e n c

• d

e c

D(R)

X(t)

sampling

Y [n]

b X(t)

enc-dec

D

e X

( R )

b e X(t)

kx b xk2

T

m m s e ( f

s

)

estimate

e X(t) = E [X(t)|Y [·]]

Decomposition:

=

+

mmse(fs)

D e

X(R)

D(fs, R)

34

/32

Appendix II: Brownian Motion to Bits

(analog-to-digital compression for the Wiener process)

is zero-mean Gaussian

X(t) EX(t)X(s) = min{t, s}

Dec Enc

fs

M ∈ {0, 1}bT Rc

YT [n] = XT (n/fs)

X(0 : T) b X(0 : T)

T

X(t)

S(t)

Z t dS(t) S(t) = µt + σX(t)

Model for assets pricing:

35

/32

digital

Analog-to-Digital Compression

(*) A. Kipnis, A. J. Goldsmith and

• Y. C. Eldar, ‘The Distortion-Rate Function of Sampled Wiener

Processes’, (under review) 2016

Theorem*

“Estimate-and-compress” is optimal for the Wiener process

e n c

• d

e c

D(fs, R)

e n c

• d

e c

D(R)

X(t)

sampling

Y [n]

b X(t)

=

+

mmse(fs)

D e

X(R)

D(fs, R)

enc-dec

D e

X(R)

b e X(t) e X(t)

m m s e ( f

s

)

estimate

≤

D(R) D(R)

[Berger ’70]:

= 2 π2 ln 2R−1

kx b xk2

T

enc-dec DY ( ¯ R)

b Y [n]

estimate

36

/32

Distortion-Bitrate-Sampling Function

(*) A. Kipnis, A. J. Goldsmith and

• Y. C. Eldar, ‘The Distortion-Rate Function of Sampled Wiener

Processes’, (under review) 2016

Theorem*

D(fs, R) = mmse(fs) + 1 fs Z 1 min n e S(φ), θ

• dφ

R(θ) = fs 2 Z 1 log+ h e S(φ)/θ i dφ

e S(φ) = 1 4 sin2(πφ/2) − 1 6

mmse(fs) = 1 6fs

θ φ

1

37

/32

Distortion vs Bitrate

38

fs=1 fixed

R [bits/sec]

distortion

1 6fs

mmse(fs)

D ( f

s

, R )

D(R)

/32

APP III: Nonuniform Sampling

t1

t2

t3 t4

t5 t6

t7

· · · · · · Λ

Sampler

X(·)

Y [n] = X(tn)

tn ∈ Λ

h(t, τ)

Theorem*

D? d−(Λ), R

• ≤ Dh,Λ(R)

(*) A. Kipnis,

• Y. C. Eldar and A. J. Goldsmith, ‘Fundamental Distortion Limits of Analog-to-Digital

Compression’, (under review) 2015

d−(Λ) = lim

r→∞ inf u∈R

|Λ ∩ [u, u + r)| r

is the lower Beurling density of Λ

Nonuniform sampling cannot improve over uniform

[Landau ’67]: necessary and sufficient condition for zero interpolation error:

d−(Λ) ≥ µ(suppSX)

39

/32

digital analog

Appendix IV: PCM Under Bitrate Constraint

X(t)

anti-aliasing filter

XLP F (t)

fs

Y [n] = XLP F (n/fs)

¯ R-bit quantizer

YQ[n] estimator b X(t)

DP CM(fs, R) = 1 T Z T E ⇣ X(t) − b X(t) ⌘2 dt

(*) A. Kipnis,

• Y. C. Eldar and A. J. Goldsmith, ‘Fundamental Distortion Limits of Analog-to-Digital

Compression’, (under review) 2015

Theorem* (stationary input, linear estimation)

DP CM(fs, R) = mmse(X|Y ) + DQ( ¯ R, fs)

DQ( ¯ R, fs) = c0 2fB fs σ22−2 ¯

R

mmse(X|Y ) = σ2 − Z

fs 2

− fs

2

SX(f)d f

R = ¯ Rfs

40

/32

PCM Under Bitrate Constraint

DP CM(fs, R) = mmse(X|Y ) + DQ( ¯ R, fs)

DQ( ¯ R, fs) = c0 2fB fs σ22−2 ¯

R

mmse(X|Y ) = σ2 − Z

fs 2

− fs

2

SX(f)d f

Optimal sampling rate in PCM is smaller than Nyquist (!)

(R is fixed) DP CM(fs, R)

fs fNyq

1

σ2

distortion

41