On the Shamai-Laroia Approximation for the Information Rate of the - - PowerPoint PPT Presentation

On the Shamai-Laroia Approximation for the Information Rate of the ISI Channel

Yair Carmon and Shlomo Shamai (Shitz)

Department of Electrical Engineering, Technion - Israel Institute of Technology

2014 Information Theory and Applications Workshop San Diego, USA February 2014

Acknowledgment: Prof. Tsachy Weissman, FP7 Network of Excellence in Wireless COMmunications NEWCOM#, Israel Science Foundation (ISF).

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 1 / 31

Outline

1

Introduction ISI Channel and I.I.D. Information Rate Analytical Lower Bounds on the Information Rate The Shamai-Laroia Conjecture (SLC)

2

Low SNR Analysis

3

Counterexamples

4

High SNR Analysis

5

Conclusion

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 2 / 31

Introduction ISI Channel and I.I.D. Information Rate

Inter-Symbol Interference Channel Model

Input-output relationship: yk =

L−1

• l=0

hlxk−l + nk Input Sequence x∞

−∞ is assumed i.i.d, with Px = Ex2

Noise Sequence n∞

−∞ is Gaussian and i.i.d., with N0 = En2

Inter-symbol interference (ISI) coefficients hL−1 Channel frequency response H (θ) = L−1

k=0 hke−jkθ

The simplest (non-discrete) model for a channel with memory Ubiquitous in wireless and wireline communications Results presented here extend straightforwardly to a complex setting

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 3 / 31

Introduction ISI Channel and I.I.D. Information Rate

Inter-Symbol Interference Channel Model

Input-output relationship: yk =

L−1

• l=0

hlxk−l + nk Input Sequence x∞

−∞ is assumed i.i.d, with Px = Ex2

Noise Sequence n∞

−∞ is Gaussian and i.i.d., with N0 = En2

Inter-symbol interference (ISI) coefficients hL−1 Channel frequency response H (θ) = L−1

k=0 hke−jkθ

The simplest (non-discrete) model for a channel with memory Ubiquitous in wireless and wireline communications Results presented here extend straightforwardly to a complex setting

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 3 / 31

Introduction ISI Channel and I.I.D. Information Rate

Mutual Information Rate

Given by I = lim

K→∞

1 2K + 1I(yK

−K ; xK −K) = I(y∞ −∞ ; x0 | x−1 −∞)

Is the rate of reliable communications achievable by a random code with codewords distributed as x∞

−∞

For Gaussian input I has a simple expression IGaussian = 1 2π π

−π

log

• 1 + Px

N0 |H(θ)|2

• dθ

When the input is distributed on a finite set (constellation), no closed form expression is known

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 4 / 31

Introduction ISI Channel and I.I.D. Information Rate

Approximating the I.I.D. Information Rate

Two main ways to investigate I:

1 Approximations and bounds based on Monte-Carlo simulations

Provide the best accuracy But little theoretic insight High computational complexity, that grows quickly with the number

• f dominant ISI taps

c.f. [Arnold-Loeliger-Vontobel-Kavcic-Zeng’06]

2 Analytical lower bounds

Not as tight as their simulation-based counterparts But much easier to compute Useful in benchmarking communication schemes and other bounds May provide theoretical insight

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 5 / 31

Introduction ISI Channel and I.I.D. Information Rate

Approximating the I.I.D. Information Rate

Two main ways to investigate I:

1 Approximations and bounds based on Monte-Carlo simulations

Provide the best accuracy But little theoretic insight High computational complexity, that grows quickly with the number

• f dominant ISI taps

c.f. [Arnold-Loeliger-Vontobel-Kavcic-Zeng’06]

2 Analytical lower bounds

Not as tight as their simulation-based counterparts But much easier to compute Useful in benchmarking communication schemes and other bounds May provide theoretical insight

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 5 / 31

Introduction Analytical Lower Bounds on the Information Rate

Data Processing Lower Bounds

For any sequence of coefficients a∞

−∞,

I = I(y∞

−∞ ; x0 | x−1 −∞) ≥ I( k aky−k ; x0 | x−1 −∞)

Can be simplified to, I ≥ I(x0 ; x0 +

k≥1 αkxk + m

• additive noise term

)

with αk =

l alh−l−k/ l alh−l

and m ∼ N(0, N0

• l a2

l /( l alh−l)2) independent of x0

Different choices of a∞

−∞ yield different bounds

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 6 / 31

Introduction Analytical Lower Bounds on the Information Rate

Data Processing with a Sample Whitened Matched Filter

a∞

−∞ are chosen so that αk = 0 for every k > 0 (non-causal ISI

eliminated) In this case the noise term is purely Gaussian The resulting bound was first proposed in [Shamai-Ozarow-Wyner’91]: I ≥ I(x0 ; x0 + m) = Ix(SNRZF-DFE) with

Ix(γ) the MI of a scalar Gaussian channel at SNR γ with input x0 SNRZF-DFE the output SNR of the zero-forcing decision feedback equalizer (DFE): SNRZF-DFE = Px N0 exp 1 2π π

−π

log

• |H(θ)|2

dθ

• Very simple, but quite loose in medium and low SNR’s
• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 7 / 31

Introduction Analytical Lower Bounds on the Information Rate

Data Processing with a Mean Square WMF

Choose a∞

−∞ so that the noise term has minimum variance

I ≥ I(x0 ; x0 +

k≥1 ˆ

αkxk + ˆ m

• min variance

) IMMSE A tight bound in many cases Still difficult to compute and analyze Some techniques for further bounding were proposed

Using probability-of-error bounds and Fano’s inequality [Shamai-Laroia’96] Using a mismatched mutual information approach [Jeong-Moon’12]

However, none of the resulting bounds is both simple and tight

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 8 / 31

Introduction Analytical Lower Bounds on the Information Rate

Data Processing with a Mean Square WMF

Choose a∞

−∞ so that the noise term has minimum variance

I ≥ I(x0 ; x0 +

k≥1 ˆ

αkxk + ˆ m

• min variance

) IMMSE A tight bound in many cases Still difficult to compute and analyze Some techniques for further bounding were proposed

Using probability-of-error bounds and Fano’s inequality [Shamai-Laroia’96] Using a mismatched mutual information approach [Jeong-Moon’12]

However, none of the resulting bounds is both simple and tight

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 8 / 31

Introduction The Shamai-Laroia Conjecture (SLC)

The Shamai-Laroia Conjecture

[Shamai-Laroia’96] conjectured that IMMSE is lower bounded by replacing x∞

1 with g∞ 1 , i.i.d. Gaussian of equal variance:

IMMSE = I(x0 ; x0 +

k≥1 ˆ

αkxk + ˆ m) ≥ I(x0 ; x0 +

k≥1 ˆ

αkgk + ˆ m) = Ix(SNRMMSE-DFE-U) ISL SNRMMSE-DFE-U is the output SNR of the unbiased MMSE DFE:

SNRMMSE-DFE-U = exp 1 2π π

−π

log

• 1 + Px

N0 |H(θ)|2

• dθ
• − 1

ISL — a simple, tight and useful approximation for IMMSE, I

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 9 / 31

Introduction The Shamai-Laroia Conjecture (SLC)

The Shamai-Laroia Conjecture — Example

BPSK input, h = [0.408, 0.817, 0.408] (moderate ISI severity)

−10 −5 5 10 15 20 0.2 0.4 0.6 0.8 1 1.2 Px/N0 [dB] Information [bits] IMMSE ISLC Gaussian upper bound Shamai-Ozarow-Wyner lower bound

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 10 / 31

Introduction The Shamai-Laroia Conjecture (SLC)

The Shamai-Laroia Conjecture — Example, cont’

BPSK input, h = [0.408, 0.817, 0.408] (moderate ISI severity)

4.6 4.8 5 5.2 5.4 5.6 5.8 6 6.2 0.72 0.74 0.76 0.78 0.8 0.82 Px/N0 [dB] Information [bits] 10 11 12 13 14 15 16 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1 Px/N0 [dB] Information [bits]

IMMSE ISLC Gaussian upper bound Shamai-Ozarow-Wyner lower bound

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 11 / 31

Introduction The Shamai-Laroia Conjecture (SLC)

The “Strong SLC” and its Refutation

A stronger version of the SLC reads I(x0 ; x0 +

k≥1 αkxk + m) ≥ I(x0 ; x0 + k≥1 αkgk + m)

for every α∞

−∞ and m (not just ˆ

α∞

−∞ and ˆ

m) [Abbe-Zheng’12] gave a counterexample

Based on a geometrical tool using the Hermite polynomials Cannot straightforwardly refute the original SLC Uses continuous-valued input distributions (finite alphabet is more interesting)

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 12 / 31

Introduction The Shamai-Laroia Conjecture (SLC)

The “Strong SLC” and its Refutation

A stronger version of the SLC reads I(x0 ; x0 +

k≥1 αkxk + m) ≥ I(x0 ; x0 + k≥1 αkgk + m)

for every α∞

−∞ and m (not just ˆ

α∞

−∞ and ˆ

m) [Abbe-Zheng’12] gave a counterexample

Based on a geometrical tool using the Hermite polynomials Cannot straightforwardly refute the original SLC Uses continuous-valued input distributions (finite alphabet is more interesting)

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 12 / 31

Introduction The Shamai-Laroia Conjecture (SLC)

Outline of Results

We present the following, Low SNR Analysis For sufficiently low SNR, ISL > IMMSE (essentially always), disproving the original SLC Counterexample Carefully constructed clearly disprove the SLC Moreover, ISL > I is shown (in a pathological case) High SNR Analysis For sufficiently high SNR, I > ISL (for finite entropy source)

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 13 / 31

Low SNR

Universal Low SNR Behavior

Define the normalized comulants of our (zero-mean) input:

Skewness — sx = Ex3

0/[Ex2 0]3/2

Excess Kurtosis — κx = Ex4

0/(Ex2 0)2 − 3

For sx = 0, IMMSE − ISL = − 1

6C3s2 xǫ3 + O(ǫ4)

For sx = 0, IMMSE − ISL = − 1

24C4κ2 xǫ4 + O(ǫ5)

where

• k≥1 ˆ

αkxk + ˆ m the minimum variance noise term Cm =

k≥1 ˆ

αm

k /

• k≥0 ˆ

α2

k

m ǫ =

k≥0 ˆ

α2

kPx/E ˆ

m2 the smallness parameter

ǫ → 0 as Px/N0 → 0 Hence, IMMSE < ISL for sufficiently low Px/N0

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 14 / 31

Low SNR

Universal Low SNR Behavior

Define the normalized comulants of our (zero-mean) input:

Skewness — sx = Ex3

0/[Ex2 0]3/2

Excess Kurtosis — κx = Ex4

0/(Ex2 0)2 − 3

For sx = 0, IMMSE − ISL = − 1

6C3s2 xǫ3 + O(ǫ4)

For sx = 0, IMMSE − ISL = − 1

24C4κ2 xǫ4 + O(ǫ5)

where

• k≥1 ˆ

αkxk + ˆ m the minimum variance noise term Cm =

k≥1 ˆ

αm

k /

• k≥0 ˆ

α2

k

m ǫ =

k≥0 ˆ

α2

kPx/E ˆ

m2 the smallness parameter

ǫ → 0 as Px/N0 → 0 Hence, IMMSE < ISL for sufficiently low Px/N0

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 14 / 31

Low SNR

Universal Low SNR Behavior

Define the normalized comulants of our (zero-mean) input:

Skewness — sx = Ex3

0/[Ex2 0]3/2

Excess Kurtosis — κx = Ex4

0/(Ex2 0)2 − 3

For sx = 0, IMMSE − ISL = − 1

6C3s2 xǫ3 + O(ǫ4)

For sx = 0, IMMSE − ISL = − 1

24C4κ2 xǫ4 + O(ǫ5)

where

• k≥1 ˆ

αkxk + ˆ m the minimum variance noise term Cm =

k≥1 ˆ

αm

k /

• k≥0 ˆ

α2

k

m ǫ =

k≥0 ˆ

α2

kPx/E ˆ

m2 the smallness parameter

ǫ → 0 as Px/N0 → 0 Hence, IMMSE < ISL for sufficiently low Px/N0

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 14 / 31

Low SNR

Universal Low SNR Behavior

Define the normalized comulants of our (zero-mean) input:

Skewness — sx = Ex3

0/[Ex2 0]3/2

Excess Kurtosis — κx = Ex4

0/(Ex2 0)2 − 3

For sx = 0, IMMSE − ISL = − 1

6C3s2 xǫ3 + O(ǫ4)

For sx = 0, IMMSE − ISL = − 1

24C4κ2 xǫ4 + O(ǫ5)

where

• k≥1 ˆ

αkxk + ˆ m the minimum variance noise term Cm =

k≥1 ˆ

αm

k /

• k≥0 ˆ

α2

k

m ǫ =

k≥0 ˆ

α2

kPx/E ˆ

m2 the smallness parameter

ǫ → 0 as Px/N0 → 0 Hence, IMMSE < ISL for sufficiently low Px/N0

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 14 / 31

Low SNR

Outline of Proof

Using the chain rule, rewrite IMMSE as IMMSE = I0

MMSE − I1 MMSE,

where Ii

MMSE I( k≥i ˆ

αkxk ;

k≥i ˆ

αkxk + ˆ m) and α0 ≡ 1 By [Guo-Wu-Shamai-Verd´ u’11], for any RV ξ and independent Gaussian ν, I(ξ ; ξ + ν) = ρ 2 − ρ2 4 + ρ3 12

• 2 −

s2

ξ

2

• − ρ4

48

• κ2

ξ − 12s2 ξ + 6

• +O
• ρ5

with ρ = Eξ2/Eν2 Apply the above series expansion to I0

MMSE, I1 MMSE and ISL to

• btain the desired result
• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 15 / 31

Low SNR

Outline of Proof

Using the chain rule, rewrite IMMSE as IMMSE = I0

MMSE − I1 MMSE,

where Ii

MMSE I( k≥i ˆ

αkxk ;

k≥i ˆ

αkxk + ˆ m) and α0 ≡ 1 By [Guo-Wu-Shamai-Verd´ u’11], for any RV ξ and independent Gaussian ν, I(ξ ; ξ + ν) = ρ 2 − ρ2 4 + ρ3 12

• 2 −

s2

ξ

2

• − ρ4

48

• κ2

ξ − 12s2 ξ + 6

• +O
• ρ5

with ρ = Eξ2/Eν2 Apply the above series expansion to I0

MMSE, I1 MMSE and ISL to

• btain the desired result
• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 15 / 31

Counterexamples

Counterexample Settings

Inter-symbol interference

Same as in the previous example (Moderate ISI severity) A three taps impulse response, h0 = h2 = 0.408, h1 = 0.817 “Channel B” from Chapter 10 of [Proakis’87]

Input distribution 1

Symmetric trinary alphabet {−1, 0, 1} Pr(x = 1) = Pr(x = −1) = 0.01, Pr(x = 0) = 0.98 Zero skewness, high excess kurtosis: κx = 47

Input distribution 2

Highly skewed binary distribution Pr(x > 0) = 1 − Pr(x < 0) = 0.002 sx ≈ −22.3, κx ≈ 495

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 16 / 31

Counterexamples

Counterexample Settings

Inter-symbol interference

Same as in the previous example (Moderate ISI severity) A three taps impulse response, h0 = h2 = 0.408, h1 = 0.817 “Channel B” from Chapter 10 of [Proakis’87]

Input distribution 1

Symmetric trinary alphabet {−1, 0, 1} Pr(x = 1) = Pr(x = −1) = 0.01, Pr(x = 0) = 0.98 Zero skewness, high excess kurtosis: κx = 47

Input distribution 2

Highly skewed binary distribution Pr(x > 0) = 1 − Pr(x < 0) = 0.002 sx ≈ −22.3, κx ≈ 495

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 16 / 31

Counterexamples

Demonstration of Low SNR Behaviour

Numerical computation agrees with low-SNR expansion for up to ǫ ≈ 0.02 At low SNR’s and moderate to high ISI, ISL − IMMSE ≈ 10−8s2

x + 10−10κ2 x

Hence, sx ≫ 1 or κx ≫ 1 is a must Explains why similar low-SNR counterexamples were not previously reported

−22 −20 −18 −16 −8 −6 −4 −2 x 10

−6

Px/N0 [dB] Difference in information [bits] −22 −20 −18 −16 0.01 0.02 0.03 0.04 IMMSE − ISL Low SNR approximation ǫ (right scale) Symmetric Trinary input −22 −20 −18 −16 −5 −4 −3 −2 −1 x 10

−4

Px/N0 [dB] Difference in information [bits] −22 −20 −18 −16 0.02 0.04 Skewed Binary Input

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 17 / 31

Counterexamples

Demonstration of Low SNR Behaviour

Numerical computation agrees with low-SNR expansion for up to ǫ ≈ 0.02 At low SNR’s and moderate to high ISI, ISL − IMMSE ≈ 10−8s2

x + 10−10κ2 x

Hence, sx ≫ 1 or κx ≫ 1 is a must Explains why similar low-SNR counterexamples were not previously reported

−22 −20 −18 −16 −8 −6 −4 −2 x 10

−6

Px/N0 [dB] Difference in information [bits] −22 −20 −18 −16 0.01 0.02 0.03 0.04 IMMSE − ISL Low SNR approximation ǫ (right scale) Symmetric Trinary input −22 −20 −18 −16 −5 −4 −3 −2 −1 x 10

−4

Px/N0 [dB] Difference in information [bits] −22 −20 −18 −16 0.02 0.04 Skewed Binary Input

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 17 / 31

Counterexamples

Demonstration of Low SNR Behaviour

Numerical computation agrees with low-SNR expansion for up to ǫ ≈ 0.02 At low SNR’s and moderate to high ISI, ISL − IMMSE ≈ 10−8s2

x + 10−10κ2 x

Hence, sx ≫ 1 or κx ≫ 1 is a must Explains why similar low-SNR counterexamples were not previously reported

−22 −20 −18 −16 −8 −6 −4 −2 x 10

−6

Px/N0 [dB] Difference in information [bits] −22 −20 −18 −16 0.01 0.02 0.03 0.04 IMMSE − ISL Low SNR approximation ǫ (right scale) Symmetric Trinary input −22 −20 −18 −16 −5 −4 −3 −2 −1 x 10

−4

Px/N0 [dB] Difference in information [bits] −22 −20 −18 −16 0.02 0.04 Skewed Binary Input

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 17 / 31

Counterexamples

Higher SNR’s — Trinary Source

−4 −2 2 4 0.145 0.15 0.155 0.16 Px/N0 [dB] Information [bits]

IMMSE ISL I

I evaluated via Monte-Carlo method ISL significantly exceeds IMMSE! (but not I) Qualitative explanation

The noise

k≥1 ˆ

αkxk + ˆ m has excess kurtosis proportional to κx The SL approximating noise term has excess kurtosis 0 (it’s Gaussian) ⇒ Quality of the approximation is expected to deteriorate as κx grows

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 18 / 31

Counterexamples

Higher SNR’s — Trinary Source

−4 −2 2 4 0.145 0.15 0.155 0.16 Px/N0 [dB] Information [bits]

IMMSE ISL I

I evaluated via Monte-Carlo method ISL significantly exceeds IMMSE! (but not I) Qualitative explanation

The noise

k≥1 ˆ

αkxk + ˆ m has excess kurtosis proportional to κx The SL approximating noise term has excess kurtosis 0 (it’s Gaussian) ⇒ Quality of the approximation is expected to deteriorate as κx grows

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 18 / 31

Counterexamples

Higher SNR’s — Trinary Source

−4 −2 2 4 0.145 0.15 0.155 0.16 Px/N0 [dB] Information [bits]

IMMSE ISL I

I evaluated via Monte-Carlo method ISL significantly exceeds IMMSE! (but not I) Qualitative explanation

The noise

k≥1 ˆ

αkxk + ˆ m has excess kurtosis proportional to κx The SL approximating noise term has excess kurtosis 0 (it’s Gaussian) ⇒ Quality of the approximation is expected to deteriorate as κx grows

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 18 / 31

Counterexamples

Higher SNR’s — Trinary Source

−4 −2 2 4 0.145 0.15 0.155 0.16 Px/N0 [dB] Information [bits]

IMMSE ISL I

I evaluated via Monte-Carlo method ISL significantly exceeds IMMSE! (but not I) Qualitative explanation

The noise

k≥1 ˆ

αkxk + ˆ m has excess kurtosis proportional to κx The SL approximating noise term has excess kurtosis 0 (it’s Gaussian) ⇒ Quality of the approximation is expected to deteriorate as κx grows

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 18 / 31

Counterexamples

Higher SNR’s — Skewed Binary Source

4 −16 −14 −12 −10 0.016 0.018 0.02 Px/N0 [dB] Information [bits]

IMMSE ISL I

I evaluated via Monte-Carlo method This time ISL > I as well as ISL > IMMSE Thus, even the relaxed SLC, I ≥ ISL, does not hold for all SNR’s

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 19 / 31

Counterexamples

Higher SNR’s — Skewed Binary Source

4 −16 −14 −12 −10 0.016 0.018 0.02 Px/N0 [dB] Information [bits]

IMMSE ISL I

I evaluated via Monte-Carlo method This time ISL > I as well as ISL > IMMSE Thus, even the relaxed SLC, I ≥ ISL, does not hold for all SNR’s

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 19 / 31

High SNR

Universal High SNR Behavior

For a finite entropy input distribution, let

R[I] = lim

Px/N0→∞ −log[H(x0) − I]

Px/N0

be the rate of convergence to the input entropy For non-trivial ISI, we show that R[ISL] ≤ 1 2 dmin 2 2 gZF-DFE < 1 2 ∆min 2 2 ≤ R[I] where

dmin is the minimum distance between (unit-power) input values gZF-DFE = ( Px

N0 )−1SNRZF-DFE is the zero-forcing DFE gain factor

∆min is a free distance associated with optimal equalization

Hence, I > ISL for sufficiently high Px/N0

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 20 / 31

High SNR

Universal High SNR Behavior

For a finite entropy input distribution, let

R[I] = lim

Px/N0→∞ −log[H(x0) − I]

Px/N0

be the rate of convergence to the input entropy For non-trivial ISI, we show that R[ISL] ≤ 1 2 dmin 2 2 gZF-DFE < 1 2 ∆min 2 2 ≤ R[I] where

dmin is the minimum distance between (unit-power) input values gZF-DFE = ( Px

N0 )−1SNRZF-DFE is the zero-forcing DFE gain factor

∆min is a free distance associated with optimal equalization

Hence, I > ISL for sufficiently high Px/N0

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 20 / 31

High SNR

Universal High SNR Behavior

For a finite entropy input distribution, let

R[I] = lim

Px/N0→∞ −log[H(x0) − I]

Px/N0

be the rate of convergence to the input entropy For non-trivial ISI, we show that R[ISL] ≤ 1 2 dmin 2 2 gZF-DFE < 1 2 ∆min 2 2 ≤ R[I] where

dmin is the minimum distance between (unit-power) input values gZF-DFE = ( Px

N0 )−1SNRZF-DFE is the zero-forcing DFE gain factor

∆min is a free distance associated with optimal equalization

Hence, I > ISL for sufficiently high Px/N0

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 20 / 31

High SNR Outline of proof — Convergence Rate of ISL

Information-Estimation Identity for H(x0) − ISL

Recall that ISL = Ix(SNRMMSE-DFE-U) By [Guo-Shamai-Verd´ u’05] (assuming Ex2 = 1), H (x) − Ix (snr) = 1 2 ∞

snr

mmsex (γ) dγ where mmsex (γ) E (x − E [x | √γx + n])2 with n ∼ N (0, 1) and independent of x We therefore need to find an exponentially tight lower bound for mmsex

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 21 / 31

High SNR Outline of proof — Convergence Rate of ISL

Information-Estimation Identity for H(x0) − ISL

Recall that ISL = Ix(SNRMMSE-DFE-U) By [Guo-Shamai-Verd´ u’05] (assuming Ex2 = 1), H (x) − Ix (snr) = 1 2 ∞

snr

mmsex (γ) dγ where mmsex (γ) E (x − E [x | √γx + n])2 with n ∼ N (0, 1) and independent of x We therefore need to find an exponentially tight lower bound for mmsex

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 21 / 31

High SNR Outline of proof — Convergence Rate of ISL

Information-Estimation Identity for H(x0) − ISL

Recall that ISL = Ix(SNRMMSE-DFE-U) By [Guo-Shamai-Verd´ u’05] (assuming Ex2 = 1), H (x) − Ix (snr) = 1 2 ∞

snr

mmsex (γ) dγ where mmsex (γ) E (x − E [x | √γx + n])2 with n ∼ N (0, 1) and independent of x We therefore need to find an exponentially tight lower bound for mmsex

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 21 / 31

High SNR Outline of proof — Convergence Rate of ISL

Lower bound on mmsex (γ)

We construct a Genie1 G distributed on {0, 1} such that given G = 1, x is uniformly distributed two values with distance dmin We have, mmsex (γ) ≥ Pr (G = 1) mmsex|G=1 (γ) and mmsex|G=1 (γ) = dmin 2 2 mmseb dmin 2 2 γ

• where b is uniformly distributed on {−1, 1}

Finally, mmseb (ρ) ≥ 2Q (√ρ)

1This is based on the analysis in [Lozano-Tulino-Verd´

u’06], but slightly altered to accommodate non-uniformly distributed inputs

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 22 / 31

High SNR Outline of proof — Convergence Rate of ISL

Lower bound on mmsex (γ)

We construct a Genie1 G distributed on {0, 1} such that given G = 1, x is uniformly distributed two values with distance dmin We have, mmsex (γ) ≥ Pr (G = 1) mmsex|G=1 (γ) and mmsex|G=1 (γ) = dmin 2 2 mmseb dmin 2 2 γ

• where b is uniformly distributed on {−1, 1}

Finally, mmseb (ρ) ≥ 2Q (√ρ)

1This is based on the analysis in [Lozano-Tulino-Verd´

u’06], but slightly altered to accommodate non-uniformly distributed inputs

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 22 / 31

High SNR Outline of proof — Convergence Rate of ISL

Lower bound on mmsex (γ)

We construct a Genie1 G distributed on {0, 1} such that given G = 1, x is uniformly distributed two values with distance dmin We have, mmsex (γ) ≥ Pr (G = 1) mmsex|G=1 (γ) and mmsex|G=1 (γ) = dmin 2 2 mmseb dmin 2 2 γ

• where b is uniformly distributed on {−1, 1}

Finally, mmseb (ρ) ≥ 2Q (√ρ)

1This is based on the analysis in [Lozano-Tulino-Verd´

u’06], but slightly altered to accommodate non-uniformly distributed inputs

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 22 / 31

High SNR Outline of proof — Convergence Rate of ISL

Putting everything together

Using the above results,

H (x0) − ISL ≥ C

• Px/N0

exp

• −1

2 dmin 2 2 SNRMMSE-DFE-U

• for some C > 0 and sufficiently high Px/N0

It is straightforward (but technical) to show that

SNRMMSE-DFE-U ≤ Px N0 gZF-DFE + K Px N0 1−ε

for some K, ε > 0 and sufficiently high Px/N0 This establishes R[ISL] = lim

Px/N0→∞ −log[H(x0) − ISL]

Px/N0 ≤ dmin 2 2 gZF-DFE

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 23 / 31

High SNR Outline of proof — Convergence Rate of ISL

Putting everything together

Using the above results,

H (x0) − ISL ≥ C

• Px/N0

exp

• −1

2 dmin 2 2 SNRMMSE-DFE-U

• for some C > 0 and sufficiently high Px/N0

It is straightforward (but technical) to show that

SNRMMSE-DFE-U ≤ Px N0 gZF-DFE + K Px N0 1−ε

for some K, ε > 0 and sufficiently high Px/N0 This establishes R[ISL] = lim

Px/N0→∞ −log[H(x0) − ISL]

Px/N0 ≤ dmin 2 2 gZF-DFE

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 23 / 31

High SNR Outline of proof — Convergence Rate of ISL

Putting everything together

Using the above results,

H (x0) − ISL ≥ C

• Px/N0

exp

• −1

2 dmin 2 2 SNRMMSE-DFE-U

• for some C > 0 and sufficiently high Px/N0

It is straightforward (but technical) to show that

SNRMMSE-DFE-U ≤ Px N0 gZF-DFE + K Px N0 1−ε

for some K, ε > 0 and sufficiently high Px/N0 This establishes R[ISL] = lim

Px/N0→∞ −log[H(x0) − ISL]

Px/N0 ≤ dmin 2 2 gZF-DFE

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 23 / 31

High SNR Outline of proof — Convergence Rate of I

Data Processing with the ML Sequence Detector

Recall that I = I(x0 ; y∞

−∞ | x−1 −∞)

Since conditioning decreases entropy H (x0) − I = H

• x0|y∞

−∞, x−1 −∞

• ≤ H
• x0|ˆ

xML

• where
• ˆ

xML

i

∞

i=−∞ is the maximum likelihood sequence estimate of

x∞

−∞ given y∞ −∞

By Fano’s inequality H

• x0|ˆ

xML

• ≤ h2
• Pr
• x0 = ˆ

xML

• + Pr
• x0 = ˆ

xML

• log |X|

with h2(x) the binary entropy function and |X| the input alphabet cardinality

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 24 / 31

High SNR Outline of proof — Convergence Rate of I

Data Processing with the ML Sequence Detector

Recall that I = I(x0 ; y∞

−∞ | x−1 −∞)

Since conditioning decreases entropy H (x0) − I = H

• x0|y∞

−∞, x−1 −∞

• ≤ H
• x0|ˆ

xML

• where
• ˆ

xML

i

∞

i=−∞ is the maximum likelihood sequence estimate of

x∞

−∞ given y∞ −∞

By Fano’s inequality H

• x0|ˆ

xML

• ≤ h2
• Pr
• x0 = ˆ

xML

• + Pr
• x0 = ˆ

xML

• log |X|

with h2(x) the binary entropy function and |X| the input alphabet cardinality

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 24 / 31

High SNR Outline of proof — Convergence Rate of I

Data Processing with the ML Sequence Detector

Recall that I = I(x0 ; y∞

−∞ | x−1 −∞)

Since conditioning decreases entropy H (x0) − I = H

• x0|y∞

−∞, x−1 −∞

• ≤ H
• x0|ˆ

xML

• where
• ˆ

xML

i

∞

i=−∞ is the maximum likelihood sequence estimate of

x∞

−∞ given y∞ −∞

By Fano’s inequality H

• x0|ˆ

xML

• ≤ h2
• Pr
• x0 = ˆ

xML

• + Pr
• x0 = ˆ

xML

• log |X|

with h2(x) the binary entropy function and |X| the input alphabet cardinality

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 24 / 31

High SNR Outline of proof — Convergence Rate of I

Forney’s Error Probability Bound

In [Forney’72] it was shown that Pr

• x0 = ˆ

xML

• ≤ K′Q

 

• Px

N0 ∆min 2 2   for some K′ > 0 and

∆2

min = inf N≥1

min xN−1 , ˜ xN−1 s.t. x0 = ˜ x0, xN−1 = ˜ xN−1 ∆2 xN−1 , ˜ xN−1

• where

∆2 xN−1 , ˜ xN−1

• =

L+N−2

• k=0
• N−1
• l=0

(xl − ˜ xl) hk−l

• 2

is the Euclidean distance between finite sequences h ∗ x, h ∗ ˜ x

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 25 / 31

High SNR Outline of proof — Convergence Rate of I

Bounding ∆min with gZF-DFE

Putting Fano and Forney together, we establish R[I] = lim

Px/N0→∞ −log[H(x0) − I]

Px/N0 ≥ 1 2 ∆min 2 2 Finally, notice that for any non-trivial ISI channel (L > 1) ∆2 xN−1 , ˜ xN−1

• ≥ |(x0 − ˜

x0) h0|2 + |(xN−1 − ˜ xN−1) hL−1|2 ≥ d2

min

• |h0|2 + |hL−1|2

> d2

min |h0|2

Assuming w.l.o.g. hL−1 is minimal phase,

h0 = exp 1 2π π

−π

log

• |H(θ)|2

dθ

• = gZF-DFE

Hence, ∆2

min > d2 mingZF-DFE

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 26 / 31

High SNR Outline of proof — Convergence Rate of I

Bounding ∆min with gZF-DFE

Putting Fano and Forney together, we establish R[I] = lim

Px/N0→∞ −log[H(x0) − I]

Px/N0 ≥ 1 2 ∆min 2 2 Finally, notice that for any non-trivial ISI channel (L > 1) ∆2 xN−1 , ˜ xN−1

• ≥ |(x0 − ˜

x0) h0|2 + |(xN−1 − ˜ xN−1) hL−1|2 ≥ d2

min

• |h0|2 + |hL−1|2

> d2

min |h0|2

Assuming w.l.o.g. hL−1 is minimal phase,

h0 = exp 1 2π π

−π

log

• |H(θ)|2

dθ

• = gZF-DFE

Hence, ∆2

min > d2 mingZF-DFE

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 26 / 31

High SNR Outline of proof — Convergence Rate of I

Bounding ∆min with gZF-DFE

Putting Fano and Forney together, we establish R[I] = lim

Px/N0→∞ −log[H(x0) − I]

Px/N0 ≥ 1 2 ∆min 2 2 Finally, notice that for any non-trivial ISI channel (L > 1) ∆2 xN−1 , ˜ xN−1

• ≥ |(x0 − ˜

x0) h0|2 + |(xN−1 − ˜ xN−1) hL−1|2 ≥ d2

min

• |h0|2 + |hL−1|2

> d2

min |h0|2

Assuming w.l.o.g. hL−1 is minimal phase,

h0 = exp 1 2π π

−π

log

• |H(θ)|2

dθ

• = gZF-DFE

Hence, ∆2

min > d2 mingZF-DFE

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 26 / 31

Conclusion

Conclusion

Conjectured in 1996, the SLC IMMSE ≥ ISL is now finally disproved I ≥ ISL also doesn’t hold in general (skewed binary input) However, it will hold above some SNR threshold In most cases, it seems this threshold is 0... Open issues

Can a threshold independent on the ISI channel be found? For conventional inputs, such as BPSK, can the threshold be proven to be very low or even 0?

A new use for ISL: upper bound on the OFDM information rate w/ i.i.d. inputs — more details at IZS 2014 (or online)!

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 27 / 31

Conclusion

Conclusion

Conjectured in 1996, the SLC IMMSE ≥ ISL is now finally disproved I ≥ ISL also doesn’t hold in general (skewed binary input) However, it will hold above some SNR threshold In most cases, it seems this threshold is 0... Open issues

Can a threshold independent on the ISI channel be found? For conventional inputs, such as BPSK, can the threshold be proven to be very low or even 0?

A new use for ISL: upper bound on the OFDM information rate w/ i.i.d. inputs — more details at IZS 2014 (or online)!

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 27 / 31

Conclusion

Conclusion

Conjectured in 1996, the SLC IMMSE ≥ ISL is now finally disproved I ≥ ISL also doesn’t hold in general (skewed binary input) However, it will hold above some SNR threshold In most cases, it seems this threshold is 0... Open issues

Can a threshold independent on the ISI channel be found? For conventional inputs, such as BPSK, can the threshold be proven to be very low or even 0?

A new use for ISL: upper bound on the OFDM information rate w/ i.i.d. inputs — more details at IZS 2014 (or online)!

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 27 / 31

References

References I

D.M. Arnold, H.A. Loeliger, P.O. Vontobel, A. Kavcic and W. Zeng Simulation-based computation of information rates for channels with memory Information Theory, IEEE Transactions on, 52(8):3498–3508, 2006.

• S. Shamai, L.H. Ozarow, and A.D. Wyner

Information rates for a discrete-time Gaussian channel with intersymbol interference and stationary inputs Information Theory, IEEE Transactions on, 37(6):1527–1539, 1991.

• S. Shamai and R. Laroia

The intersymbol interference channel: Lower bounds on capacity and channel precoding loss Information Theory, IEEE Transactions on, 42(5):1388–1404, 1996.

• S. Jeong and J. Moon

Easily computed lower bounds on the information rate of intersymbol interference channels Information Theory, IEEE Transactions on, 58(2):864–877, 2012.

• E. Abbe and L. Zheng

A coordinate system for Gaussian Networks Information Theory, IEEE Transactions on, 58(2):721–733, 2012.

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 28 / 31

References

References II

Dongning Guo, Yihong Wu, Shlomo Shamai, and Sergio Verd´ u Estimation in gaussian noise: Properties of the minimum mean-square error Information Theory, IEEE Transactions on, 57(4):2371–2385, 2011. J.G. Proakis Digital communications McGraw-hill, 1987.

• D. Guo, S. Shamai, and S. Verd´

u Mutual information and minimum mean-square error in gaussian channels Information Theory, IEEE Transactions on, 51(4):1261–1282, 2005.

• A. Lozano, A.M. Tulino, and S. Verd´

u Optimum power allocation for parallel gaussian channels with arbitrary input distributions Information Theory, IEEE Transactions on, 52(7):3033–3051, 2006.

• G. Forney Jr.

Maximum-likelihood sequence estimation of digital sequences in the presence of intersymbol interference Information Theory, IEEE Transactions on, 18(3):363–378, 1972.

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 29 / 31

Thank You

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 30 / 31

Yair Carmon and Shlomo Shamai (Shitz)

On the Shamai-Laroia Approximation for the Information Rate of the ISI Channel

Abstract: The approximation proposed by Shamai and Laroia for the achievable rate in the intersymbol interference (ISI) channel with fixed i.i.d. inputs is considered. We investigate the conjecture that this approximation is a lower bound on the achievable rate. This conjecture is a slightly relaxed version of the original Shamai-Laroia conjecture, which has recently been disproved. It is shown that for any discrete input distribution and ISI channel, the lower bound holds in the high-SNR

• regime. Numerical evidence indicates, however, that even this relaxed

version of the conjecture does not hold in general.

• Y. Carmon and S. Shamai

On the Shamai-Laroia Approximation ITA 2014 31 / 31