Strong Converse for Testing Against Independence over a Noisy - - PowerPoint PPT Presentation

Strong Converse for Testing Against Independence

• ver a Noisy Channel

Sreejith Sreekumar and Deniz Gündüz

Cornell University Imperial College London

June 8, 2020

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 1 / 15

System Model

◮ Hypothesis test (HT): testing against independence (TAI). ◮ n i.i.d. samples Un and V n available at observer and detector, respectively. ◮ Observer transmits X n = fn(Un) to detector over a DMC PY|X. ◮ Detector outputs decision ˆ H = gn(V n, Y n). ◮ Motivation: Distributed statistical inference over noisy channels.

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 2 / 15

Type I and Type II Error Probabilities Trade-off

True hypothesis: H ∈ {0, 1}. Type I and Type II Error Probabilities: αn (fn, gn) = P (gn(Y n, V n) = 1|H = 0) , βn (fn, gn) = P (gn(Y n, V n) = 0|H = 1) . Type II Error Exponent: κ(ǫ) = sup

• κ′ : (κ′, ǫ) ∈ R
• ,

R =    (κ′, ǫ) ∈ R≥0 × [0, 1] : ∃ {(fn, gn)}n∈N, αn (fn, gn) ≤ ǫ, lim inf

n→∞

−1 n log (βn (fn, gn)) ≥ κ′    .

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 3 / 15

Existing Results:

TAI over a rate-limited noiseless channel:

◮ Single-letter type II error exponent characterization [Ahlswede and Csiszár (1986)]: lim

ǫ→0 ¯

κ(ǫ, R) = ¯ θ(PUV, R) := max {I(V; W) : I(U; W) ≤ R, V − U − W} . ◮ Strong converse holds: ¯ κ(ǫ, R) = ¯ θ(PUV, R), ∀ ǫ ∈ (0, 1). Type II error exponent independent of ǫ ∈ (0, 1).

TAI over a noisy channel:

◮ Single-letter type II error exponent characterization [SS and Gündüz (2020)] lim

ǫ→0 κ(ǫ) = θ(PUV, C) := max {I(V; W) : I(U; W) ≤ C, V − U − W} .

Does strong converse hold ?

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 4 / 15

Proof of Strong Converse in Noiseless Channel Setting: Key tools

Covering lemma: [Ahlswede and Csiszar (1986)] T n

P : Type class (non-empty) of P on fi-

nite alphabet X Non-empty subset Bn ⊆ T n

P

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 5 / 15

Proof of Strong Converse in Noiseless Channel Setting: Key tools

Covering lemma: [Ahlswede and Csiszar (1986)] T n

P : Type class (non-empty) of P on fi-

nite alphabet X Non-empty subset Bn ⊆ T n

P

{πi, 1 ≤ i ≤ N}: Set of permutations

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 5 / 15

Proof of Strong Converse in Noiseless Channel Setting: Key tools

Covering lemma: [Ahlswede and Csiszar (1986)] T n

P : Type class (non-empty) of P on fi-

nite alphabet X Non-empty subset Bn ⊆ T n

P

{πi, 1 ≤ i ≤ N}: Set of permutations

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 5 / 15

Proof of Strong Converse in Noiseless Channel Setting: Key tools

Covering lemma: [Ahlswede and Csiszar (1986)] T n

P : Type class (non-empty) of P on fi-

nite alphabet X Non-empty subset Bn ⊆ T n

P

{πi, 1 ≤ i ≤ N}: Set of permutations

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 5 / 15

Proof of Strong Converse in Noiseless Channel Setting: Key tools

Covering lemma: [Ahlswede and Csiszar (1986)] T n

P : Type class (non-empty) of P on fi-

nite alphabet X Non-empty subset Bn ⊆ T n

P

{πi, 1 ≤ i ≤ N}: Set of permutations N < |T n

P |

|Bn| log |T n P | permutations exist that cover T n P .

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 5 / 15

Proof of Strong Converse in Noiseless Channel Setting: Key tools

Blowing-up lemma: [Ahlswede-Gács-Körner (1976), Marton (1996)] Let Z1, . . . , Zn be n independent r.v.’s taking values in a finite set Z. Then, for any set D ⊆ Zn with PZ n(D) > 0 and ln > Θ(√n), PZ n(Γln(D)) ≥ 1 − o(1), Γl(D) is Hamming l-neighborhood of D, i.e., Γl(D) :=

• ˜

zn ∈ Zn :

n

• i=1

✶(zi = ˜ zi) ≤ l for some zn ∈ D

• .

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 6 / 15

Strong Converse Proof Outline (Noiseless channel)

Given: Encoder fn and decision region An ∈ M × Vn such that αn(fn, gn) ≤ ǫ.

1

Blowing-up lemma is used to show existence of message m∗ ∈ M, a set Cn ⊆ f −1

n (m∗) ⊆ Un and decision rule ¯

gn = 1 − ✶ ¯

An

such that given un ∈ Cn, αn(fn, ¯ gn|Un = un)

(n)

− − → 0, βn(fn, ¯ gn|Un = un) ≤ βn(fn, gn) eno(1).

2

Covering lemma is used to guarantee existence of N ≈ enR permutations πi, i ∈ [1 : N], of Cn that cover typical set Tn(PU, δ), δ > 0.

3

M is set as index of permutation, and acceptance region as A∗

n = N i=1{i} × πi( ¯

An)

• disjoint union of sets {i} × πi( ¯

An)

• .

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 7 / 15

Strong Converse Proof Outline (Noiseless channel)

Given: Encoder fn and decision region An ∈ M × Vn such that αn(fn, gn) ≤ ǫ.

1

Blowing-up lemma is used to show existence of message m∗ ∈ M, a set Cn ⊆ f −1

n (m∗) ⊆ Un and decision rule ¯

gn = 1 − ✶ ¯

An

such that given un ∈ Cn, αn(fn, ¯ gn|Un = un)

(n)

− − → 0, βn(fn, ¯ gn|Un = un) ≤ βn(fn, gn) eno(1).

2

Covering lemma is used to guarantee existence of N ≈ enR permutations πi, i ∈ [1 : N], of Cn that cover typical set Tn(PU, δ), δ > 0.

3

M is set as index of permutation, and acceptance region as A∗

n = N i=1{i} × πi( ¯

An)

• disjoint union of sets {i} × πi( ¯

An)

• .

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 7 / 15

Strong Converse Proof Outline (Noiseless channel)

Given: Encoder fn and decision region An ∈ M × Vn such that αn(fn, gn) ≤ ǫ.

1

Blowing-up lemma is used to show existence of message m∗ ∈ M, a set Cn ⊆ f −1

n (m∗) ⊆ Un and decision rule ¯

gn = 1 − ✶ ¯

An

such that given un ∈ Cn, αn(fn, ¯ gn|Un = un)

(n)

− − → 0, βn(fn, ¯ gn|Un = un) ≤ βn(fn, gn) eno(1).

2

Covering lemma is used to guarantee existence of N ≈ enR permutations πi, i ∈ [1 : N], of Cn that cover typical set Tn(PU, δ), δ > 0.

3

M is set as index of permutation, and acceptance region as A∗

n = N i=1{i} × πi( ¯

An)

• disjoint union of sets {i} × πi( ¯

An)

• .

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 7 / 15

Strong Converse Proof Outline (Noiseless channel)

Given: Encoder fn and decision region An ∈ M × Vn such that αn(fn, gn) ≤ ǫ.

1

Blowing-up lemma is used to show existence of message m∗ ∈ M, a set Cn ⊆ f −1

n (m∗) ⊆ Un and decision rule ¯

gn = 1 − ✶ ¯

An

such that given un ∈ Cn, αn(fn, ¯ gn|Un = un)

(n)

− − → 0, βn(fn, ¯ gn|Un = un) ≤ βn(fn, gn) eno(1).

2

Covering lemma is used to guarantee existence of N ≈ enR permutations πi, i ∈ [1 : N], of Cn that cover typical set Tn(PU, δ), δ > 0.

3

M is set as index of permutation, and acceptance region as A∗

n = N i=1{i} × πi( ¯

An)

• disjoint union of sets {i} × πi( ¯

An)

• .

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 7 / 15

Strong Converse: Noisy channel

Noisy channel case: ◮ Mapping from encoder to channel output is stochastic ⇒ Message m∗ as in the noisless channel case does not exist. ⇒ Acceptance region cannot be obtained as disjoint union of set permutations.

Theorem (Strong Converse)

κ(ǫ) = θ(PUV, C) := max {I(V; W) : I(U; W) ≤ C, V − U − W} , ∀ ǫ ∈ (0, 1). Proof uses a combination of Blowing-up lemma and change of measure technique [Tyagi and Watanabe (2020)].

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 8 / 15

Strong Converse: Noisy channel

Noisy channel case: ◮ Mapping from encoder to channel output is stochastic ⇒ Message m∗ as in the noisless channel case does not exist. ⇒ Acceptance region cannot be obtained as disjoint union of set permutations.

Theorem (Strong Converse)

κ(ǫ) = θ(PUV, C) := max {I(V; W) : I(U; W) ≤ C, V − U − W} , ∀ ǫ ∈ (0, 1). Proof uses a combination of Blowing-up lemma and change of measure technique [Tyagi and Watanabe (2020)].

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 8 / 15

Proof (Outline):

Given: A sequence of (fn, gn) pairs specified by PX n|Un and An such that αn(fn, gn) ≤ ǫ.

1

Obtain reliable decoding set using Blowing-up lemma: An :=

• vn∈Vn

A(vn) × {vn}, A(vn) := {yn ∈ Yn : (yn, vn) ∈ An}. Let Bn :=    (un, vn, xn) : PX n|Un(xn|un) > 0, PY n|X n (A(vn)|xn) ≥ 1 − ǫ 2    . Then, P

• (Un, V n, X n) ∈ Bn
• H = 0
• ≥ 1−ǫ

1+ǫ.

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 9 / 15

Proof (Outline):

Given: A sequence of (fn, gn) pairs specified by PX n|Un and An such that αn(fn, gn) ≤ ǫ.

1

Obtain reliable decoding set using Blowing-up lemma: An :=

• vn∈Vn

A(vn) × {vn}, A(vn) := {yn ∈ Yn : (yn, vn) ∈ An}. Let Bn :=    (un, vn, xn) : PX n|Un(xn|un) > 0, PY n|X n (A(vn)|xn) ≥ 1 − ǫ 2    . Then, P

• (Un, V n, X n) ∈ Bn
• H = 0
• ≥ 1−ǫ

1+ǫ.

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 9 / 15

Proof (Outline)

Bn :=

• (un, vn, xn) : PX n|Un(xn|un) > 0, PY n|X n (A(vn)|xn) ≥ 1 − ǫ

2

• Since PY n|X n (A(vn)|xn) ≥ 1−ǫ

2

for (un, vn, xn) ∈ Bn, PY n|X n( ¯ A(vn)|xn) ≥ ǫ′

n (n)

− − → 1, [Blowing-up lemma] ¯ A(vn) := Γln(A(vn)), ln = O( √ n). Decision rule ¯ gn: ¯ An :=

• vn∈Vn

¯ A(vn) × {vn}. Reliable decoding set: Cn = (Bn, ¯ An).

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 10 / 15

Proof (Outline)

Bn :=

• (un, vn, xn) : PX n|Un(xn|un) > 0, PY n|X n (A(vn)|xn) ≥ 1 − ǫ

2

• Since PY n|X n (A(vn)|xn) ≥ 1−ǫ

2

for (un, vn, xn) ∈ Bn, PY n|X n( ¯ A(vn)|xn) ≥ ǫ′

n (n)

− − → 1, [Blowing-up lemma] ¯ A(vn) := Γln(A(vn)), ln = O( √ n). Decision rule ¯ gn: ¯ An :=

• vn∈Vn

¯ A(vn) × {vn}. Reliable decoding set: Cn = (Bn, ¯ An).

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 10 / 15

Proof (Outline)

Bn :=

• (un, vn, xn) : PX n|Un(xn|un) > 0, PY n|X n (A(vn)|xn) ≥ 1 − ǫ

2

• Since PY n|X n (A(vn)|xn) ≥ 1−ǫ

2

for (un, vn, xn) ∈ Bn, PY n|X n( ¯ A(vn)|xn) ≥ ǫ′

n (n)

− − → 1, [Blowing-up lemma] ¯ A(vn) := Γln(A(vn)), ln = O( √ n). Decision rule ¯ gn: ¯ An :=

• vn∈Vn

¯ A(vn) × {vn}. Reliable decoding set: Cn = (Bn, ¯ An).

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 10 / 15

Proof (Outline)

2

Change of measure via construction of a truncated distribution: P˜

Un ˜ V n ˜ X n ˜ Y n(un, vn, xn, yn)

:= PUnV n(un, vn)PX n|Un(xn|un) PUnV nX n(Bn) ✶((un, vn, xn) ∈ Bn) PY n|X n(yn|xn). Following properties hold for P˜

Un ˜ V n ˜ X n ˜ Y n:

(a) ˜ V n − ˜ Un − ˜ X n − ˜ Y n form a Markov chain; (b) D(P˜

Un ˜ V n ˜ X n ˜ Y n||PUnV nX nY n) = 1 PUnVnXn (Bn) ≤ 1+ǫ 1−ǫ;

For HT between P˜

Un ˜ V n and PUn × PV n,

(c) αn (fn, ¯ gn) → 0; (d) βn (fn, ¯ gn) ≤ βn (fn, gn)

• |Y|ne

¯ pln

ln , ln = Θ(√n).

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 11 / 15

Proof (Outline)

2

Change of measure via construction of a truncated distribution: P˜

Un ˜ V n ˜ X n ˜ Y n(un, vn, xn, yn)

:= PUnV n(un, vn)PX n|Un(xn|un) PUnV nX n(Bn) ✶((un, vn, xn) ∈ Bn) PY n|X n(yn|xn). Following properties hold for P˜

Un ˜ V n ˜ X n ˜ Y n:

(a) ˜ V n − ˜ Un − ˜ X n − ˜ Y n form a Markov chain; (b) D(P˜

Un ˜ V n ˜ X n ˜ Y n||PUnV nX nY n) = 1 PUnVnXn (Bn) ≤ 1+ǫ 1−ǫ;

For HT between P˜

Un ˜ V n and PUn × PV n,

(c) αn (fn, ¯ gn) → 0; (d) βn (fn, ¯ gn) ≤ βn (fn, gn)

• |Y|ne

¯ pln

ln , ln = Θ(√n).

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 11 / 15

Proof (Outline)

3

Bounding type II error-exponent via the weak converse: − log (βn(fn, ¯ gn)) ≤ 1 ǫ′

n

• D(P˜

V n ˜ Y n||PV n × PY n) + log 2

• . [ǫ′

n (n)

− − → 1] [αn (fn, ¯ gn) → 0 and log-sum inequality]

4

Adding divergence costs and single-letterizing: − ǫ′

n log (βn (fn, gn))

[µ, ν − Lagrange multipliers] ≤ nRs

µ,ν(PUV, C) + (ν + µ) log

1 + ǫ 1 − ǫ

• + O(

√ n log n), Rs

µ,ν(PUV, C) :=

sup

P˜

U ˜ V ˜ W

∈PUV ˜

W

• I( ˜

V; ˜ W) + µC − µI(˜ U; ˜ W) − (ν + µ)I( ˜ V; ˜ W|˜ U) − (ν + µ)D(P˜

U ˜ V||PUV)

• .

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 12 / 15

Proof (Outline)

3

Bounding type II error-exponent via the weak converse: − log (βn(fn, ¯ gn)) ≤ 1 ǫ′

n

• D(P˜

V n ˜ Y n||PV n × PY n) + log 2

• . [ǫ′

n (n)

− − → 1] [αn (fn, ¯ gn) → 0 and log-sum inequality]

4

Adding divergence costs and single-letterizing: − ǫ′

n log (βn (fn, gn))

[µ, ν − Lagrange multipliers] ≤ nRs

µ,ν(PUV, C) + (ν + µ) log

1 + ǫ 1 − ǫ

• + O(

√ n log n), Rs

µ,ν(PUV, C) :=

sup

P˜

U ˜ V ˜ W

∈PUV ˜

W

• I( ˜

V; ˜ W) + µC − µI(˜ U; ˜ W) − (ν + µ)I( ˜ V; ˜ W|˜ U) − (ν + µ)D(P˜

U ˜ V||PUV)

• .

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 12 / 15

Proof (Outline)

Rs

µ,ν(PUV, C) :=

sup

P˜

U ˜ V ˜ W

∈PUV ˜

W

• I( ˜

V; ˜ W) + µC − µI(˜ U; ˜ W) − (ν + µ)I( ˜ V; ˜ W|˜ U) − (ν + µ)D(P˜

U ˜ V||PUV)

• 5

Relating Rs

µ,ν(PUV, C) to a variational characterization of

θ(PUV, C) for ν = Θ(√n): Rs

µ,ν(PUV, C) ≤ θµ(PUV, C) + Θµ

• n− 1

4 log n

• ,

θµ(PUV, C) := sup

PW|U: V−U−W

I(V; W) + µ(C − I(U; W)).

Lemma (Variational characterization of θ(PUV, C))

θ(PUV, C) = inf

µ>0 θµ(PUV, C).

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 13 / 15

Proof (Outline)

Rs

µ,ν(PUV, C) :=

sup

P˜

U ˜ V ˜ W

∈PUV ˜

W

• I( ˜

V; ˜ W) + µC − µI(˜ U; ˜ W) − (ν + µ)I( ˜ V; ˜ W|˜ U) − (ν + µ)D(P˜

U ˜ V||PUV)

• 5

Relating Rs

µ,ν(PUV, C) to a variational characterization of

θ(PUV, C) for ν = Θ(√n): Rs

µ,ν(PUV, C) ≤ θµ(PUV, C) + Θµ

• n− 1

4 log n

• ,

θµ(PUV, C) := sup

PW|U: V−U−W

I(V; W) + µ(C − I(U; W)).

Lemma (Variational characterization of θ(PUV, C))

θ(PUV, C) = inf

µ>0 θµ(PUV, C).

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 13 / 15

Concluding Remarks and Open Problems

Strong converse holds for testing against independence over a noisy channel. Proof involves a novel combination of Blowing-up lemma and change of measure technique. Open question: Does strong converse hold in the general case of testing between arbitrary probability distributions ?

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 14 / 15

Concluding Remarks and Open Problems

Strong converse holds for testing against independence over a noisy channel. Proof involves a novel combination of Blowing-up lemma and change of measure technique. Open question: Does strong converse hold in the general case of testing between arbitrary probability distributions ?

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 14 / 15

Concluding Remarks and Open Problems

Strong converse holds for testing against independence over a noisy channel. Proof involves a novel combination of Blowing-up lemma and change of measure technique. Open question: Does strong converse hold in the general case of testing between arbitrary probability distributions ?

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 14 / 15

THANK YOU !

SS (Cornell) Strong Converse for TAI over Noisy Channel June 8, 2020 15 / 15