Learning Additive Noise Channels: Generalization Bounds and - - PowerPoint PPT Presentation

Learning Additive Noise Channels: Generalization Bounds and Algorithms

Nir Weinberger

Massachusetts Institute of Technology, MA, USA

IEEE International Symposium on Information Theory June 2020

1/22

In an nutshell

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd

2/22

In an nutshell

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Z ∼ µ, but µ is unknown and non-parametric.

2/22

In an nutshell

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Z ∼ µ, but µ is unknown and non-parametric.

2/22

In an nutshell

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Z ∼ µ, but µ is unknown and non-parametric.

✞ ✝ ☎ ✆

Can we learn to efficiently communicate from (Z1,...,Zn) i.i.d. ∼ µ ?

2/22

In an nutshell

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Z ∼ µ, but µ is unknown and non-parametric.

✞ ✝ ☎ ✆

Can we learn to efficiently communicate from (Z1,...,Zn) i.i.d. ∼ µ ? Generalization bounds for

2/22

In an nutshell

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Z ∼ µ, but µ is unknown and non-parametric.

✞ ✝ ☎ ✆

Can we learn to efficiently communicate from (Z1,...,Zn) i.i.d. ∼ µ ? Generalization bounds for

1

learning under error probability loss

2/22

In an nutshell

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Z ∼ µ, but µ is unknown and non-parametric.

✞ ✝ ☎ ✆

Can we learn to efficiently communicate from (Z1,...,Zn) i.i.d. ∼ µ ? Generalization bounds for

1

learning under error probability loss

Applies to empirical risk minimization (ERM).

2/22

In an nutshell

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Z ∼ µ, but µ is unknown and non-parametric.

✞ ✝ ☎ ✆

Can we learn to efficiently communicate from (Z1,...,Zn) i.i.d. ∼ µ ? Generalization bounds for

1

learning under error probability loss

Applies to empirical risk minimization (ERM).

2

learning under a surrogate error probability loss.

2/22

In an nutshell

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Z ∼ µ, but µ is unknown and non-parametric.

✞ ✝ ☎ ✆

Can we learn to efficiently communicate from (Z1,...,Zn) i.i.d. ∼ µ ? Generalization bounds for

1

learning under error probability loss

Applies to empirical risk minimization (ERM).

2

learning under a surrogate error probability loss.

New alternating optimization algorithm.

2/22

In an nutshell

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Z ∼ µ, but µ is unknown and non-parametric.

✞ ✝ ☎ ✆

Can we learn to efficiently communicate from (Z1,...,Zn) i.i.d. ∼ µ ? Generalization bounds for

1

learning under error probability loss

Applies to empirical risk minimization (ERM).

2

learning under a surrogate error probability loss.

New alternating optimization algorithm.

3

a “codeword-expurgating” Gibbs learning algorithm.

2/22

In an nutshell

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Z ∼ µ, but µ is unknown and non-parametric.

✞ ✝ ☎ ✆

Can we learn to efficiently communicate from (Z1,...,Zn) i.i.d. ∼ µ ? Generalization bounds for

1

learning under error probability loss

Applies to empirical risk minimization (ERM).

2

learning under a surrogate error probability loss.

New alternating optimization algorithm.

3

a “codeword-expurgating” Gibbs learning algorithm.

Caveat: a distilled learning-theoretic framework.

2/22

Motivation

Why?

3/22

Motivation

Why? Justification of learning-based methods:

3/22

Motivation

Why? Justification of learning-based methods:

1 The success of deep neural networks (DNN) [OH17; Gru+17]. 3/22

Motivation

Why? Justification of learning-based methods:

1 The success of deep neural networks (DNN) [OH17; Gru+17]. 2 Avoid channel modeling [Wan+17; OH17; FG17; Shl+19]. 3/22

Motivation

Why? Justification of learning-based methods:

1 The success of deep neural networks (DNN) [OH17; Gru+17]. 2 Avoid channel modeling [Wan+17; OH17; FG17; Shl+19].

Interference, jamming signals, non-linearities [Sch08], finite-resolution quantization.

3/22

Motivation

Why? Justification of learning-based methods:

1 The success of deep neural networks (DNN) [OH17; Gru+17]. 2 Avoid channel modeling [Wan+17; OH17; FG17; Shl+19].

Interference, jamming signals, non-linearities [Sch08], finite-resolution quantization. High-dimensional parameter, e.g. massive MIMO.

3/22

Motivation

Why? Justification of learning-based methods:

1 The success of deep neural networks (DNN) [OH17; Gru+17]. 2 Avoid channel modeling [Wan+17; OH17; FG17; Shl+19].

Interference, jamming signals, non-linearities [Sch08], finite-resolution quantization. High-dimensional parameter, e.g. massive MIMO.

3 Existing theory on learning-based quantizer design [LLZ94;

LLZ97; BLL98; Lin02].

3/22

Motivation

Why? Justification of learning-based methods:

1 The success of deep neural networks (DNN) [OH17; Gru+17]. 2 Avoid channel modeling [Wan+17; OH17; FG17; Shl+19].

Interference, jamming signals, non-linearities [Sch08], finite-resolution quantization. High-dimensional parameter, e.g. massive MIMO.

3 Existing theory on learning-based quantizer design [LLZ94;

LLZ97; BLL98; Lin02].

4 Exploit efficient optimization methods, e.g., for the design of

low latency codes [Kim+18; Jia+19].

3/22

Outline

1

Learning to Minimize Error Probability

2

Learning to Minimize a Surrogate to the Error Probability

3

Learning by Codebook Expurgation

4/22

Model

Channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd

5/22

Model

Channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Encoder: A codebook C = {xj}j∈[m] ⊂ C ⊆ (Rd)m.

5/22

Model

Channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Encoder: A codebook C = {xj}j∈[m] ⊂ C ⊆ (Rd)m. Decoder: minimal (Mahalanobis) distance decoder ˆ j(y) ∈ argmin

j∈[m]

x−yS w.r.t. inverse covariance matrix S ∈ S ⊆ Sd

+.

5/22

Expected and empirical error probability

Expected average error probability: pµ(C,S) := 1 m

m

• j=1

pµ(C,S | j), with pµ(C,S | j) := Eµ

• ✶
• min

j′∈[m],j′=j xj +Z −xj′S < ZS

• .

✶

6/22

Expected and empirical error probability

Expected average error probability: pµ(C,S) := 1 m

m

• j=1

pµ(C,S | j), with pµ(C,S | j) := Eµ

• ✶
• min

j′∈[m],j′=j xj +Z −xj′S < ZS

• .

Ultimate goal: find argminC,S pµ(C,S). ✶

6/22

Expected and empirical error probability

Expected average error probability: pµ(C,S) := 1 m

m

• j=1

pµ(C,S | j), with pµ(C,S | j) := Eµ

• ✶
• min

j′∈[m],j′=j xj +Z −xj′S < ZS

• .

Ultimate goal: find argminC,S pµ(C,S). Empirical average error probability: replace Eµ[ℓ(Z)] → EZ[ℓ(Z)] := 1 n

n

• i=1

ℓ(Zi) so that pZ(C,S) := 1 m

m

• j=1

EZ

• ✶
• min

j′∈[m]\j xj +Z −xj′S < ZS

• .

6/22

Uniform error bound and ERM

Theorem Assume that n ≥ d+1. With probability of at least 1−δ sup

C⊂(Rd)m, S∈Sd

+

|pµ(C,S)−pZ(C,S)| ≤ 4m

• 2(d+1)log
• en

d+1

• n

+

• 2log(2/δ)

n .

7/22

Uniform error bound and ERM

Theorem Assume that n ≥ d+1. With probability of at least 1−δ sup

C⊂(Rd)m, S∈Sd

+

|pµ(C,S)−pZ(C,S)| ≤ 4m

• 2(d+1)log
• en

d+1

• n

+

• 2log(2/δ)

n . Holds for the output (CZ,SZ) of any learning algorithm.

7/22

Uniform error bound and ERM

Theorem Assume that n ≥ d+1. With probability of at least 1−δ sup

C⊂(Rd)m, S∈Sd

+

|pµ(C,S)−pZ(C,S)| ≤ 4m

• 2(d+1)log
• en

d+1

• n

+

• 2log(2/δ)

n . Holds for the output (CZ,SZ) of any learning algorithm. Specifically, for ERM (CZ,SZ)ERM ∈ argminpZ(C,S), n = ˜ O(m2d+log(1/δ)

ǫ2

) samples guarantees pµ(CZ,SZ)ERM) ≤ inf

C,S pµ(C,S)+ǫ.

7/22

Uniform error bound and ERM - cont.

Open questions: The term ˜ O(log(1/δ)

ǫ2

) can be shown to be minimax tight.

8/22

Uniform error bound and ERM - cont.

Open questions: The term ˜ O(log(1/δ)

ǫ2

) can be shown to be minimax tight.

Is the dependence on ˜ O( m2d

ǫ2 ) is optimal?

8/22

Uniform error bound and ERM - cont.

Open questions: The term ˜ O(log(1/δ)

ǫ2

) can be shown to be minimax tight.

Is the dependence on ˜ O( m2d

ǫ2 ) is optimal?

Typically, m = 2dR ≫ 1, but codebook has structure.

8/22

Uniform error bound and ERM - cont.

Open questions: The term ˜ O(log(1/δ)

ǫ2

) can be shown to be minimax tight.

Is the dependence on ˜ O( m2d

ǫ2 ) is optimal?

Typically, m = 2dR ≫ 1, but codebook has structure.

What is the right complexity measure for learnability?

8/22

Outline

1

Learning to Minimize Error Probability

2

Learning to Minimize a Surrogate to the Error Probability

3

Learning by Codebook Expurgation

9/22

A Surrogate to the Error Probability

pZ(C,S) is difficult to optimize – discontinuous in (C,S). ✶ ✶

10/22

A Surrogate to the Error Probability

pZ(C,S) is difficult to optimize – discontinuous in (C,S). A hinge loss upper bound ✶(t < 0) ≤ (1−t)∨0. ✶

10/22

A Surrogate to the Error Probability

pZ(C,S) is difficult to optimize – discontinuous in (C,S). A hinge loss upper bound ✶(t < 0) ≤ (1−t)∨0. Bound the error event indicator ✶

• min

j′∈[m]\{j}

• xj −xj′2

S −2(xj −xj′)T Sz

• < 0
• ≤
• 1−

min

j′∈[m]\{j}

• xj −xj′2

S −2(xj −xj′)T Sz

• ∨0.

10/22

A Surrogate to the Error Probability

pZ(C,S) is difficult to optimize – discontinuous in (C,S). A hinge loss upper bound ✶(t < 0) ≤ (1−t)∨0. Bound the error event indicator ✶

• min

j′∈[m]\{j}

• xj −xj′2

S −2(xj −xj′)T Sz

• < 0
• ≤
• 1−

min

j′∈[m]\{j}

• xj −xj′2

S −2(xj −xj′)T Sz

• ∨0.

The corresponding expected and empirical losses are ¯ pµ(C,S) ≥ pµ(C,S), ¯ pZ(C,S) ≥ pZ(C,S).

10/22

Uniform error bound

Theorem Under some boundedness assumptions on C,S, and the support of µ with probability of at least 1−δ sup

C⊂C, S∈S

|¯ pµ(C,S)− ¯ pZ(C,S)|

• (d+1)mlog(m)

n +

• log(1/δ)

n .

11/22

Uniform error bound

Theorem Under some boundedness assumptions on C,S, and the support of µ with probability of at least 1−δ sup

C⊂C, S∈S

|¯ pµ(C,S)− ¯ pZ(C,S)|

• (d+1)mlog(m)

n +

• log(1/δ)

n . Sample complexity is n = ˜ O(dm+log(1/δ)

ǫ2

).

11/22

Uniform error bound

Theorem Under some boundedness assumptions on C,S, and the support of µ with probability of at least 1−δ sup

C⊂C, S∈S

|¯ pµ(C,S)− ¯ pZ(C,S)|

• (d+1)mlog(m)

n +

• log(1/δ)

n . Sample complexity is n = ˜ O(dm+log(1/δ)

ǫ2

).

Improvement from quadratic to linear dependency on m.

11/22

Uniform error bound

Theorem Under some boundedness assumptions on C,S, and the support of µ with probability of at least 1−δ sup

C⊂C, S∈S

|¯ pµ(C,S)− ¯ pZ(C,S)|

• (d+1)mlog(m)

n +

• log(1/δ)

n . Sample complexity is n = ˜ O(dm+log(1/δ)

ǫ2

).

Improvement from quadratic to linear dependency on m.

Boundedness assumptions are common for analytical simplicity.

11/22

Alternating optimization algorithm

Problem: ¯ pZ(C,S) is non-convex in (C,S).

12/22

Alternating optimization algorithm

Problem: ¯ pZ(C,S) is non-convex in (C,S). Idea: add auxiliary variables.

12/22

Alternating optimization algorithm

Problem: ¯ pZ(C,S) is non-convex in (C,S). Idea: add auxiliary variables. Let {αj,j′}j′∈[m]\j be such that αj,j′ ∈ [0,1], αj,j = 0 and

• j′∈[m]\j αj,j′ ≤ 1.

12/22

Alternating optimization algorithm

Problem: ¯ pZ(C,S) is non-convex in (C,S). Idea: add auxiliary variables. Let {αj,j′}j′∈[m]\j be such that αj,j′ ∈ [0,1], αj,j = 0 and

• j′∈[m]\j αj,j′ ≤ 1.

For a given z ∈ Rd, and j ∈ [m]

• 1−

min

j′∈[m]\{j}

• xj −xj′2

S −2(xj −xj′)T Sz

• ∨0

= max

{αj,j′}j′∈[m]\j

• j′∈[m]\j

αj,j′

• 1−
• xj −xj′2

S −2(xj −xj′)T Szi

• .

12/22

Alternating optimization algorithm

Problem: ¯ pZ(C,S) is non-convex in (C,S). Idea: add auxiliary variables. Let {αj,j′}j′∈[m]\j be such that αj,j′ ∈ [0,1], αj,j = 0 and

• j′∈[m]\j αj,j′ ≤ 1.

For a given z ∈ Rd, and j ∈ [m]

• 1−

min

j′∈[m]\{j}

• xj −xj′2

S −2(xj −xj′)T Sz

• ∨0

= max

{αj,j′}j′∈[m]\j

• j′∈[m]\j

αj,j′

• 1−
• xj −xj′2

S −2(xj −xj′)T Szi

• .

Given (C,S), the variables {αj,j′}j′∈[m] describe the nearest neighbors of xj w.r.t. to z.

12/22

Alternating optimization algorithm - cont.

For Z, extend the loss function ¯ pZ(C,S) = max

A

¯ pZ(C,S,A) where A := {α(i)

j,j′}j∈[m],j′∈[m]\j,i∈[n].

13/22

Alternating optimization algorithm - cont.

For Z, extend the loss function ¯ pZ(C,S) = max

A

¯ pZ(C,S,A) where A := {α(i)

j,j′}j∈[m],j′∈[m]\j,i∈[n].

Algorithm: fix (C(0),S(0)), and for t ≥ 1 alternate between:

13/22

Alternating optimization algorithm - cont.

For Z, extend the loss function ¯ pZ(C,S) = max

A

¯ pZ(C,S,A) where A := {α(i)

j,j′}j∈[m],j′∈[m]\j,i∈[n].

Algorithm: fix (C(0),S(0)), and for t ≥ 1 alternate between:

1

Optimization over A A(t) ∈ argmax

A

¯ pZ(C(t−1),S(t−1),A).

13/22

Alternating optimization algorithm - cont.

For Z, extend the loss function ¯ pZ(C,S) = max

A

¯ pZ(C,S,A) where A := {α(i)

j,j′}j∈[m],j′∈[m]\j,i∈[n].

Algorithm: fix (C(0),S(0)), and for t ≥ 1 alternate between:

1

Optimization over A A(t) ∈ argmax

A

¯ pZ(C(t−1),S(t−1),A).

2

Optimization over (C,S) (C(t),S(t)) ∈ argmin

C,S

¯ pZ(C,S,A(t)).

13/22

Alternating optimization algorithm - cont.

For Z, extend the loss function ¯ pZ(C,S) = max

A

¯ pZ(C,S,A) where A := {α(i)

j,j′}j∈[m],j′∈[m]\j,i∈[n].

Algorithm: fix (C(0),S(0)), and for t ≥ 1 alternate between:

1

Optimization over A A(t) ∈ argmax

A

¯ pZ(C(t−1),S(t−1),A).

2

Optimization over (C,S) (C(t),S(t)) ∈ argmin

C,S

¯ pZ(C,S,A(t)).

A stochastic gradient descent version of this idea appears in the paper.

13/22

Alternating optimization algorithm - cont.

For Z, extend the loss function ¯ pZ(C,S) = max

A

¯ pZ(C,S,A) where A := {α(i)

j,j′}j∈[m],j′∈[m]\j,i∈[n].

Algorithm: fix (C(0),S(0)), and for t ≥ 1 alternate between:

1

Optimization over A A(t) ∈ argmax

A

¯ pZ(C(t−1),S(t−1),A).

2

Optimization over (C,S) (C(t),S(t)) ∈ argmin

C,S

¯ pZ(C,S,A(t)).

A stochastic gradient descent version of this idea appears in the paper. Open problems:

13/22

Alternating optimization algorithm - cont.

For Z, extend the loss function ¯ pZ(C,S) = max

A

¯ pZ(C,S,A) where A := {α(i)

j,j′}j∈[m],j′∈[m]\j,i∈[n].

Algorithm: fix (C(0),S(0)), and for t ≥ 1 alternate between:

1

Optimization over A A(t) ∈ argmax

A

¯ pZ(C(t−1),S(t−1),A).

2

Optimization over (C,S) (C(t),S(t)) ∈ argmin

C,S

¯ pZ(C,S,A(t)).

A stochastic gradient descent version of this idea appears in the paper. Open problems:

Finite number of iterations and samples convergence analysis.

13/22

Alternating optimization algorithm - cont.

For Z, extend the loss function ¯ pZ(C,S) = max

A

¯ pZ(C,S,A) where A := {α(i)

j,j′}j∈[m],j′∈[m]\j,i∈[n].

Algorithm: fix (C(0),S(0)), and for t ≥ 1 alternate between:

1

Optimization over A A(t) ∈ argmax

A

¯ pZ(C(t−1),S(t−1),A).

2

Optimization over (C,S) (C(t),S(t)) ∈ argmin

C,S

¯ pZ(C,S,A(t)).

A stochastic gradient descent version of this idea appears in the paper. Open problems:

Finite number of iterations and samples convergence analysis. Tightened generalization bounds based on the algorithm.

13/22

Outline

1

Learning to Minimize Error Probability

2

Learning to Minimize a Surrogate to the Error Probability

3

Learning by Codebook Expurgation

14/22

The codebook expurgation problem

Assume a standard minimal distance decoder S = Id.

15/22

The codebook expurgation problem

Assume a standard minimal distance decoder S = Id. Let C0 be a super-codebook of m0 > m codewords.

15/22

The codebook expurgation problem

Assume a standard minimal distance decoder S = Id. Let C0 be a super-codebook of m0 > m codewords. An expurgated codebook of m codewords:

15/22

The codebook expurgation problem

Assume a standard minimal distance decoder S = Id. Let C0 be a super-codebook of m0 > m codewords. An expurgated codebook of m codewords:

Population version: C∗,µ = argmin

C={x1,...,xm}⊂C0

pµ(C)

15/22

The codebook expurgation problem

Assume a standard minimal distance decoder S = Id. Let C0 be a super-codebook of m0 > m codewords. An expurgated codebook of m codewords:

Population version: C∗,µ = argmin

C={x1,...,xm}⊂C0

pµ(C) Empirical version: C∗,Z = argmin

C={x1,...,xm}⊂C0

pZ(C).

15/22

The codebook expurgation problem

Assume a standard minimal distance decoder S = Id. Let C0 be a super-codebook of m0 > m codewords. An expurgated codebook of m codewords:

Population version: C∗,µ = argmin

C={x1,...,xm}⊂C0

pµ(C) Empirical version: C∗,Z = argmin

C={x1,...,xm}⊂C0

pZ(C).

A complex task: removing xj should take into account both errors xj → xj′ and xj′ → xj.

15/22

Gibbs selection algorithm

A relaxation: remove codewords step-by-step, k at each step.

16/22

Gibbs selection algorithm

A relaxation: remove codewords step-by-step, k at each step. Algorithm:

16/22

Gibbs selection algorithm

A relaxation: remove codewords step-by-step, k at each step. Algorithm:

1

Initialization: a codebook C0, |C0| = m0, β > 0 and Q ∈ P(Rd).

16/22

Gibbs selection algorithm

A relaxation: remove codewords step-by-step, k at each step. Algorithm:

1

Initialization: a codebook C0, |C0| = m0, β > 0 and Q ∈ P(Rd).

2

For t = 1,...T := m0−m

k

P

• Ct+1 = Ct\{xj}j∈{j1,...jk} | Z,Ct
• ∝ Q(Ct+1)·exp[−β ·pZ(Ct+1)].

16/22

Gibbs selection algorithm

A relaxation: remove codewords step-by-step, k at each step. Algorithm:

1

Initialization: a codebook C0, |C0| = m0, β > 0 and Q ∈ P(Rd).

2

For t = 1,...T := m0−m

k

P

• Ct+1 = Ct\{xj}j∈{j1,...jk} | Z,Ct
• ∝ Q(Ct+1)·exp[−β ·pZ(Ct+1)].

Finite inverse temperature β

16/22

Gibbs selection algorithm

A relaxation: remove codewords step-by-step, k at each step. Algorithm:

1

Initialization: a codebook C0, |C0| = m0, β > 0 and Q ∈ P(Rd).

2

For t = 1,...T := m0−m

k

P

• Ct+1 = Ct\{xj}j∈{j1,...jk} | Z,Ct
• ∝ Q(Ct+1)·exp[−β ·pZ(Ct+1)].

Finite inverse temperature β

β = 0: random expurgation according to Q.

16/22

Gibbs selection algorithm

A relaxation: remove codewords step-by-step, k at each step. Algorithm:

1

Initialization: a codebook C0, |C0| = m0, β > 0 and Q ∈ P(Rd).

2

For t = 1,...T := m0−m

k

P

• Ct+1 = Ct\{xj}j∈{j1,...jk} | Z,Ct
• ∝ Q(Ct+1)·exp[−β ·pZ(Ct+1)].

Finite inverse temperature β

β = 0: random expurgation according to Q. β → ∞: may not generalize well to µ.

16/22

Gibbs algorithm error types

Sequence generated by population Gibbs algorithm Cµ = (C0,Cµ,1,...,Cµ,T )

17/22

Gibbs algorithm error types

Sequence generated by population Gibbs algorithm Cµ = (C0,Cµ,1,...,Cµ,T ) Sequence generated by empirical Gibbs algorithm CZ = (C0,CZ,1,...,CZ,T )

17/22

Gibbs algorithm error types

Sequence generated by population Gibbs algorithm Cµ = (C0,Cµ,1,...,Cµ,T ) Sequence generated by empirical Gibbs algorithm CZ = (C0,CZ,1,...,CZ,T ) Excess error: pµ(CZ,T )−pµ(C∗,µ) = pµ(CZ,T )−pµ(Cµ,T )

• empirical error

+pµ(Cµ,T )−pµ(C∗,µ)

• approximation error

17/22

Gibbs algorithm error types

Sequence generated by population Gibbs algorithm Cµ = (C0,Cµ,1,...,Cµ,T ) Sequence generated by empirical Gibbs algorithm CZ = (C0,CZ,1,...,CZ,T ) Excess error: pµ(CZ,T )−pµ(C∗,µ) = pµ(CZ,T )−pµ(Cµ,T )

• empirical error

+pµ(Cµ,T )−pµ(C∗,µ)

• approximation error

Generalization error: pµ(CZ,T )−pZ(CZ,T ).

17/22

Theoretical guarantees

Theorem Under mild assumptions

18/22

Theoretical guarantees

Theorem Under mild assumptions Average empirical error E[pµ(CZ,T )−pµ(Cµ,T )] ≤ 3

• Tβ2 (logn+m0 logm0 −logk)

n

18/22

Theoretical guarantees

Theorem Under mild assumptions Average empirical error E[pµ(CZ,T )−pµ(Cµ,T )] ≤ 3

• Tβ2 (logn+m0 logm0 −logk)

n Average generalization error E[pµ(CZ,T )−pZ(CZ,T )] ≤

• T

β

n ∧ β2 4n2

• .

18/22

Theoretical guarantees

Theorem Under mild assumptions Average empirical error E[pµ(CZ,T )−pµ(Cµ,T )] ≤ 3

• Tβ2 (logn+m0 logm0 −logk)

n Average generalization error E[pµ(CZ,T )−pZ(CZ,T )] ≤

• T

β

n ∧ β2 4n2

• .

Reducing β improves empirical and generalization error.

18/22

Theoretical guarantees

Theorem Under mild assumptions Average empirical error E[pµ(CZ,T )−pµ(Cµ,T )] ≤ 3

• Tβ2 (logn+m0 logm0 −logk)

n Average generalization error E[pµ(CZ,T )−pZ(CZ,T )] ≤

• T

β

n ∧ β2 4n2

• .

Reducing β improves empirical and generalization error. Main open problem: Characterizing the approximation error and optimizing β.

18/22

Summary

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd

19/22

Summary

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Generalization bounds for

19/22

Summary

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Generalization bounds for

1

learning under error probability loss.

19/22

Summary

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Generalization bounds for

1

learning under error probability loss.

Applies to empirical risk minimization (ERM).

19/22

Summary

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Generalization bounds for

1

learning under error probability loss.

Applies to empirical risk minimization (ERM).

2

learning under a surrogate error probability loss.

19/22

Summary

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Generalization bounds for

1

learning under error probability loss.

Applies to empirical risk minimization (ERM).

2

learning under a surrogate error probability loss.

New alternating optimization algorithm.

19/22

Summary

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Generalization bounds for

1

learning under error probability loss.

Applies to empirical risk minimization (ERM).

2

learning under a surrogate error probability loss.

New alternating optimization algorithm.

3

a “codeword-expurgating” Gibbs learning algorithm.

19/22

Summary

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Generalization bounds for

1

learning under error probability loss.

Applies to empirical risk minimization (ERM).

2

learning under a surrogate error probability loss.

New alternating optimization algorithm.

3

a “codeword-expurgating” Gibbs learning algorithm.

19/22

Summary

An additive noise channel Y

• utput

= X

• input

+ Z

• noise

, Z ⊥ ⊥ X ∈ Rd Generalization bounds for

1

learning under error probability loss.

Applies to empirical risk minimization (ERM).

2

learning under a surrogate error probability loss.

New alternating optimization algorithm.

3

a “codeword-expurgating” Gibbs learning algorithm.

Questions? Comments? nir.wein@gmail.com

19/22

References I

Bartlett, Peter L., Tamás Linder, and Gábor Lugosi (1998). “The minimax distortion redundancy in empirical quantizer design”. In: IEEE Transactions on Information theory 44.5, pp. 1802–1813. Farsad, Nariman and Andrea Goldsmith (2017). “Detection algorithms for communication systems using deep learning”. In: arXiv preprint arXiv:1705.08044. Gruber, Tobias et al. (2017). “On deep learning-based channel decoding”. In: 2017 51st Annual Conference on Information Sciences and Systems (CISS). IEEE, pp. 1–6. Jiang, Yihan et al. (2019). “Learn codes: Inventing low-latency codes via recurrent neural networks”. In: ICC 2019-2019 IEEE International Conference on Communications (ICC). IEEE,

• pp. 1–7.

Kim, Hyeji et al. (2018). “Communication algorithms via deep learning”. In: arXiv preprint arXiv:1805.09317.

20/22

References II

Linder, Tamás (2002). “Learning-theoretic methods in vector quantization”. In: Principles of nonparametric learning. Springer,

• pp. 163–210.

Linder, Tamás, Gábor Lugosi, and Kenneth Zeger (1994). “Rates

• f convergence in the source coding theorem, in empirical

quantizer design, and in universal lossy source coding”. In: IEEE Transactions on Information Theory 40.6, pp. 1728–1740. – (1997). “Empirical quantizer design in the presence of source noise or channel noise”. In: IEEE Transactions on Information Theory 43.2, pp. 612–623. O’Shea, Timothy and Jakob Hoydis (2017). “An introduction to deep learning for the physical layer”. In: IEEE Transactions on Cognitive Communications and Networking 3.4, pp. 563–575. Schenk, Tim (2008). RF imperfections in high-rate wireless systems: Impact and digital compensation. Springer Science & Business Media.

21/22

References III

Shlezinger, Nir et al. (2019). “ViterbiNet: Symbol detection using a deep learning based Viterbi algorithm”. In: 2019 IEEE 20th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC). IEEE, pp. 1–5. Wang, Tianqi et al. (2017). “Deep learning for wireless physical layer: Opportunities and challenges”. In: China Communications 14.11, pp. 92–111.

22/22