Linear Models are Most Favorable among Generalized Linear Models - - PowerPoint PPT Presentation

Linear Models are Most Favorable among Generalized Linear Models

Kuan-Yun Lee and Thomas A. Courtade

Electrical Engineering and Computer Sciences University of California, Berkeley

ISIT 2020

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 1 / 20

Overview

1

Introduction and Main Results

2

Keypoints of Proof

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 2 / 20

Overview

1

Introduction and Main Results

2

Keypoints of Proof

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 3 / 20

Introduction

Given X := (X1, . . . , Xn) ∼ f (·; θ) Linear regression: X = Mθ + Z Phase retrieval: Xi = mi, θ2 + Zi Group testing: Xi = δ (mi, θ) Matrix retrieval: Xi = Tr

• M⊤

i θ

• when θ is a matrix

. . . Many other settings with sparsity, structural assumptions on M, etc.

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 4 / 20

Introduction

Given X := (X1, . . . , Xn) ∼ f (·; θ) Linear regression: X = Mθ + Z Phase retrieval: Xi = mi, θ2 + Zi Group testing: Xi = δ (mi, θ) Matrix retrieval: Xi = Tr

• M⊤

i θ

• when θ is a matrix

. . . Many other settings with sparsity, structural assumptions on M, etc.

Key Question

How well can we estimate θ from observations X ∼ f (·; θ)?

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 4 / 20

Introduction

Consider the classical linear model X = Mθ + Z under constraint θ ∈ Θ

Fundamental Question

Given a loss function L(·, ·), what is inf ˆ

θ supθ∈Θ E L(θ, ˆ

θ)? ✶

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 5 / 20

Introduction

Consider the classical linear model X = Mθ + Z under constraint θ ∈ Θ

Fundamental Question

Given a loss function L(·, ·), what is inf ˆ

θ supθ∈Θ E L(θ, ˆ

θ)? Loss functions L(θ, ˆ θ): θ − ˆ θ2

2 (estimation error)

Mθ − M ˆ θ2

2 (prediction error)

✶(supp(θ) = supp(ˆ θ)) Constraints on Θ: Θ is Lp ball Θ is a matrix space with rank constraints

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 5 / 20

Introduction

In this talk, we will focus on estimation error L(θ, ˆ θ) := θ − ˆ θ2

2

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 6 / 20

Introduction

In this talk, we will focus on estimation error L(θ, ˆ θ) := θ − ˆ θ2

2

Consider X = Mθ + Z ∈ Rn with fixed design matrix M ∈ Rn×d, Θ = Rd, Z ∼ N(0, σ2 · In). Suppose M has full column rank, then, ˆ θMLE := (M⊤M)−1M⊤X achieves the minimax error inf ˆ

θ supθ∈Rd Eθ − ˆ

θ2

2, and

Eθ − ˆ θMLE2

2 = E(M⊤M)−1M⊤Z2 2 = σ2 · Tr((M⊤M)−1).

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 6 / 20

Introduction

In this talk, we will focus on estimation error L(θ, ˆ θ) := θ − ˆ θ2

2

Consider X = Mθ + Z ∈ Rn with fixed design matrix M ∈ Rn×d, Θ = Rd, Z ∼ N(0, σ2 · In). Suppose M has full column rank, then, ˆ θMLE := (M⊤M)−1M⊤X achieves the minimax error inf ˆ

θ supθ∈Rd Eθ − ˆ

θ2

2, and

Eθ − ˆ θMLE2

2 = E(M⊤M)−1M⊤Z2 2 = σ2 · Tr((M⊤M)−1).

Follow up question

Can we generalize this? The Gaussian distribution falls into the exponential family The linear model falls into the family of generalized linear models

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 6 / 20

Exponential Family

Density of X ∈ R given natural parameter η ∈ R f (x; η) = h(x) exp ηx − Φ(η) s(σ)

• h : X ⊆ R → [0, ∞) (the base measure)

Φ : R → R (the cumulant function) s(σ) > 0: scale parameter Examples: Bernoulli

• 1

1+e−η

• : h(x) = 1, Φ(t) = log(1 + et) and s(σ) = 1

Gaussian(η, 1): h(x) =

1 √ 2πe−x2/2, Φ(t) = t2/2 and s(σ) = 1

Exponential(η): h(x) = 1, Φ(t) = − log t and s(σ) = 1 Poisson(eη): h(x) = 1/x!, Φ(t) = et and s(σ) = 1

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 7 / 20

Generalized Linear Models

Density of X ∈ Rn given parameter Mθ ∈ Rn f (x; M, θ) =

n

• i=1

h(xi) exp mi, θxi − Φ(mi, θ) s(σ)

• h : X ⊆ R → [0, ∞) (the base measure)

Φ : R → R (the cumulant function) s(σ) > 0: scale parameter Examples: Bernoulli

• 1

1+e−mi ,θ

• : h(x) = 1, Φ(t) = log(1 + et) and s(σ) = 1

Gaussian(mi, θ, 1): h(x) =

1 √ 2πe−x2/2, Φ(t) = t2/2 and s(σ) = 1

Exponential(mi, θ): h(x) = 1, Φ(t) = − log t and s(σ) = 1 Poisson(emi,θ): h(x) = 1/x!, Φ(t) = et and s(σ) = 1

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 8 / 20

Generalized Linear Models

Density of X ∈ Rn given parameter Mθ ∈ Rn f (x; M, θ) =

n

• i=1

h(xi) exp mi, θxi − Φ(mi, θ) s(σ)

• We make one common assumption: Φ′′ ≤ L.

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 9 / 20

Generalized Linear Models

Density of X ∈ Rn given parameter Mθ ∈ Rn f (x; M, θ) =

n

• i=1

h(xi) exp mi, θxi − Φ(mi, θ) s(σ)

• We make one common assumption: Φ′′ ≤ L.

The variance of Xi is s(σ) · Φ′′(mi, θ)

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 9 / 20

Generalized Linear Models

Density of X ∈ Rn given parameter Mθ ∈ Rn f (x; M, θ) =

n

• i=1

h(xi) exp mi, θxi − Φ(mi, θ) s(σ)

• We make one common assumption: Φ′′ ≤ L.

The variance of Xi is s(σ) · Φ′′(mi, θ) In the Gaussian case, Φ′′(t) = 1 In the Bernoulli case, Φ′′(t) =

et (1+et)2 ≤ 1

In the Poisson case, Φ′′(t) = et In the Exponential case, Φ′′(t) = 1

t2

Corresponds to structural assumptions on M and Θ

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 9 / 20

Generalized Linear Models

Theorem

Given observations X ∈ Rn generated from the GLM with fixed M ∈ Rn×d, f (x; M, θ) =

n

• i=1

h(xi) exp mi, θxi − Φ(mi, θ) s(σ)

• ,

with Θ := B2

d(1) := {θ : θ ∈ Rd, θ2 2 ≤ 1} and Φ′′ ≤ L,

inf

ˆ θ

sup

θ∈Θ

Eθ − ˆ θ2

2 min

• 1, s(σ)

L Tr((M⊤M)−1)

• K.-Y. Lee and T. A. Courtade (Berkeley)

New Minimax Bound for the GLM ISIT 2020 10 / 20

Generalized Linear Models

Theorem

Given observations X ∈ Rn generated from the GLM with fixed M ∈ Rn×d, f (x; M, θ) =

n

• i=1

h(xi) exp mi, θxi − Φ(mi, θ) s(σ)

• ,

with Θ := B2

d(1) := {θ : θ ∈ Rd, θ2 2 ≤ 1} and Φ′′ ≤ L,

inf

ˆ θ

sup

θ∈Θ

Eθ − ˆ θ2

2 min

• 1, s(σ)

L Tr((M⊤M)−1)

• When M⊤M has a zero eigenvalue, adopt Tr((M⊤M)−1) := +∞

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 10 / 20

Generalized Linear Models

Theorem

Given observations X ∈ Rn generated from the GLM with fixed M ∈ Rn×d, f (x; M, θ) =

n

• i=1

h(xi) exp mi, θxi − Φ(mi, θ) s(σ)

• ,

with Θ := B2

d(1) := {θ : θ ∈ Rd, θ2 2 ≤ 1} and Φ′′ ≤ L,

inf

ˆ θ

sup

θ∈Θ

Eθ − ˆ θ2

2 min

• 1, s(σ)

L Tr((M⊤M)−1)

• When M⊤M has a zero eigenvalue, adopt Tr((M⊤M)−1) := +∞

Gaussian linear model with X = LMθ + Z matches with equality and is extremal in this family of GLMs

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 10 / 20

Overview

1

Introduction and Main Results

2

Keypoints of Proof

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 11 / 20

Upper Bound on Mutual Information

Consider X ∼ f (·; θ). The Fisher information IX(θ) is defined as IX(θ) := EX |∇θf (X; θ)|2 f 2(X; θ)

• K.-Y. Lee and T. A. Courtade (Berkeley)

New Minimax Bound for the GLM ISIT 2020 12 / 20

Upper Bound on Mutual Information

Consider X ∼ f (·; θ). The Fisher information IX(θ) is defined as IX(θ) := EX |∇θf (X; θ)|2 f 2(X; θ)

• Regularity assumption:
• X ∇θf (x; θ)dλ(x) = 0 for almost every θ and

θ → f (x; θ) is (weakly) differentiable for λ-a.e. x.

Theorem (Aras, Lee, Pananjady, Courtade, 2019)

Let θ ∼ π, where π is log-concave on Rd, and let X ∼ f (·; θ). If the regularity condition is satisfied, I(θ; X) ≤ d · φ Tr(Cov(θ)) · Eθ IX(θ) d2

• where φ(x) :=

√x if 0 ≤ x < 1 1 + 1

2 log x

if x ≥ 1.

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 12 / 20

Upper Bound on Mutual Information

I(θ; X) ≤ d · φ Tr(Cov(θ)) · Eθ IX(θ) d2

• K.-Y. Lee and T. A. Courtade (Berkeley)

New Minimax Bound for the GLM ISIT 2020 13 / 20

Upper Bound on Mutual Information

I(θ; X) ≤ d · φ Tr(Cov(θ)) · Eθ IX(θ) d2

• There are few upper bounds on mutual information

bounding via channel capacity loses information of the prior

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 13 / 20

Upper Bound on Mutual Information

I(θ; X) ≤ d · φ Tr(Cov(θ)) · Eθ IX(θ) d2

• There are few upper bounds on mutual information

bounding via channel capacity loses information of the prior

No longer depends on J (π) (Fisher information of the prior). For π with density ψ(·) : Rd − → R w.r.t. Lebesgue measure, J (π) := Eθ |∇ ψ(θ)|2 ψ2(θ)

• K.-Y. Lee and T. A. Courtade (Berkeley)

New Minimax Bound for the GLM ISIT 2020 13 / 20

Upper Bound on Mutual Information

I(θ; X) ≤ d · φ Tr(Cov(θ)) · Eθ IX(θ) d2

• There are few upper bounds on mutual information

bounding via channel capacity loses information of the prior

No longer depends on J (π) (Fisher information of the prior). For π with density ψ(·) : Rd − → R w.r.t. Lebesgue measure, J (π) := Eθ |∇ ψ(θ)|2 ψ2(θ)

• Example: the van Trees inequality for θ ∈ R:

E(θ − ˆ θ)2 ≥ 1 Eθ IX(θ) + J (π) does not work for π with J (π) = +∞ (ex. uniform on convex set)

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 13 / 20

Key Ideas of Proof

Assume M⊤M is diagonal without loss of generality by rotational invariance of the L2 ball

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 14 / 20

Key Ideas of Proof

Assume M⊤M is diagonal without loss of generality by rotational invariance of the L2 ball Lower bound minimax risk by Bayes risk: inf

ˆ θ

sup

θ∈Bd

2(R)

Eθ − ˆ θ2

2 ≥ inf ˆ θ

Eθ∼πθ − ˆ θ2

2

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 14 / 20

Key Ideas of Proof

Assume M⊤M is diagonal without loss of generality by rotational invariance of the L2 ball Lower bound minimax risk by Bayes risk: inf

ˆ θ

sup

θ∈Bd

2(R)

Eθ − ˆ θ2

2 ≥ inf ˆ θ

Eθ∼πθ − ˆ θ2

2

Let π be such that θi is sampled from Unif [−ǫi/2, ǫi/2], with

d

• i=1

ǫ2

i ≤ 4

(structural constraint on θ ∈ Θ := Bd

2(1))

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 14 / 20

Key Ideas of Proof

Assume M⊤M is diagonal without loss of generality by rotational invariance of the L2 ball Lower bound minimax risk by Bayes risk: inf

ˆ θ

sup

θ∈Bd

2(R)

Eθ − ˆ θ2

2 ≥ inf ˆ θ

Eθ∼πθ − ˆ θ2

2

Let π be such that θi is sampled from Unif [−ǫi/2, ǫi/2], with

d

• i=1

ǫ2

i ≤ 4

(structural constraint on θ ∈ Θ := Bd

2(1))

The usual van Trees inequality does not work with this prior because J (π) = +∞

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 14 / 20

Key Ideas of Proof

First keypoint: break L2 error into individual estimation errors

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 15 / 20

Key Ideas of Proof

First keypoint: break L2 error into individual estimation errors Eθ − ˆ θ2

2 ≥ d

• i=1

Var(θi − ˆ θi) ≥ 1 2πe

d

• i=1

e2h(θi|ˆ

θi)

≥ 1 2πe

d

• i=1

e2h(θi)−2I(θi;X)

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 15 / 20

Key Ideas of Proof

First keypoint: break L2 error into individual estimation errors Eθ − ˆ θ2

2 ≥ d

• i=1

Var(θi − ˆ θi) ≥ 1 2πe

d

• i=1

e2h(θi|ˆ

θi)

≥ 1 2πe

d

• i=1

e2h(θi)−2I(θi;X) ≥ 1 2πe

d

• i=1

e2h(θi)−2φ(Var(θi)·Eθi IX (θi))

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 15 / 20

Key Ideas of Proof

Lemma

When the components of parameter θ ∼ π, θ ∈ Rd are independent and X is sampled from the GLM, Eθi IX(θi) ≤ Eθ ¯ IX(θ)

• ii ≤

L s(σ)[M⊤M]ii i = 1, 2, . . . , d. Here, ¯ IX(θ) is the Fisher information matrix defined for θ ∈ Rd and X ∼ f (·; θ) with terms ¯ IX(θ)

• ij := EX

∂

∂θi f (X; θ) · ∂ ∂θj f (X; θ)

f 2(X; θ)

• This lemma follows from calculus, Cauchy-Schwarz and independence

between components of θ.

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 16 / 20

Key Ideas of Proof

Second keypoint: calculate Fisher information terms

λi := λi(M⊤M)

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 17 / 20

Key Ideas of Proof

Second keypoint: calculate Fisher information terms Eθ − ˆ θ2

2 ≥

1 2πe

d

• i=1

e2h(θi)−2φ(Var(θi)·E IX (θi))

λi := λi(M⊤M)

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 17 / 20

Key Ideas of Proof

Second keypoint: calculate Fisher information terms Eθ − ˆ θ2

2 ≥

1 2πe

d

• i=1

e2h(θi)−2φ(Var(θi)·E IX (θi)) ≥ 1 2πe

d

• i=1

e2h(θi)−2φ(Var(θi)·Eθi [¯

IX (θ)]ii)

≥ 1 2πe

d

• i=1

e2h(θi)−2φ

• Var(θi)·

L s(σ) [M⊤M]ii

• λi := λi(M⊤M)

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 17 / 20

Key Ideas of Proof

Second keypoint: calculate Fisher information terms Eθ − ˆ θ2

2 ≥

1 2πe

d

• i=1

e2h(θi)−2φ(Var(θi)·E IX (θi)) ≥ 1 2πe

d

• i=1

e2h(θi)−2φ(Var(θi)·Eθi [¯

IX (θ)]ii)

≥ 1 2πe

d

• i=1

e2h(θi)−2φ

• Var(θi)·

L s(σ) [M⊤M]ii

• =

1 2πe

d

• i=1

e2h(θi)−2φ

• Var(θi)·

L s(σ) λi

• λi := λi(M⊤M)

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 17 / 20

Key Ideas of Proof

Third keypoint: Select suitable values for prior

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 18 / 20

Key Ideas of Proof

Third keypoint: Select suitable values for prior Recall prior π is Unif d

i=1[−ǫi/2, ǫi/2]

• :

h(θi) = log ǫi, Var(θi) = ǫ2

i /12

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 18 / 20

Key Ideas of Proof

Third keypoint: Select suitable values for prior Recall prior π is Unif d

i=1[−ǫi/2, ǫi/2]

• :

h(θi) = log ǫi, Var(θi) = ǫ2

i /12

Eθ − ˆ θ2

2 ≥

1 2πe

d

• i=1

exp

• 2h(θi) − 2φ
• Var(θi) ·

L s(σ)λi

• =

1 2πe

d

• i=1

ǫ2

i exp

• −2φ

ǫ2

i

12 L s(σ)λi

• .

Immediately see that a tight bound should have ǫi depend on λi

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 18 / 20

Key Ideas of Proof

Case 1: d

i=1 1 λi ≤ 1 3 L s(σ)

1 Choose ǫ2

i = 12s(σ) L 1 λi

2 Validate d

i=1 ǫ2 i = 12s(σ) L

d

i=1 1 λi ≤ 4

3 Calculate lower bound

Eθ − ˆ θ2

2 d

• i=1

ǫ2

i exp

• −2φ

ǫ2

i

12 L s(σ)λi

• s(σ)

L Tr((M⊤M)−1).

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 19 / 20

Key Ideas of Proof

Case 1: d

i=1 1 λi ≤ 1 3 L s(σ)

1 Choose ǫ2

i = 12s(σ) L 1 λi

2 Validate d

i=1 ǫ2 i = 12s(σ) L

d

i=1 1 λi ≤ 4

3 Calculate lower bound

Eθ − ˆ θ2

2 d

• i=1

ǫ2

i exp

• −2φ

ǫ2

i

12 L s(σ)λi

• s(σ)

L Tr((M⊤M)−1). Case 2: d

i=1 1 λi > 1 3 L s(σ)

Lemma

If d

i=1 1 λi > 1 3 L s(σ), then there exists a non-negative sequence (ǫi)i=1,2,...,d

such that d

i=1 ǫ2 i ≤ 4 and

Eθ − ˆ θ2

2 1.

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 19 / 20

Summary

We proved a minimax lower bound for a class of GLMs with parameter θ constrained to the L2 ball

Tight for the (Gaussian) linear model

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 20 / 20

Summary

We proved a minimax lower bound for a class of GLMs with parameter θ constrained to the L2 ball

Tight for the (Gaussian) linear model

Provided a short and concise proof which:

Uses uniform prior π (and has J (π) = +∞) Sidesteps difficulty of calculating mutual information by calculating Fisher information, which naturally extracts the dependency on M

K.-Y. Lee and T. A. Courtade (Berkeley) New Minimax Bound for the GLM ISIT 2020 20 / 20