An Instability in Variational Inference for Topic Models Behrooz - - PowerPoint PPT Presentation

An Instability in Variational Inference for Topic Models

Behrooz Ghorbani

Joint work with Hamid Javadi and Andrea Montanari

Stanford University Department of Electrical Engineering June, 2019

Behrooz Ghorbani Topic Models Stanford University 1 / 6

Problem Statement

Statistical model: X = √β d WHT + Z where W ∈ Rn×r, H ∈ Rd×r and Z is i.i.d Gaussian noise n, d ≫ 1 with n

d = δ > 0, where δ, r ∼ O(1)

Wi

i.i.d

∼ Dir(ν1) and Hj

i.i.d

∼ N(0, Ir)

Behrooz Ghorbani Topic Models Stanford University 2 / 6

Problem Statement

Statistical model: X = √β d WHT + Z where W ∈ Rn×r, H ∈ Rd×r and Z is i.i.d Gaussian noise n, d ≫ 1 with n

d = δ > 0, where δ, r ∼ O(1)

Wi

i.i.d

∼ Dir(ν1) and Hj

i.i.d

∼ N(0, Ir)

Behrooz Ghorbani Topic Models Stanford University 2 / 6

Problem Statement

Statistical model: X = √β d WHT + Z where W ∈ Rn×r, H ∈ Rd×r and Z is i.i.d Gaussian noise n, d ≫ 1 with n

d = δ > 0, where δ, r ∼ O(1)

Wi

i.i.d

∼ Dir(ν1) and Hj

i.i.d

∼ N(0, Ir)

Behrooz Ghorbani Topic Models Stanford University 2 / 6

(Naive Mean Field) Variational Inference

Goal: Use the posterior distribution, pH,W |X(·|X), to estimate W and H Variational Inference: Approximate the posterior with a simpler distribution ˆ q such that: ˆ q (H, W ) = q (H) ˜ q (W ) =

d

• a=1

qa (Ha)

n

• i=1

˜ qi (Wi) Is the output of variational inference reliable?

Behrooz Ghorbani Topic Models Stanford University 3 / 6

(Naive Mean Field) Variational Inference

Goal: Use the posterior distribution, pH,W |X(·|X), to estimate W and H Variational Inference: Approximate the posterior with a simpler distribution ˆ q such that: ˆ q (H, W ) = q (H) ˜ q (W ) =

d

• a=1

qa (Ha)

n

• i=1

˜ qi (Wi) Is the output of variational inference reliable?

Behrooz Ghorbani Topic Models Stanford University 3 / 6

(Naive Mean Field) Variational Inference

Goal: Use the posterior distribution, pH,W |X(·|X), to estimate W and H Variational Inference: Approximate the posterior with a simpler distribution ˆ q such that: ˆ q (H, W ) = q (H) ˜ q (W ) =

d

• a=1

qa (Ha)

n

• i=1

˜ qi (Wi) Is the output of variational inference reliable?

Behrooz Ghorbani Topic Models Stanford University 3 / 6

Comparison of Two Thresholds

βBayes : Information theoretic threshold

Behrooz Ghorbani Topic Models Stanford University 4 / 6

Comparison of Two Thresholds

βBayes : Information theoretic threshold If β < βBayes, then any estimator is asymptotically uncorrelated with the truth

Behrooz Ghorbani Topic Models Stanford University 4 / 6

Comparison of Two Thresholds

βBayes : Information theoretic threshold If β < βBayes, then any estimator is asymptotically uncorrelated with the truth βinst : Threshold for variational inference to return a non-trivial estimate

Behrooz Ghorbani Topic Models Stanford University 4 / 6

Comparison of Two Thresholds

βBayes : Information theoretic threshold If β < βBayes, then any estimator is asymptotically uncorrelated with the truth βinst : Threshold for variational inference to return a non-trivial estimate If β < βinst, ˆ Wi = 1

r 1r ⇒ No signal found in the data!

Behrooz Ghorbani Topic Models Stanford University 4 / 6

Comparison of Two Thresholds

βBayes : Information theoretic threshold If β < βBayes, then any estimator is asymptotically uncorrelated with the truth βinst : Threshold for variational inference to return a non-trivial estimate If β < βinst, ˆ Wi = 1

r 1r ⇒ No signal found in the data!

If β > βinst, ˆ Wi = 1

r 1r ⇒ variational algorithm declares that it has

found a signal!

Behrooz Ghorbani Topic Models Stanford University 4 / 6

Comparison of Two Thresholds

We want βBayes ≈ βinst

Behrooz Ghorbani Topic Models Stanford University 5 / 6

Comparison of Two Thresholds

We want βBayes ≈ βinst

1.0 1.5 2.0 2.5 3.0 Aspect Ratio, ± = n

d

2 4 6 8 10 12 14 Signal Strength, ¯ Comparisons of ¯Bayes and ¯inst ¯Bayes ¯inst

Behrooz Ghorbani Topic Models Stanford University 5 / 6

Comparison of Two Thresholds

We want βBayes ≈ βinst

1.0 1.5 2.0 2.5 3.0 Aspect Ratio, ± = n

d

2 4 6 8 10 12 14 Signal Strength, ¯ Comparisons of ¯Bayes and ¯inst ¯Bayes ¯inst

Behrooz Ghorbani Topic Models Stanford University 5 / 6

Credible intervals: Nominal coverage 90%

20 40 60 80 100 Weight Index, i 0.0 0.2 0.4 0.6 0.8 1.0 Wi, 1

• 2. 0 = β < βinst = 2. 2
• Wi, 1

Wi, 1

20 40 60 80 100 Weight Index, i 0.0 0.2 0.4 0.6 0.8 1.0 Wi, 1 β = 4. 1 20 40 60 80 100 Weight Index, i 0.0 0.2 0.4 0.6 0.8 1.0 Wi, 1 β = βBayes = 6. 0

Empirical coverage β = 2 < βinst: 0.87 β = 4.1 ∈ (βinst, βBayes): 0.65 β = 6 = βBayes: 0.51

Behrooz Ghorbani Topic Models Stanford University 6 / 6