Automatic relevance determination in nonnegative matrix - - PowerPoint PPT Presentation

Background ARD in NMF Results

Automatic relevance determination in nonnegative matrix factorization with the β-divergence

Vincent Y. F. Tan1 and C´ edric F´ evotte2

1University of Wisconsin-Madison 2CNRS LTCI; T´

el´ ecom ParisTech Paris, France

NIPS “Sparse Low Rank” workshop December 2011

• V. Y. F. Tan (Univ. of Wisconsin) and C. F´

evotte (CNRS) Automatic relevance determination in β-NMF

Background ARD in NMF Results

Nonnegative matrix factorization (NMF)

Given a nonnegative matrix V of dimensions F × N, NMF is the problem of finding a factorization V ≈ WH where W and H are nonnegative matrices of dimensions F × K and K × N, respectively.

• V. Y. F. Tan (Univ. of Wisconsin) and C. F´

evotte (CNRS) Automatic relevance determination in β-NMF

Background ARD in NMF Results

Nonnegative matrix factorization (NMF)

Given a nonnegative matrix V of dimensions F × N, NMF is the problem of finding a factorization V ≈ WH where W and H are nonnegative matrices of dimensions F × K and K × N, respectively. Constrained optimization problem: min

W,H≥0 D(V|WH) =

• fn

d([V]fn|[WH]fn) where d(x|y) is a scalar cost function. Objective of this work is to identify the “right” value of K.

• V. Y. F. Tan (Univ. of Wisconsin) and C. F´

evotte (CNRS) Automatic relevance determination in β-NMF

Background ARD in NMF Results

Automatic relevance determination in NMF

Inspired by Bayesian PCA (Bishop, 1999): each “component” k is assigned a relevance (= variance) parameter φk.

+ ... +

φK ≈ V w1 h1 wK hK φ1

Half-Gaussian or exponential priors on wk and hk.

E.g., p(wk|φk) =

• f

φ−1

k

exp −φ−1

k wfk,

p(hk|φk) =

• n

φ−1

k

exp −φ−1

k hkn

• V. Y. F. Tan (Univ. of Wisconsin) and C. F´

evotte (CNRS) Automatic relevance determination in β-NMF

Background ARD in NMF Results

Automatic relevance determination in NMF

After a few manipulations, we are essentially left with the minimization of C(W, H) = Dβ(V|WH) + ρ

K

• k=1

log (wk + hk + ε) where

◮ Dβ(V|WH) is the measure of fit (in this work, β-divergence) ◮ x = 1 2x2 2 (half-Gaussian priors) or x = x1

(exponential priors).

• V. Y. F. Tan (Univ. of Wisconsin) and C. F´

evotte (CNRS) Automatic relevance determination in β-NMF

Background ARD in NMF Results

Swimmer decomposition results

8 data samples (among 256) Estimated W using with exponential priors /ℓ1 penalization

• V. Y. F. Tan (Univ. of Wisconsin) and C. F´

evotte (CNRS) Automatic relevance determination in β-NMF

Background ARD in NMF Results

Audio decomposition results

• 2

4 6 8 10 12 14 −0.5 0.5 audio signal time (s) log power spectrogram frame frequency 100 200 300 400 500 600 100 200 300 400 500

• V. Y. F. Tan (Univ. of Wisconsin) and C. F´

evotte (CNRS) Automatic relevance determination in β-NMF

Background ARD in NMF Results

Audio decomposition results

5 10 15 20 0.01 0.02 0.03 0.04 0.05 IS−NMF 5 10 15 20 0.01 0.02 0.03 0.04 0.05 ARD IS−NMF

Figure: Histograms of standard deviation values of all K = 18 components produced by Itakura-Saito NMF and ARD Itakura-Saito NMF (with ℓ2 penalization). ARD IS-NMF only retains the 6 “right” components.

Check our full-length technical report available on arxiv.

• V. Y. F. Tan (Univ. of Wisconsin) and C. F´

evotte (CNRS) Automatic relevance determination in β-NMF