Learning Mixtures of Truncated Basis Functions from Data Helge - - PowerPoint PPT Presentation

Learning Mixtures of Truncated Basis Functions from Data

Helge Langseth, Thomas D. Nielsen, and Antonio Salmerón PGM 2012

This work is supported by an Abel grant from Iceland, Liechtenstein, and Norway through the EEA Financial Mechanism (Nils mobility project). Supported and Coordinated by Universidad Complutense de Madrid, by the Spanish Ministry of Science and Innovation through projects TIN2010-20900-C04-02-03, and by ERDF (FEDER) funds. Learning MoTBFs from data

1

Background: Approximations

Learning MoTBFs from data

Background: Approximations 2

Geometry of approximations

A quick recall of how of how to do approximations in Rn:

1 2 3 4 5 0 1 2 3 4 5 1 2 3 4 5

We want to approximate the vector f = (3, 2, 5) with A vector along e1 = (1, 0, 0).

Learning MoTBFs from data

Background: Approximations 2

Geometry of approximations

A quick recall of how of how to do approximations in Rn:

1 2 3 4 5 0 1 2 3 4 5 1 2 3 4 5

We want to approximate the vector f = (3, 2, 5) with A vector along e1 = (1, 0, 0). Best choice is f, e1 · e1 = (3, 0, 0).

Learning MoTBFs from data

Background: Approximations 2

Geometry of approximations

A quick recall of how of how to do approximations in Rn:

1 2 3 4 5 0 1 2 3 4 5 1 2 3 4 5

We want to approximate the vector f = (3, 2, 5) with A vector along e1 = (1, 0, 0). Best choice is f, e1 · e1 = (3, 0, 0). Now, add a vector along e2.

Best choice is f, e2 · e2, independently of the choice made for e1. Also, the choice we made for e1 is still optimal since e1 ⊥ e2. Best approximation is in general

ℓf, eℓ · eℓ.

Learning MoTBFs from data

Background: Approximations 2

Geometry of approximations

A quick recall of how of how to do approximations in Rn:

1 2 3 4 5 0 1 2 3 4 5 1 2 3 4 5

We want to approximate the vector f = (3, 2, 5) with A vector along e1 = (1, 0, 0). Best choice is f, e1 · e1 = (3, 0, 0). Now, add a vector along e2.

Best choice is f, e2 · e2, independently of the choice made for e1. Also, the choice we made for e1 is still optimal since e1 ⊥ e2. Best approximation is in general

ℓf, eℓ · eℓ.

All of this maps over to approximations of functions! We only need a definition of the inner product and the equivalent to

• rthonormal basis vectors.

Learning MoTBFs from data

Background: Approximations 2

Geometry of approximations

A quick recall of how of how to do approximations in Rn:

1 2 3 4 5 0 1 2 3 4 5 1 2 3 4 5

We want to approximate the vector f = (3, 2, 5) with A vector along e1 = (1, 0, 0). Best choice is f, e1 · e1 = (3, 0, 0). Now, add a vector along e2.

Best choice is f, e2 · e2, independently of the choice made for e1. Also, the choice we made for e1 is still optimal since e1 ⊥ e2. Best approximation is in general

ℓf, eℓ · eℓ.

Inner product for functions For two functions u(·) and v(·) defined on Ω ⊆ R, we use u, v =

Ω u(x) v(x) dx.

Learning MoTBFs from data

Background: Approximations 2

Generalised Fourier series

Definition (Legal set of basis functions) Let Ψ = {ψi}∞

i=0 be an indexed set of basis functions. Let Q be the set of all

linear combination of functions in Ψ. Ψ is a legal set of basis functions if:

1

ψ0 is constant;

2

u ∈ Q and v ∈ Q implies that (u · v) ∈ Q;

3

For any pair of real numbers s and t, s = t, there exists a function ψi ∈ Ψ s.t. ψi(s) = ψi(t). Legal basis functions {1, x, x2, x3, . . .} is a legal set of basis functions. {1, exp(−x), exp(x), exp(−2x), exp(2x), . . .} is also legal. {1, log(x), log(2x), log(3x), . . .} is not a legal set of basis functions.

Learning MoTBFs from data

Background: Approximations 3

Generalised Fourier series

Definition (Legal set of basis functions) Let Ψ = {ψi}∞

i=0 be an indexed set of basis functions. Let Q be the set of all

linear combination of functions in Ψ. Ψ is a legal set of basis functions if:

1

ψ0 is constant;

2

u ∈ Q and v ∈ Q implies that (u · v) ∈ Q;

3

For any pair of real numbers s and t, s = t, there exists a function ψi ∈ Ψ s.t. ψi(s) = ψi(t). Generalized Fourier series Assume Ψ is legal and contains orthonormal basis functions (if not, they can be made orthonormal through a Gram-Schmidt process). Then, the Generalized Fourier Series approximation to a function f is defined as ˆ f(·) =

ℓf, ψℓ ψℓ(·).

Learning MoTBFs from data

Background: Approximations 3

Generalised Fourier series

Definition (Legal set of basis functions) Let Ψ = {ψi}∞

i=0 be an indexed set of basis functions. Let Q be the set of all

linear combination of functions in Ψ. Ψ is a legal set of basis functions if:

1

ψ0 is constant;

2

u ∈ Q and v ∈ Q implies that (u · v) ∈ Q;

3

For any pair of real numbers s and t, s = t, there exists a function ψi ∈ Ψ s.t. ψi(s) = ψi(t). Important properties

1

Any function – including density functions – can be approximated arbitrarily well by this approach.

2

Ω

• f(x) − k

ℓ=0 ci ψℓ(x)

2 dx ≥

• Ω
• f(x) − k

ℓ=0f, ψℓ ψℓ(x)

2 dx, so the generalized Fourier series approximation is optimal in L2 sense.

Learning MoTBFs from data

Background: Approximations 3

MoTBFs

Learning MoTBFs from data

MoTBFs 4

The marginal MoTBF potential

Definition Let Ψ = {ψi}∞

i=0 with ψi : R → R define a legal set of basis functions on

Ω ⊆ R. Then gk : Ω → R+

0 is an MoTBF potential at level k wrt. Ψ . . .

1

if gk(x) =

k

• i=0

ai ψi(x) for all x ∈ Ω, where ai are real constants;

2

. . . or there is a partition of Ω into intervals I1, . . . , Im s.t. gk is defined as above on each Ij. Special cases An MoTBFs potential at level k = 0 is simply a standard discretisation. MoPs (original definition) and MTEs are also special cases of MoTBFs.

Learning MoTBFs from data

MoTBFs 4

The marginal MoTBF potential

Definition Let Ψ = {ψi}∞

i=0 with ψi : R → R define a legal set of basis functions on

Ω ⊆ R. Then gk : Ω → R+

0 is an MoTBF potential at level k wrt. Ψ . . .

1

if gk(x) =

k

• i=0

ai ψi(x) for all x ∈ Ω, where ai are real constants;

2

. . . or there is a partition of Ω into intervals I1, . . . , Im s.t. gk is defined as above on each Ij. Simplification We do not utilize the option to split the domain into subdomains here.

Learning MoTBFs from data

MoTBFs 4

Example: Polynomials vs. the Std. Gaussian

• 3
• 2
• 1

1 2 3 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

• 3
• 2
• 1

1 2 3 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

• 3
• 2
• 1

1 2 3 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

g0 = 0.4362 · ψ0 g2 = 0.4362 · ψ0 + · ψ1 + −0.1927 · ψ2 g8 = 0.4362 · ψ0 + · ψ1 + . . . . . . 0.0052 · ψ8 Use orthonormal polynomials (shifted & scaled Legendre polynomials). Approximation always integrates to unity. Direct computations give the gk closest in L2-norm. Positivity constraint and KL minimisation convex optimization.

Learning MoTBFs from data

MoTBFs 5

Learning Univariate Distributions

Learning MoTBFs from data

Learning Univariate Distributions 6

Relationship between KL and ML

Idea for learning MoTBFs from data Generate a kernel density for a (marginal) probability distribution, and use the translation-scheme to approximate it with an MoTBF.

Learning MoTBFs from data

Learning Univariate Distributions 6

Relationship between KL and ML

Idea for learning MoTBFs from data Generate a kernel density for a (marginal) probability distribution, and use the translation-scheme to approximate it with an MoTBF. Setup Let f(x) be the density generating {x1, . . . , xN}. Let gk(x|θ) = k

i=0 θi · ψi(x) be an MoTBF of order k.

Let hN(x) be a kernel density estimator. Result: KL minimization is likelihood maximization in the limit Let ˆ θN = arg minθ D ( hN(·) gk(·|θ) ). Then ˆ θN converges to the maximum likelihood estimator of θ as N → ∞ (given certain regularity conditions).

Learning MoTBFs from data

Learning Univariate Distributions 6

Example: Learning the standard Gaussian

• 3
• 2
• 1

1 2 3 0.2 0.4 0.6 Density estimate; 50 samples.

Learning MoTBFs from data

Learning Univariate Distributions 7

Example: Learning the standard Gaussian

• 3
• 2
• 1

1 2 3 0.2 0.4 0.6 Density estimate; 50 samples. g0: BIC = −91.54.

Learning MoTBFs from data

Learning Univariate Distributions 7

Example: Learning the standard Gaussian

• 3
• 2
• 1

1 2 3 0.2 0.4 0.6 Density estimate; 50 samples. g0: BIC = −91.54. g2: BIC = −83.21.

Learning MoTBFs from data

Learning Univariate Distributions 7

Example: Learning the standard Gaussian

• 3
• 2
• 1

1 2 3 0.2 0.4 0.6 Density estimate; 50 samples. g0: BIC = −91.54. g2: BIC = −83.21. g4: BIC = −76.13.

Learning MoTBFs from data

Learning Univariate Distributions 7

Example: Learning the standard Gaussian

• 3
• 2
• 1

1 2 3 0.2 0.4 0.6 Density estimate; 50 samples. g0: BIC = −91.54. g2: BIC = −83.21. g4: BIC = −76.13. ⇐ Best BIC score. g12: BIC = −88.78.

Learning MoTBFs from data

Learning Univariate Distributions 7

Comparison to “State-of-the-art”

Direct ML optimization At PGM’08/IJAR’10 we presented ML-learning of univariate MTEs: Divides support of function up into intervals. Direct ML optimization inside each interval. Computationally difficult. Summary of results Precision of the new method in terms of log likelihood is comparable to (but slightly poorer than) previous results. Speedup factor from 10 to 15. Fewer parameters chosen by BIC selection criteria.

Learning MoTBFs from data

Learning Univariate Distributions 8

Conditional Distributions

Learning MoTBFs from data

Conditional Distributions 9

Definition of conditional distributions

Assume we have x ∈ Im, and want to define g(m)

k

(y|x) there. We define conditional MoTBFs to only depend on their conditioning variable(s) through the relevant hypercube, and not the numerical value: g(m)

k

(y|x) = k

j=0 θ(m) j

ψj(y) for x ∈ Im.

g(3,1)(y) X1 g(2,1)(y) g(2,2)(y) g(3,3)(y) g(1,2)(y) g(1,3)(y) X2 g(2,3)(y) g(3,2)(y) g(1,1)(y)

Conditioning hypercubes learned by optimizing BIC-score.

Learning MoTBFs from data

Conditional Distributions 9

Results: X ∼ N(0, 1), Y |{X = x} ∼ N(x/2, 1)

50 cases 500 cases 2500 cases 5000 cases

Learning MoTBFs from data

Conditional Distributions 10

Concluding Remarks

Learning MoTBFs from data

Concluding Remarks 11

Summary

Conclusions: KL-guided learning is much faster than the current implementations of direct ML optimization. There is – however – a loss in precision. The KL-guided learning results do not use splitpoints for the head

• variable. This can be exploited by inference algorithms.

Future work:

1

Look for improvements with respect to computational speed and numerical stability of the learning algorithm.

2

Investigate the formal properties of the estimators.

3

Compare our approach to López-Cruz et al. (2012): Learning mixtures of polynomials from data using B-spline interpolation.

Learning MoTBFs from data

Concluding Remarks 11