Estimation with Infinite Dimensional Kernel Exponential Families - - PowerPoint PPT Presentation

1

Kenji Fukumizu

The Institute of Statistical Mathematics

Joint work with Bharath Sriperumbudur (Penn State U), Arthur Gretton (UCL), Aapo Hyvarinen (U Helsinki), Revant Kumar (Georgia Tech)

IGAIA IV. June 12-17, 2016. Liblice, Czech Republic

Estimation with Infinite Dimensional Kernel Exponential Families

Introduction

2

Infinite dimensional exponential family

 (Finite dim.) exponential family

𝑞𝜄 𝑦 = exp ෍

𝑘=1 𝑛

𝜄

𝑘𝑈 𝑘 𝑦 − 𝐵 𝜄

𝑟0(𝑦)

 Infinite dimensional extension?

𝑞𝑔 𝑦 = exp 𝑔 𝑦 − 𝐵 𝑔 𝑟0(𝑦) where 𝐵 𝑔 ≔ log ∫ 𝑓𝑔(𝑦)𝑟0 𝑦 𝑒𝑦 𝑔 is a natural parameter in an infinite dimensional function class. – Maximal exponential model (Pistone & Sempi AoS 1995):

• Orlicz space (Banach sp.) is used.
• Estimation is not at all obvious.

“Empirical” mean parameter cannot be defined.

 Kernel exponential manifold (Fukumizu 2009; Canu & Smola 2005)

Reproducing kernel Hilbert space is used.

• 𝑞𝑔 𝑦 = exp 〈𝑔, 𝑙 ⋅, 𝑦 〉 − 𝐵 𝑔

𝑟0(𝑦)

• Empirical estimation is possible

– Mean parameter: 𝑛𝑔 = 𝐹𝑞𝑔[𝑙 ⋅, 𝑌 ] – Maximum likelihood estimator: ෝ 𝑛𝑔 =

1 𝑜 σ𝑗=1 𝑜

𝑙(⋅, 𝑌𝑗)

• Manifold structure can be defined (Fukumizu 2009)

4

Parameter Infinite dimensional sufficient statistics

Problems in estimation

 Normalization constant / partition function

– Even in finite dim. cases 𝐵 𝜄 ≔ log ∫ 𝑓σ𝑘=1

𝑛

𝜄𝑘𝑈𝑘 𝑦 𝑟0 𝑦 𝑒𝑦

is not easy to compute. – MLE: “Mean parameter  natural parameter” needs to solve 𝜖𝐵 𝜄 𝜖𝜄 = 1 𝑜 ෍

𝑗=1 𝑜

𝑈 𝑌𝑗 . – Even more difficult for an infinite dimensional exponential family

 This talk  score matching (Hyvarinen, JMLR 2005)

– Estimation method without normalization constants. – Introducing a new method for (unnormalized) density estimation.

5

Score Matching

6

Score matching for exponential family

(Hyvärinen, JMLR2005)

 Fisher divergence

𝑞, 𝑟: two p.d.f.’s on Ω = ς𝑏=1

𝑒

(𝑡𝑏, 𝑢𝑏) ⊂ 𝐒 ∪ ±∞

𝑒.

𝐾 𝑞||𝑟 ≔ 1 2 න ෍

𝑏=1 𝑒

𝜖 log 𝑞 𝑦 𝜖𝑦𝑏 − 𝜖 log 𝑟 𝑦 𝜖𝑦𝑏

2

𝑞(𝑦)𝑒𝑦 – 𝐾(𝑞| 𝑟 ≥ 0. Equality holds iff 𝑞 = 𝑟 (under mild conditions). – Derivative w.r.t. 𝑦, not parameter.

• For location parameter 𝑞(𝑦) = 𝑔 𝑦 − 𝜄 ,

𝜖 log 𝑞 𝑦 𝜖𝑦𝑏 = − 𝜖 log 𝑔

𝜄 𝑦

𝜖𝜄𝑏 𝐾(𝑞||𝑟) = squared 𝑀2-distance of Fisher scores.

7

Set 𝑞 = 𝑞0 (true), and 𝑟 = 𝑞𝜄 to be estimated. 𝐾 𝜄 ≔ 𝐾 𝑞0||𝑞𝜄

=

1 2 ∫σ𝑏=1

𝑒 𝜖 log 𝑞𝜄 𝑦 𝜖𝑦𝑏

− 𝜖 log 𝑞0 𝑦

𝜖𝑦𝑏 2

𝑞0(𝑦)𝑒𝑦

= 1 2 න ෍

𝑏=1 𝑒

𝜖 log 𝑞𝜄 𝑦 𝜖𝑦𝑏

2

𝑞0(𝑦)𝑒𝑦 + න ෍

𝑏=1 𝑒 𝜖2 log 𝑞𝜄 𝑦

𝜖𝑦𝑏

2

𝑞0 𝑦 𝑒𝑦 + const.

• Assume

lim

𝑦𝑏→𝑡𝑏 or 𝑢𝑏 𝑞0(𝑦) 𝜖 log 𝑞𝜄 𝑦 𝜖𝑦𝑏

= 0, and use partial integral

∫

𝜖 log 𝑞𝜄 𝑦 𝜖𝑦𝑏 𝜖 log 𝑞0 𝑦 𝜖𝑦𝑏

𝑞0 𝑦 𝑒𝑦 = 𝑞0 𝑦

𝜖 log 𝑞𝜄 𝑦 𝜖𝑦𝑏 𝑡𝑏 𝑢𝑏

− ∫

𝜖2 log 𝑞𝜄 𝑦 𝜖𝑦𝑏

2

𝑞0(𝑦)𝑒𝑦

8

≡ ሚ 𝐾 𝜄

𝜖𝑞0 𝑦 𝜖𝑦𝑏

 Empirical estimation

ሚ 𝐾 𝜄 = 1 2 න ෍

𝑏=1 𝑒

𝜖 log 𝑞𝜄 𝑦 𝜖𝑦𝑏

2

𝑞0(𝑦)𝑒𝑦 + න ෍

𝑏=1 𝑒 𝜖2 log 𝑞𝜄 𝑦

𝜖𝑦𝑏

2

𝑞0 𝑦 𝑒𝑦 𝑌1, … , 𝑌𝑜: i.i.d. sample ~ 𝑞0. ሚ 𝐾𝑜 𝜄 = 1 𝑜 ෍

𝑏=1 𝑒

෍

𝑗=1 𝑜

1 2 𝜖 log 𝑞𝜄 𝑌𝑗 𝜖𝑦𝑏

2

+ 𝜖2 log 𝑞𝜄 𝑌𝑗 𝜖𝑦𝑏

2

መ 𝜄 = arg min ሚ 𝐾𝑜(𝜄) : Score matching estimator

9

Score matching for exponential family

– For exponential family 𝑞𝜄 𝑦 = exp σ𝑘 𝜄

𝑘𝑈 𝑘 𝑦 − 𝐵 𝜄

𝑟0 𝑦 ,

ሚ 𝐾𝑜 𝜄 = ෍

𝑗=1 𝑜

෍

𝑏=1 𝑒 1

2 ෍

𝑘=1 𝑛

𝜄

𝑘

𝜖𝑈

𝑘 𝑌𝑗

𝜖𝑦𝑏 + 𝜖 log 𝑟0 𝑌𝑗 𝜖𝑦𝑏

2

+ ෍

𝑘=1 𝑛

𝜄

𝑘

𝜖2𝑈

𝑘 𝑌𝑗

𝜖𝑦𝑏

2

+ 𝜖2 log 𝑟0 𝑌𝑗 𝜖𝑦𝑏

2

• No need of 𝐵 𝜄 ! (derivative w.r.t. 𝑦)
• Quadratic form w.r.t. 𝜄  Solvable!
• In the Gaussian case, ෠

𝜄 is the same as MLE.

10

Kernel Exponential Family

11

Reproducing kernel Hilbert space

– Def. Ω: set. 𝐼: Hilbert space consisting of functions on Ω. 𝐼: reproducing kernel Hilbert space (RKHS), if for any 𝑦 ∈ Ω there is 𝑙𝑦 ∈ 𝐼 s.t.

𝑔, 𝑙𝑦 = 𝑔 𝑦

for ∀𝑔 ∈ 𝐼 [reproducing property] – 𝑙 𝑦, 𝑧 ≔ 𝑙𝑦(𝑧). 𝑙 is a positive definite kernel, i.e., 𝑙 𝑦, 𝑧 = 𝑙(𝑧, 𝑦) and the Gram matrix 𝑙 𝑦𝑗, 𝑦𝑘

𝑗𝑘 is positive

semidefinite for any 𝑦1, … , 𝑦𝑜. – Moore-Aronszajn theorem: for any positive definite kernel on Ω, there uniquely exists an RKHS s.t. its reproducing kernel is 𝑙(⋅, 𝑦). (One-to-one correspondence between p.d. kernel and RKHS) – Example of pos. def. kernel on 𝐒𝑒: 𝑙 𝑦, 𝑧 = exp −

‖𝑦−𝑧‖2 2𝜏2

.

12

Kernel exponential family

• Def. 𝑙: pos. def. kernel on Ω = ς𝑏=1

𝑒

(𝑡𝑏, 𝑢𝑏) ⊂ 𝐒 ∪ ±∞

𝑒.

𝐼𝑙: RKHS. 𝑟0: p.d.f. on Ω with supp 𝑟0 = Ω. 𝐺𝑙 ≔ {𝑔 ∈ 𝐼𝑙 ∣ ∫ 𝑓𝑔 𝑦 𝑟0 𝑦 𝑒𝑦 < ∞} (functional) parameter space

𝑄

𝑙 ≔ {𝑞𝑔: Ω → 0, ∞ ∣

𝑞𝑔 𝑦 = 𝑓𝑔 𝑦 −𝐵 𝑔 𝑟0 𝑦 , 𝑔 ∈ 𝐺𝑙}

where 𝐵 𝑔 ≔ ∫ 𝑓𝑔(𝑦)𝑟0 𝑦 𝑒𝑦 𝑄𝑙: kernel exponential family (KEF) – With finite dimensional 𝐼𝑙, KEF is reduced to a finite dim. exponential family. e.g. 𝑙 𝑦, 𝑧 = 1 + 𝑦𝑈𝑧 2  Gaussian distributions.

13

Score matching for KEF

Assume 𝑙 is of class 𝐷2 (𝜖𝑏+𝑐𝑙(𝑦, 𝑧)/𝜖𝑏𝑦𝜖𝑐𝑧 exists and is continuous for 𝑏 + 𝑐 ≤ 2) and lim

𝑦𝑏→𝑡𝑏 or 𝑢𝑏

ฬ

𝜖2𝑙 𝑦,𝑧 𝜖𝑦𝑏𝜖𝑧𝑏 𝑧=𝑦

𝑞0 𝑦 = 0 (for partial integral). – Score matching objective function ሚ 𝐾𝑜 𝑔 ≔ ෍

𝑗=1 𝑜

෍

𝑏=1 𝑒 1

2 𝜖𝑔 𝑌𝑗 𝜖𝑦𝑏 + 𝜖 log 𝑟0 𝑌𝑗 𝜖𝑦𝑏

2

+ 𝜖2𝑔 𝑌𝑗 𝜖𝑦𝑏

2

+ 𝜖2 log 𝑟0 𝑌𝑗 𝜖𝑦𝑏

2

Note 𝑔 𝑌𝑗 = 𝑔, 𝑙 ⋅, 𝑌𝑗 ,

𝜖𝑔 𝑌𝑗 𝜖𝑦𝑏 = 𝑔, 𝜖𝑙 ⋅,𝑌𝑗 𝜖𝑦𝑏

,

𝜖2𝑔 𝑌𝑗 𝜖𝑦𝑏

2

= 𝑔,

𝜖2𝑙 ⋅,𝑌𝑗 𝜖𝑦𝑏

2

. ሚ 𝐾𝑜 𝑔 is a quadratic form w.r.t. 𝑔 ∈ 𝐼.

14

– Estimation መ 𝐷𝑜𝑔 = 𝜊𝑜 where መ 𝐷𝑜 ≔ 1 𝑜 ෍

𝑗=1 𝑜

෍

𝑏=1 𝑒 𝜖𝑙 ⋅, 𝑌𝑗

𝜖𝑦𝑏 𝜖𝑙 ⋅, 𝑌𝑗 𝜖𝑦𝑏 ,∗ ∶ 𝐼𝑙 → 𝐼𝑙 መ 𝜊𝑜 ≔ 1 𝑜 ෍

𝑗=1 𝑜

෍

𝑏=1 𝑒

𝜖𝑙 ⋅, 𝑌𝑗 𝜖𝑦𝑏 𝜖 log 𝑟0 𝑌𝑗 𝜖𝑦𝑏 + 𝜖2𝑙 ⋅, 𝑌𝑗 𝜖𝑦𝑏

2

∈ 𝐼𝑙 – Regularized estimator ෡ 𝑔

𝑜 =

መ 𝐷𝑜 + 𝜇𝑜𝐽

−1 መ

𝜊𝑜 i.e., ෡ 𝑔

𝑜 = argmin𝑔 ሚ

𝐾𝑜 𝑔 + 𝜇𝑜 𝑔 𝐼𝑙

2

15

– Estimator: መ 𝑔

𝑜 = 𝛽 መ

𝜊𝑜 + ෍

𝑘=1 𝑜

෍

𝑐=1 𝑒

𝛾𝑘𝑐 𝜖𝑙 ⋅, 𝑌

𝑘

𝜖𝑦𝑐

where

1 𝑜 σ𝑏,𝑗 ℎ𝑗 𝑏 2 + 𝜇

መ 𝜊𝑜

2 1 𝑜 σ𝑏,𝑗 ℎ𝑗 𝑏 𝐻𝑗𝑘 𝑏𝑐 + 𝜇 ℎ𝑘 𝑐 1 𝑜 σ𝑏,𝑗 ℎ𝑗 𝑏 𝐻𝑗𝑘 𝑏𝑐 + 𝜇 ℎ𝑘 𝑐 1 𝑜 σ𝑑,𝑛 𝐻𝑗𝑛 𝑏𝑑𝐻𝑛𝑘 𝑐𝑑 + 𝜇 𝐻𝑗𝑘 𝑏𝑐

𝛽 𝛾𝑗𝑏 = − መ 𝜊𝑜

2

ℎ𝑘

𝑐

ℎ𝑘

𝑐 = 1 𝑜 σ𝑗,𝑏 𝜖3𝑙 𝑌𝑗,𝑌𝑘 𝜖𝑦𝑏

2𝜖𝑧𝑐 +

𝜖2𝑙 𝑌𝑗,𝑌𝑘 𝜖𝑦𝑏𝜖𝑧𝑐 𝜖ℓ 𝑌𝑗 𝜖𝑦𝑏 , 𝜖ℓ 𝑌𝑗 𝜖𝑦𝑏 = 𝜖 log 𝑟0 𝑌𝑗 𝜖𝑦𝑏

𝐻𝑗𝑘

𝑏𝑐 = 𝜖2𝑙 𝑌𝑗,𝑌𝑘 𝜖𝑦𝑏𝜖𝑧𝑐 , መ

𝜊𝑜

2 = 1 𝑜2 σ𝑗𝑘,𝑏𝑐 𝜖4𝑙 𝑌𝑗,𝑌𝑘 𝜖𝑦𝑏

2𝜖𝑧𝑐 2 + 2

𝜖3𝑙 𝑌𝑗,𝑌𝑘 𝜖𝑦𝑏

2𝜖𝑧𝑐

𝜖ℓ(𝑌𝑘) 𝜖𝑦𝑐 + 𝜖2𝑙 𝑌𝑗,𝑌𝑘 𝜖𝑦𝑏𝜖𝑧𝑐 𝜖ℓ 𝑌𝑗 𝜖𝑦𝑏 𝜖ℓ(𝑌𝑘) 𝜖𝑦𝑐

• መ

𝑔

𝑜 can be taken in Span 𝜖𝑙 ⋅,𝑌𝑘 𝜖𝑦𝑐

, መ 𝜊𝑜 .

• Estimator is simply given by solving 1 + 𝑜𝑒 -dimensional linear

equation.

16

Explicit Solution

(from representer theorem)

Unnormalized p.d.f.

– Score matching for KEF gives only 𝑔(𝑦) or 𝑓𝑔 𝑦 , unnormalized p.d.f.

• Estimation of 𝐵 𝑔 ≔ ∫ 𝑓𝑔(𝑦)𝑟0 𝑦 𝑒𝑦 is yet nontrivial.

– There are interesting applications. 1) Nonparametric structure learning for graphical model given data (Sun, Kolar, Xu NIPS2015) 𝑞 𝑌 ∝ ෑ

𝑗𝑘∈𝐹

𝑞𝑗𝑘 𝑌𝑗, 𝑌

𝑘 ,

𝐻 = (𝑊, 𝐹) 𝑞𝑗𝑘 is estimated nonparametrically with KEF (with sparse edges).

17

b c d e a

2) Hamiltonian Monte Carlo with intractable gradient

(Strathmann et al. NIPS 2015)

Estimate

𝜖 log 𝜌 𝑦 𝜖𝑦

with EKF, assuming it does not allow a closed form expression (intractable cases).

• Hamiltonian Monte Carlo (Neal 2012)

Goal: sample from 𝜌 𝑉 𝑦 = − log 𝜌 𝑦 𝐿 𝑨 : auxiliary momentum, e.g. −𝑨2/𝜐2 Hamiltonian 𝐼 𝑨, 𝑦 ≔ 𝑉 𝑦 + 𝐿(𝑨) Hamiltonian flow:

𝑒𝑦 𝑒𝑢 = 𝜖𝐼 𝑒𝑨 = 𝜖𝐿 𝜖𝑨, 𝑒𝑨 𝑒𝑢 = − 𝜖𝐼 𝑒𝑦 = 𝜖 log 𝜌 𝑦 𝜖𝑦

This flow is used in proposal of MCMC

18

Convergence

 Misspecification

True parameter 𝑔

∗ is taken from a wider space than 𝐼𝑙.

Extended parameter space 𝑋

2 0 𝑞0 ≔

ൗ 𝑔 ∈ 𝐷1 Ω ∣

𝜖𝑔 𝑦 𝜖𝑦𝑏 ∈ 𝑀2 Ω; 𝑞0 , 𝑏 = 1, … , 𝑒

∼ where 𝑔 ∼ 𝑕 ⟺ σ𝑏=1

𝑒

𝜖𝑔/𝜖𝑦𝑏 − 𝜖𝑕/𝜖𝑦𝑏

𝑀2 𝑞0 2

= 0 𝑔 , [𝑕] 𝑋

2 0(𝑞0): = σ𝑏=1

𝑒

∫

𝜖𝑔 𝑦 𝜖𝑦𝑏 𝜖𝑕 𝑦 𝜖𝑦𝑏 𝑞0 𝑦 𝑒𝑦.

𝑋

2 𝑞0 : completion of the pre-Hilbert space 𝑋 2 0 𝑞0 .

• With 𝑙 is of class 𝐷2 (and other technical conditions), the

canonical map 𝐽𝑙: 𝐼𝑙 → 𝑋

2 𝑞0 ,

𝑔 ↦ 𝑔 defines a (up to constant) embedding of 𝐼𝑙.

19

Theorem (convergence rate) Under some assumptions, (i) If 𝑔

∗ ≔ log(𝑞0/𝑟0) ∈ 𝑆 𝐽𝑙𝐽𝑙 ∗ , with 𝜇𝑜 → 0, 𝑜𝜇𝑜 → ∞

𝐾(𝑞0||𝑞 መ

𝑔

𝑜) → 0

𝑜 → ∞ . (ii) If 𝑔

∗ ∈ 𝑆( 𝐽𝑙𝐽𝑙 ∗ 𝛾) (0 < 𝛾 ≤ 1), then with 𝜇𝑜 = 𝑜 − max 1

3, 1 2𝛾+1 ,

𝐾(𝑞0‖𝑞 መ

𝑔

𝑜) = 𝑃𝑞 𝑜

− min 2 3, 2𝛾 2𝛾+1

. 𝐽𝑙𝐽𝑙

∗: operator on 𝑋 2 𝑞0 , given by

𝐽𝑙𝐽𝑙

∗ 𝑔 = න ෍ 𝑏=1 𝑒 𝜖𝑙 ⋅, 𝑦

𝜖𝑦𝑏 𝜖𝑔 𝑦 𝜖𝑦𝑏 𝑞0 𝑦 𝑒𝑦

20

Hyperparameter selection

– Hyperparameters

• Kernel / kernel parameter (𝑙 𝑦, 𝑧 = exp −

1 2𝜏2 𝑦 − 𝑧 2 )

• regularization coefficient

– Cross-validation is possible with the objective function

ሚ 𝐾𝑜 𝑔 ≔ ෍

𝑗=1 𝑜

෍

𝑏=1 𝑒 1

2 𝜖𝑔 𝑌𝑗 𝜖𝑦𝑏 + 𝜖 log 𝑟0 𝑌𝑗 𝜖𝑦𝑏

2

+ 𝜖2𝑔 𝑌𝑗 𝜖𝑦𝑏

2

+ 𝜖2 log 𝑟0 𝑌𝑗 𝜖𝑦𝑏

2

.

21

Experiments

22

Kernel Density Estimation

– KDE: standard nonparametric method for estimating p.d.f. Given i.i.d. sample 𝑌1, … , 𝑌𝑜 ∼ 𝑄 Ƹ 𝑞𝑜 𝑦 = 1 𝑜 ෍

𝑗=1 𝑜

𝐿 𝑦 − 𝑌𝑗 ℎ𝑜 𝐿 𝑦 : p.d.f. e.g. 𝐿 𝑦 =

1 2𝜌 𝑒/2 exp − 𝑦 2 2

• KDE works well for one-dimensional cases, but weak for high

(say, 10) dimensional cases.

• Sensitive to the choice of ℎ𝑜, (though CV and other methods

are applicable).

23

Comparison: EKF vs KDE

24

5 10 15 20 1 2 3 4 Dimension Socre objective function Gaussian distribution: n = 500 Score match KDE

 Evaluated by score objective function 𝐾

kernel: 𝑙 𝑦, 𝑧 = exp − 𝑦 − 𝑧 2/2𝜏2 + 0.1 𝑦𝑈𝑧 + 0.5 2

5 10 15 20 1 2 3 4 5 Dimension Score objective function Gaussian mixture: n = 300 Score match KDE

5 10 15 20 1 2 3 4 5 Dimension Score objective function Gaussian mixture: n = 300 Score match KDE

・ Gaussian Mixture

𝑞0 = 0.5𝜚𝑒 𝑦; 4𝟐𝑒, 𝐽𝑒 + 0.5𝜚𝑒(𝑦; −4𝟐𝑒, 𝐽𝑒)

・ Gaussian 𝑞0 = 𝜚𝑒 𝑦; 0, 𝐽𝑒

25

 Evaluated by correlation

𝐷𝑝𝑠 𝑞, 𝑞0 ≔

𝐹𝑆 𝑞 𝑎 𝑞0 𝑎 𝐹𝑆 𝑞 𝑎 2 𝐹𝑆[𝑞0 𝑎 2], 𝑎 ∼ 1 104 σ𝑗=1 104 𝜀𝑌𝑗 , 𝑌𝑗 ∼ 𝑞0𝑒𝑦

– Gaussian

𝑞0 = 𝜚𝑒 𝑦; 0, 𝐽𝑒

5 10 15 20 0.65 0.7 0.75 0.8 0.85 0.9 0.95 Dimension Socre objective function Gaussian distribution: n = 500 Score match KDE

𝑗. 𝑗. 𝑒. Correlation

5 10 15 20 0.5 0.6 0.7 0.8 0.9 1 Dimension Correlation Gaussian mixture: n = 300 Score match KDE

– Gaussian Mixture

𝑞0 = 0.5𝜚𝑒 𝑦; 41𝑒, 𝐽𝑒 + 0.5𝜚𝑒(𝑦; −41𝑒, 𝐽𝑒)

Conclusions

26

 Infinite dimensional exponential family with positive definite kernel

– A natural extension of finite dimensional exponential family – Sufficient statistics and parameter are given by feature vector 𝑙(⋅, 𝑦) and function 𝑔, respectively.

 Score matching method gives a tractable estimator for kernel exponential family.

– No need of computing normalization constants. – The estimator is given as a solution to a linear equation. – Non-normalized density function is estimated nonparametrically.

Thank you.

Reference

• B. Sriperumbudur, K. Fukumizu, R. Kumar, A. Gretton, and A.
• Hyvarinen. Density Estimation in Infinite Dimensional Exponential
• Families. arXiv:1312.3516.

27