Kenji Fukumizu The Institute of Statistical Mathematics / Graduate - - PowerPoint PPT Presentation
Kernel Methods for Statistical Learning Kenji Fukumizu The Institute of Statistical Mathematics / Graduate University for Advanced Studies September 6-7, 2012 Machine Learning Summer School 2012, Kyoto Version 2012.09.04 The latest version of
1 1
The Institute of Statistical Mathematics / Graduate University for Advanced Studies
September 6-7, 2012
Machine Learning Summer School 2012, Kyoto Version 2012.09.04 The latest version of slides is downloadable at http://www.ism.ac.jp/~ fukumizu/MLSS2012/
kernel PCA, kernel CCA, kernel ridge regression, etc
A brief introduction to SVM
Mathematical aspects of positive definite kernels
Recent advances of kernel methods
2
each section)
– Schölkopf, B. and A. Smola. Learning with Kernels. MIT Press. 2002. – Lecture slides (more detailed than this course) This page contains Japanese information, but the slides are written in English. Slides: 1, 2, 3, 4, 5, 6, 7, 8 – For Japanese only (Sorry!):
－正定値カーネルによるデータ解析」 朝倉書店（2010）
―非線形データ解析の新しい展開」 岩波書店（2008）
3
1
The Institute of Statistical Mathematics / Graduate University for Advanced Studies September 6-7 Machine Learning Summer School 2012, Kyoto
I-1
2
I-2
3
I-3
– Analysis of data is a process of inspecting, cleaning, transforming, and modeling data with the goal of highlighting useful information, suggesting conclusions, and supporting decision making. –Wikipedia
I-4
– ‘Table’ of numbers Matrix expression – Linear algebra is used for methods of analysis
etc.
) ( ) ( 1 ) 2 ( ) 2 ( 1 ) 1 ( ) 1 ( 1 N m N m m
m dimensional, N data
I-5
PCA: project data onto the subspace with largest variance. 1st direction =
1 || ||
a
N i N j j i T T
1 2 1 ) ( ) (
XX T
N i T N j j i N j j i
X N X X N X N
1 1 ) ( ) ( 1 ) ( ) (
1 1 1
XX
(Empirical) covariance matrix of where
I-6
– 1st principal direction – p-th principal direction = PCA Eigenproblem of covariance matrix
XX T a 1 || ||
1
unit eigenvector w.r.t. the largest eigenvalue of
unit eigenvector w.r.t. the p-th largest eigenvalue of
I-7
– Binary classification Find a linear classifier so that – Example: Fisher’s linear discriminant analyzer, Linear SVM, etc.
) ( ) ( 1 ) 2 ( ) 2 ( 1 ) 1 ( ) 1 ( 1 N m N m m
X X X X X X X
Input data
N N
Y Y Y Y } 1 {
) ( ) 2 ( ) 1 (
Class label
T
) ( ) ( )
i i
for all (or most) i.
I-8
2 4 6
2 4 6
x1 x2
Watch the following movie! http://jp.youtube.com/watch?v=3liCbRZPrZA linearly inseparable
5 10 15 20 5 10 15 20
5 10 15
z1 z3 z2
transform
2 1 2 2 2 1 3 2 1
linearly separable
I-9
] [ ] [ ] , [ Y Var X Var Y X Cov
XY
ρ
2 2
] [ ] [ ] [ ] [ Y E Y E X E X E Y E Y X E X E
1 2 3
1 2 3
X Y ρ = 0.94
I-10
0.5 1 1.5 2 2.5
0.5 1 1.5 2 2.5
0.5 1 1.5
0.5 1 1.5 2 2.5
X Y ρ(X, Y ) = 0.17 ρ(X2,Y ) = 0.96 transform (X, Y) (X2, Y)
I-11
Analysis of data is a process of inspecting, cleaning, transforming,
and modeling data with the goal of highlighting useful information, suggesting conclusions, and supporting decision making. – Wikipedia. Kernel method = a systematic way of transforming data into a high- dimensional feature space to extract nonlinearity or higher-order moments of data.
I-12
I-13
– Idea of kernel method – What kind of space is appropriate as a feature space?
many linear methods.
Feature space xi Hk xｊ
,
i
x
j
x
Space of original data feature map Do linear analysis in the feature space! e.g. SVM
I-14
– For example, how about using power series expansion? (X, Y, Z) (X, Y, Z, X2, Y2, Z2, XY, YZ, ZX, …) – But, many recent data are high-dimensional. e.g. microarray, images, etc... The above expansion is intractable! e.g. Up to 2nd moments, 10,000 dimension: Dim of feature space:
10000C1 + 10000C2 = 50,005,000 (!)
– Need a cleverer way Kernel method.
I-15
– Feature map: from original space to feature space – With special choice of feature space, we have a function (positive definite kernel) k(x, y) such that – Many linear methods use only the inner products of data, and do not need the explicit form of the vector Φ. (e.g. PCA)
j i j i
kernel trick.
1 1 n n
I-16
Definition.
Ω: set. : Ω Ω → is a positive definite kernel if
1) (symmetry) 2) (positivity) for arbitrary x1, …, xn ∈ Ω
) , (
1 ,
n j i j i j i
x x k c c ) , ( ) , ( ) , ( ) , (
1 1 1 1 n n n n
x x k x x k x x k x x k
is positive semidefinite, i.e., for any
i
(Gram matrix)
I-17
Examples: positive definite kernels on Rm (proof is give in Section IV)
2 2
d T P
) , ( N d c
) (
m i i i L
y x y x k
1
| | exp ) , ( ) (
T
I-18
Proposition 1.1 Let be a vector space with inner product ⋅,⋅ and Φ: Ω → be a map (feature map). If : Ω Ω → is defined by then k(x,y) is necessarily positive definite. – Positive definiteness is necessary. – Proof)
I-19
– Positive definite kernel is sufficient. Theorem 1.2 (Moore-Aronszajn) For a positive definite kernel on Ω, there is a Hilbert space
(reproducing kernel Hilbert space, RKHS) that consists of
functions on Ω such that 1)
⋅, ∈ for any
2) span ⋅,
∈ Ω
is dense in 3) (reproducing property)
for any ∈ , ∈ Ω
* Hilbert space: vector space with inner product such that the topology is complete.
I-20
– Feature space = RKHS. Feature map:
– Kernel trick: by reproducing property
– Prepare only positive definite kernel: We do not need an explicit form of feature vector or feature space. All we need for kernel methods are kernel values ,
.
I-21
1
The Institute of Statistical Mathematics / Graduate University for Advanced Studies
September 6-7
Machine Learning Summer School 2012, Kyoto
1
II-2
II-3
– Linear method for dimension reduction of data – Project the data in the directions of large variance. 1st principal axis =
1 || ||
a
n i n j j i T T
1 2 1
XX T
n i T n j j i n j j i
X n X X n X n
1 1 1
1 1 1
XX
where
II-4
– Kernel PCA: nonlinear dimension reduction of data (Schölkopf et al.
1998).
– Do PCA in feature space
1 || ||
H
f
1 || || a
II-5
n i n s s i T T
1 2 1
n i n s s i
X n X f n X f
1 2 1
) ( 1 ) ( , 1 ] ) ( , [ Var
It is sufficient to assume
Orthogonal directions to the data can be neglected, since for
𝑔 = ∑ 𝑑𝑗(Φ 𝑌𝑗 −
1 𝑜 ∑
Φ 𝑌𝑡 ) + 𝑔
⊥ 𝑜 𝑡=1 𝑜 𝑗=1
, where 𝑔
⊥ is orthogonal to
the span{Φ 𝑌𝑗 −
1 𝑜 ∑
Φ 𝑌𝑡 }
𝑜 𝑡=1 𝑗=1 𝑜
, the objective function of kernel PCA does not depend on 𝑔
⊥.
Then,
c K c X f
X T 2
~ ) ( , Var
, j i ij X
n s s n i i
1 1
with
(centered Gram matrix)
where
c K c f
X T H
~ || ||
2
(centered feature vector)
II-6
𝑔 = 𝑑𝑗 Φ 𝑌𝑗 − 1 𝑜 Φ 𝑌𝑡
𝑜 𝑡=1 𝑜 𝑗=1
[Exercise]
X T 2
subject to
X T
n s s i j i ij X
X X k n X X k K
1
) , ( 1 ) , ( ~
N s t s t n t j t
X X k n X X k n
1 , 2 1
) , ( 1 ) , ( 1
The centered Gram matrix 𝐿
𝑌 is expressed with Gram matrix
𝐿𝑌 = 𝑙 𝑌𝑗, 𝑌
𝑘 𝑗𝑘 as
𝐿 𝑌 = 𝐽𝑜 − 1 𝑜 𝟐𝑜𝟐𝑜
𝑈 𝑈
𝐿 𝐽𝑜 − 1 𝑜 𝟐𝑜𝟐𝑜
𝑈
𝟐𝑜 = 1 ⋮ 1 ∈ 𝐒𝑜
[Exercise]
II-7
𝐽𝑜 = Unit matrix
– Kernel PCA can be solved by eigen-decomposition. – Kernel PCA algorithm
X
N i T i i i X
1
2 1
N
eigenvalues unit eigenvectors
N
2 1
X
i T p p i
II-8
– Consider feature vectors with kernels. – Linear method is applied to the feature vectors. (Kernelization) – Typically, only the inner products are used to express the objective function of the method. – The solution is given in the form (representer theorem, see Sec.IV), and thus everything is written by Gram matrices. These are common in derivation of any kernel methods.
) , ( ) ( ), (
j i j i
X X k X X ) ( ,
i
X f
, ) (
1
n i i i
X c f
II-9
13 dim. chemical measurements of for three types of wine. 178 data. Class labels are NOT used in PCA, but shown in the figures. First two principal components:
5
1 2 3 4
0.2 0.4 0.6
0.2 0.4 0.6
Linear PCA Kernel PCA (Gaussian kernel)
( = 3)
II-10
0.2 0.4 0.6
0.2 0.4 0.6 = 4
0.1 0.2 0.3 0.4
0.1 0.2 0.3 0.4 0.5
Kernel PCA (Gaussian)
0.2 0.4 0.6
0.2 0.4 0.6
= 5 = 2
2 2
II-11
– PCA can be used for noise reduction (principal directions represent signal, other directions noise). – Apply kernel PCA for noise reduction:
Gx: Projection ofx onto Vd = noise reduced feacure vector.
data sapce for the noise reduced feature vector Gx.
2
x x
′
5 10 15 20
20
2 4 6 8 10
(x) Gx
Note: Gxis not necessariy
given as an image of .
II-12
Noise reduced images（linear PCA） Noisy images Noise reduced images （kernel PCA, Gaussian kernel） Original data (NOT used for PCA) Generated by Matlab stprtool (by V. Franc)
II-13
– Nonlinear features can be extracted. – Can be used for a preprocessing of other analysis like
– The results depend on the choice of kernel and kernel parameters. Interpreting the results may not be straightforward. – How to choose a kernel and kernel parameter?
should be maximized.
II-14
II-15
– Canonical correlation analysis (CCA, Hotelling 1936) Linear dependence of two multivariate random vectors.
1 , … , (𝑌𝑂, 𝑍 𝑂)
𝑗: ℓ-dimensional
Find the directions 𝑏 and 𝑐 so that the correlation of 𝑏𝑈𝑌 and
𝑐𝑈𝑍 is maximized.
, ] [ ] [ ] , [ max ] , [ max
, ,
Y b X a Y b X a Y b X a
T T T T b a T T b a
Var Var Cov Corr ρ
X Y a aTX bTY b
II-16
– Rewritten as a generalized eigenproblem: [Exercise: derive this. (Hint. Use Lagrange multiplier method.)] – Solution:
𝑏,𝑐 𝑏𝑈𝑊
𝑃 𝑊 𝑌𝑌 𝑊 𝑌𝑌 𝑃 𝑏 𝑐 = 𝜍 𝑊 𝑌𝑌 𝑃 𝑃 𝑊 𝑌𝑌 𝑏 𝑐 𝑏 = 𝑊
𝑌𝑌 1/2𝑣1, 𝑐 = 𝑊 𝑌𝑌 1/2𝑤1
where 𝑣1 (𝑤1, resp.) is the left (right, resp.) first eigenvector for the SVD of 𝑊
−1/2𝑊
𝑌𝑌𝑊
−1/2.
II-17
– Kernel CCA (Akaho 2000, Melzer et al. 2002, Bach et al 2002)
1 , … , (𝑌𝑂, 𝑍 𝑂) arbitrary variables
𝑌1, … , 𝑌𝑂 ↦ Φ𝑌 𝑌1 , … , , Φ𝑌 𝑌𝑂 ∈ 𝐼𝑌, 𝑍
1, … , 𝑍 𝑂 ↦ Φ𝑌 𝑍 1 , … , Φ𝑌 𝑍 𝑂 ∈ 𝐼𝑌.
X Y f x(X)
) (X f
x y(Y) y g
) (Y g
max
𝑔∈𝐼𝑌,∈𝐼𝑍
Cov[𝑔 𝑌 , 𝑍 ] Var 𝑔 𝑌 Var[ 𝑍 ] = max
𝑔∈𝐼𝑌,∈𝐼𝑍
∑ 𝑔, Φ 𝑌 𝑌𝑗 〈Φ 𝑌 𝑍
𝑗 , 〉 𝑂 𝑗
∑ 𝑔, Φ 𝑌 𝑌𝑗
2 𝑂 𝑗
∑ , Φ 𝑌 𝑍
𝑗 2 𝑂 𝑗
II-18
– We can assume 𝑔 = ∑𝑗=1
𝑂 𝛽𝑗Φ
𝑌(𝑌𝑗) and = ∑𝑗=1
𝑂 𝛾𝑗Φ
𝑌(𝑍
𝑗).
max𝛽∈R𝑂,𝛾∈𝑆𝑂 𝛽𝑈 𝐿 𝑌𝐿 𝑌𝛾 𝛽𝑈𝐿 𝑌
2𝛽 𝛾𝑈𝐿
𝑌
2𝛾
– Regularization: to avoid trivial solution, – Solution: generalized eigenproblem
(same as kernel PCA)
max
𝑔∈𝐼𝑌,∈𝐼𝑍
∑ 𝑔, Φ 𝑌 𝑌𝑗 〈Φ 𝑌 𝑍
𝑗 , 〉 𝑂 𝑗=1
∑ 𝑔, Φ 𝑌 𝑌𝑗
2 𝑂 𝑗=1
+ 𝜁𝑂 𝑔 𝐼𝑌
2 ∑
, Φ 𝑌 𝑍
𝑗 2 +𝜁𝑂
𝐼𝑍
2 𝑂 𝑗=1
II-19
𝐿 𝑌 and 𝐿 𝑌 : centered
Gram matrices.
𝑃 𝐿 𝑌𝐿 𝑌 𝐿 𝑌𝐿 𝑌 𝑃 𝛽 𝛾 = 𝜍 𝐿 𝑌
2 + 𝜁𝑂𝐿
𝑌 𝑃 𝑃 𝐿 𝑌
2 + 𝜁𝑂𝐿
𝑌 𝛽 𝛾
– Application to image retrieval (Hardoon, et al. 2004).
Yi: corresponding texts (extracted from the same webpages).
spaces which contain the dependence between X and Y.
𝑜𝑜𝑥, compute its feature vector, and find
the image whose feature has the highest inner product.
Xi Y x(Xi) x y(Y) y g ) ( , ) ( ,
1
Y g Y g
y d y
) ( , ) ( ,
1 i x d i x
X f X f
2s7, n907wa, ...”
II-20
f
– Example:
Text -- “height: 6-11, weight: 235 lbs, position: forward, born: september 18, 1968, split, croatia college: none” Extracted images
II-21
II-22
𝑌1, 𝑍
1 , … , 𝑌𝑜, 𝑍 𝑜 : data (𝑌𝑗 ∈ 𝐒𝑛, 𝑍 𝑗 ∈ 𝐒)
Ridge regression: linear regression with 𝑀2 penalty.
min
𝑏
|𝑍
𝑗 2 − 𝑏𝑈𝑌𝑗|2 𝑜 𝑗=1
+ 𝜇 𝑏 2
– Solution (quadratic function):
𝑏 = 𝑊
𝑌𝑌 + 𝜇𝐽𝑜 −1𝑌𝑈𝑍
– Ridge regression is preferred, when 𝑊
𝑌𝑌 is (close to) singular.
(The constant term is omitted for simplicity)
𝑊
𝑌𝑌 = 1 𝑜 𝑌𝑈𝑌, 𝑌 =
𝑌1
𝑈
⋮ 𝑌𝑜
𝑈
∈ 𝐒𝑜×𝑛, 𝑍 = 𝑍
1
⋮ 𝑍
𝑜
∈ 𝐒𝑜
where
II-23
– 𝑌1, 𝑍
1 , … , 𝑌𝑜, 𝑍 𝑜 : 𝑌 arbitrary data, 𝑍 ∈ 𝐒.
– Kernelization of ridge regression: positive definite kernel 𝑙 for 𝑌
min
𝑔∈𝐼 |𝑍 𝑗 − 𝑔, Φ 𝑌𝑗 𝐼|2 𝑜 𝑗=1
+ 𝜇 𝑔 𝐼
2
min
𝑔∈𝐼 |𝑍 𝑗 − 𝑔(𝑌𝑗)|2 𝑜 𝑗=1
+ 𝜇 𝑔 𝐼
2
equivalently,
Ridge regression on 𝐼 Nonlinear ridge regr.
II-24
– Solution is given by the form
Let 𝑔 = ∑
𝑑𝑗Φ 𝑌𝑗 + 𝑔
⊥ = 𝑔 Φ + 𝑔 ⊥ 𝑜 𝑗=1
(𝑔
Φ ∈ 𝑇𝑇𝑏𝑜 Φ 𝑌𝑗 𝑗=1 𝑜
, 𝑔
⊥: orthogonal complement)
Objective function = ∑
𝑍
𝑗 − 𝑔 Φ + 𝑔 ⊥, Φ 𝑌𝑗 2 + 𝜇 𝑔 Φ + 𝑔 ⊥ 2 𝑜 𝑗=1
= ∑
𝑍
𝑗 − 𝑔 Φ, Φ 𝑌𝑗 2 + 𝜇 ( 𝑔 Φ 2+ 𝑔 ⊥ 2 𝑜 𝑗=1
) The 1st term does not depend on 𝑔
⊥, and 2nd term is minimized in the
case 𝑔
⊥ = 0.
– Objective function :
– Solution: 𝑑̂ = 𝐿𝑌 + 𝜇𝐽𝑜 −1𝑍 Function: 𝑔
̂ 𝑦 = 𝑍𝑈 𝐿𝑌 + 𝜇𝐽𝑜 −1𝐥 𝑦
, ) (
1
n j j j
X c f
II-25
𝐥 𝑦 = 𝑙 𝑦, 𝑌1 ⋮ 𝑙 𝑦, 𝑌𝑜
– The minimization min
𝑔
𝑗 − 𝑔 𝑌𝑗 2
may be attained with zero errors. But the function may not be unique. – Regularization min
𝑔∈𝐼 ∑
𝑗 − 𝑔(𝑌𝑗)|2 𝑜 𝑗=1
2
penalty is preferred for uniqueness and smoothness.
smoothness is discussed in Sec. IV.
II-26
𝑍 = 1/ 1.5 + | 𝑌| 2 + 𝑎, 𝑌 ~ 𝑂 0, 𝐽𝑒 , 𝑎~𝑂 0, 0.12
𝑜 = 100, 500 runs
Kernel ridge regression with Gaussian kernel Local linear regression with Epanechnikov kernel (‘locfit’ in R is used) Bandwidth parameters are chosen by CV.
II-27
1 10 100 0.002 0.004 0.006 0.008 0.01 0.012 Dimension of X Mean square errors Kernel method Local linear
– 𝐿: smoothing kernel (𝐿 𝑦 ≥ 0, ∫ 𝐿 𝑦 𝑒𝑦 = 1, not necessarily
positive definite) – Local linear regression
𝐹 𝑍 𝑌 = 𝑦0 is estimated by
min
𝑏,𝑐 𝑍 𝑗 − 𝑏 − 𝑐𝑈(𝑌𝑗−𝑦0) 2 𝑜 𝑗
𝐿ℎ(𝑌𝑗 − 𝑦0) 𝐿ℎ(𝑦) = ℎ−𝑒𝐿
𝑦 ℎ
II-28
II-29
𝑌1, 𝑍
1 , … , 𝑌𝑜, 𝑍 𝑜 : data
𝑙: positive definite kernel for 𝑌, 𝐼: corresponding RKHS. Ψ: monotonically increasing function on 𝑆+.
Theorem 2.1 (representer theorem, Kimeldorf & Wahba 1970) The solution to the minimization problem: min
𝑔∈𝐼 𝐺 (𝑌1, 𝑍 1, 𝑔 𝑌1
, … , (𝑌𝑜, 𝑍
𝑜, 𝑔 𝑌𝑜 )) + Ψ(| 𝑔 |)
is attained by
𝑔 = ∑ 𝑑𝑗Φ 𝑌𝑗
𝑜 𝑗=1
with some 𝑑1, … , 𝑑𝑜 ∈ 𝑆𝑜 . The proof is essentially the same as the one for the kernel ridge
II-30
– Structured data: non-vectorial data with some structure.
DNA sequence, Protein (sequence of amino acid) Text (sequence of words)
Chemical compound etc.
Parse tree
measures – Many kernels uses counts of substructures (Haussler 1999).
S NP VP Det N The cat chased the mouse. V Det N NP
II-31
– p-spectrum kernel (Leslie et al 2002): positive definite kernel for string.
𝑙𝑞 𝑡, 𝑢 = Occurrences of common subsequences of length p.
– Example: s = “statistics” t = “pastapistan” 3-spectrum
s: sta, tat, ati, tis, ist, sti, tic, ics
t : pas, ast, sta, tap, api, pis, ist, sta, tan
K3(s, t) ＝ 1・2 + 1・1 = 3
– Linear time (𝑃(𝑇(|𝑇| + |𝑢|) ) algorithm with suffix tree is known
(Vishwanathan & Smola 2003). sta tat ati tis ist sti tic ics pas ast tap api pis tan
(s) 1 1 1 1 1 1 1 1 (t) 2 1 1 1 1 1 1 1
II-32
– Application: kernel PCA of ‘words’ with 3-spectrum kernel
2 4 6 8
2 4 6 8 mathematics engineering pioneering cybernetics physics psychology metaphysics methodology biology biostatistics statistics informatics bioinformatics II-33
– Choice of kernel (polyn or Gauss) – Choice of parameters (bandwidth parameter in Gaussian kernel)
– Reflect the structure of data (e.g., kernels for structured data) – For supervised learning (e.g., SVM) Cross-validation – For unsupervised learning (e.g. kernel PCA)
CV. – Learning a kernel: Multiple kernel learning (MKL)
𝑙 𝑦, 𝑧 = ∑ 𝑑𝑗𝑙𝑗(𝑦, 𝑧)
𝑁 𝑗=1
II-34
– Gram matrix: 𝑜 × 𝑜 where 𝑜 is the sample size. Large 𝑜 causes computational problems. e.g. Inversion, eigendecomposition costs 𝑃(𝑜3) in time. – Low-rank approximation
– The decay of eigenvalues of a Gram matrix is often quite fast
(See Widom 1963, 1964; Bach & Jordan 2002).
II-35
𝐿
𝑆 𝑆
𝑠 𝑜 𝑠 𝑜 𝑜 𝑜
– Two major methods
𝑃(𝑜𝑠2) in time and 𝑃(𝑜𝑠) in space
Random sampling + eigendecomposition – Example: kernel ridge regression
𝑍𝑈 𝐿𝑌 + 𝜇𝐽𝑜 −1𝐥 𝑦
time : 𝑃(𝑜3) Low rank approximation: 𝐿𝑌 ≈ 𝑆𝑆𝑈. With Woodbury formula*
𝑍𝑈 𝐿𝑌 + 𝜇𝐽𝑜 −1𝐥 𝑦 ≈ 𝑍𝑈 𝑆𝑆𝑈 + 𝜇𝐽𝑜 −1𝐥 𝑦
=
1 𝜇 𝑍𝑈𝐥 𝑦 − 𝑍𝑈𝑆 𝑆𝑈𝑆 + 𝜇𝐽𝑠 −1𝑆𝑈𝐥 𝑦
time : 𝑃(𝑠2𝑜 + 𝑠3) * Woodbury (Sherman–Morrison–Woodbury) formula:
𝐵 + 𝑉𝑊 −1 = 𝐵−1 − 𝐵−1𝑉 𝐽 + 𝑊𝐵−1𝑉 −1𝑊𝐵−1.
II-36
– Kernel Fisher discriminant analysis （kernel FDA） (Mika et al. 1999) – Kernel logistic regression （Roth 2001, Zhu&Hastie 2005） – Kernel partial least square（kernel PLS）（Rosipal&Trejo 2001） – Kernel K-means clustering（Dhillon et al 2004） etc, etc, ... – Variants of SVM Section III.
II-37
– Various classical linear methods can be kernelized
Linear algorithms on RKHS.
– The solution typically has the form – The problem is reduced to manipulation of Gram matrices of size n (sample size).
effectively. – Structured data:
. ) (
1
n i i i
X c f
(representer theorem)
II-38
Induction-based Information Sciences (IBIS2000). (in Japanese) Bach, F.R. and M.I. Jordan. (2002) Kernel independent component analysis. Journal of Machine Learning Research, 3:1–48. Dhillon, I. S., Y. Guan, and B. Kulis. (2004) Kernel k-means, spectral clustering and normalized cuts. Proc. 10th ACM SIGKDD Intern. Conf. Knowledge Discovery and Data Mining (KDD), 551–556. Fan, J. and I. Gijbels. Local Polynomial Modelling and Its Applications. Chapman Hall/CRC, 1996. Fine, S. and K. Scheinberg. (2001) Efficient SVM Training Using Low-Rank Kernel
Gökhan, B., T. Hofmann, B. Schölkopf, A.J. Smola, B. Taskar, S.V.N. Vishwanathan. (2007) Predicting Structured Data. MIT Press. Hardoon, D.R., S. Szedmak, and J. Shawe-Taylor. (2004) Canonical correlation analysis: An overview with application to learning methods. Neural Computation, 16:2639– 2664. Haussler, D. Convolution kernels on discrete structures. Tech Report UCSC-CRL-99-10, Department of Computer Science, University of California at Santa Cruz, 1999.
II-39
Leslie, C., E. Eskin, and W.S. Noble. (2002) The spectrum kernel: A string kernel for SVM protein classification, in Proc. Pacific Symposium on Biocomputing, 564–575. Melzer, T., M. Reiter, and H. Bischof. (2001) Nonlinear feature extraction using generalized canonical correlation analysis. Proc. Intern. Conf. .Artificial Neural Networks (ICANN 2001), 353–360. Mika, S., G. Rätsch, J. Weston, B. Schölkopf, and K.-R. Müller. (1999) Fisher discriminant analysis with kernels. In Y.-H. Hu, J. Larsen, E. Wilson, and S. Douglas, edits, Neural Networks for Signal Processing, volume IX, 41–48. IEEE. Rosipal, R. and L.J. Trejo. (2001) Kernel partial least squares regression in reproducing kernel Hilbert space. Journal of Machine Learning Research, 2: 97–123. Roth, V. (2001) Probabilistic discriminative kernel classifiers for multi-class problems. In Pattern Recognition: Proc. 23rd DAGM Symposium, 246–253. Springer. Schölkopf, B., A. Smola, and K-R. Müller. (1998) Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10:1299–1319. Schölkopf, B. and A. Smola. Learning with Kernels. MIT Press. 2002. Vishwanathan, S. V. N. and A.J. Smola. (2003) Fast kernels for string and tree matching. Advances in Neural Information Processing Systems 15, 569–576. MIT Press. Williams, C. K. I. and M. Seeger. (2001) Using the Nyström method to speed up kernel
II-40
Widom, H. (1963) Asymptotic behavior of the eigenvalues of certain integral equations. Transactions of the American Mathematical Society, 109:278{295, 1963. Widom, H. (1964) Asymptotic behavior of the eigenvalues of certain integral equations
II-41
II-42
– –
II-43
c K c f
X T H
~ || ||
2
𝑔 𝐼
2 = 𝑑𝑗Φ
𝑌𝑗
𝑜 𝑗=1
, 𝑑
𝑘Φ
𝑌
𝑘 𝑜 𝑘=1
= 𝑑𝑗𝑑
𝑘 Φ
𝑌𝑗 , Φ 𝑌
𝑘 𝑜 𝑗
= 𝑑𝑈𝐿 𝑌𝑑.
c K c X f
X T 2
~ ) ( , Var
Var 𝑔, Φ 𝑌 = 1 𝑜 𝑑𝑗Φ 𝑌𝑗
𝑜 𝑗=1
, Φ 𝑌
𝑘 2 𝑜 𝑘=1
= 1 𝑜 𝑑𝑗Φ 𝑌𝑗
𝑜 𝑗=1
, Φ 𝑌
𝑘
𝑑ℎΦ 𝑌ℎ
𝑜 ℎ=1
, Φ 𝑌
𝑘 𝑜 𝑘=1
= 1 𝑜 𝑑𝑗𝐿 𝑗𝑘
𝑜 𝑗
𝑑ℎ𝐿 ℎ𝑘
𝑜 ℎℎ 𝑜 𝑘=1
= 1 𝑜 𝑑𝑈𝐿 𝑌
2𝑑
∎
1
The Institute of Statistical Mathematics / Graduate University for Advanced Studies
September 6-7
Machine Learning Summer School 2012, Kyoto
1
– 𝑌1, 𝑍
1 , … , (𝑌𝑜, 𝑍 𝑜): training data
𝑗 ∈ {±1}: binary teaching data,
– Linear classifier
𝑔
𝑥(𝑦) = 𝑥𝑈𝑦 + 𝑐
ℎ 𝑦 = sgn 𝑔
𝑥 𝑦
We wish to make 𝑔
𝑥(𝑦) with
the training data so that a new data 𝑦 can be correctly classified.
y = fw(x) fw(x)≧0 fw(x) < 0
III-2
2 4 6 8
2 4 6 8
Assumption: the data is linearly separable. Among infinite number of separating hyperplanes, choose the one to give the largest margin. – Margin = distance of two classes measured along the direction of 𝑥. – Support vector machine: Hyperplane to give the largest margin. – The classifier is the middle of the margin. – “Supporting points”
called support vectors.
𝑥
margin support vector
III-3
– Fix the scale (rescaling of (𝑥, 𝑐) does not change the plane) Then Margin =
2 𝑥
[Exercise] Prove this.
1 ) max( 1 ) min( b X w b X w
i T i T
if Yi = 1, if Yi = − 1
III-4
(Boser, Guyon, Vapnik 1992)
Objective function:
max
𝑥,𝑐 1 𝑥 subject to 𝑥𝑈𝑌𝑗 + 𝑐 ≥ 1 if 𝑍 𝑗 = 1,
𝑥𝑈𝑌𝑗 + 𝑐 ≥ 1 if 𝑍
𝑗 = −1.
– Quadratic program (QP):
subject to 𝑍
𝑗(𝑥𝑈𝑌𝑗 + 𝑐) ≥ 1 (∀𝑗)
min
𝑥,𝑐 𝑥 2
SVM (hard margin)
III-5
– “Linear separability” is too strong. Relax it. – This is also QP. – The coefficient C must be given. Cross-validation is often used. Hard margin Soft margin
𝑍
𝑗(𝑥𝑈𝑌𝑗 + 𝑐) ≥ 1
𝑍
𝑗(𝑥𝑈𝑌𝑗 + 𝑐) ≥ 1 − 𝜊𝑗,
𝜊𝑗 ≥ 0 𝑍
𝑗(𝑥𝑈𝑌𝑗 + 𝑐) ≥ 1 − 𝜊𝑗,
𝜊𝑗≥ 0 min
𝑥,𝑐 𝑥 2 + 𝐷 ∑
𝜊𝑗
𝑜 𝑗=1
SVM (soft margin) subject to (∀𝑗)
III-6
III-7
2 4 6 8
2 4 6 8
w 𝑥𝑈𝑦 + 𝑐 = 1 𝑥𝑈𝑦 + 𝑐 = −1 𝒙𝑼𝒀𝒋 + 𝒄 = 𝟐 − 𝝄𝒋 𝒙𝑼𝒀𝒌 + 𝒄 = −𝟐 + 𝝄𝒌
– Soft-margin SVM is equivalent to the regularization problem:
[Exercise] Confirm the above equivalence.
min
𝑥,𝑐 1 − 𝑍 𝑗 𝑥𝑈𝑌𝑗 + 𝑐 + + 𝜇 𝑥 2 𝑜 𝑗=1
loss regularization term 𝑨 + = max{0, 𝑨} ℓ 𝑔 𝑦 , 𝑧 = 1 − 𝑔 𝑦
+
III-8
min
𝑥,𝑐 ∑
𝑍
𝑗 − 𝑥𝑈𝑌𝑗 + 𝑐 2
+ 𝜇 𝑥 2
𝑜 𝑗=1
– 𝑌1, 𝑍
1 , … , (𝑌𝑜, 𝑍 𝑜): training data
𝑗 ∈ {±1}: binary teaching data,
– Kernelization: Positive definite kernel 𝑙 on Ω (RKHS 𝐼), Feature vectors Φ 𝑌1 , … , Φ 𝑌𝑜 – Linear classifier on 𝐼 nonlinear classifier on Ω
III-9
By representer theorem, 𝑔 = ∑
𝑥
𝑘Φ 𝑌 𝑘 𝑜 𝑘=1
.
min
𝑔,𝑐 1 − 𝑍 𝑗(𝑔 𝑌𝑗 + 𝑐) + + 𝜇 𝑔 𝐼 2 𝑜 𝑗=1
𝑍
𝑗(〈𝑔, Φ 𝑌𝑗 〉 + 𝑐) ≥ 1,
𝜊𝑗≥ 0 min
𝑔,𝑐 𝑔 2 + 𝐷 𝜊𝑗 𝑜 𝑗=1
subject to (∀𝑗)
𝑍
𝑗( 𝐿𝑥 𝑗 + 𝑐) ≥ 1,
𝜊𝑗≥ 0 min
𝑥,𝑐 𝑥𝑈𝐿𝑥 + 𝐷 𝜊𝑗 𝑜 𝑗=1
nonlinear SVM (soft margin) subject to (∀𝑗)
III-10
– By Lagrangian multiplier method, the dual problem is SVM (dual problem)
𝑔
∗ 𝑦 + 𝑐∗ = 𝛽𝑗∗𝑍 𝑗 𝑜 𝑗=1
𝐿 𝑦, 𝑌𝑗 + 𝑐∗
– Sparse expression: Only the data with 0 < 𝛽𝑗∗ ≤ 𝐷 appear in the summation. Support vectors.
𝑍
𝑗( 𝐿𝑥 𝑗 + 𝑐) ≥ 1,
𝜊𝑗≥ 0 min
𝑥,𝑐 𝑥𝑈𝐿𝑥 + 𝐷 ∑
𝜊𝑗
𝑜 𝑗=1
subject to (∀𝑗) 0 ≤ 𝛽𝑗 ≤ 𝐷, ∑ 𝑍
𝑗𝛽𝑗 𝑜 𝑗=1
= 0 max
𝛽
∑ 𝛽𝑗
𝑜 𝑗=1
− ∑ 𝛽𝑗𝛽𝑘𝑍
𝑗𝑍 𝑘𝐿𝑗𝑘 𝑜 𝑗,𝑘=1
subject to (∀𝑗)
III-11
Theorem The solution of the primal and dual problem of SVM is given by the following equations: (1) 1 − 𝑍
𝑗 𝑔∗ 𝑌𝑗 + 𝑐∗ − 𝜊𝑗 ∗ ≤ 0 (∀ 𝑗)
[primal constraint]
(2) 𝜊𝑗
∗ ≥ 0 (∀ 𝑗)
[primal constraint]
(3) 0 ≤ 𝛽𝑗
∗ ≤ 𝐷, (∀ 𝑗)
[dual constraint]
(4) 𝛽𝑗
∗(1 − 𝑍 𝑗 (𝑔∗ 𝑌𝑗 + 𝑐∗) − 𝜊𝑗 ∗) = 0 (∀ 𝑗)
[complementary]
(6) 𝜊𝑗
∗ 𝐷 − 𝛽𝑗 ∗ = 0 (∀ 𝑗),
[complementary]
(7) ∑𝑘=1
𝑜
𝐿𝑗𝑘𝑥
𝑘 ∗ − ∑ 𝛽𝑘 ∗𝑍 𝑘𝐿𝑗𝑘 𝑜 𝑘
= 0,
(8) ∑ 𝑘=1
𝑜
𝛽𝑘
∗𝑍 𝑘 = 0,
III-12
– Two types of support vectors.
2 4 6 8
2 4 6 8
w support vectors 0 < i < C (Yi 𝜒(Xi) = 1) support vectors i = C (Yi𝜒(Xi) 1)
𝜒∗(𝑦) = 𝑔
∗ 𝑦 + 𝑐∗ =
𝑗 𝑌𝑗: support vectors
𝐿 𝑦, 𝑌𝑗 + 𝑐∗
III-13
– One of the kernel methods:
– Quadratic program:
(e.g. SMO, Plat 1999) – Sparse representation
– Regularization
hinge loss function.
III-14
– NOT discussed on SVM in this lecture are
– Combination of binary classifiers (1-vs-1, 1-vs-rest) – Generalization of large margin criterion
Crammer & Singer (2001), Mangasarian & Musicant (2001), Lee, Lin, & Wahba (2004), etc
– Support vector regression (Vapnik 1995)
– 𝜉-SVM (Schölkopf et al 2000)
– Structured-output (Collins & Duffty 2001, Taskar et al 2004, Altun
et al 2003, etc)
– One-class SVM (Schökopf et al 2001)
– Solving primal problem
III-15
Altun, Y., I. Tsochantaridis, and T. Hofmann. Hidden Markov support vector machines. In Proc. 20th Intern. Conf. Machine Learning, 2003. Boser, B.E., I.M. Guyon, and V.N. Vapnik. A training algorithm for optimal margin
Learning Theory, pages 144–152, Pittsburgh, PA, 1992. ACM Press. Crammer, K. and Y. Singer. On the algorithmic implementation of multiclass kernel- based vector machines. Journal of Machine Learning Research, 2:265–292, 2001. Collins, M. and N. Duffy. Convolution kernels for natural language. In Advances in Neural Information Processing Systems 14, pages 625–632. MIT Press, 2001. Mangasarian, O. L. and David R. Musicant. Lagrangian support vector machines. Journal of Machine Learning Research, 1:161–177, 2001 Lee, Y., Y. Lin, and G. Wahba. Multicategory support vector machines, theory, and application to the classification of microarray data and satellite radiance data. Journal of the American Statistical Association, 99: 67–81, 2004. Schölkopf, B., A. Smola, R.C. Williamson, and P.L. Bartlett. (2000) New support vector algorithms. Neural Computation, 12(5):1207–1245.
III-16
Schölkopf, B., J.C. Platt, J. Shawe-Taylor, R.C. Williamson, and A.J.Smola. (2001) Estimating the support of a high-dimensional distribution. Neural Computation, 13(7):1443–1471. Vapnik, V.N. The Nature of Statistical Learning Theory. Springer 1995. Platt, J. Fast training of support vector machines using sequential minimal optimization. In
Support Vector Learning, pages 185–208. MIT Press, 1999. Books on Application domains: – Lee, S.-W., A. Verri (Eds.) Pattern Recognition with Support Vector Machines: First International Workshop, SVM 2002, Niagara Falls, Canada, August 10, 2002.
– Joachims, T. Learning to Classify Text Using Support Vector Machines: Methods, Theory and Algorithms. Springer, 2002. – Schölkopf, B., K. Tsuda, J.-P. Vert (Eds.). Kernel Methods in Computational
III-17
1
The Institute of Statistical Mathematics / Graduate University for Advanced Studies
September 6-7
Machine Learning Summer School 2012, Kyoto
1
Definition.
: set. is a positive definite kernel if for
arbitrary 𝑦1, … , 𝑦𝑜 ∈ Ω and 𝑑1, … , 𝑑𝑜 ∈ 𝐃, Remark: From the above condition, the Gram matrix 𝑙 𝑦𝑗, 𝑦𝑘
𝑗𝑘is
necessarily Hermitian, i.e. 𝑙 𝑧, 𝑦 = 𝑙(𝑦, 𝑧). [Exercise]
. ) , (
1 ,
n j i j i j i
x x k c c
IV-2
Proposition 4.1 If 𝑙𝑗: 𝑌 × 𝑌 → 𝐃 (𝑗 = 1,2, … ,) are positive definite kernels, then so are the following: 1. (positive combination) 𝑏𝑙1 + 𝑐𝑙2 (𝑏, 𝑐 ≥ 0). 2. (product) 𝑙1𝑙2 𝑙1 𝑦, 𝑧 𝑙2 𝑦, 𝑧 3. (limit) lim
𝑗→∞ 𝑙𝑗(𝑦, 𝑧), assuming the limit exists.
prove that Hadamard product (element-wise) of two positive semidefinite matrices is positive semidefinite. Remark: The set of positive definite kernels on 𝑌 is a closed cone, where the topology is defined by the point-wise convergence.
IV-3
Proposition 4.2 Let 𝐵 and 𝐶 be positive semidefinite Hermitian matrices. Then, Hadamard product 𝐿 = 𝐵 ∗ 𝐶 (element-wise product) is positive semidefinite. Proof) Eigendecomposition of 𝐵: 𝐵 = 𝑉Λ𝑉
𝑈 =
i.e., Then,
n p j p i p p ij
U U A
1λ
（λp ≧ 0 by the positive semidefiniteness）.
n j i ij j i
K c c
1 ,
.
1 , 1 , 1 1 1
n j i ij j n j i n i n n j i ij j j i i
B U c U c B U c U c λ λ
n p n j i ij j p i p p j i
B U U c c
1 1 ,
λ
IV-4
𝑉𝑞
𝑗
𝜇1 0 ⋯ 0 0 𝜇2 ⋯ ⋮ ⋱ ⋮ 0 ⋯ 0 𝜇𝑜 𝑉𝑞
𝑘 𝑈
Proposition 4.3 Let 𝑙 be a positive definite kernel on Ω, and 𝑔: Ω → 𝐃 be an arbitrary function. Then,
𝑙 𝑦, 𝑧 : = 𝑔 𝑦 𝑙 𝑦, 𝑧 𝑔(𝑧)
is positive definite. In particular,
𝑔 𝑦 𝑔(𝑧)
is a positive definite kernel. – Proof [Exercise] – Example. Normalization: is positive definite, and Φ(𝑦)
= 1 for any 𝑦 ∈ Ω. 𝑙 𝑦, 𝑧 = 𝑙 𝑦, 𝑧 𝑙(𝑦, 𝑦)𝑙(𝑧, 𝑧)
IV-5
– Euclidean inner product 𝑦𝑈𝑧: easy (Prop. 1.1). – Polynomial kernel 𝑦𝑈𝑧 + 𝑑 𝑒 (𝑑 ≥ 0):
𝑦𝑈𝑧 + 𝑑 𝑒 = 𝑦𝑈𝑧 𝑒 + 𝑏1 𝑦𝑈𝑧 𝑒−1 + ⋯ + 𝑏𝑒, 𝑏𝑗 ≥ 0.
Product and non-negative combination of p.d. kernels. – Gaussian RBF kernel exp −
𝑦−𝑧 2 𝜏2
:
exp − 𝑦 − 𝑧 2 𝜏2 = 𝑓− 𝑦 2/𝜏2𝑓𝑦𝑈𝑧/𝜏2 𝑓− 𝑧 2/𝜏2
Note
𝑓𝑦𝑈𝑧/𝜏2 = 1 + 1 1! 𝜏2 𝑦𝑈𝑧 + 1 2! 𝜏4 𝑦𝑈𝑧 2 + ⋯
is positive definite (Prop. 4.1). Proposition 4.3 then completes the proof. – Laplacian kernel is discussed later.
IV-6
– A positive definite kernel 𝑙 𝑦, 𝑧 on 𝐒𝑛 is called shift-invariant if the kernel is of the form 𝑙 𝑦, 𝑧 = 𝜔 𝑦 − 𝑧 . – Examples: Gaussian, Laplacian kernel – Fourier kernel (C-valued positive definite kernel): for each 𝜕,
𝑙𝐺 𝑦, 𝑧 = exp −1𝜕𝑈 𝑦 − 𝑧 = exp −1𝜕𝑈𝑦 exp −1𝜕𝑈𝑧
(Prop. 4.3) – If 𝑙 𝑦, 𝑧 = 𝜔 𝑦 − 𝑧 is positive definite, the function 𝜔 is called positive definite.
IV-7
Theorem 4.4 (Bochner) Let 𝜔 be a continuous function on 𝐒𝑛. Then, 𝜔 is (𝐃-valued) positive definite if and only if there is a finite non-negative Borel measure Λ on 𝐒𝑛 such that
𝜔 𝑨 = exp −1𝜕𝑈𝑨 𝑒Λ(𝜕)
Bochner’s theorem characterizes all the continuous shift-invariant positive definite kernels. exp
−1𝜕𝑈𝑨 𝜕 ∈ 𝐒𝑛 is the generator
– Λ is the inverse Fourier (or Fourier-Stieltjes) transform of 𝜔.
– Roughly speaking, the shift invariant functions are the class that have non-negative Fourier transform. – Sufficiency is easy: ∑
𝑑𝑗𝑑
𝑘
𝜔 𝑨𝑗 − 𝑨
𝑘 = ∫ | ∑ 𝑑𝑗𝑓 −1𝜕𝑈𝑨𝑗 |2𝑒Λ 𝜕 𝑗
.
Necessity is difficult.
IV-8
Suppose (shift-invariant) kernel 𝑙 has a form
𝑙 𝑦, 𝑧 = exp −1𝜕𝑈 𝑦 − 𝑧 𝜍 𝜕 𝑒𝜕 .
Then, RKHS 𝐼𝑙 is given by
𝐼𝑙 = 𝑔 ∈ 𝑀2 𝐒, 𝑒𝑦 𝑔 ̂ 𝜕
2
𝜍 𝜕 𝑒𝜕 < ∞ , 𝑔, = 𝑔 ̂ 𝜕 𝜕 𝜍 𝜕 𝑒𝜕
where 𝑔
̂ is the Fourier transform of 𝑔: 𝑔 ̂(𝜕) =
1 2𝜌 𝑛 ∫ 𝑔(𝑦)exp − −1𝜕𝑈𝑦 𝑒𝜕 .
𝜍 𝜕 > 0.
IV-9
𝑙𝐻 𝑦, 𝑧 = exp − 𝑦 − 𝑧 2 2𝜏2 , 𝜍𝐻 𝜕 = 1 2𝜌 𝑛 exp − 𝜏2 𝜕 2 2 𝐼𝑙𝐻 = 𝑔 ∈ 𝑀2 𝐒, 𝑒𝑦 𝑔 ̂ 𝜕
2 exp 𝜏2 𝜕 2
2 𝑒𝜕 < ∞ 𝑔, = 2𝜌 𝑛 𝑔 ̂ 𝜕 𝜕 exp 𝜏2 𝜕 2 2 𝑒𝜕
𝑙𝑀 𝑦, 𝑧 = exp −𝛾|𝑦 − 𝑧| , 𝜍𝑀 𝜕 = 1 2𝜌 𝜕2 + 𝛾 𝐼𝑙𝑀 = 𝑔 ∈ 𝑀2 𝐒, 𝑒𝑦 𝑔 ̂ 𝜕
2 𝜕2 + 𝛾 𝑒𝜕 < ∞
𝑔, = 2𝜌 𝑔 ̂ 𝜕 𝜕 𝜕2 + 𝛾 𝑒𝜕
– Decay of 𝑔 ∈ 𝐼 for high-frequency is different for Gaussian and Laplacian.
IV-10
– Polynomial kernel on 𝐒:
𝑙𝑞 𝑦, 𝑧 = 𝑦𝑈𝑧 + 𝑑 𝑒, 𝑑 ≥ 0, 𝑒 ∈ 𝐎 𝑙𝑞 𝑦, 𝑨0 = 𝑨0
𝑒𝑦𝑒 + 𝑒
1 𝑑𝑨0
𝑒−1𝑦𝑒−1 + 𝑒
2 𝑑2𝑨0
𝑒−2𝑦𝑒−1 + ⋯ + 𝑑𝑒.
Span of these functions are polynomials of degree 𝑒. Proposition 4.5 If 𝑑 ≠ 0, the RKHS is the space of polynomials of degree at most 𝑒. [Proof: exercise. Hint. Find 𝑐𝑗 to satisfy ∑
𝑐𝑗𝑙 𝑦, 𝑨𝑗
𝑒 𝑗=0
= ∑ 𝑏𝑗𝑦𝑗
𝑒 𝑗=0
as a solution to a linear equation.]
IV-11
𝐼1, 𝑙1 , 𝐼2, 𝑙2 : two RKHS’s and positive definite kernels on Ω.
RKHS for 𝑙1 + 𝑙2:
𝐼1 + 𝐼2 = 𝑔: Ω → 𝑆 ∃𝑔
1 ∈ 𝐼1, ∃𝑔 2 ∈ 𝐼2, 𝑔 = 𝑔 1 + 𝑔 2}
𝑔 2 = 𝑔
1 𝐼1 2 + 𝑔 2 𝐼2 2 𝑔 = 𝑔 1 + 𝑔 2, 𝑔 1 ∈ 𝐼1, 𝑔 2 ∈ 𝐼2}
RKHS for 𝑙1𝑙2:
𝐼1 ⊗ 𝐼2 =tensor product as a vector space. 𝑔 = ∑ 𝑔
𝑗𝑗 𝑜 𝑗=1
𝑔
𝑗 ∈ 𝐼1, 𝑗 ∈ 𝐼2} is dense in 𝐼1 ⊗ 𝐼2.
∑ 𝑔
𝑗 1 𝑗 1 , 𝑜 𝑗=1
∑ 𝑔
𝑘 2 𝑘 2 𝑛 𝑘=1
= ∑ ∑ 𝑔
𝑗 1 , 𝑔 𝑘 2 𝐼1 𝑗 1 , 𝑘 2 𝐼2 𝑛 𝑘=1 𝑜 𝑗=1
.
IV-12
– Positive definiteness of kernels are preserved by
– Bochner’s theorem: characterization of the continuous shift- invariance kernels on 𝐒𝑛. – Explicit form of RKHS
frequency domain.
IV-13
Aronszajn., N. Theory of reproducing kernels. Trans. American Mathematical Society, 68(3):337–404, 1950. Saitoh., S. Integral transforms, reproducing kernels, and their applications. Addison Wesley Longman, 1997.
IV-14
Using the definition for one point, we have 𝑙 𝑦, 𝑦 is real and non- negative for all 𝑦. For any 𝑦 and 𝑧, applying the definition with coefficient (𝑑, 1) where 𝑑 ∈ 𝐃, we have
𝑑 2𝑙 𝑦, 𝑦 + 𝑑𝑙 𝑦, 𝑧 + 𝑑̅𝑙 𝑧, 𝑦 + 𝑙 𝑧, 𝑧 ≥ 0.
Since the right hand side is real, its complex conjugate also satisfies
𝑑 2𝑙 𝑦, 𝑦 + 𝑑̅𝑙 𝑦, 𝑧 + 𝑑𝑙 𝑧, 𝑦 + 𝑙 𝑧, 𝑧 ≥ 0.
The difference of the left hand side of the above two inequalities is real, so that
𝑑̅ 𝑙 𝑧, 𝑦 − 𝑙 𝑦, 𝑧 − 𝑑(𝑙 𝑧, 𝑦 − 𝑙 𝑦, 𝑧 )
is a real number. On the other hand, since 𝛽 − 𝛽
must be pure
imaginary for any complex number 𝛽,
𝑑̅ 𝑙 𝑧, 𝑦 − 𝑙 𝑦, 𝑧 = 0
holds for any 𝑑 ∈ 𝐃. This implies 𝑙 𝑧, 𝑦 = 𝑙(𝑦, 𝑧).
IV-15
1
The Institute of Statistical Mathematics / Graduate University for Advanced Studies September 6-7 Machine Learning Summer School 2012, Kyoto
1
V-2
– We have seen that kernelization of linear methods provides nonlinear methods, which capture ‘nonlinearity’ or ‘high-order moments’ of original data. e.g. nonlinear SVM, kernel PCA, kernel CCA, etc. – This section discusses more basic statistics on RKHS
(original space) feature map H (RKHS) X (X) = k( , X)
mean, covariance, conditional covariance
V-3
V-4
X: random variable taking value on a measurable space Ω k: measurable positive definite kernel on Ω. H: RKHS.
Always assumes “bounded” kernel for simplicity: sup
: feature vector = random variable on RKHS. – Definition. The kernel mean ∈ of X on H is defined by – Reproducing expectation:
* Notation: depends on , also. But, we do not show it for simplicity.
5
) ( , X f E f mX ) , ( ) ( X k X
∀ ∈
V-5
, : random variable taking values on Ω, Ω, resp. , , , : RKHS given by kernels on Ω and Ω, resp.
– Simply, covariance of feature vectors. c.f. Euclidean case VYX = E[YXT] – E[Y]E[X]T : covariance matrix – Reproducing covariance:
Y X YX
V-6
∗
∗ denotes the linear functional ,⋅ : ↦ 〈, 〉
)]) ( ), ( [ Cov ( )] ( [ )] ( [ )] ( ) ( [ , Y g X f X f E Y g E X f Y g E f g
YX
for all ∈ , ∈ .
– Standard identification:
∗ ≅ : ,⋅, ↔ .
– The operator is regarded as an element in ⊗ , i.e., can be regarded as
V-7
Σ ≡ Φ ⊗ Φ ∗ Φ ⊗ Φ
∗
X Y X Y HX HY X Y X(X) Y(Y)
YX
(Fukumizu, Bach, Jordan 2004, 2009; Sriperumbudur et al 2010)
– Kernel mean can capture higher-order moments of the variable. Example
X: R-valued random variablek: pos.def. kernel on R.
Suppose k admits a Taylor series expansion on R. The kernel mean mX works as a moment generating function:
2 2 1
) ( ) ( ) , ( xu c xu c c x u k
2 2 2 1
] [ ] [ )] , ( [ ) ( u X E c u X E c c X u k E u mX
] [ ) ( 1
X E u m du d c
u X
) exp( ) , ( xu u x k
e.g.) (ci > 0)
V-8
H: RKHS on with a bounded measurable kernel k. mP: kernel mean on H for a variable with probability
characteristic (w.r.t. P) if the mapping is one-to-one. – The kernel mean with a characteristic kernel uniquely determines a probability.
P
P
m P H ,
Q P
i.e.
V-9
– Analogy to “characteristic function” With Fourier kernel
probability on Rm.
kernel uniquely determines a probability on (, B). Note: may not be Euclidean. – The characteristic RKHS must be large enough! Examples for Rm (proved later): Gaussian, Laplacian kernel. Polynomial kernels are not characteristic.
y x y x F
T
1 exp ) , ( )]. , ( [ ) ( . . u X F E u
X
f Ch
)] , ( [ ) ( X u k E u mX
V-10
– Statistical inference: inference on some properties of probabilities. – With characteristic kernels, they can be cast into the inference
Two sample problem: ? Independence test:
⊗ ?
? ⊗ ?
V-11
– An advantage of RKHS approach is easy empirical estimation. – : i.i.d.
: sample on RKHS Empirical mean: Theorem 5.1 (strong -consistency) Assume
n
1
n
X X , ,
1
n i i n i i n X
1 1 ) (
). ( 1 ˆ
) (
n n O m m
p X n X
. )] , ( [ X X k E
V-12
,
, … , , : i.i.d. sample on Ω Ω.
An estimator of YX is defined by Theorem 5.2 – Hilbert-Schmidt norm: same as Frobenius norm of a matrix, but
| |
∑ ,
n i X i X Y i Y n YX
m X k m Y k n
1 ) (
ˆ ) , ( ˆ ) , ( 1 ˆ
) ( 1 ˆ
) (
n n Op
HS YX n YX
(sum of squares in matrix expression)
V-13
V-14
– Two sample homogeneity test Two i.i.d. samples are given; Q: Are they sampled from the same distribution? Null hypothesis H0: . Alternative H1: . – Practically important: we often wish to distinguish two things.
different?
– If then means of and are different, we may use it for test. If they are identical, we need to look at higher order information.
P X X
n ~
,...,
1
. ~ ,...,
1
Q Y Y
and
V-15
– If mean and variance are the same, it is a very difficult problem. – Example: do they have the same distribution?
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
1 2 3 4
ℓ 100
V-16
(Gretton et al. NIPS 2007, 2010, JMLR2012).
– Comparison of
and
comparison of and .
– In population – With characteristic kernel, MMD = 0 if and only if
.
)]. , ( [ 2 )] ~ , ( [ )] ~ , ( [
2 2
Y X k E Y Y k E X X k E m m MMD
H Y X
V-17
) ( ) ( ) ( ) ( sup , sup
1 || :|| 1 || :||
x dQ x f x dP x f m m f m m
H H
f f Y X f f H Y X
( , : independent copy of , )
hence, MMD.
– Test statistics: Empirical estimator MMD2
emp
– Asymptotic distributions under H0 and H1 are known (see Appendix)
– Approximation of null distribution
V-18 2
ˆ ˆ
H Y X
m m
1 , 2 1 1 1 , 2
) , ( 1 ) , ( 2 ) , ( 1
b a b a n i a a i n j i j i
Y Y k Y X k n X X k n
2 emp
MMD
Data set Attr. MMD-B WW KS Neural I
Same
96.5 97.0 95.0
Different
0.0 0.0 10.0 Neural II
Same
94.6 95.0 94.5
Different
3.3 0.8 31.8 Health
Same
95.5 94.7 96.1
Different
1.0 2.8 44.0 Subtype
Same
99.1 94.6 97.3
Different
0.0 0.0 28.4
Percentage of accepting . Significance level .
Data size / Dim
Neural I: 4000 / 63 Neural II: 1000 / 100 Health: 25 / 12600 Subtype: 25 / 2118 Comparison of two databases.
(Gretton et al. JMLR 2012)
V-19
WW: Wald-Walfovitz test KS: Kolmogorov-Smirnov test Classical methods (see Appendix)
20
0.2 0.4 0.6 0.8 1 0.002 0.004 0.006 0.008 0.01 0.012 0.014 0.2 0.4 0.6 0.8 1 0.005 0.01 0.015 0.02 0.025 0.03 0.2 0.4 0.6 0.8 1 0.005 0.01 0.015 0.02
NX = NY = 100 NX = NY = 200 NX = NY = 500 0,1 vs Unif 1 0,1 Average values of MMD
0,1 vs 0,1
– and are independent if and only if
, .
Given i.i.d. sample ,
, … , , , are and independent?
– Null hypothesis H0:
⊗ (independent)
– Alternative H1:
⊗
(not independent)
V-21
V-22
1 2 3
1 2 3
1 2 3
1 2 3
Dependent
Theorem 5.3. (Fukumizu, Bach, Jordan, JMLR 2004) If the product kernel is characteristic, then Recall Σ ≅ ⊗ ∈ ⊗ . Comparison between (kernel mean of ) and ⊗ (kernel mean of
).
– Dependence measure: Hilbert-Schmidt independence criterion
O
YX
V-23
, ≔ Σ
⊗
,
: independent copy of , . [Exercise]
(Gretton, Fukumizu, Teo, Song, Schölkopf, Smola. NIPS 2008)
– Test statistic: – This is a special case of MMD comparing
and ⊗ with
product kernel . – Asymptotic distributions are given similarly to MMD (Gretton et al
2009).
V-24
n k j i k i Y j i X n j i j i Y j i X emp
Y Y k X X k n Y Y k X X k n Y X
1 , , 3 1 , 2
) , ( ) , ( 2 ) , ( ) , ( 1 ) , ( HSIC
n k k Y n j i j i X
Y Y k X X k n
1 , 1 , 4
) , ( ) , ( 1
Or equivalently,
HSIC, ≔ Σ
Tr , : centered Gram matrices
– Distance covariance / distance correlation has gained popularity in statistical community as a dependence measure beyond Pearson correlation.
where , and , are i.i.d. ~
.
V-25
Proposition 5.4 (Sejdinovic, Gretton, Sriperumbudur, F. ICML2012) A kernel on Euclidean spaces
, .
is positive definite, and
HSIC
, , .
– Distance covariance is a specific instance of MMD. – Positive definite kernel approach is more general in choosing kernels, and thus may perform better. – Extension
, is positive definite for 0 2. We can define
,
V-26
∑
V-27
2
Dependence
Ratio of accepting independence
Original distance covariance HSIC with Gaussian kernel
– Heuristics for Gaussian kernel:
median
– Using performance of statistical test: Type II error of the test statistics (Gretton et al NIPS 2012). Challenging open questions.
V-28
V-29
– Conditional probabilities appear in many machine learning problems
already seen in Section II.
′
– Graphical modeling Conditional independent relations among variables. – Causality – Dimension reduction / feature extraction
V-30
X1 X4 X2 X3 X5
Definition. – Simply, kernel mean of |. – It determines the conditional probability with a characteristic kernel. – Again, inference problems on conditional probabilities can be solved as inference on conditional kernel means. – But, how can we estimate it?
Φ ⋅,
V-31
, : random variables on Ω, Ω, , : positive definite kernels on Ω, Ω, , , ∞.
– Reproducing property:
, ∀ ∈ , ∈
∀ , ∈
(Or ∈ ⊗
∗ ≅ ⊗ )
V-32
– Recall for zero-mean Gaussian random variable , ,
,
– For the feature vector
Φ
Φ .
Φ Φ
regularized estimator can be justified.
V-33
– Empirical estimation: given ,
, … , , ,
In Gram matrix expression,
∗
Proposition 5.5 (Consistency) If is characteristic,
, ∈ ⊗ , and → 0, →
∞ as → ∞, then for every
in in probability.
, ⋮ , , ∗ ∗,
∗,
V-34
Conditional covariance matrix:
| ≔
for any
Definition: , , : random variables on Ω, Ω, Ω.
, , : positive definite kernel on Ω, Ω, Ω.
Conditional cross-covariance operator
Proposition 5.6 If is characteristic, then for ∀ ∈ , ∈ ,
V-35
, | Cov , |
– An interpretation: Compare the conditional kernel means for
, and |. Φ ⊗ Φ Φ | ⊗ Φ
Dependence on is not easy to handle Average it out.
Φ ⊗ Φ Φ| ⊗ Φ ]
– Empirical estimator:
⋅ ⋅
– Recall: for Gaussian random variable
– By average over , | does not imply conditional independence, which requires , | for each . – Trick: consider
| ≔
,
where , and the product kernel is used for . Theorem 5.7 (Fukumizu et al. JMLR 2004) Assume , , are characteristic, then
| ⇔ | . – |, | can be similarly used.
V-37
– Squared HS-norm
|
independence test. – Unlike the independence test, the asymptotic null distribution is not available. Permutation test is needed. – Background: The conditional independent test with continuous non-Gaussian variables is not easy, and a challenging open problem.
V-38
– Directional acyclic graph (DAG) is used for representing the causal structure among variables. – The structure can be learned by conditional independence tests. The above test can be applied (Sun, Janzing, Schölkopf, F. ICML2007).
V-39
and
X Y Z
Input: X = (X1, ... , Xm), Output: Y (either continuous or discrete) Goal: find an effective dimension reduction space (EDR space) spanned by an m x d matrix B s.t. No further assumptions on cond. p.d.f. p.
| |
( | ) ( | )
T
T Y X Y B X
p Y X p Y B X
BTX = (b1
TX, ..., bd TX)
linear feature vector where X U V Y
X Y | BTX ,
(Samarov 1993; Hristache et al 2001)
– Assumptions:
– Gradient of the regression function lies in the EDR space at each x
= average or PCA
T
T )
x B z
T
V-41
– Weakness of ADE:
ADE uses local polynomial regression. » Sensitive to bandwidth
2 2 1
e.g.
– Kernel method
Φ
V-42
– Reproducing the derivative (e.g. Steinwart & Christmann, Chap. 4): Assume is differentiable and then – Combining with the estimation of conditional kernel mean, The top eigenvectors of
estimates the EDR space
X X
X
V-43
for any ∈
j Y i Y ij
x x X Y E x x X Y E x M ] | ) ( [ ˆ , ] | ) ( [ ˆ ) ( ˆ
j X n XX YX i X n XX YX
x x k I C C x x k I C C
) , ( ) ˆ ( ˆ , ) , ( ) ˆ ( ˆ
1 1
(Fukumizu & Leng, NIPS 2012)
– Method
Estimator – gKDR estimate the subspace to realize the conditional independence – Choice of kernel: Cross-validation with some regressor/classifier, e.g. kNN method.
. ) ( ) ( ) ( ) ( 1 ~
1 1 1
n i i X n X Y n X T i X n
X I n G G I n G X n M k k
T X x n X X i X
i
x x X k x x X k X
) , ( ,..., ) , ( ) (
1
k
. ~
n
M . ˆ B
) , (
j i X X
X X k G
V-44
– Speech signals of 26 alphabets – 617 dim. (continuous) – 6238 training data / 1559 test data – Results
Dim
gKDR gKDR-v CCA
c.f. C4.5 + ECOC: 6.61% Neural Networks (best): 3.27% Classification errors for test data by SVM (%)
V-45
Data from UCI repository.
Dim
gKDR gKDR-v CCA 10 15 20 25 30 35 40 45 50
14.43 7.50 5.00 4.75
7.57 4.75 4.30 3.85 3.85 3.59 3.53 3.08 13.09 8.66 6.54 6.09
– Author identification for Amazon commerce reviews. – Dim = 10000 (linguistic style: e.g. usage of digit, punctuation, words and sentences' length, and frequency of words, etc)
– = # authors x 30
(Data from UCI repository.)
V-46
– Example of 2-dim plots for 3 authors – gKDR (dim = # authors) vs correlation-based variable selection (dim= 500/2000)
10-fold cross-validation errors (%)
# Authors gKDR + 5NN gKDR + SVM Corr (500) + SVM Corr (2000) + SVM
10 9.3 12.0 15.7 8.3 20 16.2 16.2 30.2 18.0 30 20.1 18.0 29.2 24.0 40 22.8 21.8 35.4 25.0 50 22.7 19.5 41.1 29.0
V-47
(Fukumizu, Song, Gretton NIPS 2011)
, .
–
: kernel mean of prior, ∑ ⋅,
,
, … , , ~, i.i.d.
– Goal: compute kernel mean of posterior |.
V-48
| | Λ Λ
Λ
Λ Diag
– NO PARAMETRIC MODELS, BUT SAMPLES! – When is it useful?
is unavailable, but sampling is easy. – Approximate Bayesian Computation (ABC), Process prior.
given in training phase. – Example: nonparametric HMM (shown later).
sampling methods, such as MCMC, SMC, etc. But, they may take long time. KBR uses matrix operations.
V-49
Model: , ∏
∏
|
| and/or | is not known.
But, data ,
in training phase. Examples:
– Testing phase (e.g., filtering, e.g.): given
, … , , estimate hidden state .
– Sequential filtering/prediction uses Bayes’ rule KBR applied.
X0 X1 X2 X3 XT Y0 Y1 Y2 Y3 YT …
V-50
– Hidden : angles of a video camera located at a corner of a room. – Observed
: movie frame of a room + additive Gaussian noise.
– : 3600 downsampled frames of 20 x 20 RGB pixels (1200 dim. ).
– The first 1800 frames for training, and the second half for testing
V-51
noise KBR (Trace) Kalman filter(Q) 2 = 10-4 0.15 0.01 0.56 0.02 2 = 10-3 0.210.01 0.54 0.02
Average MSE for camera angles (10 runs) To represent SO(3) model, Tr[AB-1] for KBR, and quaternion expression for Kalman filter are used .
– Kernel mean gives a representation of probability distribution. – Inference on probabilities can be cast into inference on kernel
– Conditional probabilities can be handled with kernel means and covariances
V-52
Fukumizu, K., Bach, F.R., and Jordan, M.I. (2004) Dimensionality reduction for supervised learning with reproducing kernel Hilbert spaces. Journal of Machine Learning Research. 5:73-99, Fukumizu, K., F.R. Bach and M. Jordan. (2009) Kernel dimension reduction in regression. Annals of Statistics. 37(4), pp.1871-1905 Fukumizu, K., L. Song, A. Gretton (2011) Kernel Bayes' Rule. Advances in Neural Information Processing Systems 24 (NIPS2011) 1737-1745. Gretton, A., K.M. Borgwardt, M.Rasch, B. Schölkopf, A.J. Smola (2007) A Kernel Method for the Two-Sample-Problem. Advances in Neural Information Processing Systems 19, 513-520. Gretton, A., Z. Harchaoui, K. Fukumizu, B. Sriperumbudur (2010) A Fast, Consistent Kernel Two-Sample Test. Advances in Neural Information Processing Systems 22, 673-681. Gretton, A., K. Fukumizu, C.-H. Teo, L. Song, B. Schölkopf, A. Smola. (2008) A Kernel Statistical Test of Independence. Advances in Neural Information Processing Systems 20, 585-592. Gretton, A., K. Fukumizu, C.-H. Teo, L. Song, B. Schölkopf, A. Smola. (2008) A Kernel Statistical Test of Independence. Advances in Neural Information Processing Systems 20, 585-592.
V-53
Hristache, M., A. Juditsky, J. Polzehl, and V. Spokoiny. (2001) Structure Adaptive Approach for Dimension Reduction. Annals of Statsitics, 29(6):1537-1566. Samarov, A. (1993). Exploring regression structure using nonparametric functional
Sejdinovic, D., A. Gretton, B. Sriperumbudur, K. Fukumizu. (2012) Hypothesis testing using pairwise distances and associated kernels. Proc. 29th International Conference on Machine Learning (ICML2012). Sriperumbudur, B.K., A. Gretton, K. Fukumizu, B. Schölkopf, G.R.G. Lanckriet. (2010) Hilbert Space Embeddings and Metrics on Probability Measures. Journal
Sun, X., D. Janzing, B. Schölkopf and K. Fukumizu. (2007) A kernel-based causal learning algorithm. Proc. 24th Annual International Conference on Machine Learning (ICML2007), 855-862. Székely, G.J., M.L. Rizzo, and N.K. Bakirov. (2007) Measuring and testing dependence by correlation of distances. Annals of Statistics, 35(6): 2769-2794. Székely, G.J. and M.L. Rizzo. (2009) Brownian distance covariance. Annals of Applied Statistics, 3(4):1236-1265.
V-54
V-55
Example) Based on MMD, we wish to make a decision whether the two variables have the same distribution. Simple-minded idea: Set a small value like t = 0.001
MMD(X,Y) < t
Perhaps, same
MMD(X,Y) t
Different But, the threshold should depend on the properties of X and Y.
– A statistical way of deciding whether a hypothesis is true or not. – The decision is based on sample We cannot be 100% certain.
V-56
“X and Y have the same distribution”
e.g. TN = MMDemp
2
The probability that MMDemp
2 > t under
H0 is very small.
V-57
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
p.d.f. of Null distribution area = (5%, 1% etc) threshold t
critical region One-sided test TN area = p-value TN > t p-value < significance level
V-58
– Type I error = false positive (e.g. “ ” = positive) – Type II error = false negative
TRUTH H0 Alternative TEST RESULT Reject H0 Accept H0 Type I error Type II error
give small type II error. True negative True positive False positive False negative
V-59
, … , ~,
, … , ℓ ~: i.i.d. Let ℓ.
Assume
ℓ → 1
(0 1) as → ∞. Under the null hypothesis of , where , , … are i.i.d. with law 0; 1/ 1 , and
the eigenvalues of the integral operator on
,
with
the centered kernel
, ,
MMD
1
V-60
Under the alternative , where
4
, ,
,
– The asymptotic distributions are derived by the general theory of U-statistics (e.g. see van der Vaart 1998, Chapter 12). – Estimation of the null distribution:
MMD
V-61
One-dimensional variables – Empirical distribution function – KS test statistics – Asymptotic null distribution is known (not shown here).
N i i N
t X I N t F
1
) ( 1 ) ( ) ( ) ( sup
2 1
t F t F D
N N t N
R
DN FN
1(t)
FN
2(t)
t
V-62
One-dimensional samples – Combine the samples and plot the points in ascending order. – Label the points based on the original two groups. – Count the number of “runs”, i.e. consecutive sequences of the same label. – Test statistics – In one-dimensional case, less powerful than KS test
– Minimum spanning tree is used (Friedman Rafsky 1979)
R = Number of runs
) 1 , ( ] [ ] [ N R Var R E R TN
R = 10
V-63