Structured Multicategory Support Vector Machine with ANOVA - - PowerPoint PPT Presentation

Structured Multicategory Support Vector Machine with ANOVA decomposition

www.stat.ohio-state.edu/

Yoonkyung Lee Department of Statistics The Ohio State University

Structured Multicategory Support Vector Machine with ANOVA decomposition – p.1/15

Predictive learning

A training data set

• .

Functional relationship

• and

Regression: continuous

Classification: categorical

Goodness Prediction accuracy for a given loss

• .

Interpretation.

Structured Multicategory Support Vector Machine with ANOVA decomposition – p.2/15

Support Vector Machines

Vapnik (1995), http://www.kernel-machines.org Find

• minimizing
• Then
• Competitive classification accuracy.

Flexibility - implicit embedding through kernel. Handle high dimensional data. A black box unless the embedding is explicit.

Structured Multicategory Support Vector Machine with ANOVA decomposition – p.3/15

Feature Selection

The best subset selection. Nonnegative garrote [Breiman, Technometrics (1995)] Least Absolute Shrinkage and Selection Operator [Tibshirani, JRSS (1996)] COmponent Selection and Smoothing Operator [Lin & Zhang, Technical Report (2003)] Structural modelling with sparse kernels [Gunn & Kandola, Machine Learning (2002)]

Structured Multicategory Support Vector Machine with ANOVA decomposition – p.4/15

ANOVA decomposition

Wahba (1990) Function:

• Functional space:
• ,
• Reproducing kernel (r.k.):
• Modification of r.k. by rescaling parameters
• Structured Multicategory Support Vector Machine with ANOVA decomposition – p.5/15

• penalty on
• Truncating

• , find
• minimizing
• Then
• For sparsity, minimize
• subject to
• Structured Multicategory Support Vector Machine with ANOVA decomposition – p.6/15

Structured MSVM

Lee, Lin & Wahba, JASA (2004) Find

• with

the sum-to-zero constraint minimizing

• subject to
• for
• By the representer theorem,
• Structured Multicategory Support Vector Machine with ANOVA decomposition – p.7/15

Updating Algorithm

Denoting the objective function by

• ,

Initialize

• and
• .

At the

• th step (
• )
• step:

Find

• minimizing
• with
• fixed.
• step:

Find

• minimizing
• with
• fixed.

Structured Multicategory Support Vector Machine with ANOVA decomposition – p.8/15

A toy example

scatter plot and visualization of the

• step

x1 x2 0.0 0.5 1.0 0.0 0.5 1.0

0.6 0.7 0.8 0.9 0.0 0.5 1.0 1.5 0.0 0.5 1.0 1.5 θ1 θ2

Structured Multicategory Support Vector Machine with ANOVA decomposition – p.9/15

The trajectory of

• two-way interaction spline kernel with
• tuned by GCKL

log2(λθ) θ −15 −10 −5 0.0 0.5 1.0 1 2 1 x 2

Structured Multicategory Support Vector Machine with ANOVA decomposition – p.10/15

Classification boundaries

• rdinary MSVM (0.3970) vs. structured MSVM (0.3967)

x1 x2 0.0 0.5 1.0 0.0 0.5 1.0 x1 x2 0.0 0.5 1.0 0.0 0.5 1.0

Structured Multicategory Support Vector Machine with ANOVA decomposition – p.11/15

An “apple” example

• depends only on
• .

additive spline kernel with 5-fold CV (

• ,
• , and
• ).

log2(λθ) θ −10 −8 −6 −4 −2 0.0 0.5 1.0

Structured Multicategory Support Vector Machine with ANOVA decomposition – p.12/15

Gene selection: microarray data

2308 genes and four tumor types [Khan et al. Nature

Medicine (2001)]

46 positive rescaling parameters out of 500.

gene rank θ 100 200 300 400 500 0.0 0.5 1.0

Structured Multicategory Support Vector Machine with ANOVA decomposition – p.13/15

Concluding remarks

Integrate feature selection with learning classification rule. Enhance interpretation without compromising prediction accuracy. Characterize the solution path for effective computation and tuning. Tailor the structure of component penalty for refined selection.

Structured Multicategory Support Vector Machine with ANOVA decomposition – p.14/15

Joint work with Yuwon Kim (SNU), Ja-Yong Koo (Inha Univ.), and Sangjun Lee (SNU) in Korea. Manuscript to be posted at www.stat.ohio-state.edu/ yklee.

Structured Multicategory Support Vector Machine with ANOVA decomposition – p.15/15