Incorporating Detractors into SVM Marcin Orchel AGH University of - - PowerPoint PPT Presentation

Agenda Support Vector Machines Generalized SVC Reduced Models

Incorporating Detractors into SVM

Marcin Orchel

AGH University of Science and Technology

Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models

1 Support Vector Machines 2 Generalized SVC 3 Reduced Models

Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models

Support Vector Machines

Support Vector Machines (SVM) – a set of supervised learning methods used mainly for classification and regression problems SVM maximal margin classifier – classifies data with the hyperplane that has the largest distance to the closest training vectors SVM soft margin classifier - extended version able to classify nonseparable data H

ξ1

• wξ2
• w

Figure:

Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models

Support Vector Classification (SVC) Formulation

SVC primal problem: minimization of f

• w, b,

ξ

• = 1

2 w2 + C · ξ with constraints: yih ( ai) ≥ 1 − ξi, ξ ≥ 0 for i ∈ {1..l}, where C ≫ 0, h ( ai) = w · ai + b. SVC dual problem: maximization of W ( α) = 1 · α − 1 2

• αTQ

α with constraints

• α ·

y = 0, 0 ≤ α ≤ C where Qij = yiyj ( ai · aj) for all i, j ∈ {1 . . . l}.

Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models

SVC Solution

decision boundary h∗ ( x) =

l

• i=1

yi

cα∗ i K (

ai, x) + b∗

c = 0

(1) non support vectors have α∗

i = 0

they are ignored in the solution H

ξ1 w ξ2 w

Figure:

Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models

Generalized SVC Idea

How to make non support vectors useful and include them in the solution? generalized SVC – add possibility for points outside margins to be support vectors H

ξ1 w ξ2 w

Figure:

Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models

Generalized SVC Formulation

Use generalized SVC formulation: minimization of f

• w, b,

ξ

• = 1

2 w2 + C · ξ with constraints: yih ( ai) ≥ 1 − ξi + ϕi, ξ ≥ 0 for i ∈ {1..l}, where C ≫ 0, ϕi ∈ R, h ( ai) = w · ai + b.

Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models

Generalized SVC – Interpretation of the Weights

H

ξ1 w ξ2 w ϕ3 w ϕ4 w

Figure:

Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models

Reduced Models

How to use these new weights? Proposed reduced method:

1 run the SVC on the original data 2 compute values of the new weights based on the found solution 3 reduce the data by removing some of support vectors 4 run generalized SVC on the reduced data with values of the

new weights

the proposed method preserves additional knowledge about the original solution in the new weights

Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models

Properties of Reduced Models

reduced models are models with increased sparsity, reduced models are faster for postprocessing (because of simpler solutions), e.g. testing new examples reduction ratio parameter is the trade-off between generalization performance and simplicity to some extent reducing data with the proposed method does not decrease generalization performance result: we are able to generate reduced models with similar number of support vectors and better generalization performance than for the simple reduction

Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models

Results

increased generalization performance of reduced models ranging from 5% to 70% comparing to the simple reduction with similar number of support vectors

• 120
• 100
• 80
• 60
• 40
• 20

10 20 30 40 50 60 70

• 120
• 100
• 80
• 60
• 40
• 20

10 20 30 40 50 60 70

Figure:

Marcin Orchel Incorporating Detractors into SVM

Agenda Support Vector Machines Generalized SVC Reduced Models

Conclusions

with the generalized SVC we can create simpler SVC solutions – with less support vectors even without compromising generalization performance reduced models are faster in post-processing the research is financed by the Polish Ministry of Science and Higher Education project No NN519579338 I would like to express my sincere gratitude to Professor Witold Dzwinel (AGH University of Science and Technology, Department of Computer Science). Questions

Marcin Orchel Incorporating Detractors into SVM