Support Vector Machines Marco Chiarandini Department of Mathematics - - PowerPoint PPT Presentation

▶
support vector machines
SMART_READER_LITE
LIVE PREVIEW

Support Vector Machines Marco Chiarandini Department of Mathematics - - PowerPoint PPT Presentation

Feb 12, 2023 32 likes •287 views

DM825 Introduction to Machine Learning Lecture 8 Support Vector Machines Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Functional and Geometric Ma Optimal Margin Classifier Lagrange

slide-1
SLIDE 1

DM825 Introduction to Machine Learning Lecture 8

Support Vector Machines

Marco Chiarandini

Department of Mathematics & Computer Science University of Southern Denmark

slide-2
SLIDE 2

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Overview

Support Vector Machines:

  • 1. Functional and Geometric Margins
  • 2. Optimal Margin Classifier
  • 3. Lagrange Duality
  • 4. Karush Kuhn Tucker Conditions
  • 5. Solving the Optimal Margin
  • 6. Kernels
  • 7. Soft margins
  • 8. SMO Algorithm

2

slide-3
SLIDE 3

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

In This Lecture

  • 1. Functional and Geometric Margins
  • 2. Optimal Margin Classifier
  • 3. Lagrange Duality
  • 4. Karush Kuhn Tucker Conditions
  • 5. Solving the Optimal Margin

3

slide-4
SLIDE 4

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Introduction

◮ Binary classification. ◮ y ∈ {−1, 1} (instead of {0, 1} like in GLM) ◮ Let’s have h(

θ, x) output values {−1, 1}: f(z) = sign(z)

  • 1

ifz ≥ 0 −1 ifz < 0 (hence no probabilities like in logistic regression)

◮ h(

θ, x) = f( θ x + θ0), x ∈ Rn, θ ∈ Rn, θ0 ∈ R

◮ Assume for now training set is linearly separable

4

slide-5
SLIDE 5

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

SVM determine model parameters by solving a convex optimization problem and hence a local optimal solution is also global optimal. Margin: smallest distance between the decision boundary and any of the samples. The location of the boundary is determined by a subset of the data points, known as support vectors

5

slide-6
SLIDE 6

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Outline

  • 1. Functional and Geometric Margins
  • 2. Optimal Margin Classifier
  • 3. Lagrange Duality
  • 4. Karush Kuhn Tucker Conditions
  • 5. Solving the Optimal Margin

6

slide-7
SLIDE 7

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Resume

◮ functional margin:

ˆ γi = yi( θT xi + θ0) = ⇒ ˆ γ = min

i

ˆ γi requires a normalization condition

◮ geometric margin:

γi = yi

  • θ

θ

T

xi + θ0 θ

  • =

⇒ γ = min

i

γi scale invariant

◮ γ = ˆ γ

  • θ

◮ if

θ = 1 then ˆ γi = γi the two margins correspond

7

slide-8
SLIDE 8

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Outline

  • 1. Functional and Geometric Margins
  • 2. Optimal Margin Classifier
  • 3. Lagrange Duality
  • 4. Karush Kuhn Tucker Conditions
  • 5. Solving the Optimal Margin

8

slide-9
SLIDE 9

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Optimization Problem

Looking at the geometric margin: (OPT1) : max

γ, θ,θ0

γ γ ≤ yi( θT xi + θ0) ∀i = 1, . . . , m θ = 1 Alternatively, looking at functional margins and recalling that γ =

ˆ γ

  • θ:

(OPT2) : max

ˆ γ, θ,θ0

ˆ γ θ ˆ γ ≤ yi( θT xi + θ0) ∀i = 1, . . . , m

9

slide-10
SLIDE 10

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

For the functional margins we can fix the scale, for the geometric margin no scaling problem. Then we can arbitrary fix ˆ γ = 1 (OPT3) : min

  • θ,θ0

1 2 θ 2 1 ≤ yi( θT xi + θ0) ∀i = 1, . . . , m where we used that: max 1/ θ = min θ and removed the square root because monotonous in θ =

  • θT

θ. This problem is a convex optimization problem, it has convex quadratic

  • bjective function and linear constraints, hence it can be solved optimally and

efficiently

10

slide-11
SLIDE 11

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Convex optimization problem

minimize f0(x) subject to fi(x) ≤ bi, i = 1, . . . , m

  • bjective and constraint functions are convex:

fi(αx + βy) ≤ αfi(x) + βfi(y) if α + β = 1, α ≥ 0, β ≥ 0

11

slide-12
SLIDE 12

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Outline

  • 1. Functional and Geometric Margins
  • 2. Optimal Margin Classifier
  • 3. Lagrange Duality
  • 4. Karush Kuhn Tucker Conditions
  • 5. Solving the Optimal Margin

12

slide-13
SLIDE 13

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Lagrangian

standard form problem (not necessarily convex) minimize f0(x) subject to fi(x) ≤ 0, i = 1, ..., m hi(x) = 0, i = 1, ..., p variable x ∈ Rn, domain D, optimal value p∗ Lagrangian: L : Rn × Rm × Rp → R, with dom L = D × Rm × Rp, L(x, α, β) = f0(x) +

m

  • i=1

αifi(x) +

p

  • i=1

βhi(x)

◮ weighted sum of objective and constraint functions ◮ αi is Lagrange multiplier associated with fi(x) ≤ 0 ◮ βi is Lagrange multiplier associated with hi(x) = 0 ◮

α and β are dual or Lagrangian variables

13

slide-14
SLIDE 14

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Lagrange dual function

Lagrange dual function: LD : Rm × Rp → R LD(α, β) = min

x∈D L(x, α, β) = min x∈D

  • f0(x) +

m

  • i=1

αifi(x) +

p

  • i=1

βhi(x)

  • LD is concave, can be −∞ for some α and β

Lower bound property: for a feasible ˜ x

  • 1. ∀α ≥ 0, β

LD(α, β) ≤ p∗

  • 2. LP (x) = maxα≥0,β (LD(α, β)) ≤ p∗ (best lower bound, it may be = p∗)

Proof of (1): for any ˜ x feasible and α ≥ 0 : L(˜ x, α, β) = f0(˜ x) +

m

  • i=1

αifi(˜ x) +

p

  • i=1

βhi(˜ x) ≤ f0(˜ x) hence LD(α, β) = min

x∈D L(x, α, β) ≤ L(˜

x, α, β) ≤ f0(˜ x) (2) is true because (1) true for any α, β.

14

slide-15
SLIDE 15

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

If f0 and gi are convex and hi affine, d∗ = max

α≥0,β (LD(α, β)) = p∗

so we can solve the dual in place of the primal.

15

slide-16
SLIDE 16

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Outline

  • 1. Functional and Geometric Margins
  • 2. Optimal Margin Classifier
  • 3. Lagrange Duality
  • 4. Karush Kuhn Tucker Conditions
  • 5. Solving the Optimal Margin

16

slide-17
SLIDE 17

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Karush Kuhn Tucker Conditions

standard form problem (not necessarily convex) minimize f0(x) subject to gi(x) ≤ bi, i = 1, ..., m variable x ∈ Rn, f, g nonlinear, f : Rn → R, g : Rn → Rm Necessary conditions for optimality (local validity):          ∇f(x0) = m

i=1 λi∇gi(x0)

λi ≥ 0∀i m

i=1 λi(gi(x0) − bi) = 0

gi(x0) − bi ≤ 0

17

slide-18
SLIDE 18

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Outline

  • 1. Functional and Geometric Margins
  • 2. Optimal Margin Classifier
  • 3. Lagrange Duality
  • 4. Karush Kuhn Tucker Conditions
  • 5. Solving the Optimal Margin

18

slide-19
SLIDE 19

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Let’s go back to our problem: (OPT3) : min

  • θ,θ0

1 2 θ 2 1 ≤ yi( θT xi + θ0) ∀i = 1, . . . , m L( θ, θ0, α) = 1 2 θ 2 −

m

  • i=1

αi

  • yi(

θT xi + θ0) − 1

  • we find the dual form by solving in

θ, θ0 LD( α) = min

  • θ,θ0

L( θ, θ0, α) ∇

θL(

θ, θ0, α) = θ −

m

  • i=1

αiyi xi = 0 = ⇒ θ =

m

  • i=1

αiyi xi ∂L( θ, θ0, α) ∂θ0 = −

m

  • i=1

αiyiαi = 0 = ⇒

m

  • i=1

αiyiαi = 0

19

slide-20
SLIDE 20

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Substituting in L( θ, θ0, α): LD( θ) = 1 2 m

  • i=1

αiyi xi  

m

  • j=1

αjyj xj   −

m

  • i=1

αi  yi  (

m

  • j=1

αjyj xj) xi + θ0   − 1   =

m

  • i=1

αi − 1 2

m

  • i=1

m

  • j=1

yiyjαiαj xi xj

20

slide-21
SLIDE 21

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

We are left with the dual problem: max

  • α

W( α) =

m

  • i=1

αi − 1 2

m

  • i=1

m

  • j=1

yiyjαiαj xi xj s.t. αi ≥ 0 ∀i = 1 . . . m

m

  • i=1

αiyi = 0

◮ This problem is in m variables. Problem (OPT3) has D variables and

quandratic programming can be solved in O(D3). If D ≪ n then it seems we did not earned a lot

◮ the form above allows us to use kernel trick and have even infinite

dimensions (D ≫ m)

◮ the use of the kernel and its constraint of being positive semidefinite

ensures that the problem is bounded from below.

21

slide-22
SLIDE 22

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

In addition, an optimal solution satisfies the KKT conditions on (OPT3): yi( θT xi + θ0) ≥ 1 αi[yi( θT xi + θ0) − 1] = 0 ∀i From these we can see that

◮ if αi > 0, then yi(

θT xi + θ0) = 1 ( xi is on the boundary)

◮ if yi(

θT xi + θ0) > 1, xi is not on the boundary and αi = 0

22

slide-23
SLIDE 23

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Points where yi( θT xi + θ0) > 1 are the support vectors:

23

slide-24
SLIDE 24

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Prediction

For a new pint x predict by: h( θ, x) = f( θ x + θ0) = sign( θ x + θ0) = sign   m

  • i=1

αiyi xi T

  • x + θ0

  = sign m

  • i=1

αiyi xi, x + θ0

  • For the KKT conditions, most trainig data can be discarded after training

and only the points that are support vectors need to be retained for this computation

24

slide-25
SLIDE 25

Functional and Geometric Ma Optimal Margin Classifier Lagrange Duality Karush Kuhn Tucker Conditions Solving the Optimal Margin

Intercept

We can derive θ0 by: θ0 = −maxi:yi=−1 θT xi + mini:yi=1 θT xi 2

25