Support Vector Machines Machine Learning 1 Big picture Linear - - PowerPoint PPT Presentation

Machine Learning

Support Vector Machines

1

Big picture

2

Linear models

Big picture

3

Linear models How good is a learning algorithm?

Big picture

4

Linear models How good is a learning algorithm? Online learning Perceptron, Winnow

Big picture

5

Linear models How good is a learning algorithm? Online learning PAC, Agnostic learning Perceptron, Winnow

Big picture

6

Linear models How good is a learning algorithm? Online learning PAC, Agnostic learning Perceptron, Winnow Support Vector Machines

Big picture

7

Linear models How good is a learning algorithm? Online learning PAC, Agnostic learning Perceptron, Winnow Support Vector Machines

…. ….

This lecture: Support vector machines

• Training by maximizing margin
• The SVM objective
• Solving the SVM optimization problem
• Support vectors, duals and kernels

8

This lecture: Support vector machines

• Training by maximizing margin
• The SVM objective
• Solving the SVM optimization problem
• Support vectors, duals and kernels

9

VC dimensions and linear classifiers

What we know so far 1. If we have 𝑛 examples, then with probability 1 - 𝜀, the true error of a hypothesis ℎ with training error 𝑓𝑠𝑠

! ℎ is

bounded by

10

Generalization error Training error A function of VC dimension. Low VC dimension gives tighter bound

𝑓𝑠𝑠! ℎ ≤ 𝑓𝑠𝑠

" ℎ +

𝑊𝐷 𝐼 ln 2𝑛 𝑊𝐷(𝐼) + 1 + ln 4 𝜀 𝑛

VC dimensions and linear classifiers

What we know so far 1. If we have 𝑛 examples, then with probability 1 - 𝜀, the true error of a hypothesis ℎ with training error 𝑓𝑠𝑠

! ℎ is

bounded by

11

Generalization error Training error

𝑓𝑠𝑠! ℎ ≤ 𝑓𝑠𝑠

" ℎ +

𝑊𝐷 𝐼 ln 2𝑛 𝑊𝐷(𝐼) + 1 + ln 4 𝜀 𝑛

A function of VC dimension. Low VC dimension gives tighter bound

What we know so far 1. If we have 𝑛 examples, then with probability 1 - 𝜀, the true error of a hypothesis ℎ with training error 𝑓𝑠𝑠

! ℎ is

bounded by

VC dimensions and linear classifiers

What we know so far 1. If we have 𝑛 examples, then with probability 1 - 𝜀, the true error of a hypothesis ℎ with training error 𝑓𝑠𝑠

! ℎ is

bounded by 2. VC dimension of a linear classifier in 𝑒 dimensions = 𝑒 + 1

12

Generalization error Training error

𝑓𝑠𝑠! ℎ ≤ 𝑓𝑠𝑠

" ℎ +

𝑊𝐷 𝐼 ln 2𝑛 𝑊𝐷(𝐼) + 1 + ln 4 𝜀 𝑛

A function of VC dimension. Low VC dimension gives tighter bound

VC dimensions and linear classifiers

What we know so far 1. If we have 𝑛 examples, then with probability 1 - 𝜀, the true error of a hypothesis ℎ with training error 𝑓𝑠𝑠

! ℎ is

bounded by 2. VC dimension of a linear classifier in 𝑒 dimensions = 𝑒 + 1

13

But are all linear classifiers the same?

Generalization error Training error

𝑓𝑠𝑠! ℎ ≤ 𝑓𝑠𝑠

" ℎ +

𝑊𝐷 𝐼 ln 2𝑛 𝑊𝐷(𝐼) + 1 + ln 4 𝜀 𝑛

A function of VC dimension. Low VC dimension gives tighter bound

Recall: Margin

The margin of a hyperplane for a dataset is the distance between the hyperplane and the data point nearest to it.

14

+ + + + + ++ +

• -
• -
• -

Recall: Margin

The margin of a hyperplane for a dataset is the distance between the hyperplane and the data point nearest to it.

15

+ + + + + ++ +

• -
• -
• -
• Margin with respect to this hyperplane

Which line is a better choice? Why?

+ + + + + ++ +

• -
• -
• -
• +

+ + + + ++ +

• -
• -
• -
• h2

h1

16

Which line is a better choice? Why?

+ + + + + ++ +

• -
• -
• -
• +

+ + + + ++ +

• -
• -
• -
• h2

+ +

A new example, not from the training set might be misclassified if the margin is smaller

h1

17

Data dependent VC dimension

• Intuitively, larger margins are better
• Suppose we only consider linear separators with

margins 𝜈! and 𝜈"

– 𝐼" = linear separators that have a margin 𝜈" – 𝐼# = linear separators that have a margin 𝜈# – And 𝜈" > 𝜈#

• The entire set of functions 𝐼! is “better”

18

Data dependent VC dimension

Theorem (Vapnik):

– Let H be the set of linear classifiers that separate the training set by a margin at least 𝛿 – Then 𝑊𝐷 𝐼 ≤ min 𝑆# 𝛿# , 𝑒 + 1 – 𝑆 is the radius of the smallest sphere containing the data

19

Data dependent VC dimension

Theorem (Vapnik):

– Let H be the set of linear classifiers that separate the training set by a margin at least 𝛿 – Then 𝑊𝐷 𝐼 ≤ min 𝑆# 𝛿# , 𝑒 + 1 – 𝑆 is the radius of the smallest sphere containing the data Larger margin ⇒ Lower VC dimension Lower VC dimension ⇒ Better generalization bound

20

Learning strategy

Find the linear separator that maximizes the margin

21

This lecture: Support vector machines

• Training by maximizing margin
• The SVM objective
• Solving the SVM optimization problem
• Support vectors, duals and kernels

22

Support Vector Machines

• Lower VC dimension → Better generalization
• Vapnik: For linear separators, the VC dimension depends inversely
• n the margin

– That is, larger margin → better generalization

• For the separable case:

– Among all linear classifiers that separate the data, find the one that maximizes the margin – Maximizing the margin by minimizing 𝒙!𝒙 if for all examples 𝑧𝒙!𝒚 ≥ 1

• General case:

– Introduce slack variables – one 𝜊" for each example – Slack variables allow the margin constraint above to be violated

23

So far

Recall: The geometry of a linear classifier

24

Prediction = sgn(𝑐 + 𝑥1 𝑦1 + 𝑥2𝑦2)

+ + + + + ++ +

• -
• -
• -
• |𝑥! 𝑦! + 𝑥"𝑦" + 𝑐|

𝑥!

" + 𝑥" "

= 𝑧(𝑥!𝑦! + 𝑥"𝑦" + 𝑐) | 𝐱 | 𝑐 + 𝑥1 𝑦1 + 𝑥2𝑦2 = 0

Recall: The geometry of a linear classifier

25

Prediction = sgn(𝑐 + 𝑥1 𝑦1 + 𝑥2𝑦2)

+ + + + + ++ +

• -
• -
• -
• We only care about

the sign, not the magnitude

|𝑥! 𝑦! + 𝑥"𝑦" + 𝑐| 𝑥!

" + 𝑥" "

= 𝑧(𝑥!𝑦! + 𝑥"𝑦" + 𝑐) | 𝐱 | 𝑐 + 𝑥1 𝑦1 + 𝑥2𝑦2 = 0

Recall: The geometry of a linear classifier

26

𝑐 + 𝑥1 𝑦1 + 𝑥2𝑦2 = 0 𝑐 2 + 𝑥1 2 𝑦1 + 𝑥2 2 𝑦2 = 0 1000𝑐 + 1000𝑥1 𝑦1 + 1000𝑥2𝑦2 = 0

+ + + + + ++ +

• -
• -
• -
• We only care about

the sign, not the magnitude

All these are equivalent. We could multiply or divide the coefficients by any positive number and the sign of the prediction will not change |𝑥! 𝑦! + 𝑥"𝑦" + 𝑐| 𝑥!

" + 𝑥" "

= 𝑧(𝑥!𝑦! + 𝑥"𝑦" + 𝑐) | 𝐱 |

Prediction = sgn(𝑐 + 𝑥1 𝑦1 + 𝑥2𝑦2)

Maximizing margin

• Margin of a hyperplane = distance of the closest

point from the hyperplane 𝛿𝐱,% = max

&

𝑧&(𝐱'𝐲& + 𝑐) | 𝐱 |

• We want maxw °

27

Some people call this the geometric margin The numerator alone is called the functional margin

Maximizing margin

• Margin of a hyperplane = distance of the closest

point from the hyperplane 𝛿𝐱,% = max

&

𝑧&(𝐱'𝐲& + 𝑐) | 𝐱 |

• We want to maximize this margin: max

𝐱,% 𝛿𝐱,%

28

Sometimes this is called the geometric margin The numerator alone is called the functional margin

Recall: The geometry of a linear classifier

29

b +w1 x1 + w2x2=0

We only care about the sign, not the magnitude

+ + + + + ++ +

• -
• -
• -
• Prediction = sgn(𝑐 + 𝑥1 𝑦1 + 𝑥2𝑦2)

|𝑥! 𝑦! + 𝑥"𝑦" + 𝑐| 𝑥!

" + 𝑥" "

Towards maximizing the margin

30

𝑥! 𝑑 𝑦! + 𝑥" 𝑑 𝑦" + 𝑐 𝑑 𝑥! 𝑑

"

+ 𝑥" 𝑑

"

+ + + + + ++ +

• -
• -
• -
• |𝑥! 𝑦! + 𝑥"𝑦" + 𝑐|

𝑥!

" + 𝑥" "

We only care about the sign, not the magnitude

We can scale the weights to make the optimization easier b +w1 x1 + w2x2=0

Towards maximizing the margin

31

𝑥! 𝑑 𝑦! + 𝑥" 𝑑 𝑦" + 𝑐 𝑑 𝑥! 𝑑

"

+ 𝑥" 𝑑

"

+ + + + + ++ +

• -
• -
• -
• |𝑥! 𝑦! + 𝑥"𝑦" + 𝑐|

𝑥!

" + 𝑥" "

Key observation: We can scale the 𝑑 so that the numerator is 1 for points that define the margin.

We only care about the sign, not the magnitude

We can scale the weights to make the optimization easier b +w1 x1 + w2x2=0

Towards maximizing the margin

32

𝑥! 𝑑 𝑦! + 𝑥" 𝑑 𝑦" + 𝑐 𝑑 𝑥! 𝑑

"

+ 𝑥" 𝑑

"

+ + + + + ++ +

• -
• -
• -
• |𝑥! 𝑦! + 𝑥"𝑦" + 𝑐|

𝑥!

" + 𝑥" "

1 𝑥! 𝑑

"

+ 𝑥" 𝑑

"

We only care about the sign, not the magnitude

We can scale the weights to make the optimization easier b +w1 x1 + w2x2=0 Key observation: We can scale the 𝑑 so that the numerator is 1 for points that define the margin.

Towards maximizing the margin

33

𝑥! 𝑑 𝑦! + 𝑥" 𝑑 𝑦" + 𝑐 𝑑 𝑥! 𝑑

"

+ 𝑥" 𝑑

"

+ + + + + ++ +

• -
• -
• -
• |𝑥! 𝑦! + 𝑥"𝑦" + 𝑐|

𝑥!

" + 𝑥" "

1 𝑣!

" + 𝑣" "

We only care about the sign, not the magnitude

We can scale the weights to make the optimization easier b +w1 x1 + w2x2=0 Key observation: We can scale the 𝑑 so that the numerator is 1 for points that define the margin.

Towards maximizing the margin

34

b +w1 x1 + w2x2=0

+ + + + + ++ +

• -
• -
• -
• |𝑥! 𝑦! + 𝑥"𝑦" + 𝑐|

𝑥!

" + 𝑥" "

Maximizing margin

• Margin of a hyperplane = distance of the closest point from

the hyperplane 𝛿𝐱,4 = max

5

𝑧5(𝐱6𝐲5 + 𝑐) | 𝐱 |

• We want to maximize this margin: max

𝐱,4 𝛿𝐱,4

• We only care about the sign of 𝐱 and b in the end and not the

magnitude

– Set the absolute score (functional margin) of the closest point to be 1 and allow 𝐱 to adjust itself

35

max

𝐱

𝛿 is equivalent to max

𝐱 " 𝐱 in this setting

Max-margin classifiers

Learning problem:

36

Max-margin classifiers

Learning problem:

37

Mimimizing gives us max

𝐱 ! 𝐱

Max-margin classifiers

Learning problem:

38

This condition is true for every example, specifically, for the example closest to the separator Mimimizing gives us max

𝐱 ! 𝐱

Max-margin classifiers

Learning problem:

• This is called the “hard” Support Vector Machine

We will look at how to solve this optimization problem later

39

This condition is true for every example, specifically, for the example closest to the separator Mimimizing gives us max

𝐱 ! 𝐱

Support Vector Machines

• Lower VC dimension → Better generalization
• Vapnik: For linear separators, the VC dimension depends inversely
• n the margin

– That is, larger margin → better generalization

• For the separable case:

– Among all linear classifiers that separate the data, find the one that maximizes the margin – Maximizing the margin by minimizing 𝒙!𝒙 if for all examples 𝑧𝒙!𝒚 ≥ 1

• General case:

– Introduce slack variables – one 𝜊" for each example – Slack variables allow the margin constraint above to be violated

40

So far

What if the data is not separable?

41

Maximize margin Every example has an functional margin of at least 1

Hard SVM

• This is a constrained optimization problem
• If the data is not separable, there is no w that will

classify the data

• Infeasible problem, no solution!

What if the data is not separable?

Hard SVM

• This is a constrained optimization problem
• If the data is not separable, there is no w that will

classify the data

• Infeasible problem, no solution!

42

Maximize margin Every example has an functional margin of at least 1

Dealing with non-separable data

Key idea: Allow some examples to “break into the margin”

43

+ + + + + ++ +

• -
• -
• -
• +

Dealing with non-separable data

Key idea: Allow some examples to “break into the margin”

44

+ + + + + ++ +

• -
• -
• -
• +

Dealing with non-separable data

Key idea: Allow some examples to “break into the margin”

45

+ + + + + ++ +

• -
• -
• -
• +
• This separator has a large

enough margin that it should generalize well.

Dealing with non-separable data

Key idea: Allow some examples to “break into the margin”

46

So, when computing margin, ignore the examples that make the margin smaller or the data inseparable.

+ + + + + ++ +

• -
• -
• -
• +
• This separator has a large

enough margin that it should generalize well.

Soft SVM

• Hard SVM:
• Introduce one slack variable »i per example

– And require yiwTxi ¸ 1 - »i and »i ¸ 0

47

Maximize margin Every example has an functional margin of at least 1 Intuition: The slack variable allows examples to “break” into the margin If the slack value is zero, then the example is either on or outside the margin

Soft SVM

• Hard SVM:
• Introduce one slack variable 𝜊5 per example

– And require 𝑧$𝑥%𝑦$ ≥ 1 − 𝜊$ and 𝜊$ ≥ 0

48

Maximize margin Every example has an functional margin of at least 1 Intuition: The slack variable allows examples to “break” into the margin If the slack value is zero, then the example is either on or outside the margin

Soft SVM

• Hard SVM:
• Introduce one slack variable 𝜊5 per example

– And require 𝑧$𝑥%𝑦$ ≥ 1 − 𝜊$ and 𝜊$ ≥ 0

49

Maximize margin Every example has an functional margin of at least 1 Intuition: The slack variable allows examples to “break” into the margin If the slack value is zero, then the example is either on or outside the margin

Soft SVM

• Hard SVM:
• Introduce one slack variable 𝜊5 per example

– And require 𝑧$𝑥%𝑦$ ≥ 1 − 𝜊$ and 𝜊$ ≥ 0

• New optimization problem for learning

50

Maximize margin Every example has an functional margin of at least 1

Soft SVM

• Hard SVM:
• Introduce one slack variable 𝜊5 per example

– And require 𝑧$𝑥%𝑦$ ≥ 1 − 𝜊$ and 𝜊$ ≥ 0

• New optimization problem for learning

51

Maximize margin Every example has an functional margin of at least 1

Soft SVM

52

Soft SVM

53

Maximize margin

Soft SVM

54

Maximize margin Minimize total slack (i.e allow as few examples as possible to violate the margin)

Soft SVM

55

Maximize margin Minimize total slack (i.e allow as few examples as possible to violate the margin) Tradeoff between the two terms

Support Vector Machines

• Lower VC dimension → Better generalization
• Vapnik: For linear separators, the VC dimension depends inversely
• n the margin

– That is, larger margin → better generalization

• For the separable case:

– Among all linear classifiers that separate the data, find the one that maximizes the margin – Maximizing the margin by minimizing 𝒙!𝒙 if for all examples 𝑧𝒙!𝒚 ≥ 1

• General case:

– Introduce slack variables – one 𝜊" for each example – Slack variables allow the margin constraint above to be violated

56

So far

Support Vector Machines

Eliminate the slack variables to rewrite this This form is more interpretable

57

Maximize margin Minimize total slack (i.e allow as few examples as possible to violate the margin) Tradeoff between the two terms

Support Vector Machines

Eliminate the slack variables to rewrite this This form is more interpretable

58

Maximize margin Minimize total slack (i.e allow as few examples as possible to violate the margin) Tradeoff between the two terms

Maximizing margin and minimizing loss

Three cases

• Example is correctly classified and is outside the margin: penalty = 0
• Example is incorrectly classified: penalty = 1 – yi wTxi
• Example is correctly classified but within the margin: penalty = 1 – yi wTxi

This is the hinge loss function

59

Maximize margin Penalty for the prediction

Maximizing margin and minimizing loss

We can consider three cases

• Example is correctly classified and is outside the margin: penalty = 0
• Example is incorrectly classified: penalty = 1 − 𝑧!𝐱"𝐲!
• Example is correctly classified but within the margin: penalty =1 − 𝑧!𝐱"𝐲!

This is the hinge loss function

60

Maximize margin Penalty for the prediction

Maximizing margin and minimizing loss

We can consider three cases

• Example is correctly classified and is outside the margin: penalty = 0
• Example is incorrectly classified: penalty = 1 − 𝑧!𝐱"𝐲!
• Example is correctly classified but within the margin: penalty =1 − 𝑧!𝐱"𝐲!

This is the hinge loss function

61

Maximize margin Penalty for the prediction

Maximizing margin and minimizing loss

We can consider three cases

• Example is correctly classified and is outside the margin: penalty = 0
• Example is incorrectly classified: penalty = 1 − 𝑧!𝐱"𝐲!
• Example is correctly classified but within the margin: penalty =1 − 𝑧!𝐱"𝐲!

This is the hinge loss function

62

Maximize margin Penalty for the prediction

Maximizing margin and minimizing loss

We can consider three cases

• Example is correctly classified and is outside the margin: penalty = 0
• Example is incorrectly classified: penalty = 1 − 𝑧!𝐱"𝐲!
• Example is correctly classified but within the margin: penalty =1 − 𝑧!𝐱"𝐲!

This is the hinge loss function

63

Maximize margin Penalty for the prediction

Maximizing margin and minimizing loss

We can consider three cases

• Example is correctly classified and is outside the margin: penalty = 0
• Example is incorrectly classified: penalty = 1 − 𝑧!𝐱"𝐲!
• Example is correctly classified but within the margin: penalty =1 − 𝑧!𝐱"𝐲!

This is the hinge loss function

64

Maximize margin Penalty for the prediction

The Hinge Loss

65

ywTx Loss

The Hinge Loss

66

0-1 loss Hinge loss

ywTx Loss

The Hinge Loss

67

0-1 loss

ywTx Loss

0-1 loss: If the sign of y and wTx is the same, then no penalty 0-1 loss: If the sign of y and wTx are different, then penalty = 1 Hinge loss

The Hinge Loss

68

Hinge: Penalize predictions even if they are correct, but too close to the margin

ywTx Loss

Hinge: No penalty if wTx is far away from 1 (-1 for negative examples) Hinge: Incorrect predictions get a linearly increasing penalty with wTx

Maximizing margin and minimizing loss

We can consider three cases

• Example is correctly classified and is outside the margin: penalty = 0
• Example is incorrectly classified: penalty = 1 − 𝑧!𝐱"𝐲!
• Example is correctly classified but within the margin: penalty =1 − 𝑧!𝐱"𝐲!

69

Maximize margin Penalty for the prediction

General learning principle

Risk minimization

70

Define the notion of “loss” over the training data as a function of a hypothesis Learning = find the hypothesis that has lowest loss on the training data

General learning principle

Regularized risk minimization

71

Define the notion of “loss” over the training data as a function of a hypothesis Learning = find the hypothesis that has lowest [Regularizer + loss on the training data] Define a regularization function that penalizes

• ver-complex hypothesis.

Capacity control gives better generalization

SVM objective function

72

Regularization term:

• Maximize the margin
• Imposes a preference over the

hypothesis space and pushes for better generalization

• Can be replaced with other

regularization terms which impose

• ther preferences

Empirical Loss:

• Hinge loss
• Penalizes weight vectors that make

mistakes

• Can be replaced with other loss

functions which impose other preferences

SVM objective function

73

Regularization term:

• Maximize the margin
• Imposes a preference over the

hypothesis space and pushes for better generalization

• Can be replaced with other

regularization terms which impose

• ther preferences

Empirical Loss:

• Hinge loss
• Penalizes weight vectors that make

mistakes

• Can be replaced with other loss

functions which impose other preferences A hyper-parameter that controls the tradeoff between a large margin and a small hinge-loss