The Perceptron Algorithm Machine Learning 1 Some slides based on - - PowerPoint PPT Presentation

Machine Learning

The Perceptron Algorithm

1

Some slides based on lectures from Dan Roth, Avrim Blum and others

Outline

• The Perceptron Algorithm
• Variants of Perceptron
• Perceptron Mistake Bound

2

Where are we?

• The Perceptron Algorithm
• Variants of Perceptron
• Perceptron Mistake Bound

3

Recall: Linear Classifiers

Inputs are 𝑒 dimensional vectors, denoted by 𝐲 Output is a label 𝑧 ∈ {−1, 1}

Linear Threshold Units classify an example 𝐲 using

parameters 𝐱 (a 𝑒 dimensional vector) and 𝑐 (a real number) according the following classification rule Output = sign(𝐱!𝐲 + 𝑐) = sign(∑" 𝑥"𝑦" + 𝑐) 𝐱!𝐲 + 𝑐 ≥ 0 ⇒ 𝑧 = +1 𝐱!𝐲 + 𝑐 < 0 ⇒ 𝑧 = −1

4

𝑐 is called the bias term

Recall: Linear Classifiers

Inputs are 𝑒 dimensional vectors, denoted by 𝐲 Output is a label 𝑧 ∈ {−1, 1}

Linear Threshold Units classify an example 𝐲 using

parameters 𝐱 (a 𝑒 dimensional vector) and 𝑐 (a real number) according the following classification rule Output = sign(𝐱!𝐲 + 𝑐) = sign(∑" 𝑥"𝑦" + 𝑐) 𝐱!𝐲 + 𝑐 ≥ 0 ⇒ 𝑧 = +1 𝐱!𝐲 + 𝑐 < 0 ⇒ 𝑧 = −1

5

𝑐 is called the bias term

∑ sgn 𝑥! 𝑥" 𝑥# 𝑥$ 𝑥% 𝑥& 𝑥' 𝑥( 𝑦! 𝑦" 𝑦# 𝑦$ 𝑦% 𝑦& 𝑦' 𝑦( 1 𝑐

The geometry of a linear classifier

6

sgn(b +w1 x1 + w2x2)

In higher dimensions, a linear classifier represents a hyperplane that separates the space into two half-spaces

x1 x2 + + + + +++ +

• -
• -
• -
• b +w1 x1 + w2x2=0

We only care about the sign, not the magnitude [w1 w2]

The Perceptron

7

The Perceptron algorithm

• Rosenblatt 1958

– (Though there were some hints of a similar idea earlier, eg: Agmon 1954)

• The goal is to find a separating hyperplane

– For separable data, guaranteed to find one

• An online algorithm

– Processes one example at a time

• Several variants exist

– We will see these briefly at towards the end

8

The Perceptron algorithm

Input: A sequence of training examples 𝐲+, 𝑧+ , 𝐲,, 𝑧, , ⋯ where all 𝐲" ∈ ℜ-, 𝑧" ∈ {−1, 1}

• 1. Initialize 𝐱. = 0 ∈ ℜ-
• 2. For each training example 𝐲", 𝑧" :
• 1. Predict y/ = sgn(𝐱0

1𝐲")

• 2. If y/ ≠ 𝑧":
• Update 𝐱02+ ← 𝐱0 + 𝑠(𝑧"𝐲")
• 3. Return final weight vector

9

The Perceptron algorithm

Input: A sequence of training examples 𝐲+, 𝑧+ , 𝐲,, 𝑧, , ⋯ where all 𝐲" ∈ ℜ-, 𝑧" ∈ {−1, 1}

• 1. Initialize 𝐱. = 0 ∈ ℜ-
• 2. For each training example 𝐲", 𝑧" :
• 1. Predict y/ = sgn(𝐱0

1𝐲")

• 2. If y/ ≠ 𝑧":
• Update 𝐱02+ ← 𝐱0 + 𝑠(𝑧"𝐲")
• 3. Return final weight vector

10

Remember: Prediction = sgn(wTx) There is typically a bias term also (wTx + b), but the bias may be treated as a constant feature and folded into w

The Perceptron algorithm

Input: A sequence of training examples 𝐲+, 𝑧+ , 𝐲,, 𝑧, , ⋯ where all 𝐲" ∈ ℜ-, 𝑧" ∈ {−1, 1}

• 1. Initialize 𝐱. = 0 ∈ ℜ-
• 2. For each training example 𝐲", 𝑧" :
• 1. Predict y/ = sgn(𝐱0

1𝐲")

• 2. If y/ ≠ 𝑧":
• Update 𝐱02+ ← 𝐱0 + 𝑠(𝑧"𝐲")
• 3. Return final weight vector

11

Remember: Prediction = sgn(wTx) There is typically a bias term also (wTx + b), but the bias may be treated as a constant feature and folded into w Footnote: For some algorithms it is mathematically easier to represent False as -1, and at other times, as 0. For the Perceptron algorithm, treat -1 as false and +1 as true.

The Perceptron algorithm

Input: A sequence of training examples 𝐲+, 𝑧+ , 𝐲,, 𝑧, , ⋯ where all 𝐲" ∈ ℜ-, 𝑧" ∈ {−1, 1}

• 1. Initialize 𝐱. = 0 ∈ ℜ-
• 2. For each training example 𝐲", 𝑧" :
• 1. Predict y/ = sgn(𝐱0

1𝐲")

• 2. If y/ ≠ 𝑧":
• Update 𝐱02+ ← 𝐱0 + 𝑠(𝑧"𝐲")
• 3. Return final weight vector

12

Mistake on positive: 𝐱)*! ← 𝐱) + 𝑠𝐲+ Mistake on negative: 𝐱)*! ← 𝐱) − 𝑠𝐲+

The Perceptron algorithm

Input: A sequence of training examples 𝐲+, 𝑧+ , 𝐲,, 𝑧, , ⋯ where all 𝐲" ∈ ℜ-, 𝑧" ∈ {−1, 1}

• 1. Initialize 𝐱. = 0 ∈ ℜ-
• 2. For each training example 𝐲", 𝑧" :
• 1. Predict y/ = sgn(𝐱0

1𝐲")

• 2. If y/ ≠ 𝑧":
• Update 𝐱02+ ← 𝐱0 + 𝑠(𝑧"𝐲")
• 3. Return final weight vector

13

r is the learning rate, a small positive number less than 1 Mistake on positive: 𝐱)*! ← 𝐱) + 𝑠𝐲+ Mistake on negative: 𝐱)*! ← 𝐱) − 𝑠𝐲+

The Perceptron algorithm

Input: A sequence of training examples 𝐲+, 𝑧+ , 𝐲,, 𝑧, , ⋯ where all 𝐲" ∈ ℜ-, 𝑧" ∈ {−1, 1}

• 1. Initialize 𝐱. = 0 ∈ ℜ-
• 2. For each training example 𝐲", 𝑧" :
• 1. Predict y/ = sgn(𝐱0

1𝐲")

• 2. If y/ ≠ 𝑧":
• Update 𝐱02+ ← 𝐱0 + 𝑠(𝑧"𝐲")
• 3. Return final weight vector

14

r is the learning rate, a small positive number less than 1 Update only on error. A mistake-driven algorithm Mistake on positive: 𝐱)*! ← 𝐱) + 𝑠𝐲+ Mistake on negative: 𝐱)*! ← 𝐱) − 𝑠𝐲+

The Perceptron algorithm

Input: A sequence of training examples 𝐲+, 𝑧+ , 𝐲,, 𝑧, , ⋯ where all 𝐲" ∈ ℜ-, 𝑧" ∈ {−1, 1}

• 1. Initialize 𝐱. = 0 ∈ ℜ-
• 2. For each training example 𝐲", 𝑧" :
• 1. Predict y/ = sgn(𝐱0

1𝐲")

• 2. If y/ ≠ 𝑧":
• Update 𝐱02+ ← 𝐱0 + 𝑠(𝑧"𝐲")
• 3. Return final weight vector

15

r is the learning rate, a small positive number less than 1 Update only on error. A mistake-driven algorithm Mistake on positive: 𝐱)*! ← 𝐱) + 𝑠𝐲+ Mistake on negative: 𝐱)*! ← 𝐱) − 𝑠𝐲+ Mistake can be written as y+𝐱)

,𝐲+ ≤ 0

The Perceptron algorithm

Input: A sequence of training examples 𝐲+, 𝑧+ , 𝐲,, 𝑧, , ⋯ where all 𝐲" ∈ ℜ-, 𝑧" ∈ {−1, 1}

• 1. Initialize 𝐱. = 0 ∈ ℜ-
• 2. For each training example 𝐲", 𝑧" :
• 1. Predict y/ = sgn(𝐱0

1𝐲")

• 2. If y/ ≠ 𝑧":
• Update 𝐱02+ ← 𝐱0 + 𝑠(𝑧"𝐲")
• 3. Return final weight vector

16

r is the learning rate, a small positive number less than 1 Update only on error. A mistake-driven algorithm This is the simplest version. We will see more robust versions shortly Mistake can be written as y+𝐱)

,𝐲+ ≤ 0

Mistake on positive: 𝐱)*! ← 𝐱) + 𝑠𝐲+ Mistake on negative: 𝐱)*! ← 𝐱) − 𝑠𝐲+

Intuition behind the update

Suppose we have made a mistake on a positive example That is, 𝑧 = +1 and 𝐱!

"𝐲 ≤ 0

Call the new weight vector 𝐱!#$ = 𝐱! + 𝐲 (say r = 1) The new dot product is 𝐱%#$

" 𝐲 = 𝐱! + 𝐲 "𝐲 = 𝐱! "𝐲 + 𝐲𝐔𝐲 ≥ 𝐱𝐮 𝐔𝐲

For a positive example, the Perceptron update will increase the score assigned to the same input Similar reasoning for negative examples

17

Mistake on positive: 𝐱)*! ← 𝐱) + 𝑠𝐲+ Mistake on negative: 𝐱)*! ← 𝐱) − 𝑠𝐲+

Geometry of the perceptron update

18

wold Predict Mistake on positive: 𝐱)*! ← 𝐱) + 𝑠𝐲+ Mistake on negative: 𝐱)*! ← 𝐱) − 𝑠𝐲+

Geometry of the perceptron update

19

wold (x, +1) Predict Mistake on positive: 𝐱)*! ← 𝐱) + 𝑠𝐲+ Mistake on negative: 𝐱)*! ← 𝐱) − 𝑠𝐲+

Geometry of the perceptron update

20

wold (x, +1)

For a mistake on a positive example

Predict Mistake on positive: 𝐱)*! ← 𝐱) + 𝑠𝐲+ Mistake on negative: 𝐱)*! ← 𝐱) − 𝑠𝐲+

Geometry of the perceptron update

21

wold (x, +1) 𝐱 ← 𝐱 + 𝑧𝐲

For a mistake on a positive example

(x, +1) Predict Update Mistake on positive: 𝐱)*! ← 𝐱) + 𝑠𝐲+ Mistake on negative: 𝐱)*! ← 𝐱) − 𝑠𝐲+

Geometry of the perceptron update

22

wold (x, +1) 𝐱 ← 𝐱 + 𝑧𝐲

For a mistake on a positive example

(x, +1) Predict Update y x Mistake on positive: 𝐱)*! ← 𝐱) + 𝑠𝐲+ Mistake on negative: 𝐱)*! ← 𝐱) − 𝑠𝐲+

Geometry of the perceptron update

23

wold (x, +1) 𝐱 ← 𝐱 + 𝑧𝐲

For a mistake on a positive example

(x, +1) Predict Update y x Mistake on positive: 𝐱)*! ← 𝐱) + 𝑠𝐲+ Mistake on negative: 𝐱)*! ← 𝐱) − 𝑠𝐲+

Geometry of the perceptron update

24

wold (x, +1) (x, +1) wnew 𝐱 ← 𝐱 + 𝑧𝐲

For a mistake on a positive example

(x, +1) Predict Update After y x Mistake on positive: 𝐱)*! ← 𝐱) + 𝑠𝐲+ Mistake on negative: 𝐱)*! ← 𝐱) − 𝑠𝐲+

Geometry of the perceptron update

25

wold Predict

Geometry of the perceptron update

26

wold (x, -1) Predict

For a mistake on a negative example

Geometry of the perceptron update

27

wold (x, -1) 𝐱 ← 𝐱 + 𝑧𝐲 (x, -1) Predict Update

For a mistake on a negative example

y x

Geometry of the perceptron update

28

wold (x, -1) 𝐱 ← 𝐱 + 𝑧𝐲 (x, -1) Predict Update

For a mistake on a negative example

y x

Geometry of the perceptron update

29

wold (x, -1) 𝐱 ← 𝐱 + 𝑧𝐲 (x, -1) Predict Update

For a mistake on a negative example

y x

Geometry of the perceptron update

30

wold (x, -1) (x, -1) wnew 𝐱 ← 𝐱 + 𝑧𝐲 (x, -1) Predict Update After

For a mistake on a negative example

y x

Where are we?

• The Perceptron Algorithm
• Variants of Perceptron
• Perceptron Mistake Bound

31

Practical use of the Perceptron algorithm

• 1. Using the Perceptron algorithm with a finite dataset
• 2. Voting and Averaging
• 3. Margin Perceptron

32

• 1. The “standard” algorithm

Given a training set 𝐸 = 𝐲(, 𝑧( where 𝐲( ∈ ℜ), 𝑧( ∈ −1, 1 1. Initialize 𝐱 = 𝟏 ∈ ℜ) 2. For epoch in 1 ⋯ 𝑈: 1. Shuffle the data 2. For each training example 𝐲(, 𝑧( ∈ 𝐸:

• If 𝑧(𝐱*𝐲( ≤ 0, then:

– update 𝐱 ← 𝐱 + 𝑠𝑧(𝐲( 3. Return 𝐱 Prediction on a new example with features 𝐲: sgn 𝐱*𝐲

33

• 1. The “standard” algorithm

Given a training set 𝐸 = 𝐲(, 𝑧( where 𝐲( ∈ ℜ), 𝑧( ∈ −1, 1 1. Initialize 𝐱 = 𝟏 ∈ ℜ) 2. For epoch in 1 ⋯ 𝑈: 1. Shuffle the data 2. For each training example 𝐲(, 𝑧( ∈ 𝐸:

• If 𝑧(𝐱*𝐲( ≤ 0, then:

– update 𝐱 ← 𝐱 + 𝑠𝑧(𝐲( 3. Return 𝐱 Prediction on a new example with features 𝐲: sgn 𝐱*𝐲

34

T is a hyper-parameter to the algorithm Another way of writing that there is an error

• 2. Voting and Averaging
• So far: We return the final weight vector
• Voted perceptron

– Remember every weight vector in your sequence of updates. – At final prediction time, each weight vector gets to vote on the label. The number of votes it gets is the number of iterations it survived before being updated – Comes with strong theoretical guarantees about generalization, impractical because of storage issues

• Averaged perceptron

– Instead of using all weight vectors, use the average weight vector (i.e longer surviving weight vectors get more say) – More practical alternative and widely used

35

• 2. Voting and Averaging
• So far: We return the final weight vector
• Voted perceptron

– Remember every weight vector in your sequence of updates. – At final prediction time, each weight vector gets to vote on the label. The number of votes it gets is the number of iterations it survived before being updated – Comes with strong theoretical guarantees about generalization, impractical because of storage issues

• Averaged perceptron

– Instead of using all weight vectors, use the average weight vector (i.e longer surviving weight vectors get more say) – More practical alternative and widely used

36

Averaged Perceptron

Given a training set 𝐸 = 𝐲(, 𝑧( where 𝐲( ∈ ℜ), 𝑧( ∈ −1, 1 1. Initialize 𝐱 = 𝟏 ∈ ℜ) and 𝐛 = 𝟏 ∈ ℜ) 2. For epoch in 1 ⋯ 𝑈: 1. Shuffle the data 2. For each training example 𝐲(, 𝑧( ∈ 𝐸:

• If 𝑧(𝐱*𝐲( ≤ 0, then:

– update 𝐱 ← 𝐱 + 𝑠𝑧(𝐲(

• 𝐛 ← 𝒃 + 𝐱

3. Return 𝐛 Prediction on a new example with features 𝐲: sgn 𝐛*𝐲

37

Averaged Perceptron

Given a training set 𝐸 = 𝐲(, 𝑧( where 𝐲( ∈ ℜ), 𝑧( ∈ −1, 1 1. Initialize 𝐱 = 𝟏 ∈ ℜ) and 𝐛 = 𝟏 ∈ ℜ) 2. For epoch in 1 ⋯ 𝑈: 1. Shuffle the data 2. For each training example 𝐲(, 𝑧( ∈ 𝐸:

• If 𝑧(𝐱*𝐲( ≤ 0, then:

– update 𝐱 ← 𝐱 + 𝑠𝑧(𝐲(

• 𝐛 ← 𝒃 + 𝐱

3. Return 𝐛 Prediction on a new example with features 𝐲: sgn 𝐛*𝐲

38

This is the simplest version of the averaged perceptron There are some easy programming tricks to make sure that a is also updated

• nly when there is an error

Averaged Perceptron

Given a training set 𝐸 = 𝐲(, 𝑧( where 𝐲( ∈ ℜ), 𝑧( ∈ −1, 1 1. Initialize 𝐱 = 𝟏 ∈ ℜ) and 𝐛 = 𝟏 ∈ ℜ) 2. For epoch in 1 ⋯ 𝑈: 1. Shuffle the data 2. For each training example 𝐲(, 𝑧( ∈ 𝐸:

• If 𝑧(𝐱*𝐲( ≤ 0, then:

– update 𝐱 ← 𝐱 + 𝑠𝑧(𝐲(

• 𝐛 ← 𝒃 + 𝐱

3. Return 𝐛 Prediction on a new example with features 𝐲: sgn 𝐛*𝐲

39

This is the simplest version of the averaged perceptron There are some easy programming tricks to make sure that a is also updated

• nly when there is an error

If you want to use the Perceptron algorithm, use averaging

• 3. Margin Perceptron
• Perceptron makes updates only when the prediction is

incorrect 𝑧"𝐱1𝐲" ≤ 0

• What if the prediction is close to being incorrect? That is, Pick

a small positive 𝜃 and update when 𝑧"𝐱1𝐲" ≤ 𝜃

• Can generalize better, but need to choose 𝜃

Exercise: Why is the margin perceptron a good idea?

40

The Perceptron

41

The hype

42

The New Yorker, December 6, 1958 P. 44 The New York Times, July 8 1958

The hype

43

The New Yorker, December 6, 1958 P. 44 The New York Times, July 8 1958 The IBM 704 computer

What you need to know

• The Perceptron algorithm
• The geometry of the update
• What can it represent
• Variants of the Perceptron algorithm

44