Lecture 3: Perceptron Princeton University COS 495 Instructor: - - PowerPoint PPT Presentation

Machine Learning Basics Lecture 3: Perceptron

Princeton University COS 495 Instructor: Yingyu Liang

Perceptron

Overview

• Previous lectures: (Principle for loss function) MLE to derive loss
• Example: linear regression; some linear classification models
• This lecture: (Principle for optimization) local improvement
• Example: Perceptron; SGD

Task

(𝑥∗)𝑈𝑦 = 0 Class +1 Class -1 𝑥∗ (𝑥∗)𝑈𝑦 > 0 (𝑥∗)𝑈𝑦 < 0

Attempt

• Given training data 𝑦𝑗, 𝑧𝑗 : 1 ≤ 𝑗 ≤ 𝑜 i.i.d. from distribution 𝐸
• Hypothesis 𝑔

𝑥 𝑦 = 𝑥𝑈𝑦

• 𝑧 = +1 if 𝑥𝑈𝑦 > 0
• 𝑧 = −1 if 𝑥𝑈𝑦 < 0
• Prediction: 𝑧 = sign(𝑔

𝑥 𝑦 ) = sign(𝑥𝑈𝑦)

• Goal: minimize classification error

Perceptron Algorithm

• Assume for simplicity: all 𝑦𝑗 has length 1

Perceptron: figure from the lecture note of Nina Balcan

Intuition: correct the current mistake

• If mistake on a positive example

𝑥𝑢+1

𝑈 𝑦 = 𝑥𝑢 + 𝑦 𝑈𝑦 = 𝑥𝑢 𝑈𝑦 + 𝑦𝑈𝑦 = 𝑥𝑢 𝑈𝑦 + 1

• If mistake on a negative example

𝑥𝑢+1

𝑈 𝑦 = 𝑥𝑢 − 𝑦 𝑈𝑦 = 𝑥𝑢 𝑈𝑦 − 𝑦𝑈𝑦 = 𝑥𝑢 𝑈𝑦 − 1

The Perceptron Theorem

• Suppose there exists 𝑥∗ that correctly classifies

𝑦𝑗, 𝑧𝑗

• W.L.O.G., all 𝑦𝑗 and 𝑥∗ have length 1, so the minimum distance of any

example to the decision boundary is 𝛿 = min

𝑗

| 𝑥∗ 𝑈𝑦𝑗|

• Then Perceptron makes at most

1 𝛿 2

mistakes

The Perceptron Theorem

• Suppose there exists 𝑥∗ that correctly classifies

𝑦𝑗, 𝑧𝑗

• W.L.O.G., all 𝑦𝑗 and 𝑥∗ have length 1, so the minimum distance of any

example to the decision boundary is 𝛿 = min

𝑗

| 𝑥∗ 𝑈𝑦𝑗|

• Then Perceptron makes at most

1 𝛿 2

mistakes

Need not be i.i.d. ! Do not depend on 𝑜, the length of the data sequence!

Analysis

• First look at the quantity 𝑥𝑢

𝑈𝑥∗

• Claim 1: 𝑥𝑢+1

𝑈 𝑥∗ ≥ 𝑥𝑢 𝑈𝑥∗ + 𝛿

• Proof: If mistake on a positive example 𝑦

𝑥𝑢+1

𝑈 𝑥∗ = 𝑥𝑢 + 𝑦 𝑈𝑥∗ = 𝑥𝑢 𝑈𝑥∗ + 𝑦𝑈𝑥∗ ≥ 𝑥𝑢 𝑈𝑥∗ + 𝛿

• If mistake on a negative example

𝑥𝑢+1

𝑈 𝑥∗ = 𝑥𝑢 − 𝑦 𝑈𝑥∗ = 𝑥𝑢 𝑈𝑥∗ − 𝑦𝑈𝑥∗ ≥ 𝑥𝑢 𝑈𝑥∗ + 𝛿

Analysis

• Next look at the quantity 𝑥𝑢
• Claim 2: 𝑥𝑢+1

2 ≤

𝑥𝑢

2 + 1

• Proof: If mistake on a positive example 𝑦

𝑥𝑢+1

2 =

𝑥𝑢 + 𝑦

2 =

𝑥𝑢

2 +

𝑦

2 + 2𝑥𝑢 𝑈𝑦

Negative since we made a mistake on x

Analysis: putting things together

• Claim 1: 𝑥𝑢+1

𝑈 𝑥∗ ≥ 𝑥𝑢 𝑈𝑥∗ + 𝛿

• Claim 2: 𝑥𝑢+1

2 ≤

𝑥𝑢

2 + 1

After 𝑁 mistakes:

• 𝑥𝑁+1

𝑈

𝑥∗ ≥ 𝛿𝑁

• 𝑥𝑁+1

≤ √𝑁

• 𝑥𝑁+1

𝑈

𝑥∗ ≤ 𝑥𝑁+1 So 𝛿𝑁 ≤ √𝑁, and thus 𝑁 ≤

1 𝛿 2

Intuition

• Claim 1: 𝑥𝑢+1

𝑈 𝑥∗ ≥ 𝑥𝑢 𝑈𝑥∗ + 𝛿

• Claim 2: 𝑥𝑢+1

2 ≤

𝑥𝑢

2 + 1

The correlation gets larger. Could be:

• 1. 𝑥𝑢+1 gets closer to 𝑥∗
• 2. 𝑥𝑢+1 gets much longer

Rules out the bad case “2. 𝑥𝑢+1 gets much longer”

Some side notes on Perceptron

History

Figure from Pattern Recognition and Machine Learning, Bishop

Note: connectionism vs symbolism

• Symbolism: AI can be achieved by representing concepts as symbols
• Example: rule-based expert system, formal grammar
• Connectionism: explain intellectual abilities using connections

between neurons (i.e., artificial neural networks)

• Example: perceptron, larger scale neural networks

Symbolism example: Credit Risk Analysis

Example from Machine learning lecture notes by Tom Mitchell

Connectionism example

Figure from Pattern Recognition and machine learning, Bishop

Neuron/perceptron

Note: connectionism v.s. symbolism

• Formal theories of logical reasoning, grammar, and other higher

mental faculties compel us to think of the mind as a machine for rule- based manipulation of highly structured arrays of symbols. What we know of the brain compels us to think of human information processing in terms of manipulation of a large unstructured set of numbers, the activity levels of interconnected neurons.

• --- The Central Paradox of Cognition (Smolensky et al., 1992)

Note: online vs batch

• Batch: Given training data 𝑦𝑗, 𝑧𝑗 : 1 ≤ 𝑗 ≤ 𝑜 , typically i.i.d.
• Online: data points arrive one by one
• 1. The algorithm receives an unlabeled example 𝑦𝑗
• 2. The algorithm predicts a classification of this example.
• 3. The algorithm is then told the correct answer 𝑧𝑗, and update its model

Stochastic gradient descent (SGD)

Gradient descent

• Minimize loss ෠

𝑀 𝜄 , where the hypothesis is parametrized by 𝜄

• Gradient descent
• Initialize 𝜄0
• 𝜄𝑢+1 = 𝜄𝑢 − 𝜃𝑢𝛼෠

𝑀 𝜄𝑢

Stochastic gradient descent (SGD)

• Suppose data points arrive one by one
• ෠

𝑀 𝜄 =

1 𝑜 σ𝑢=1 𝑜

𝑚(𝜄, 𝑦𝑢, 𝑧𝑢), but we only know 𝑚(𝜄, 𝑦𝑢, 𝑧𝑢) at time 𝑢

• Idea: simply do what you can based on local information
• Initialize 𝜄0
• 𝜄𝑢+1 = 𝜄𝑢 − 𝜃𝑢𝛼𝑚(𝜄𝑢, 𝑦𝑢, 𝑧𝑢)

Example 1: linear regression

• Find 𝑔

𝑥 𝑦 = 𝑥𝑈𝑦 that minimizes ෠

𝑀 𝑔

𝑥 = 1 𝑜 σ𝑢=1 𝑜

𝑥𝑈𝑦𝑢 − 𝑧𝑢 2

• 𝑚 𝑥, 𝑦𝑢, 𝑧𝑢 =

1 𝑜 𝑥𝑈𝑦𝑢 − 𝑧𝑢 2

• 𝑥𝑢+1 = 𝑥𝑢 − 𝜃𝑢𝛼𝑚 𝑥𝑢, 𝑦𝑢, 𝑧𝑢 = 𝑥𝑢 −

2𝜃𝑢 𝑜

𝑥𝑢

𝑈𝑦𝑢 − 𝑧𝑢 𝑦𝑢

Example 2: logistic regression

• Find 𝑥 that minimizes

෠ 𝑀 𝑥 = − 1 𝑜 ෍

𝑧𝑢=1

log𝜏(𝑥𝑈𝑦𝑢) − 1 𝑜 ෍

𝑧𝑢=−1

log[1 − 𝜏 𝑥𝑈𝑦𝑢 ] ෠ 𝑀 𝑥 = − 1 𝑜 ෍

𝑢

log𝜏(𝑧𝑢𝑥𝑈𝑦𝑢) 𝑚 𝑥, 𝑦𝑢, 𝑧𝑢 = −1 𝑜 log𝜏(𝑧𝑢𝑥𝑈𝑦𝑢)

Example 2: logistic regression

• Find 𝑥 that minimizes

𝑚 𝑥, 𝑦𝑢, 𝑧𝑢 = −1 𝑜 log𝜏(𝑧𝑢𝑥𝑈𝑦𝑢) 𝑥𝑢+1 = 𝑥𝑢 − 𝜃𝑢𝛼𝑚 𝑥𝑢, 𝑦𝑢, 𝑧𝑢 = 𝑥𝑢 +

𝜃𝑢 𝑜 𝜏 𝑏 1−𝜏 𝑏 𝜏(𝑏)

𝑧𝑢𝑦𝑢 Where 𝑏 = 𝑧𝑢𝑥𝑢

𝑈𝑦𝑢

Example 3: Perceptron

• Hypothesis: 𝑧 = sign(𝑥𝑈𝑦)
• Define hinge loss

𝑚 𝑥, 𝑦𝑢, 𝑧𝑢 = −𝑧𝑢𝑥𝑈𝑦𝑢 𝕁[mistake on 𝑦𝑢] ෠ 𝑀 𝑥 = − ෍

𝑢

𝑧𝑢𝑥𝑈𝑦𝑢 𝕁[mistake on 𝑦𝑢] 𝑥𝑢+1 = 𝑥𝑢 − 𝜃𝑢𝛼𝑚 𝑥𝑢, 𝑦𝑢, 𝑧𝑢 = 𝑥𝑢 + 𝜃𝑢𝑧𝑢𝑦𝑢 𝕁[mistake on 𝑦𝑢]

Example 3: Perceptron

• Hypothesis: 𝑧 = sign(𝑥𝑈𝑦)

𝑥𝑢+1 = 𝑥𝑢 − 𝜃𝑢𝛼𝑚 𝑥𝑢, 𝑦𝑢, 𝑧𝑢 = 𝑥𝑢 + 𝜃𝑢𝑧𝑢𝑦𝑢 𝕁[mistake on 𝑦𝑢]

• Set 𝜃𝑢 = 1. If mistake on a positive example

𝑥𝑢+1 = 𝑥𝑢 + 𝑧𝑢𝑦𝑢 = 𝑥𝑢 + 𝑦

• If mistake on a negative example

𝑥𝑢+1 = 𝑥𝑢 + 𝑧𝑢𝑦𝑢 = 𝑥𝑢 − 𝑦

Pros & Cons

Pros:

• Widely applicable
• Easy to implement in most cases
• Guarantees for many losses
• Good performance: error/running time/memory etc.

Cons:

• No guarantees for non-convex opt (e.g., those in deep learning)
• Hyper-parameters: initialization, learning rate

Mini-batch

• Instead of one data point, work with a small batch of 𝑐 points

(𝑦𝑢𝑐+1,𝑧𝑢𝑐+1),…, (𝑦𝑢𝑐+𝑐,𝑧𝑢𝑐+𝑐)

• Update rule

𝜄𝑢+1 = 𝜄𝑢 − 𝜃𝑢𝛼 1 𝑐 ෍

1≤𝑗≤𝑐

𝑚 𝜄𝑢, 𝑦𝑢𝑐+𝑗, 𝑧𝑢𝑐+𝑗

• Other variants: variance reduction etc.

Homework

Homework 1

• Assignment online
• Course website:

http://www.cs.princeton.edu/courses/archive/spring16/cos495/

• Piazza: https://piazza.com/princeton/spring2016/cos495
• Due date: Feb 17th (one week)
• Submission
• Math part: hand-written/print; submit to TA (Office: EE, C319B)
• Coding part: in Matlab/Python; submit the .m/.py file on Piazza

Homework 1

• Grading policy: every late day reduces the attainable credit for the

exercise by 10%.

• Collaboration:
• Discussion on the problem sets is allowed
• Students are expected to finish the homework by himself/herself
• The people you discussed with on assignments should be clearly detailed:

before the solution to each question, list all people that you discussed with on that particular question.