Machine Learning: Chenhao Tan University of Colorado Boulder - - PowerPoint PPT Presentation

Machine Learning: Chenhao Tan

University of Colorado Boulder

LECTURE 2 Slides adapted from Jordan Boyd-Graber, Thorsten Joachims, Kilian Weinberger

Machine Learning: Chenhao Tan | Boulder | 1 of 31

Logistics

• Piazza: https://piazza.com/colorado/fall2017/csci5622/
• Moodle:

https://moodle.cs.colorado.edu/course/view.php?id=507

• Prerequisite quiz
• Final project
• iCliker

Machine Learning: Chenhao Tan | Boulder | 2 of 31

Outline

Supervised Learning Data representation K-nearest neighbors Overview Performance Guarantee Curse of Dimensionality

Machine Learning: Chenhao Tan | Boulder | 3 of 31

Supervised Learning

Outline

Supervised Learning Data representation K-nearest neighbors Overview Performance Guarantee Curse of Dimensionality

Machine Learning: Chenhao Tan | Boulder | 4 of 31

Supervised Learning

Supervised Learning

Data

X

Labels

Y

• Supervised methods find patterns in fully observed data and then try to

predict something from partially observed data.

• For example, in sentiment analysis, after learning something from annotated

reviews, we want to take new reviews and automatically identify sentiments.

Machine Learning: Chenhao Tan | Boulder | 5 of 31

Supervised Learning

Formal Definitions

• Labels Y, e.g., binary labels y ∈ {+1, −1}
• Instance space X, all the possible instances (based on data representation)
• Target function f: X → Y (f is unknown)

Machine Learning: Chenhao Tan | Boulder | 6 of 31

Supervised Learning

Formal Definitions

• Labels Y, e.g., binary labels y ∈ {+1, −1}
• Instance space X, all the possible instances (based on data representation)
• Target function f: X → Y (f is unknown)
• Example/instance (x, y)
• Training data Strain: collection of examples observed by the algorithm

Machine Learning: Chenhao Tan | Boulder | 6 of 31

Supervised Learning

Formal Definitions

• Goal of a learning algorithm:

Find a function h : X → Y from training data Strain so that h approximates f

Machine Learning: Chenhao Tan | Boulder | 7 of 31

Supervised Learning

Supervised learning in a nutshell

Strain = {(x, y)} → h

Machine Learning: Chenhao Tan | Boulder | 8 of 31

Supervised Learning

No Free Lunch Theorems

• No free lunch for supervised machine learning [Wolpert, 1996]:

in a noise-free scenario where the loss function is the misclassification rate, if

• ne is interested in off-training-set error, then there are no a priori distinctions

between learning algorithms.

Machine Learning: Chenhao Tan | Boulder | 9 of 31

Supervised Learning

No Free Lunch Theorems

• No free lunch for supervised machine learning [Wolpert, 1996]:

in a noise-free scenario where the loss function is the misclassification rate, if

• ne is interested in off-training-set error, then there are no a priori distinctions

between learning algorithms. Corollary I: there is no single ML algorithm that works for everything. Corollary II: every successful ML algorithm makes assumptions.

Machine Learning: Chenhao Tan | Boulder | 9 of 31

Supervised Learning

No Free Lunch Theorems

• No free lunch for supervised machine learning [Wolpert, 1996]:

in a noise-free scenario where the loss function is the misclassification rate, if

• ne is interested in off-training-set error, then there are no a priori distinctions

between learning algorithms. Corollary I: there is no single ML algorithm that works for everything. Corollary II: every successful ML algorithm makes assumptions.

• No free lunch for search/optimization [Wolpert and Macready, 1997]: All

algorithms that search for an extremum of a cost function perform exactly the same when averaged over all possible cost functions.

Machine Learning: Chenhao Tan | Boulder | 9 of 31

Data representation

Outline

Supervised Learning Data representation K-nearest neighbors Overview Performance Guarantee Curse of Dimensionality

Machine Learning: Chenhao Tan | Boulder | 10 of 31

Data representation

Data representation

Republican nominee George Bush said he felt nervous as he voted today in his adopted home state of Texas, where he ended...

Machine Learning: Chenhao Tan | Boulder | 11 of 31

Data representation

Data representation

Let us have an interactive example to think through data representation!

Machine Learning: Chenhao Tan | Boulder | 12 of 31

Data representation

Data representation

Let us have an interactive example to think through data representation! Auto insurance quotes id rent income urban state car value car year 1 yes 50,000 no CO 20,000 2010 2 yes 70,000 no CO 30,000 2012 3 no 250,000 yes CO 55,000 2017 4 yes 200,000 yes NY 50,000 2016

Machine Learning: Chenhao Tan | Boulder | 12 of 31

Data representation

Understanding assumptions in data representation

Akaike information criterion

• distributed. Let n be the number of observations and RSS be

• ther images of cats

• F. silvestris

• commands. Individual cats have also been known to learn to manipulate simple mechanisms, such as doorknobs. Cats use

• pets. This hobby is known as the "Cat Fancy".

• nly, to its present role as the administrative center of the university. Originally, the sculptures in front of the building were lions, as a

• C. lupus

• species. The domestic dog has been (and continues to be) one of the most widely-kept

Akaike information criterion

• distributed. Let n be the number of observations and RSS be

• The methods we’ll study make assumptions about the data on which they are
• applied. E.g.,
• Documents can be analyzed as a sequence of words;
• or, as a “bag” of words.
• Independent of each other;
• or, as connected to each other
• What are the assumptions behind the methods?
• When/why are they appropriate?
• Much of this is an art, and it is inherently dynamic

Machine Learning: Chenhao Tan | Boulder | 13 of 31

K-nearest neighbors

Outline

Supervised Learning Data representation K-nearest neighbors Overview Performance Guarantee Curse of Dimensionality

Machine Learning: Chenhao Tan | Boulder | 14 of 31

K-nearest neighbors | Overview

K-nearest neighbors

Find the K-nearest neighbors of x in training data and predict the majority label of those K points.

Machine Learning: Chenhao Tan | Boulder | 15 of 31

K-nearest neighbors | Overview

K-nearest neighbors

Find the K-nearest neighbors of x in training data and predict the majority label of those K points. h(x) = arg max

y∈{+1,−1}

• (x′,y′)∈NN(x,Strain,k)

I(y = y′)

Machine Learning: Chenhao Tan | Boulder | 15 of 31

K-nearest neighbors | Overview

K-nearest neighbors

Find the K-nearest neighbors of x in training data and predict the majority label of those K points. h(x) = arg max

y∈{+1,−1}

• (x′,y′)∈NN(x,Strain,k)

I(y = y′) Assumptions in the algorithm: nearby instances share similar labels.

Machine Learning: Chenhao Tan | Boulder | 15 of 31

K-nearest neighbors | Overview

A Simple Example

• Suppose you’re a big company monitoring the web
• Someone says something about your product (x)
• You want to know whether they’re positive (y = +1) or negative (y = −1)

Machine Learning: Chenhao Tan | Boulder | 16 of 31

K-nearest neighbors | Overview

k = 1

Train

Apple makes great laptops → (+1)

Machine Learning: Chenhao Tan | Boulder | 17 of 31

K-nearest neighbors | Overview

k = 1

Train

Apple makes great laptops → (+1)

Test

Apple makes great laptops

Machine Learning: Chenhao Tan | Boulder | 17 of 31

K-nearest neighbors | Overview

k = 1

Train

Apple makes great laptops → (+1)

Test

Apple really makes great laptops

Machine Learning: Chenhao Tan | Boulder | 17 of 31

K-nearest neighbors | Overview

Parameters in the algorithm

• Distance function

Machine Learning: Chenhao Tan | Boulder | 18 of 31

K-nearest neighbors | Overview

Parameters in the algorithm

• Distance function

Discrete

d(x1, x2) = 1 − |x1 ∩ x2| |x1 ∪ x2| (1)

Continuous

Euclidean distance d(x1, x2) = x1 − x22 (2)

Machine Learning: Chenhao Tan | Boulder | 18 of 31

K-nearest neighbors | Overview

Parameters in the algorithm

• Distance function

Discrete

d(x1, x2) = 1 − |x1 ∩ x2| |x1 ∪ x2| (1)

Continuous

Euclidean distance d(x1, x2) = x1 − x22 (2) Manhattan distance d(x1, x2) = x1 − x21 (3)

Machine Learning: Chenhao Tan | Boulder | 18 of 31

K-nearest neighbors | Overview

Parameters in the algorithm

• Distance function

Discrete

d(x1, x2) = 1 − |x1 ∩ x2| |x1 ∪ x2| (1)

Continuous

Euclidean distance d(x1, x2) = x1 − x22 (2) Manhattan distance d(x1, x2) = x1 − x21 (3)

• Number of nearest neighbors k

Machine Learning: Chenhao Tan | Boulder | 18 of 31

K-nearest neighbors | Overview

KNN Classification

K = 1

What is the prediction

• f y1?

Closest points: Prediction:

1 2 3 4

Machine Learning: Chenhao Tan | Boulder | 19 of 31

K-nearest neighbors | Overview

KNN Classification

K = 1

What is the prediction

• f y2?

Closest points: Prediction:

1 2 3 4

Machine Learning: Chenhao Tan | Boulder | 19 of 31

K-nearest neighbors | Overview

KNN Classification

K = 1

What is the prediction

• f y3?

Closest points: Prediction:

1 2 3 4

Machine Learning: Chenhao Tan | Boulder | 19 of 31

K-nearest neighbors | Overview

KNN Classification

K = 1

What is the prediction

• f y4?

Closest points: Prediction:

1 2 3 4

Machine Learning: Chenhao Tan | Boulder | 19 of 31

K-nearest neighbors | Overview

KNN Classification

K = 2

What is the prediction

• f y1?

Closest points: Prediction:

1 2 3 4

Machine Learning: Chenhao Tan | Boulder | 19 of 31

K-nearest neighbors | Overview

KNN Classification

K = 2

What is the prediction

• f y2?

Closest points: Prediction:

1 2 3 4

Machine Learning: Chenhao Tan | Boulder | 19 of 31

K-nearest neighbors | Overview

KNN Classification

K = 2

What is the prediction

• f y3?

Closest points: Prediction:

1 2 3 4

Machine Learning: Chenhao Tan | Boulder | 19 of 31

K-nearest neighbors | Overview

KNN Classification

K = 2

What is the prediction

• f y4?

Closest points: Prediction:

1 2 3 4

Machine Learning: Chenhao Tan | Boulder | 19 of 31

K-nearest neighbors | Overview

KNN Classification

K = 3

What is the prediction

• f y1?

Closest points: Prediction:

1 2 3 4

Machine Learning: Chenhao Tan | Boulder | 19 of 31

K-nearest neighbors | Overview

KNN Classification

K = 3

What is the prediction

• f y2?

Closest points: Prediction:

1 2 3 4

Machine Learning: Chenhao Tan | Boulder | 19 of 31

K-nearest neighbors | Overview

KNN Classification

K = 3

What is the prediction

• f y3?

Closest points: Prediction:

1 2 3 4

Machine Learning: Chenhao Tan | Boulder | 19 of 31

K-nearest neighbors | Overview

KNN Classification

K = 3

What is the prediction

• f y4?

Closest points: Prediction:

1 2 3 4

Machine Learning: Chenhao Tan | Boulder | 19 of 31

K-nearest neighbors | Overview

Machine Learning: Chenhao Tan | Boulder | 20 of 31

K-nearest neighbors | Performance Guarantee

How good is K-NN in theory? (Performance guarantee)

Machine Learning: Chenhao Tan | Boulder | 21 of 31

K-nearest neighbors | Performance Guarantee

Performance Guarantee for 1-NN

What if we have close to infinite training data (n → ∞), how well can 1-NN perform?

Machine Learning: Chenhao Tan | Boulder | 22 of 31

K-nearest neighbors | Performance Guarantee

Performance Guarantee for 1-NN

What if we have close to infinite training data (n → ∞), how well can 1-NN perform? Bayes optimal classifier Assuming that we know P(y|x), y ∈ {+1, −1}, best prediction: y∗ = arg maxy P(y|x)

Machine Learning: Chenhao Tan | Boulder | 22 of 31

K-nearest neighbors | Performance Guarantee

Performance Guarantee for 1-NN

What if we have close to infinite training data (n → ∞), how well can 1-NN perform? Bayes optimal classifier Assuming that we know P(y|x), y ∈ {+1, −1}, best prediction: y∗ = arg maxy P(y|x) Example: P(+1|x) = 0.9, P(−1|x) = 0.1, what do you predict for x?

Machine Learning: Chenhao Tan | Boulder | 22 of 31

K-nearest neighbors | Performance Guarantee

Performance Guarantee for 1-NN

What if we have close to infinite training data (n → ∞), how well can 1-NN perform? Bayes optimal classifier Assuming that we know P(y|x), y ∈ {+1, −1}, best prediction: y∗ = arg maxy P(y|x) Example: P(+1|x) = 0.9, P(−1|x) = 0.1, what do you predict for x? ErrBayesOpt = 1 − P(y∗|x)

Machine Learning: Chenhao Tan | Boulder | 22 of 31

K-nearest neighbors | Performance Guarantee

Performance Guarantee for 1-NN

Theorem

As N = |Strain| → ∞, the 1-NN error is no more than twice the error of the Bayes Optimal Classifier. [Cover and Hart, 1967]

Proof.

Let xNN be the nearest neighbor of our test point x. As N → ∞, dist(xNN, x) → 0, thus P(y∗|xNN) → P(y∗|x).

Machine Learning: Chenhao Tan | Boulder | 23 of 31

K-nearest neighbors | Performance Guarantee

Performance Guarantee for 1-NN

Theorem

As N = |Strain| → ∞, the 1-NN error is no more than twice the error of the Bayes Optimal Classifier. [Cover and Hart, 1967]

Proof.

Let xNN be the nearest neighbor of our test point x. As N → ∞, dist(xNN, x) → 0, thus P(y∗|xNN) → P(y∗|x). Errnn = P(yx = yxNN) = P(y∗|x)(1 − P(y∗|xNN)) + P(y∗|xNN)(1 − P(y∗|x)) ≤ (1 − P(y∗|xNN)) + (1 − P(y∗|x)) = 2 ∗ (1 − P(y∗|x)) = 2 ErrBayesOpt

Machine Learning: Chenhao Tan | Boulder | 23 of 31

K-nearest neighbors | Curse of Dimensionality

How does the algorithm scale? (Curse of Dimensionality)

Machine Learning: Chenhao Tan | Boulder | 24 of 31

K-nearest neighbors | Curse of Dimensionality

Curse of Dimensionality

Given N points in [0, 1], what is the size of the smallest interval to contain k-nearest neighbor for x?

Machine Learning: Chenhao Tan | Boulder | 25 of 31

K-nearest neighbors | Curse of Dimensionality

Curse of Dimensionality

Given N points in [0, 1], what is the size of the smallest interval to contain k-nearest neighbor for x? x

l

Machine Learning: Chenhao Tan | Boulder | 25 of 31

K-nearest neighbors | Curse of Dimensionality

Curse of Dimensionality

Given N points in [0, 1], what is the size of the smallest interval to contain k-nearest neighbor for x? x

l

N ∗ l ≈ k ⇒ l ≈ k N

Machine Learning: Chenhao Tan | Boulder | 25 of 31

K-nearest neighbors | Curse of Dimensionality

Curse of Dimensionality

In general, for d dimensions, what is the length of the smallest hypercube to contain k-nearest neighbor for x?

Machine Learning: Chenhao Tan | Boulder | 26 of 31

K-nearest neighbors | Curse of Dimensionality

Curse of Dimensionality

In general, for d dimensions, what is the length of the smallest hypercube to contain k-nearest neighbor for x? N ∗ ld ≈ k ⇒ l ≈ k N 1

d

Machine Learning: Chenhao Tan | Boulder | 26 of 31

K-nearest neighbors | Curse of Dimensionality

Curse of Dimensionality

If N = 1000, k = 10, d l 2 0.1 10 0.63 100 0.955 1000 0.9954 We almost need the entire space to find 10 nearest neighbors.

Machine Learning: Chenhao Tan | Boulder | 27 of 31

K-nearest neighbors | Curse of Dimensionality

How does the algorithm scale? (Memory and Efficiency of the naive implementation)

Machine Learning: Chenhao Tan | Boulder | 28 of 31

K-nearest neighbors | Curse of Dimensionality

How does the algorithm scale? (Memory and Efficiency of the naive implementation) Training: N/A Testing

• memory: O(Nd)
• time: O(Nd)

Machine Learning: Chenhao Tan | Boulder | 28 of 31

K-nearest neighbors | Curse of Dimensionality

Summary

• Supervised learning: learn h from Strain
• Data representation can be tricky in many cases
• K-NN gives a simple classifier with nice guarantee

Machine Learning: Chenhao Tan | Boulder | 29 of 31

K-nearest neighbors | Curse of Dimensionality

First Homework

• Implement k-nearest neighbors
• Cross validation to search hyperparameters
• Acclimate you to the Python programming environment
• Introduce you to using Moodle to submit assignments

Machine Learning: Chenhao Tan | Boulder | 30 of 31

K-nearest neighbors | Curse of Dimensionality

References (1)

Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1):21–27, 1967. David H Wolpert. The lack of a priori distinctions between learning algorithms. Neural computation, 8(7): 1341–1390, 1996. David H Wolpert and William G Macready. No free lunch theorems for optimization. IEEE transactions on evolutionary computation, 1(1):67–82, 1997.

Machine Learning: Chenhao Tan | Boulder | 31 of 31