Introduction to Machine Learning COMPSCI 371D Machine Learning - - PowerPoint PPT Presentation

Introduction to Machine Learning

COMPSCI 371D — Machine Learning

COMPSCI 371D — Machine Learning Introduction to Machine Learning 1 / 12

Outline

1 Nearest Neighbor Prediction 2 Complexity Considerations 3 The Voronoi Diagram 4 Overfitting and k Nearest Neighbors

COMPSCI 371D — Machine Learning Introduction to Machine Learning 2 / 12

Nearest Neighbor Prediction

Nearest Neighbor Prediction

• NN is very simple: This is why we start here
• NN is very unusual:
• No training!
• Slow inference (using the predictor)
• Y can be anything
• Almost no difference between regression and classification
• Hypothesis space hard to define

COMPSCI 371D — Machine Learning Introduction to Machine Learning 3 / 12

Nearest Neighbor Prediction

How it Works

• Given T = {(x1, y1), . . . , (xN, yN)}
• Just store T (memorization)
• Need a distance in the data space X
• Perhaps ∆(x, x′) = x − x′2
• Then, h(x) = yν(x)

where ν(x) ∈ arg minn=1,...,N ∆(x, xn)

• Return the value yn for the training point xn

that is nearest to x

COMPSCI 371D — Machine Learning Introduction to Machine Learning 4 / 12

Nearest Neighbor Prediction COMPSCI 371D — Machine Learning Introduction to Machine Learning 5 / 12

Complexity Considerations

How to find ν(x)?

ν(x) = arg minn=1,...,N ∆(x, xn)

• Compute all ∆(x, xn) and find the smallest
• O(Nd) (where x ∈ Rd)
• Cannot do better exactly
• Can do better if we accept ∆(x, xν(x)) < (1 + ǫ)∆(x, xν∗(x))

for some ǫ > 0

• “Approximate NN” uses k-d trees, R-trees,

locality sensitive hashing

COMPSCI 371D — Machine Learning Introduction to Machine Learning 6 / 12

The Voronoi Diagram

The Voronoi Diagram

• Only conceptual, or for d = 2, 3, maybe 4
• Θ(N log N + N⌈d/2⌉)

COMPSCI 371D — Machine Learning Introduction to Machine Learning 7 / 12

The Voronoi Diagram

Decision Boundary

COMPSCI 371D — Machine Learning Introduction to Machine Learning 8 / 12

Overfitting and k Nearest Neighbors

Overfitting

COMPSCI 371D — Machine Learning Introduction to Machine Learning 9 / 12

Overfitting and k Nearest Neighbors

k Nearest Neighbors

• Retrieve the k nearest neighbors x1, . . . , xk of x
• Return a summary of the corresponding y1, . . . , yk
• Classification summary: majority
• Regression summary: Mean, median

COMPSCI 371D — Machine Learning Introduction to Machine Learning 10 / 12

Overfitting and k Nearest Neighbors

Less Overfitting (k = 9)

COMPSCI 371D — Machine Learning Introduction to Machine Learning 11 / 12

Overfitting and k Nearest Neighbors

A Simple Regression Example, R → R

1000 2000 3000 4000 5000 6000 100 200 300 400 500 600 700 800 1000 2000 3000 4000 5000 6000 100 200 300 400 500 600 700 800 k = 1 k = 10 k = 100

COMPSCI 371D — Machine Learning Introduction to Machine Learning 12 / 12