Machine Learning Basics Lecture slides for Chapter 5 of Deep - - PowerPoint PPT Presentation

Machine Learning Basics

Lecture slides for Chapter 5 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2016-09-26

(Goodfellow 2016)

Linear Regression

−1.0 −0.5 0.0 0.5 1.0 x1 −3 −2 −1 1 2 3 y

Linear regression example

0.5 1.0 1.5 w1 0.20 0.25 0.30 0.35 0.40 0.45 0.50 0.55 MSE(train)

Optimization of w

Figure 5.1

(Goodfellow 2016)

Underfitting and Overfitting in Polynomial Estimation

x0

• x0
• x0
• Figure 5.2

(Goodfellow 2016)

Generalization and Capacity

Optimal Capacity Capacity Error Underfitting zone Overfitting zone Generalization gap

Training error Generalization error

Figure 5.3

(Goodfellow 2016)

Training Set Size

100 101 102 103 104 105 Number of training examples 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 Error (MSE)

Bayes error Train (quadratic) Test (quadratic) Test (optimal capacity) Train (optimal capacity)

100 101 102 103 104 105 Number of training examples 5 10 15 20 Optimal capacity (polynomial degree)

Figure 5.4

(Goodfellow 2016)

Weight Decay

x( y

Underfitting (Excessive λ)

x( y

Appropriate weight decay (Medium λ)

x( y

Overfitting (λ →()

Figure 5.5

(Goodfellow 2016)

Bias and Variance

Capacity Bias Generalization error Variance Optimal capacity Overfitting zone Underfitting zone

Figure 5.6

(Goodfellow 2016)

Decision Trees

1 1110 1111 110 10 01 00 010 011 11 111

1 01 111 011 1111 1110 110 10 010 00 11

Figure 5.7

(Goodfellow 2016)

Principal Components Analysis

−20 −10 10 20 x1 −20 −10 10 20 x2 −20 −10 10 20 z1 −20 −10 10 20 z2

Figure 5.8

(Goodfellow 2016)

Curse of Dimensionality

Figure 5.9: As the number of relevant dimensions of the data increases (from

Figure 5.9

(Goodfellow 2016)

Nearest Neighbor

Figure 5.10

(Goodfellow 2016)

Manifold Learning

0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 −1.0 −0.5 0.0 0.5 1.0 1.5 2.0 2.5

Figure 5.11

(Goodfellow 2016)

Uniformly Sampled Images

Figure 5.12

(Goodfellow 2016)

QMUL Dataset

Figure 5.13