Regression and generalization CE-717: Machine Learning Sharif - - PowerPoint PPT Presentation

Regression and generalization

CE-717: Machine Learning

Sharif University of Technology

• M. Soleymani

Fall 2018

Topics

} Beyond linear regression models } Evaluation & model selection } Regularization }

Bias-Variance

2

Recall: Linear regression (squared loss)

} Linear regression functions

𝑔 ∶ ℝ → ℝ 𝑔(𝑦; 𝒙) = 𝑥- + 𝑥/𝑦 𝑔 ∶ ℝ0 → ℝ 𝑔(𝒚; 𝒙) = 𝑥- + 𝑥/𝑦/ + . . . 𝑥3𝑦3

} Minimizing the squared loss for linear regression

𝐾(𝒙) = 𝒛 − 𝒀𝒙 8

8

} We obtain 𝒙

9 = 𝒀:𝒀 ;𝟐 𝒀:𝒛

3

𝒙 = 𝑥-,𝑥/,...,𝑥3 : are the parameters we need to set.

Beyond linear regression

} How to extend the linear regression to non-linear

functions?

} Transform the data using basis functions } Learn a linear regression on the new feature vectors (obtained

by basis functions)

4

Beyond linear regression

} 𝑛?@ order polynomial regression (univariate 𝑔 ∶ ℝ ⟶ ℝ)

𝑔 𝑦; 𝒙 = 𝑥- + 𝑥/𝑦 + . . . +𝑥B;/𝑦B;/ +𝑥B 𝑦B

} Solution: 𝒙

C = 𝒀′:𝒀′

;𝟐 𝒀′:𝒛

𝒛 = 𝑧/ ⋮ 𝑧G 𝒀′ = 1 𝑦 / I 𝑦 / J ⋯ 𝑦 / L 1 𝑦 8 I 𝑦 8 J ⋯ 𝑦 8 L ⋮ 1 ⋮ 𝑦 G I ⋮ 𝑦 G J ⋮ ⋯ ⋮ 𝑦 G I 𝒙 = 𝒙 9 - 𝒙 9/ ⋮ 𝒙 9 B

5

Polynomial regression: example

6

𝑛 = 1 𝑛 = 3 𝑛 = 5 𝑛 = 7

Generalized linear

} Linear combination of fixed non-linear function of the

input vector

𝑔(𝒚; 𝒙) = 𝑥- + 𝑥/𝜚/(𝒚)+ . . . 𝑥B𝜚B(𝒚)

{𝜚/(𝒚), . . . , 𝜚B(𝒚)}: set of basis functions (or features) 𝜚S 𝒚 : ℝ3 → ℝ

7

Basis functions: examples

} Linear } Polynomial (univariate)

8

Basis functions: examples

} Gaussian: 𝜚U 𝒚 = 𝑓𝑦𝑞 −

𝒚;𝒅Y

J

8ZY

J

} Sigmoid: 𝜚U 𝒚 = 𝜏

𝒚;𝒅Y ZY

𝜏 𝑏 =

/ /]^_` (;a)

9

Radial Basis Functions: prototypes

} Predictions based on similarity to “prototypes”:

𝜚U 𝒚 = 𝑓𝑦𝑞 − 1 2𝜏

U 8 𝒚 − 𝒅U 8

} Measuring the similarity to the prototypes 𝒅/, … , 𝒅B

} σ8 controls how quickly it vanishes as a function of the

distance to the prototype.

} Training examples themselves could serve as prototypes

10

Generalized linear: optimization

11

𝐾 𝒙 = e 𝑧 S − 𝑔 𝒚 S ; 𝒙

8 G Sf/

= e 𝑧 S − 𝒙:𝝔 𝒚 S

8 G Sf/

𝒛 = 𝑧(/) ⋮ 𝑧(G) 𝚾 = 1 𝜚/(𝒚(/)) ⋯ 𝜚B(𝒚(/)) 1 ⋮ 𝜚/(𝒚(8)) ⋮ ⋯ ⋱ 𝜚B(𝒚(8)) ⋮ 1 𝜚/(𝒚(G)) ⋯ 𝜚B(𝒚(G)) 𝒙 = 𝑥- 𝑥/ ⋮ 𝑥B 𝒙

j = 𝚾:𝚾

;𝟐 𝚾:𝒛

Model complexity and overfitting

} With limited training data, models may achieve zero

training error but a large test error.

} Over-fitting: when the training loss no longer bears any

relation to the test (generalization) loss.

} Fails to generalize to unseen examples.

12

1 𝑜 e 𝑧 S − 𝑔 𝒚 S ; 𝜾

8

≈ 0

G Sf/

Training (empirical) loss Expected (true) loss E𝐲,q 𝑧 − 𝑔 𝒚; 𝜾

8 ≫ 0

Polynomial regression

13

𝑛 = 0 𝑛 = 1 𝑛 = 3 𝑛 = 9 𝑧 𝑧 𝑧 𝑧 [Bishop]

Polynomial regression: training and test error

14

𝑛 𝑆𝑁𝑇𝐹 = ∑ 𝑧 S − 𝑔 𝒚 S ; 𝜾

8 G Sf/

𝑜

• [Bishop]

Over-fitting causes

15

} Model complexity

} E.g., Model with a large number of parameters (degrees of

freedom)

} Low number of training data

} Small data size compared to the complexity of the model

Model complexity

16

} Example:

} Polynomials with larger 𝑛 are becoming increasingly tuned to

the random noise on the target values.

16

𝑛 = 0 𝑛 = 1 𝑛 = 3 𝑛 = 9 𝑧 𝑧 𝑧 𝑧 [Bishop]

Number of training data & overfitting

17

} Over-fitting problem becomes less severe as the size of

training data increases.

𝑛 = 9 𝑛 = 9 𝑜 = 15 𝑜 = 100 [Bishop]

How to evaluate the learner’s performance?

18

} Generalization error: true (or expected) error that we

would like to optimize

} Two ways to assess the generalization error is:

} Practical: Use a separate data set to test the model } Theoretical: Law of Large numbers

} statistical bounds on the difference between training and expected

errors

Avoiding over-fitting

19

} Determine a suitable value for model complexity (Model

Selection)

} Simple hold-out method } Cross-validation

} Regularization (Occam’s Razor)

} Explicit preference towards simple models } Penalize for the model complexity in the objective function

} Bayesian approach

Evaluation and model selection

20

} Evaluation:

} We need to measure how well the learned function can

predicts the target for unseen examples

} Model selection:

} Most of the time we need to select among a set of models

} Example: polynomials with different degree 𝑛

} and thus we need to evaluate these models first

Model Selection

21

} learning algorithm defines the data-driven search over

the hypothesis space (i.e. search for good parameters)

} hyperparameters are the tunable aspects of the model,

that the learning algorithm does not select

This slide has been adopted from CMU ML course: http://www.cs.cmu.edu/~mgormley/courses/10601-s18/

Model Selection

22

} Model selection is the process by which we choose the

“best” model from among a set of candidates

} assume access to a function capable of measuring the quality of

a model

} typically done “outside” the main training algorithm

} Model selection / hyperparameter optimization is just

another form of learning

This slide has been adopted from CMU ML course: http://www.cs.cmu.edu/~mgormley/courses/10601-s18/

Simple hold-out: model selection

23

} Steps:

} Divide training data into training and validation set 𝑤_𝑡𝑓𝑢 } Use only the training set to train a set of models } Evaluate each learned model on the validation set

} 𝐾~ 𝒙 =

/ ~_•€? ∑

𝑧(S) − 𝑔 𝒚(S); 𝒙

8

• S∈~_•€?

} Choose the best model based on the validation set error

} Usually, too wasteful of valuable training data

} Training data may be limited. } On the other hand, small validation set give a relatively noisy

estimate of performance.

Simple hold out: training, validation, and test sets

24

} Simple hold-out chooses the model that minimizes error on

validation set.

} 𝐾~ 𝒙

9 is likely to be an optimistic estimate of generalization error.

} extra parameter (e.g., degree of polynomial) is fit to this set.

} Estimate generalization error for the test set

} performance of the selected model is finally evaluated on the test set

Training Validation Test

Cross-Validation (CV): Evaluation

25

} 𝑙-fold cross-validation steps:

} Shuffle the dataset and randomly partition training data into 𝑙 groups of

approximately equal size

} for 𝑗 = 1 to 𝑙

} Choose the 𝑗-th group as the held-out validation group } Train the model on all but the 𝑗-th group of data } Evaluate the model on the held-out group

} Performance scores of the model from 𝑙 runs are averaged.

} The average error rate can be considered as an estimation of the true

performance. … … … … … First run Second run (k-1)th run k-th run

Cross-Validation (CV): Model Selection

26

} For each model we first find the average error find by CV. } The model with the best average performance is

selected.

Cross-validation: polynomial regression example

} 5-fold CV } 100 runs

} average

27

𝑛 = 1 CV: 𝑁𝑇𝐹 = 0.30 𝑛 = 3 CV: 𝑁𝑇𝐹 = 1.45 𝑛 = 5 CV: 𝑁𝑇𝐹 = 45.44 𝑛 = 7 CV: 𝑁𝑇𝐹 = 31759

Leave-One-Out Cross Validation (LOOCV)

28

} When data is particularly scarce, cross-validation with

𝑙 = 𝑂

} Leave-one-out treats each training sample in turn as a test

example and all other samples as the training set.

} Use for small datasets

} When training data is valuable } LOOCV can be time expensive as 𝑂

training steps are required.

Regularization

29

} Adding a penalty term in the cost function to discourage

the coefficients from reaching large values.

} Ridge regression (weight decay):

𝐾 𝒙 = e 𝑧 S − 𝒙:𝝔 𝒚 S

8

+ 𝜇𝒙:𝒙

G Sf/

𝒙

C = 𝚾:𝚾 + 𝜇𝑱

;𝟐 𝚾:𝒛

Polynomial order

30

} Polynomials with larger 𝑛

are becoming increasingly tuned to the random noise on the target values.

} magnitude of the coefficients typically gets larger by increasing

𝑛.

[Bishop]

Regularization parameter

31

𝑥 9/ 𝑥 98 𝑥 9ˆ 𝑥 9‰ 𝑥 9Š 𝑥 9‹ 𝑥 9Œ 𝑥 9• 𝑥 9Ž 𝑛 = 9 𝑥 9- 𝑚𝑜𝜇 = −∞ 𝑚𝑜𝜇 = −18 [Bishop]

Regularization parameter

32

} Generalization

} 𝜇 now controls the effective complexity of the model and

hence determines the degree of over-fitting

[Bishop]

Choosing the regularization parameter

33

} A set of models with different values of 𝜇. } Find 𝒙

9 for each model based on training data

} Find 𝐾~(𝒙

9) (or 𝐾’~(𝒙 9)) for each model

} 𝐾~ 𝒙 =

/ G_~ ∑

𝑧(S) − 𝑔 𝑦(S); 𝒙

8

• S∈~_•€?

} Select the model with the best 𝐾~(𝒙

9) (or 𝐾’~(𝒙 9))

The approximation-generailization trade-off

34

} Small true error shows good approximation of 𝑔 out of

sample

} More complex ℋ ⇒ better chance of approximating 𝑔 } Less complex ℋ ⇒ better chance of generalization out of 𝑔

Complexity of Hypothesis Space: Example

35

Price Size Price Size Price Size

𝑥- + 𝑥/𝑦 𝑥- + 𝑥/𝑦 + 𝑥8𝑦8 𝑥- + 𝑥/𝑦 + 𝑥8𝑦8 + 𝑥ˆ𝑦ˆ + 𝑥‰𝑦‰ This example has been adapted from: Prof. Andrew Ng’s slides More complex ℋ Less complex ℋ

Complexity of Hypothesis Space: Example

36

Price Size Price Size Price Size

𝑥- + 𝑥/𝑦 𝑥- + 𝑥/𝑦 + 𝑥8𝑦8 𝑥- + 𝑥/𝑦 + 𝑥8𝑦8 + 𝑥ˆ𝑦ˆ + 𝑥‰𝑦‰ This example has been adapted from: Prof. Andrew Ng’s slides Overfitting Underfitting

Complexity of Hypothesis Space: Example

37

𝑛 egree of polynomial d error 𝐾~ 𝐾?•aSG 𝐾~ 𝒙 = 1 𝑜_𝑤 e 𝑧(S) − 𝑔 𝒚(S); 𝒙

8

• S∈~a–_•€?

𝐾?•aSG 𝒙 = 1 𝑜_𝑢𝑠𝑏𝑗𝑜 e 𝑧(S) − 𝑔 𝒚(S); 𝒙

8

• S∈?•aSG_•€?

Complexity of Hypothesis Space

38

} Less complex ℋ:

} 𝐾?•aSG(𝒙

9) ≈ 𝐾~(𝒙 9) and 𝐾?•aSG(𝒙 9) is very high

} More complex ℋ:

} 𝐾?•aSG(𝒙

9) ≪ 𝐾~(𝒙 9) and 𝐾?•aSG(𝒙 9) is low

𝑛 egree of polynomial d error 𝐾~(𝒙 9) 𝐾?•aSG(𝒙 9)

Size of training set

39

𝑔 𝑦; 𝒙 = 𝑥- + 𝑥/𝑦 + 𝑥8𝑦8 (training set size) error 𝑜 𝐾~ 𝒙 = 1 𝑜_𝑤 e 𝑧(S) − 𝑔 𝑦(S); 𝒙

8

• S∈~a–_•€?

𝐾?•aSG 𝒙 = 1 𝑜_𝑢𝑠𝑏𝑗𝑜 e 𝑧(S) − 𝑔 𝑦(S); 𝒙

8

• S∈?•aSG_•€?

𝐾~ 𝐾?•aSG This slide has been adapted from: Prof. Andrew Ng’s slides

Less complex ℋ

40

size price size price 𝑔 𝑦; 𝒙 = 𝑥- + 𝑥/𝑦

If model is very simple, getting more training data will not (by itself) help much.

(training set size) 𝑜 error 𝐾~ 𝐾?•aSG High error This slide has been adapted from: Prof. Andrew Ng’s slides

More complex ℋ

41

(training set size) error 𝑜 𝐾~ 𝐾?•aSG Gap size price size price

For more complex models, getting more training data is usually helps.

𝑔 𝑦; 𝒙 = 𝑥- + 𝑥/𝑦 + ⋯ 𝑥/-𝑦/- This slide has been adapted from: Prof. Andrew Ng’s slides

Regularization: Example

42

𝑔 𝑦; 𝒙 = 𝑥- + 𝑥/𝑦 + 𝑥8𝑦8 +𝑥ˆ 𝑦ˆ +𝑥‰ 𝑦‰ 𝐾 𝒙 = 1 𝑜 e 𝑧 S − 𝑔 𝑦 S ; 𝒙

8

+ 𝜇𝒙:𝒙

G Sf/

Large 𝜇x

(Prefer to more simple models)

Intermediate 𝜇

Price Size Price Size Price Size

Small 𝜇

(Prefer to more complex models)

𝑥/ = 𝑥8 ≈ 0 𝜇 = 0 This example has been adapted from: Prof. Andrew Ng’s slides

Theoretical Part

43

Model complexity: Bias-variance trade-off

44

} Least squares, can lead to severe over-fitting if complex models

are trained using data sets of limited size.

} A frequentist viewpoint of the model complexity issue, known

as the bias-variance trade-off.

Formal discussion on bias, variance, and noise

45

} Best unrestricted regression function } Noise } Bias and variance

The learning diagram: deterministic target

46

ℎ: 𝒴 → 𝒵 𝑔: 𝒴 → 𝒵 𝑦 / , 𝑧(/) , … , 𝑦 œ , 𝑧(œ) 𝑦 / , … , 𝑦 œ

[Y.S. Abou Mostafa, 2012]

The learning diagram including noisy target

47

} Type equation here.

ℎ: 𝒴 → 𝒵 𝑔: 𝒴 → 𝒵 𝑦 / , 𝑧(/) , … , 𝑦 œ , 𝑧(œ) 𝑦 / , … , 𝑦 œ 𝑔 𝒚 = ℎ(𝒚)

𝑄 𝑦, 𝑧 = 𝑄 𝑦 𝑄(𝑧|𝑦) Target distribution Distribution

• n features

[Y.S. Abou Mostafa, 2012]

Best unrestricted regression function

} If we know the joint distribution 𝑄(𝒚, 𝑧)

and no constraints on the regression function?

} cost function: mean squared error

ℎ∗ = argmin

@:ℝ¯→ℝ

𝔽𝒚,±

𝑧 − ℎ 𝒚

8

ℎ∗ 𝒚 = 𝔽±|𝒚[𝑧]

48

Best unrestricted regression function: Proof

𝔽𝒚,±

𝑧 − ℎ 𝒚

8 = ´ 𝑧 − ℎ 𝒚 8𝑞 𝒚, 𝑧 𝑒𝒚𝑒𝑧

• } For each 𝒚 separately minimize loss since ℎ(𝒚) can be chosen

independently for each different 𝒚: 𝜀𝔽𝒚,± 𝑧 − ℎ 𝒚

8

𝜀ℎ(𝒚) = − · 2 𝑧 − ℎ 𝒚 𝑞 𝒚, 𝑧 𝑒𝑧

• = 0

⇒ ℎ 𝒚 = ∫ 𝑧𝑞 𝒚, 𝑧 𝑒𝑧

• ∫ 𝑞 𝒚, 𝑧 𝑒𝑧
• =

∫ 𝑧𝑞 𝒚, 𝑧 𝑒𝑧

• 𝑞 𝒚

= · 𝑧𝑞 𝑧|𝒚 𝑒𝑧 = 𝔽±|𝒚

• 𝑧

⟹ ℎ∗ 𝒚 = 𝔽±|𝒚[𝑧]

49

Error decomposition

50

𝐹?•º€ 𝑔

𝒠 𝒚

= 𝔽𝒚,± 𝑔

𝒠 𝒚 − 𝑧 8

= 𝔽𝒚,± 𝑔

𝒠 𝒚 − ℎ 𝒚 + ℎ 𝒚 − 𝑧 8

= 𝔽𝒚 𝑔

𝒠 𝒚 − ℎ 𝒚 8 + 𝔽𝒚,±

ℎ 𝒚 − 𝒛 8 +2𝔽𝒚,± 𝑔

𝒠 𝒚 − ℎ 𝒚

ℎ 𝒚 − 𝑧

ℎ 𝒚 : minimizes the expected loss 𝒚, 𝑧 ~𝑄

𝔽𝒚 𝑔

𝒠 𝒚 − ℎ 𝒚

𝔽±|𝒚 ℎ 𝒚 − 𝑧

Expected loss

Error decomposition

51

𝐹?•º€ 𝑔

𝒠 𝒚

= 𝔽𝒚,± 𝑔

𝒠 𝒚 − 𝑧 8

= 𝔽𝒚,± 𝑔

𝒠 𝒚 − ℎ 𝒚 + ℎ 𝒚 − 𝑧 8

= 𝔽𝒚 𝑔

𝒠 𝒚 − ℎ 𝒚 8 + 𝔽𝒚,±

ℎ 𝒚 − 𝒛 8 +2𝐹𝒚,± 𝑔 𝒚; 𝒙 9 − ℎ 𝒚 ℎ 𝒚 − 𝑧

} Noise shows the irreducible minimum value of the loss

function

ℎ 𝒚 : minimizes the expected loss 𝒚, 𝑧 ~𝑄 noise

Expectation of true error

52

𝐹?•º€ 𝑔

𝒠 𝒚

= 𝔽𝒚,± 𝑔

𝒠 𝒚 − 𝑧 8

= 𝔽𝒚 𝑔

𝒠 𝒚 − ℎ 𝒚 8 + 𝑜𝑝𝑗𝑡𝑓

𝔽𝒠 𝔽𝒚

𝑔

𝒠 𝒚 − ℎ 𝒚 8

= 𝔽𝒚 𝔽𝒠 𝑔

𝒠 𝒚 − ℎ 𝒚 8

We now want to focus on 𝔽𝒠 𝑔

𝒠 𝒚 − ℎ 𝒚 8 .

The average hypothesis

53

𝑔̅ 𝒚 ≡ 𝐹𝒠 𝑔

𝒠 𝒚

𝑔̅ 𝒚 ≈ 1 𝐿 e 𝑔𝒠 Á 𝒚

 Ãf/

𝐿 training sets (of size 𝑂) sampled from 𝑄(𝒚, 𝑧): 𝒠(/), 𝒠(8), … , 𝒠(Â)

Using the average hypothesis

54

𝔽𝒠

𝑔

𝒠 𝒚 − ℎ 𝒚 8

= 𝔽𝒠 𝑔

𝒠 𝒚 − 𝑔̅ 𝒚 + 𝑔̅ 𝒚 − ℎ 𝒚 8

= 𝔽𝒠 Ä 𝑔

𝒠 𝒚 − 𝑔̅ 𝒚 8

+ 𝑔̅ 𝒚 − ℎ 𝒚

8

+ 2 𝑔

𝒠 𝒚 − 𝑔̅ 𝒚

𝑔̅ 𝒚 − ℎ 𝒚 Å = 𝔽𝒠 𝑔

𝒠 𝒚 − 𝑔̅ 𝒚 8

+ 𝑔̅ 𝒚 − ℎ 𝒚

8

Bias and variance

55

𝔽𝒠 𝑔

𝒠 𝒚 − ℎ 𝒚 8 = 𝔽𝒠

𝑔

𝒠 𝒚 − 𝑔̅ 𝒚 8

+ 𝑔̅ 𝒚 − ℎ 𝒚

8

𝔽𝒚 𝔽𝒠 𝑔

𝒠 𝒚 − ℎ 𝒚 8

= 𝔽𝒚 var 𝒚 + bias(𝒚) = var + bias

var(𝒚) bias(𝒚)

Bias-variance trade-off

56

var = 𝔽𝒚 𝔽𝒠 𝑔

𝒠 𝒚 − 𝑔̅ 𝒚 8

bias = 𝔽𝒚 𝑔̅ 𝒚 − ℎ 𝒚

More complex ℋ ⇒ lower bias but higher variance

ℎ ℎ

[Y.S. Abou Mostafa, 2012]

Example: sin target

57

} Only two training example 𝑂 = 2 } Two models used for learning:

} ℋ-: 𝑔 𝑦 = 𝑐 } ℋ/: 𝑔 𝑦 = 𝑏𝑦 + 𝑐

} Which is better ℋ- or ℋ/?

Learning from a training set

58

ℋ- ℋ/ [Y.S. Abou Mostafa, 2012]

Variance ℋ-

59

𝑔̅(𝑦)

[Y.S. Abou Mostafa, et. al]

Variance ℋ/

60

𝑔̅(𝑦)

[Y.S. Abou Mostafa, et. al]

Which is better?

61

𝑔̅(𝑦) 𝑔̅(𝑦)

[Y.S. Abou Mostafa, 2012]

Lesson

62

Match the model complexity to the data sources not to the complexity of the target function.

Expected training and true error curves

63

} Errors vary with the number of training samples

𝐹ÊËÌÍÎ 𝐹ÊËÌÍÎ 𝐹ÊËÏ^ 𝐹ÊËÏ^ expected true error: 𝔽𝒠 𝐹?•º€ 𝑔

𝒠 𝒚

expected training error: 𝔽𝒠 𝐹?•aSG 𝑔

𝒠 𝒚

[Y.S. Abou Mostafa, 2012]

Regularization

64

[Y.S. Abou Mostafa, 2012]

Regularization: bias and variance

65

𝑔̅(𝑦) 𝑔̅(𝑦)

[Y.S. Abou Mostafa, 2012]

Winner of ℋ-, ℋ/, and ℋ/ with regularization

66

[Y.S. Abou Mostafa, 2012]

𝑔̅(𝑦) 𝑔̅(𝑦)

ℋ/

𝑔̅(𝑦)

Regularization and bias/variance

67

𝑀 = 100 data sets 𝑜 = 25 𝑛 = 25 𝜇 is large 𝜇 is intermediate 𝜇 is small [Bishop]

Learning curves of bias, variance, and noise

68

[Bishop]

Bias-variance decomposition: summary

69

} The noise term is unavoidable. } The terms we are interested in are bias and variance. } The approximation-generalization trade-off is seen in the

bias-variance decomposition.

Resources

70

} C. Bishop, “Pattern Recognition and Machine Learning”,

Chapter 1.1,1.3, 3.1, 3.2.

} Yaser S. Abu-Mostafa, Malik Maghdon-Ismail, and Hsuan

Tien Lin,“Learning from Data”, Chapter 2.3, 3.2, 3.4.