SLIDE 1
Regression and generalization
CE-717: Machine Learning
Sharif University of Technology
- M. Soleymani
Topics
} Beyond linear regression models } Evaluation & model selection } Regularization }
Bias-Variance
2
Regression and generalization CE-717: Machine Learning Sharif - - PowerPoint PPT Presentation
Regression and generalization CE-717: Machine Learning Sharif - - PowerPoint PPT Presentation
Regression and generalization CE-717: Machine Learning Sharif University of Technology M. Soleymani Fall 2019 Topics } Beyond linear regression models } Evaluation & model selection } Regularization } Bias-Variance 2 Recall: Linear
SLIDE 2
SLIDE 3
Recall: Linear regression (squared loss)
} Linear regression functions
๐ โถ โ โ โ ๐(๐ฆ; ๐) = ๐ฅ- + ๐ฅ/๐ฆ ๐ โถ โ0 โ โ ๐(๐; ๐) = ๐ฅ- + ๐ฅ/๐ฆ/ + . . . ๐ฅ3๐ฆ3
} Minimizing the squared loss for linear regression
๐พ(๐) = ๐ โ ๐๐ 8
8
} We obtain ๐
9 = ๐:๐ ;๐ ๐:๐
3
๐ = ๐ฅ-,๐ฅ/,...,๐ฅ3 : are the parameters we need to set.
SLIDE 4
Beyond linear regression
} How to extend the linear regression to non-linear
functions?
} Transform the data using basis functions } Learn a linear regression on the new feature vectors (obtained
by basis functions)
4
SLIDE 5
Beyond linear regression
} ๐?@ order polynomial regression (univariate ๐ โถ โ โถ โ)
๐ ๐ฆ; ๐ = ๐ฅ- + ๐ฅ/๐ฆ + . . . +๐ฅB;/๐ฆB;/ +๐ฅB ๐ฆB
} Solution: ๐
C = ๐โฒ:๐โฒ
;๐ ๐โฒ:๐
๐ = ๐ง/ โฎ ๐งG ๐โฒ = 1 ๐ฆ / I ๐ฆ / J โฏ ๐ฆ / L 1 ๐ฆ 8 I ๐ฆ 8 J โฏ ๐ฆ 8 L โฎ 1 โฎ ๐ฆ G I โฎ ๐ฆ G J โฎ โฏ โฎ ๐ฆ G I ๐ = ๐ 9 - ๐ 9/ โฎ ๐ 9 B
5
SLIDE 6
Polynomial regression: example
6
๐ = 1 ๐ = 3 ๐ = 5 ๐ = 7
SLIDE 7
Generalized linear
} Linear combination of fixed non-linear function of the
input vector
๐(๐; ๐) = ๐ฅ- + ๐ฅ/๐/(๐)+ . . . ๐ฅB๐B(๐)
{๐/(๐), . . . , ๐B(๐)}: set of basis functions (or features) ๐S ๐ : โ3 โ โ
7
SLIDE 8
Basis functions: examples
} Linear } Polynomial (univariate)
8
SLIDE 9
Basis functions: examples
} Gaussian: ๐U ๐ = ๐๐ฆ๐ โ
๐;๐ Y
J
8ZY
J
} Sigmoid: ๐U ๐ = ๐
๐;๐ Y ZY
๐ ๐ =
/ /]^_` (;a)
9
SLIDE 10
Radial Basis Functions: prototypes
} Predictions based on similarity to โprototypesโ:
๐U ๐ = ๐๐ฆ๐ โ 1 2๐
U 8 ๐ โ ๐ U 8
} Measuring the similarity to the prototypes ๐ /, โฆ , ๐ B
} ฯ8 controls how quickly it vanishes as a function of the
distance to the prototype.
} Training examples themselves could serve as prototypes
10
SLIDE 11
Generalized linear: optimization
11
๐พ ๐ = e ๐ง S โ ๐ ๐ S ; ๐
8 G Sf/
= e ๐ง S โ ๐:๐ ๐ S
8 G Sf/
๐ = ๐ง(/) โฎ ๐ง(G) ๐พ = 1 ๐/(๐(/)) โฏ ๐B(๐(/)) 1 โฎ ๐/(๐(8)) โฎ โฏ โฑ ๐B(๐(8)) โฎ 1 ๐/(๐(G)) โฏ ๐B(๐(G)) ๐ = ๐ฅ- ๐ฅ/ โฎ ๐ฅB ๐
j = ๐พ:๐พ
;๐ ๐พ:๐
SLIDE 12
Model complexity and overfitting
} With limited training data, models may achieve zero
training error but a large test error.
} Over-fitting: when the training loss no longer bears any
relation to the test (generalization) loss.
} Fails to generalize to unseen examples.
12
1 ๐ e ๐ง S โ ๐ ๐ S ; ๐พ
8
โ 0
G Sf/
Training (empirical) loss Expected (true) loss E๐ฒ,q ๐ง โ ๐ ๐; ๐พ
8 โซ 0
SLIDE 13
Polynomial regression
13
๐ = 0 ๐ = 1 ๐ = 3 ๐ = 9 ๐ง ๐ง ๐ง ๐ง [Bishop]
SLIDE 14
Polynomial regression: training and test error
14
๐ ๐๐๐๐น = โ ๐ง S โ ๐ ๐ S ; ๐พ
8 G Sf/
๐
- [Bishop]
SLIDE 15
Over-fitting causes
15
} Model complexity
} E.g., Model with a large number of parameters (degrees of
freedom)
} Low number of training data
} Small data size compared to the complexity of the model
SLIDE 16
Model complexity
16
} Example:
} Polynomials with larger ๐ are becoming increasingly tuned to
the random noise on the target values.
16
๐ = 0 ๐ = 1 ๐ = 3 ๐ = 9 ๐ง ๐ง ๐ง ๐ง [Bishop]
SLIDE 17
Number of training data & overfitting
17
} Over-fitting problem becomes less severe as the size of
training data increases.
๐ = 9 ๐ = 9 ๐ = 15 ๐ = 100 [Bishop]
SLIDE 18
How to evaluate the learnerโs performance?
18
} Generalization error: true (or expected) error that we
would like to optimize
} Two ways to assess the generalization error are:
} Practical: Use a separate data set to test the model } Theoretical: Law of Large numbers
} statistical bounds on the difference between training and expected
errors
SLIDE 19
Avoiding over-fitting
19
} Determine a suitable value for model complexity (Model
Selection)
} Simple hold-out method } Cross-validation
} Regularization (Occamโs Razor)
} Explicit preference towards simple models } Penalize for the model complexity in the objective function
} Bayesian approach
SLIDE 20
Avoiding over-fitting
20
} Determine a suitable value for model complexity (Model
Selection)
} Simple hold-out method } Cross-validation
} Regularization (Occamโs Razor)
} Explicit preference towards simple models } Penalize for the model complexity in the objective function
} Bayesian approach
SLIDE 21
Evaluation and model selection
21
} Evaluation:
} We need to measure how well the learned function can
predict the target for unseen examples
} Model selection:
} Most of the time we need to select among a set of models
} Example: polynomials with different degree ๐
} and thus we need to evaluate these models first
SLIDE 22
Model Selection
22
} Learning algorithm defines the data-driven search over
the hypothesis space
} search for good parameters
} Hyper-parameters are the tunable aspects of the
model, that the learning algorithm does not select
This slide has been adopted from CMU ML course: http://www.cs.cmu.edu/~mgormley/courses/10601-s18/
SLIDE 23
Model Selection
23
} Model selection is the process by which we choose the
โbestโ model among a set of candidates
} assume access to a function capable of measuring the quality of
a model
} typically done โoutsideโ the main training algorithm
} Model selection / hyper-parameter optimization is just
another form of learning
This slide has been adopted from CMU ML course: http://www.cs.cmu.edu/~mgormley/courses/10601-s18/
SLIDE 24
Simple hold-out: model selection
24
} Steps:
} Divide training data into training and validation set ๐ค_๐ก๐๐ข } Use only the training set to train a set of models } Evaluate each learned model on the validation set
} ๐พ~ ๐ =
/ ~_โขโฌ? โ
๐ง(S) โ ๐ ๐(S); ๐
8
- Sโ~_โขโฌ?
} Choose the best model based on the validation set error
} Usually, too wasteful of valuable training data
} Training data may be limited. } On the other hand, small validation set obtains a relatively
noisy estimate of performance.
SLIDE 25
Simple hold out: training, validation, and test sets
25
} Simple hold-out chooses the model that minimizes error on
validation set.
} ๐พ~ ๐
9 is likely to be an optimistic estimate of generalization error.
} extra parameter (e.g., degree of polynomial) is fit to this set.
} Estimate generalization error for the test set
} performance of the selected model is finally evaluated on the test set
Training Validation Test
SLIDE 26
Cross-Validation (CV): Evaluation
26
} ๐-fold cross-validation steps:
} Shuffle the dataset and randomly partition training data into ๐ groups of
approximately equal size
} for ๐ = 1 to ๐
} Choose the ๐-th group as the held-out validation group } Train the model on all but the ๐-th group of data } Evaluate the model on the held-out group
} Performance scores of the model from ๐ runs are averaged.
} The average error rate can be considered as an estimation of the true
performance of the model. โฆ โฆ โฆ โฆ โฆ First run Second run (k-1)th run k-th run
SLIDE 27
Cross-Validation (CV): Model Selection
27
} For each model, we first find the average error by CV. } The model with the best average performance is
selected.
SLIDE 28
Cross-validation: polynomial regression example
} 5-fold CV } 100 runs
} average
28
๐ = 1 CV: ๐๐๐น = 0.30 ๐ = 3 CV: ๐๐๐น = 1.45 ๐ = 5 CV: ๐๐๐น = 45.44 ๐ = 7 CV: ๐๐๐น = 31759
SLIDE 29
Leave-One-Out Cross Validation (LOOCV)
29
} When data is particularly scarce, cross-validation with
๐ = ๐
} Leave-one-out treats each training sample in turn as a test
example and all other samples as the training set.
} Use for small datasets
} When training data is valuable } LOOCV can be time expensive as ๐
training steps are required.
SLIDE 30
Avoiding over-fitting
30
} Determine a suitable value for model complexity (Model
Selection)
} Simple hold-out method } Cross-validation
} Regularization (Occamโs Razor)
} Explicit preference towards simple models } Penalize for the model complexity in the objective function
} Bayesian approach
SLIDE 31
Regularization
31
} Adding a penalty term in the cost function to discourage
the coefficients from reaching large values.
} Ridge regression (weight decay):
๐พ ๐ = e ๐ง S โ ๐:๐ ๐ S
8
+ ๐๐:๐
G Sf/
๐
C = ๐พ:๐พ + ๐๐ฑ
;๐ ๐พ:๐
๐ = ๐ง(/) โฎ ๐ง(G) ๐พ = 1 ๐/(๐(/)) โฏ ๐B(๐(/)) 1 โฎ ๐/(๐(8)) โฎ โฏ โฑ ๐B(๐(8)) โฎ 1 ๐/(๐(G)) โฏ ๐B(๐(G)) ๐ = ๐ฅ- ๐ฅ/ โฎ ๐ฅB
SLIDE 32
Polynomial order
32
} Polynomials with larger ๐
are becoming increasingly tuned to the random noise on the target values.
} magnitude of the coefficients typically gets larger by increasing
๐.
[Bishop]
SLIDE 33
Regularization parameter
33
๐ฅ 9/ ๐ฅ 98 ๐ฅ 9ห ๐ฅ 9โฐ ๐ฅ 9ล ๐ฅ 9โน ๐ฅ 9ล ๐ฅ 9โข ๐ฅ 9ลฝ ๐ = 9 ๐ฅ 9- ๐๐๐ = โโ ๐๐๐ = โ18 [Bishop]
SLIDE 34
Regularization parameter
34
} Generalization
} ๐ now controls the effective complexity of the model and
hence determines the degree of over-fitting
[Bishop]
SLIDE 35
Choosing the regularization parameter
35
} A set of models with different values of ๐. } Find ๐
9 for each model based on training data
} Find ๐พ~(๐
9) (or ๐พโ~(๐ 9)) for each model
} ๐พ~ ๐ =
/ G_~ โ
๐ง(S) โ ๐ ๐ฆ(S); ๐
8
- Sโ~_โขโฌ?
} Select the model with the best ๐พ~(๐
9) (or ๐พโ~(๐ 9))
SLIDE 36
The approximation-generalization trade-off
36
} Small true error shows good approximation of ๐ out of
sample
} More complex โ โ better chance of approximating ๐ } Less complex โ โ better chance of generalization out of ๐
SLIDE 37
Complexity of Hypothesis Space: Example
37
Price Size Price Size Price Size
๐ฅ- + ๐ฅ/๐ฆ ๐ฅ- + ๐ฅ/๐ฆ + ๐ฅ8๐ฆ8 ๐ฅ- + ๐ฅ/๐ฆ + ๐ฅ8๐ฆ8 + ๐ฅห๐ฆห + ๐ฅโฐ๐ฆโฐ This example has been adapted from: Prof. Andrew Ngโs slides More complex โ Less complex โ
SLIDE 38
Complexity of Hypothesis Space: Example
38
Price Size Price Size Price Size
๐ฅ- + ๐ฅ/๐ฆ ๐ฅ- + ๐ฅ/๐ฆ + ๐ฅ8๐ฆ8 ๐ฅ- + ๐ฅ/๐ฆ + ๐ฅ8๐ฆ8 + ๐ฅห๐ฆห + ๐ฅโฐ๐ฆโฐ This example has been adapted from: Prof. Andrew Ngโs slides Overfitting Underfitting
SLIDE 39
Complexity of Hypothesis Space: Example
39
๐ egree of polynomial d error ๐พ~ ๐พ?โขaSG ๐พ~ ๐ = 1 ๐_๐ค e ๐ง(S) โ ๐ ๐(S); ๐
8
- Sโ~aโ_โขโฌ?
๐พ?โขaSG ๐ = 1 ๐_๐ข๐ ๐๐๐ e ๐ง(S) โ ๐ ๐(S); ๐
8
- Sโ?โขaSG_โขโฌ?
SLIDE 40
Complexity of Hypothesis Space
40
} Less complex โ:
} ๐พ?โขaSG(๐
9) โ ๐พ~(๐ 9) and ๐พ?โขaSG(๐ 9) is very high
} More complex โ:
} ๐พ?โขaSG(๐
9) โช ๐พ~(๐ 9) and ๐พ?โขaSG(๐ 9) is low
๐ egree of polynomial d error ๐พ~(๐ 9) ๐พ?โขaSG(๐ 9)
SLIDE 41
Size of training set
41
๐ ๐ฆ; ๐ = ๐ฅ- + ๐ฅ/๐ฆ + ๐ฅ8๐ฆ8 (training set size) error ๐ ๐พ~ ๐ = 1 ๐_๐ค e ๐ง(S) โ ๐ ๐ฆ(S); ๐
8
- Sโ~aโ_โขโฌ?
๐พ?โขaSG ๐ = 1 ๐_๐ข๐ ๐๐๐ e ๐ง(S) โ ๐ ๐ฆ(S); ๐
8
- Sโ?โขaSG_โขโฌ?
๐พ~ ๐พ?โขaSG This slide has been adapted from: Prof. Andrew Ngโs slides
SLIDE 42
Less complex โ
42
size price size price ๐ ๐ฆ; ๐ = ๐ฅ- + ๐ฅ/๐ฆ
If model is very simple, getting more training data will not (by itself) help much.
(training set size) ๐ error ๐พ~ ๐พ?โขaSG High error This slide has been adapted from: Prof. Andrew Ngโs slides
SLIDE 43
More complex โ
43
(training set size) error ๐ ๐พ~ ๐พ?โขaSG Gap size price size price
For more complex models, getting more training data is usually helps.
๐ ๐ฆ; ๐ = ๐ฅ- + ๐ฅ/๐ฆ + โฏ ๐ฅ/-๐ฆ/- This slide has been adapted from: Prof. Andrew Ngโs slides
SLIDE 44
Regularization: Example
44
๐ ๐ฆ; ๐ = ๐ฅ- + ๐ฅ/๐ฆ + ๐ฅ8๐ฆ8 +๐ฅห ๐ฆห +๐ฅโฐ ๐ฆโฐ ๐พ ๐ = 1 ๐ e ๐ง S โ ๐ ๐ฆ S ; ๐
8
+ ๐๐:๐
G Sf/
Large ๐x
(Prefer to more simple models)
Intermediate ๐
Price Size Price Size Price Size
Small ๐
(Prefer to more complex models)
๐ฅ/ = ๐ฅ8 โ 0 ๐ = 0 This example has been adapted from: Prof. Andrew Ngโs slides
SLIDE 45
Theoretical Part
45
SLIDE 46
Model complexity: Bias-variance trade-off
46
} Least squares can lead to severe over-fitting if complex models
are trained using data sets of limited size.
} A frequentist viewpoint of the model complexity issue, known
as the bias-variance trade-off.
SLIDE 47
Formal discussion on bias, variance, and noise
47
} Best unrestricted regression function } Noise } Bias and variance
SLIDE 48
The learning diagram: deterministic target
48
โ: ๐ด โ ๐ต ๐: ๐ด โ ๐ต ๐ฆ / , ๐ง(/) , โฆ , ๐ฆ ล , ๐ง(ล) ๐ฆ / , โฆ , ๐ฆ ล
[Y.S. Abou Mostafa, 2012]
SLIDE 49
The learning diagram including noisy target
49
} Type equation here.
โ: ๐ด โ ๐ต ๐: ๐ด โ ๐ต ๐ฆ / , ๐ง(/) , โฆ , ๐ฆ ล , ๐ง(ล) ๐ฆ / , โฆ , ๐ฆ ล ๐ ๐ = โ(๐)
๐ ๐ฆ, ๐ง = ๐ ๐ฆ ๐(๐ง|๐ฆ) Target distribution Distribution
- n features
[Y.S. Abou Mostafa, 2012]
SLIDE 50
Best unrestricted regression function
} If we know the joint distribution ๐(๐, ๐ง)
and no constraints on the regression function?
} cost function: mean squared error
โโ = argmin
@:โยฏโโ
๐ฝ๐,ยฑ
๐ง โ โ ๐
8
โโ ๐ = ๐ฝยฑ|๐[๐ง]
50
SLIDE 51
Best unrestricted regression function: Proof
๐ฝ๐,ยฑ
๐ง โ โ ๐
8 = ยด ๐ง โ โ ๐ 8๐ ๐, ๐ง ๐๐๐๐ง
- } For each ๐, separately minimize loss since โ(๐) can be chosen
independently for each different ๐: ๐๐ฝ๐,ยฑ ๐ง โ โ ๐
8
๐โ(๐) = โ ยท 2 ๐ง โ โ ๐ ๐ ๐, ๐ง ๐๐ง
- = 0
โ โ ๐ = โซ ๐ง๐ ๐, ๐ง ๐๐ง
- โซ ๐ ๐, ๐ง ๐๐ง
- =
โซ ๐ง๐ ๐, ๐ง ๐๐ง
- ๐ ๐
= ยท ๐ง๐ ๐ง|๐ ๐๐ง = ๐ฝยฑ|๐
- ๐ง
โน โโ ๐ = ๐ฝยฑ|๐[๐ง]
51
SLIDE 52
Error decomposition
52
๐น?โขยบโฌ ๐
๐ ๐
= ๐ฝ๐,ยฑ ๐
๐ ๐ โ ๐ง 8
= ๐ฝ๐,ยฑ ๐
๐ ๐ โ โ ๐ + โ ๐ โ ๐ง 8
= ๐ฝ๐ ๐
๐ ๐ โ โ ๐ 8 + ๐ฝ๐,ยฑ
โ ๐ โ ๐ 8 +2๐ฝ๐,ยฑ ๐
๐ ๐ โ โ ๐
โ ๐ โ ๐ง
โ ๐ : minimizes the expected loss ๐, ๐ง ~๐
๐ฝ๐ ๐
๐ ๐ โ โ ๐
๐ฝยฑ|๐ โ ๐ โ ๐ง
Expected loss
SLIDE 53
Error decomposition
53
๐น?โขยบโฌ ๐
๐ ๐
= ๐ฝ๐,ยฑ ๐
๐ ๐ โ ๐ง 8
= ๐ฝ๐,ยฑ ๐
๐ ๐ โ โ ๐ + โ ๐ โ ๐ง 8
= ๐ฝ๐ ๐
๐ ๐ โ โ ๐ 8 + ๐ฝ๐,ยฑ
โ ๐ โ ๐ 8 +2๐น๐,ยฑ ๐ ๐; ๐ 9 โ โ ๐ โ ๐ โ ๐ง
} Noise shows the irreducible minimum value of the loss
function
โ ๐ : minimizes the expected loss ๐, ๐ง ~๐ noise
SLIDE 54
Expectation of true error
54
๐น?โขยบโฌ ๐
๐ ๐
= ๐ฝ๐,ยฑ ๐
๐ ๐ โ ๐ง 8
= ๐ฝ๐ ๐
๐ ๐ โ โ ๐ 8 + ๐๐๐๐ก๐
๐ฝ๐ ๐ฝ๐
๐
๐ ๐ โ โ ๐ 8
= ๐ฝ๐ ๐ฝ๐ ๐
๐ ๐ โ โ ๐ 8
We now want to focus on ๐ฝ๐ ๐
๐ ๐ โ โ ๐ 8 .
SLIDE 55
The average hypothesis
55
๐ฬ ๐ โก ๐น๐ ๐
๐ ๐
๐ฬ ๐ โ 1 ๐ฟ e ๐๐ ร ๐
ร รf/
๐ฟ training sets (of size ๐) sampled from ๐(๐, ๐ง): ๐ (/), ๐ (8), โฆ , ๐ (ร)
SLIDE 56
Using the average hypothesis
56
๐ฝ๐
๐
๐ ๐ โ โ ๐ 8
= ๐ฝ๐ ๐
๐ ๐ โ ๐ฬ ๐ + ๐ฬ ๐ โ โ ๐ 8
= ๐ฝ๐ ร ๐
๐ ๐ โ ๐ฬ ๐ 8
+ ๐ฬ ๐ โ โ ๐
8
+ 2 ๐
๐ ๐ โ ๐ฬ ๐
๐ฬ ๐ โ โ ๐ ร = ๐ฝ๐ ๐
๐ ๐ โ ๐ฬ ๐ 8
+ ๐ฬ ๐ โ โ ๐
8
SLIDE 57
Bias and variance
57
๐ฝ๐ ๐
๐ ๐ โ โ ๐ 8 = ๐ฝ๐
๐
๐ ๐ โ ๐ฬ ๐ 8
+ ๐ฬ ๐ โ โ ๐
8
๐ฝ๐ ๐ฝ๐ ๐
๐ ๐ โ โ ๐ 8
= ๐ฝ๐ var ๐ + bias(๐) = var + bias
var(๐) bias(๐)
SLIDE 58
Bias-variance trade-off
58
var = ๐ฝ๐ ๐ฝ๐ ๐
๐ ๐ โ ๐ฬ ๐ 8
bias = ๐ฝ๐ ๐ฬ ๐ โ โ ๐
More complex โ โ lower bias but higher variance
โ โ
[Y.S. Abou Mostafa, 2012]
SLIDE 59
Example: sin target
59
} Only two training example ๐ = 2 } Two models used for learning:
} โ-: ๐ ๐ฆ = ๐ } โ/: ๐ ๐ฆ = ๐๐ฆ + ๐
} Which is better โ- or โ/?
SLIDE 60
Learning from a training set
60
โ- โ/ [Y.S. Abou Mostafa, 2012]
SLIDE 61
Variance โ-
61
๐ฬ (๐ฆ)
[Y.S. Abou Mostafa, et. al]
SLIDE 62
Variance โ/
62
๐ฬ (๐ฆ)
[Y.S. Abou Mostafa, et. al]
SLIDE 63
Which is better?
63
๐ฬ (๐ฆ) ๐ฬ (๐ฆ)
[Y.S. Abou Mostafa, 2012]
SLIDE 64
Lesson
64
Match the model complexity to the data sources not to the complexity of the target function.
SLIDE 65
Expected training and true error curves
65
} Errors vary with the number of training samples
๐นรรรรร ๐นรรรรร ๐นรรร^ ๐นรรร^ expected true error: ๐ฝ๐ ๐น?โขยบโฌ ๐
๐ ๐
expected training error: ๐ฝ๐ ๐น?โขaSG ๐
๐ ๐
[Y.S. Abou Mostafa, 2012]
SLIDE 66
Regularization
66
[Y.S. Abou Mostafa, 2012]
SLIDE 67
Regularization: bias and variance
67
๐ฬ (๐ฆ) ๐ฬ (๐ฆ)
[Y.S. Abou Mostafa, 2012]
SLIDE 68
Winner of โ-, โ/, and โ/ with regularization
68
[Y.S. Abou Mostafa, 2012]
๐ฬ (๐ฆ) ๐ฬ (๐ฆ)
โ/
๐ฬ (๐ฆ)
SLIDE 69
Regularization and bias/variance
69
๐ = 100 data sets ๐ = 25 ๐ = 25 ๐ is large ๐ is intermediate ๐ is small [Bishop] v v
SLIDE 70
Learning curves of bias, variance, and noise
70
[Bishop]
SLIDE 71
Bias-variance decomposition: summary
71
} The noise term is unavoidable. } The terms we are interested in are bias and variance. } The approximation-generalization trade-off is seen in the
bias-variance decomposition.
SLIDE 72
Resources
72