Learning: Linear Methods CE417: Introduction to Artificial - PowerPoint PPT Presentation

Square error loss function for classification! Square error loss is not suitable for classification: Least square loss penalizes ‘too correct’ predictions (that they lie a long way on the correct side } of the decision) Least square loss also lack robustness to noise } - 𝐿 = 2 * 𝐾 𝒙 = @ 𝑥𝑦 : + 𝑥 E − 𝑧 : :B' 36

Notation } 𝒙 = 𝑥 E , 𝑥 ' , . . . , 𝑥 Y \ } 𝒚 = 1, 𝑦 ' , … , 𝑦 Y \ } 𝑥 E + 𝑥 ' 𝑦 ' + ⋯ + 𝑥 Y 𝑦 Y = 𝒙 \ 𝒚 } We show input by 𝒚 or 𝑔(𝒚) 37

SSE cost function for classification 𝐿 = 2 } Is it more suitable if we set 𝑔 𝒚; 𝒙 = 𝑕 𝒙 \ 𝒚 ? - sign 𝒙 \ 𝒚 − 𝑧 * * 𝐾 𝒙 = @ sign 𝒙 \ 𝒚 : − 𝑧 : 𝑧 = 1 :B' sign 𝑨 = j− 1, 𝑨 < 0 1, 𝑨 ≥ 0 𝒙 \ 𝒚 } 𝐾 𝒙 is a piecewise constant function shows the number of misclassifications 𝐾(𝒙) Training error incurred in classifying training samples 38

Perceptron algorithm } Linear classifier } Two-class: 𝑧 ∈ {−1,1} } 𝑧 = −1 for 𝐷 * , 𝑧 = 1 for 𝐷 ' } Goal: ∀𝑗, 𝒚 (:) ∈ 𝐷 ' ⇒ 𝒙 \ 𝒚 (:) > 0 ∀𝑗, 𝒚 : ∈ 𝐷 * ⇒ 𝒙 \ 𝒚 : < 0 } } 𝑕 𝒚; 𝒙 = sign(𝒙 \ 𝒚) 39

� Perceptron criterion 𝐾 o 𝒙 = − @ 𝒙 \ 𝒚 : 𝑧 : :∈ℳ ℳ : subset of training data that are misclassified Many solutions? Which solution among them? 40

Cost function 𝐾(𝒙) 𝐾 o (𝒙) 𝑥 E 𝑥 E 𝑥 ' 𝑥 ' # of misclassifications Perceptron’s as a cost function cost function There may be many solutions in these cost functions 41 [Duda, Hart, and Stork, 2002]

� � Batch Perceptron “Gradient Descent” to solve the optimization problem: 𝒙 MN' = 𝒙 M − 𝜃𝛼 𝒙 𝐾 o (𝒙 M ) 𝒙 𝐾 o 𝒙 = − @ 𝒚 : 𝑧 : 𝛼 :∈ℳ Batch Perceptron converges in finite number of steps for linearly separable data: Initialize 𝒙 Repeat 𝒚 : 𝑧 : 𝒙 = 𝒙 + 𝜃 ∑ :∈ℳ Until convergence 42

Stochastic gradient descent for Perceptron } Single-sample perceptron: } If 𝒚 (:) is misclassified: 𝒙 MN' = 𝒙 M + 𝜃𝒚 (:) 𝑧 (:) } Perceptron convergence theorem: for linearly separable data } If training data are linearly separable, the single-sample perceptron is also guaranteed to find a solution in a finite number of steps Fixed-Increment single sample Perceptron Initialize 𝒙, 𝑢 ← 0 repeat 𝜃 can be set to 1 and 𝑢 ← 𝑢 + 1 proof still works 𝑗 ← 𝑢 mod 𝑂 if 𝒚 (:) is misclassified then 𝒙 = 𝒙 + 𝒚 (:) 𝑧 (:) Until all patterns properly classified 43

Weight Updates 44

Learning: Binary Perceptron Start with weights = 0 } For each training instance: } Classify with current weights } If correct (i.e., y=y*), no change! } If wrong: adjust the weight vector } 𝒙 MN' = 𝒙 M + 𝜃𝒚 (:) 𝑧 (:) 45

Example 46

Perceptron: Example Change 𝒙 in a direction that corrects the error 47 [Bishop]

Learning: Binary Perceptron } Start with weights = 0 } For each training instance: } Classify with current weights } If correct (i.e., y=y*), no change! } If wrong: adjust the weight vector by adding or subtracting the feature vector. Subtract if y* is -1. 48

Examples: Perceptron } Separable Case 49

Convergence of Perceptron [Duda, Hart & Stork, 2002] } For data sets that are not linearly separable, the single-sample perceptron learning algorithm will never converge 50

Multiclass Decision Rule } If we have multiple classes: A weight vector for each class: } Score (activation) of a class y: } Prediction highest score wins } Binary = multiclass where the negative class has weight zero 51

Learning: Multiclass Perceptron } Start with all weights = 0 } Pick up training examples one by one } Predict with current weights } If correct, no change! } If wrong: lower score of wrong answer, raise score of right answer 52

Example: Multiclass Perceptron “win the vote” “win the election” “win the game” BIAS : 1 BIAS : 0 BIAS : 0 win : 0 win : 0 win : 0 game : 0 game : 0 game : 0 vote : 0 vote : 0 vote : 0 the : 0 the : 0 the : 0 ... ... ... 53

Properties of Perceptrons } Separability: true if some parameters get the training set perfectly classified Separable } Convergence: if the training is separable, perceptron will eventually converge (binary case) } Mistake Bound: the maximum number of mistakes (binary case) related to the margin or degree of separability Non-Separable 54

Examples: Perceptron } Non-Separable Case 55

Examples: Perceptron } Non-Separable Case 56

Discriminative approach: logistic regression 𝐿 = 2 𝑕 𝒚; 𝒙 = 𝜏(𝒙 \ 𝒚) 𝒚 = 1, 𝑦 ' , … , 𝑦 Y 𝒙 = 𝑥 E , 𝑥 ' , … , 𝑥 Y 𝜏 . is an activation function } Sigmoid (logistic) function } Activation function 1 𝜏 𝑨 = 1 + 𝑓 wx 57

Logistic regression: cost function 𝒙 F = argmin 𝐾(𝒙) 𝒙 𝐾 𝒙 = A = @ −𝑧 (:) log 𝜏 𝒙 \ 𝒚 (:) − (1 − 𝑧 (:) )log 1 − 𝜏 𝒙 \ 𝒚 (:) :B' } 𝐾(𝒙) is convex w.r.t. parameters. 58

Logistic regression: loss function Loss 𝑧, 𝑔 𝒚; 𝒙 = −𝑧×log 𝜏 𝒚; 𝒙 − (1 − 𝑧)×log(1 − 𝜏 𝒚; 𝒙 ) −log(𝜏(𝒚; 𝒙)) if 𝑧 = 1 Loss 𝑧, 𝜏 𝒚; 𝒙 = j Since 𝑧 = 1 or 𝑧 = 0 −log(1 − 𝜏 𝒚; 𝒙 ) if 𝑧 = 0 ⇒ How is it related to zero-one loss? } = j1 𝑧 ≠ 𝑧 } Loss 𝑧, 𝑧 0 𝑧 = 𝑧 } 1 𝜏 𝒚; 𝒙 = 1 + 𝑓𝑦𝑞(−𝒙 \ 𝒚) 59

Logistic regression: Gradient descent 𝒙 MN' = 𝒙 M − 𝜃𝛼 𝒙 𝐾(𝒙 M ) A 𝜏 𝒚 : ; 𝒙 − 𝑧 : 𝒚 : 𝛼 𝒙 𝐾 𝒙 = @ :B' } Is it similar to gradient of SSE for linear regression? A 𝒙 \ 𝒚 : − 𝑧 : 𝒚 : 𝛼 𝒙 𝐾 𝒙 = @ :B' 60

Multi-class logistic regression \ } 𝑕 𝒚; 𝑿 = 𝑕 ' 𝒚, 𝑿 , … , 𝑕 • 𝒚, 𝑿 } 𝑿 = 𝒙 ' ⋯ 𝒙 • contains one vector of parameters for each class \ 𝒚 ) exp (𝒙 ‚ 𝑕 ‚ 𝒚; 𝑿 = • \ 𝒚 ) ∑ exp (𝒙 … …B' 61

Logistic regression: multi-class † = argmin 𝑿 𝐾(𝑿) 𝑿 A • : log 𝑕 ‚ 𝒚 (:) ; 𝑿 𝐾 𝑿 = − @ @ 𝑧 ‚ :B' ‚B' 𝒛 is a vector of length 𝐿 (1-of-K coding) 𝑿 = 𝒙 ' ⋯ 𝒙 • e.g., 𝒛 = 0,0,1,0 \ when the target class is 𝐷 ˆ 62

Logistic regression: multi-class MN' = 𝒙 … M − 𝜃𝛼 𝑿 𝐾(𝑿 M ) 𝒙 … A 𝑕 … 𝒚 : ; 𝑿 − 𝑧 … : 𝒚 : 𝛼 𝒙 ‰ 𝐾 𝑿 = @ :B' 63

Multi-class classifier } 𝑕 𝒚; 𝑿 = 𝑕 ' 𝒚, 𝑿 , … , 𝑕 • 𝒚, 𝑿 } 𝑿 = 𝒙 ' ⋯ 𝒙 • contains one vector of parameters for each class } In linear classifiers, 𝑿 is 𝑒×𝐿 where 𝑒 shows number of features } 𝑿 \ 𝒚 provides us a vector } 𝑕 𝒚; 𝑿 contains K numbers giving class scores for the input 𝒚 64

Example } Output obtained from 𝑿 \ 𝒚 + 𝒄 𝑦 ' ⋮ 𝒚 = 𝑦 •Ž• 28 𝒙 ' ×28 𝑿 \ = ⋮ 𝒙 'E 'EוŽ• 𝑐 ' ⋮ 𝒄 = 𝑐 'E 65 This slide has been adopted from Fei Fei Li and colleagues lectures, cs231n, Stanford 2017

Example 𝑿 \ How can we tell whether this W and b is good or bad? 66 This slide has been adopted from Fei Fei Li and colleagues lectures, cs231n, Stanford 2017

Bias can also be included in the W matrix 67 This slide has been adopted from Fei Fei Li and colleagues lectures, cs231n, Stanford 2017

Softmax classifier loss: example ” •(–) 𝑓 𝑀 (:) = − log • 𝑓 ” ‰ ∑ …B' 𝑀 (') = − log 0.13 = 0.89 68 This slide has been adopted from Fei Fei Li and colleagues lectures, cs231n, Stanford 2017

Support Vector Machines } Maximizing the margin: good according to intuition, theory, practice } Support vector machines (SVMs) find the separator with max margin 69

Hard-margin SVM: Optimization problem 2 max 𝒙 𝒙,— ˜ s. t. 𝒙 \ 𝒚 A + 𝑥 E ≥ 1 ∀𝑧 A = 1 𝒙 \ 𝒚 A + 𝑥 E ≤ −1 ∀𝑧 A = −1 𝒙 \ 𝒚 + 𝑥 E = 0 𝑦 2 1 𝒙 * Margin: 𝒙 𝒙 \ 𝒚 + 𝑥 E = 1 𝒙 𝒙 \ 𝒚 + 𝑥 E = −1 70 𝑦 1

Distance between an 𝒚 (A) and the plane distance = 𝒙 \ 𝒚 (A) + 𝑥 E 𝒙 𝒚 (A) 71

Hard-margin SVM: Optimization problem We can equivalently optimize: 1 2 𝒙 \ 𝒙 min 𝒙,— ˜ 𝒙 \ 𝒚 A + 𝑥 E ≥ 1 𝑜 = 1, … , 𝑂 s. t. 𝑧 A } It is a convex Quadratic Programming (QP) problem } There are computationally efficient packages to solve it. } It has a global minimum (if any). 72

Error measure } Margin violation amount 𝜊 A ( 𝜊 A ≥ 0 ): 𝒙 \ 𝒚 A + 𝑥 E ≥ 1 − 𝜊 A } 𝑧 A - } Total violation: ∑ 𝜊 A AB' 73

Soft-margin SVM: Optimization problem } SVM with slack variables: allows samples to fall within the margin, but penalizes them - 1 2 𝒙 * + 𝐷 @ 𝜊 A min ž 𝒙,— ˜ , š › ›œ• AB' 𝒙 \ 𝒚 A + 𝑥 E ≥ 1 − 𝜊 A 𝑜 = 1, … , 𝑂 s. t. 𝑧 A 𝜊 A ≥ 0 𝜊 A : slack variables 𝑦 2 𝜊 < 1 0 < 𝜊 A < 1 : if 𝒚 A is correctly classified but inside margin 𝜊 A > 1 : if 𝒚 A is misclassifed 𝜊 > 1 𝑦 1 74

Soft-margin SVM: Cost function - 1 2 𝒙 * + 𝐷 @ 𝜊 A min ž 𝒙,— ˜ , š › ›œ• AB' 𝒙 \ 𝒚 A + 𝑥 E ≥ 1 − 𝜊 A 𝑜 = 1, … , 𝑂 s. t. 𝑧 A 𝜊 A ≥ 0 } It is equivalent to the unconstrained optimization problem: - 1 2 𝒙 * + 𝐷 @ max (0,1 − 𝑧 (A) (𝒙 \ 𝒚 (A) + 𝑥 E )) min 𝒙,— ˜ AB' 75

Multi-class SVM - 𝐾 𝑿 = 1 𝑂 @ 𝑀 : + 𝜇 𝑿 * :B' … ≡ 𝑕 … 𝒚 : ; 𝑿 𝑡 𝑀 : = @ max 0,1 + 𝑡 … − 𝑡 ¡ (–) Hinge loss: \ 𝒚 (:) = 𝒙 … …¢¡ (–) \ 𝒚 (:) − 𝒙 ¡ (–) \ 𝒚 (:) = @ max 0,1 + 𝒙 … …¢¡ (–) • Y * 𝑆 𝑿 = @ @ 𝑥 ¤‚ L2 regularization: ‚B' ¤B' 76

Multi-class SVM loss: Example 3 training examples, 3 classes. With some W the scores are 𝑋 \ 𝑦 \ 𝒚 (:) 𝑡 … = 𝒙 … 𝑀 : = @ max 0,1 + 𝑡 … − 𝑡 ¡ (–) …¢¡ (–) - 1 = 1 𝑂 @ 𝑀 : 3 2.9 + 0 + 12.9 = 5.7 :B' 𝑀 (ˆ) = max 𝑀 (') = max 0,1 + 5.1 − 3.2 𝑀 (*) = max 0,1 + 1.3 − 4.9 (0, 2.2 − (−3.1) + 1) +max (0, 2.5 − (−3.1) + 1) + max 0,1 − 1.7 − 3.2 + max 0,1 + 2 − 4.9 = max (0, 6.3) + max (0, 6.6) = max 0,2.9 + max(0, −3.9) = max 0, −2.6 + max(0, −1.9) = 2.9 + 0 = 6.3 + 6.6 = 12.9 = 0 + 0 77 This slide has been adopted from Fei Fei Li and colleagues lectures, cs231n, Stanford 2017

Recap We need 𝛼 ª 𝑀 to update weights • Y * L2 regularization 𝑆 𝑋 = ∑ ∑ 𝑥 ¤‚ ‚B' ¤B' • Y 𝑆 𝑋 = ∑ ∑ L1 regularization 𝑥 ¤‚ ‚B' ¤B' 78 This slide has been adopted from Fei Fei Li and colleagues lectures, cs231n, Stanford 2017

Generalized linear } Linear combination of fixed non-linear function of the input vector 𝑔(𝒚; 𝒙) = 𝑥 E + 𝑥 ' 𝜚 ' (𝒚)+ . . . 𝑥 ¬ 𝜚 ¬ (𝒚) {𝜚 ' (𝒚), . . . , 𝜚 ¬ (𝒚)} : set of basis functions (or features) 𝜚 : 𝒚 : ℝ Y → ℝ 79

Basis functions: examples } Linear } Polynomial (univariate) 80

Polynomial regression: example 𝑛 = 3 𝑛 = 1 𝑛 = 5 𝑛 = 7 81

Generalized linear classifier } Assume a transformation 𝜚: ℝ Y → ℝ ¬ on the feature space 𝝔 𝒚 = [𝜚 ' (𝒚), . . . , 𝜚 ¬ (𝒚)] } 𝒚 → 𝝔 𝒚 {𝜚 ' (𝒚), . . . , 𝜚 ¬ (𝒚)} : set of basis functions (or features) 𝜚 : 𝒚 : ℝ Y → ℝ } Find a hyper-plane in the transformed feature space: 𝜚 * (𝒚) 𝑦 2 𝜚: 𝒚 → 𝝔 𝒚 𝒙 \ 𝝔 𝒚 + 𝑥 E = 0 𝑦 1 𝜚 ' (𝒚) 82

Model complexity and overfitting } With limited training data, models may achieve zero training error but a large test error. 1 A * 𝑧 : − 𝑔 𝒚 : ; 𝜾 Training 𝑜 @ ≈ 0 :B' (empirical) loss * ≫ 0 Expected E 𝐲,´ 𝑧 − 𝑔 𝒚; 𝜾 (true) loss } Over-fitting: when the training loss no longer bears any relation to the test (generalization) loss. } Fails to generalize to unseen examples. 83

Polynomial regression 𝑛 = 0 𝑛 = 1 𝑧 𝑧 𝑛 = 9 𝑛 = 3 𝑧 𝑧 84 [Bishop]

Over-fitting causes } Model complexity } E.g., Model with a large number of parameters (degrees of freedom) } Low number of training data } Small data size compared to the complexity of the model 85

Model complexity } Example: } Polynomials with larger 𝑛 are becoming increasingly tuned to the random noise on the target values. 𝑛 = 0 𝑛 = 1 𝑧 𝑧 𝑛 = 3 𝑛 = 9 𝑧 𝑧 86 86 [Bishop]

Number of training data & overfitting } Over-fitting problem becomes less severe as the size of training data increases. 𝑛 = 9 𝑛 = 9 𝑜 = 15 𝑜 = 100 [Bishop] 87

How to evaluate the learner’s performance? } Generalization error: true (or expected) error that we would like to optimize } Two ways to assess the generalization error is: } Practical: Use a separate data set to test the model } Theoretical: Law of Large numbers } statistical bounds on the difference between training and expected errors 88

Avoiding over-fitting } Determine a suitable value for model complexity (Model Selection) } Simple hold-out method } Cross-validation } Regularization (Occam’s Razor) } Explicit preference towards simple models } Penalize for the model complexity in the objective function 89

Model Selection } learning algorithm defines the data-driven search over the hypothesis space (i.e. search for good parameters) } hyperparameters are the tunable aspects of the model, that the learning algorithm does not select This slide has been adopted from CMU ML course: 90 http://www.cs.cmu.edu/~mgormley/courses/10601-s18/

Model Selection } Model selection is the process by which we choose the “best” model from among a set of candidates } assume access to a function capable of measuring the quality of a model } typically done “outside” the main training algorithm } Model selection / hyperparameter optimization is just another form of learning This slide has been adopted from CMU ML course: 91 http://www.cs.cmu.edu/~mgormley/courses/10601-s18/

� Simple hold-out: model selection } Steps: } Divide training data into training and validation set 𝑤_𝑡𝑓𝑢 } Use only the training set to train a set of models } Evaluate each learned model on the validation set * 𝑧 (:) − 𝑔 𝒚 (:) ; 𝒙 ' ¸_”¹M ∑ } 𝐾 ¸ 𝒙 = :∈¸_”¹M } Choose the best model based on the validation set error } Usually, too wasteful of valuable training data } Training data may be limited. } On the other hand, small validation set give a relatively noisy estimate of performance. 92

Simple hold out: training, validation, and test sets } Simple hold-out chooses the model that minimizes error on validation set. } 𝐾 ¸ 𝒙 F is likely to be an optimistic estimate of generalization error. } extra parameter (e.g., degree of polynomial) is fit to this set. } Estimate generalization error for the test set } performance of the selected model is finally evaluated on the test set Training Validation 93 Test

Cross-Validation (CV): Evaluation } 𝑙 -fold cross-validation steps: } Shuffle the dataset and randomly partition training data into 𝑙 groups of approximately equal size } for 𝑗 = 1 to 𝑙 } Choose the 𝑗 -th group as the held-out validation group } Train the model on all but the 𝑗 -th group of data } Evaluate the model on the held-out group } Performance scores of the model from 𝑙 runs are averaged . } The average error rate can be considered as an estimation of the true performance. … First run … Second run … … (k-1)th run … k-th run 94

Cross-Validation (CV): Model Selection } For each model we first find the average error find by CV. } The model with the best average performance is selected. 95

Cross-validation: polynomial regression example } 5-fold CV } 100 runs } average 𝑛 = 3 𝑛 = 1 CV: 𝑁𝑇𝐹 = 1.45 CV: 𝑁𝑇𝐹 = 0.30 𝑛 = 5 𝑛 = 7 CV: 𝑁𝑇𝐹 = 45.44 CV: 𝑁𝑇𝐹 = 31759 96

Regularization } Adding a penalty term in the cost function to discourage the coefficients from reaching large values. } Ridge regression (weight decay): A * 𝑧 : − 𝒙 \ 𝝔 𝒚 : + 𝜇𝒙 \ 𝒙 𝐾 𝒙 = @ :B' w𝟐 𝚾 \ 𝒛 ¾ = 𝚾 \ 𝚾 + 𝜇𝑱 𝒙 97

Polynomial order } Polynomials with larger 𝑛 are becoming increasingly tuned to the random noise on the target values. } magnitude of the coefficients typically gets larger by increasing 𝑛 . [Bishop] 98

Regularization parameter 𝑛 = 9 𝑥 F E 𝑥 F ' 𝑥 F * 𝑥 F ˆ 𝑥 F • 𝑥 F Â 𝑥 F Ã 𝑥 F • [Bishop] 𝑥 F Ž 𝑥 F Ä 𝑚𝑜𝜇 = −∞ 𝑚𝑜𝜇 = −18 99

Regularization parameter } Generalization } 𝜇 now controls the effective complexity of the model and hence determines the degree of over-fitting 100 [Bishop]