machine learning for signal
play

Machine Learning for Signal Processing Detecting faces (& other - PowerPoint PPT Presentation

Dec 10, 2022 •362 likes •1.79k views

Machine Learning for Signal Processing Detecting faces (& other objects) in images Class 7. 22 Sep 2015 11755/18979 1 Last Lecture: How to describe a face The typical face A typical face that captures the essence of

  1. Boosting • The basic idea: Can a “weak” learning algorithm that performs just slightly better than a random guess be boosted into an arbitrarily accurate “strong” learner • This is a “meta” algorithm, that poses no constraints on the form of the weak learners themselves 11755/18979 40

  2. Boosting: A Voting Perspective • Boosting is a form of voting – Let a number of different classifiers classify the data – Go with the majority – Intuition says that as the number of classifiers increases, the dependability of the majority vote increases • Boosting by majority • Boosting by weighted majority – A (weighted) majority vote taken over all the classifiers – How do we compute weights for the classifiers? – How do we actually train the classifiers 11755/18979 41

  3. ADA Boost • Challenge: how to optimize the classifiers and their weights? – Trivial solution: Train all classifiers independently – Optimal: Each classifier focuses on what others missed – But joint optimization becomes impossible • Ada ptive Boost ing: Greedy incremental optimization of classifiers – Keep adding classifiers incrementally, to fix what others missed 11755/18979 42

  4. AdaBoost ILLUSTRATIVE EXAMPLE 11755/18979 43

  5. AdaBoost First WEAK Learner 11755/18979 44

  6. AdaBoost The First Weak Learner makes Errors 11755/18979 45

  7. AdaBoost Reweighted data 11755/18979 46

  8. AdaBoost SECOND Weak Learner FOCUSES ON DATA “MISSED” BY FIRST LEARNER 11755/18979 47

  9. AdaBoost SECOND STRONG Learner Combines both Weak Learners 11755/18979 48

  10. AdaBoost RETURNING TO THE SECOND WEAK LEARNER 11755/18979 49

  11. AdaBoost The SECOND Weak Learner makes Errors 11755/18979 50

  12. AdaBoost Reweighting data 11755/18979 51

  13. THIRD Weak AdaBoost Learner FOCUSES ON DATA “MISSED” BY FIRST AND SECOND LEARNERs 11755/18979 52

  14. AdaBoost THIRD STRONG Learner 11755/18979 53

  15. Boosting: An Example • Red dots represent training data from Red class • Blue dots represent training data from Blue class 11755/18979 54

  16. Boosting: An Example • The final strong learner has learnt a complicated decision boundary 11755/18979 55

  17. Boosting: An Example • The final strong learner h as learnt a complicated decision boundary • Decision boundaries in areas with low density of training points assumed inconsequential 11755/18979 56

  18. Overall Learning Pattern  Strong learner increasingly accurate with increasing number of weak learners  Residual errors increasingly difficult to correct ‒ Additional weak learners less and less effective Error of n th weak learner Error of n th strong learner number r of weak k learn rners rs 11755/18979 57

  19. Overfitting  Note: Can continue to add weak learners EVEN after strong learner error goes to 0!  Shown to IMPROVE generalization! Error of n th weak learner This may go to 0 Error of n th strong learner number r of weak k learn rners rs 11755/18979 58

  20. AdaBoost: Summary • No relation to Ada Lovelace • Adaptive Boosting • Adaptively Selects Weak Learners • ~8K citations for just one paper for Freund and Schapire 11755/18979 59

  21. The ADABoost Algorithm • Initialize D 1 ( x i ) = 1/ N • For t = 1, …, T – Train a weak classifier h t using distribution D t – Compute total error on training data • e t = Sum { D t (x i ) ½(1 – y i h t ( x i ))} – Set  t = ½ ln ((1 – e t ) / e t ) – For i = 1… N • set D t +1 ( x i ) = D t ( x i ) exp(-  t y i h t ( x i )) – Normalize D t +1 to make it a distribution • The final classifier is – H ( x ) = sign( S t  t h t ( x )) 11755/18979 60

  22. First, some example data = 0.3 E1 - 0.6 E2 = 0.2 E1 + 0.4 E2 = 0.5 E1 - 0.5 E2 = -0.8 E1 - 0.1 E2 E 1 = 0.7 E1 - 0.1 E2 = 0.4 E1 - 0.9 E2 = 0.6 E1 - 0.4 E2 = 0.2 E1 + 0.5 E2 E 2 Image = a*E1 + b*E2  a = Image.E1 • Face detection with multiple Eigen faces • Step 0: Derived top 2 Eigen faces from Eigen face training data • Step 1: On a (different) set of examples, express each image as a linear combination of Eigen faces – Examples include both faces and non faces – Even the non-face images are explained in terms of the Eigen faces 11755/18979 61

  23. Training Data D A = 0.3 E1 - 0.6 E2 = 0.2 E1 + 0.4 E2 E B = 0.5 E1 - 0.5 E2 = -0.8 E1 - 0.1 E2 F C = 0.7 E1 - 0.1 E2 = 0.4 E1 - 0.9 E2 G D = 0.6 E1 - 0.4 E2 = 0.2 E1 + 0.5 E2 ID E1 E2. Class A 0.3 -0.6 +1 Face = +1 B 0.5 -0.5 +1 Non-face = -1 C 0.7 -0.1 +1 D 0.6 -0.4 +1 E 0.2 0.4 -1 F -0.8 -0.1 -1 G 0.4 -0.9 -1 H 0.2 0.5 -1 11755/18979 62

  24. The ADABoost Algorithm • Initialize D 1 ( x i ) = 1/ N • For t = 1, …, T – Train a weak classifier h t using distribution D t – Compute total error on training data • e t = Sum { D t (x i ) ½(1 – y i h t ( x i ))} – Set  t = ½ ln ((1 – e t ) / e t ) – For i = 1… N • set D t +1 ( x i ) = D t ( x i ) exp(-  t y i h t ( x i )) – Normalize D t +1 to make it a distribution • The final classifier is – H ( x ) = sign( S t  t h t ( x )) 11755/18979 63

  25. Initialize D 1 ( x i ) = 1/ N 11755/18979 64

  26. Training Data = 0.3 E1 - 0.6 E2 = 0.2 E1 + 0.4 E2 = 0.5 E1 - 0.5 E2 = -0.8 E1 - 0.1 E2 = 0.7 E1 - 0.1 E2 = 0.4 E1 - 0.9 E2 = 0.6 E1 - 0.4 E2 = 0.2 E1 + 0.5 E2 ID E1 E2. Class Weight A 0.3 -0.6 +1 1/8 B 0.5 -0.5 +1 1/8 C 0.7 -0.1 +1 1/8 D 0.6 -0.4 +1 1/8 E 0.2 0.4 -1 1/8 F -0.8 -0.1 -1 1/8 G 0.4 -0.9 -1 1/8 H 0.2 0.5 -1 1/8 11755/18979 65

  27. The ADABoost Algorithm • Initialize D 1 ( x i ) = 1/ N • For t = 1, …, T – Train a weak classifier h t using distribution D t – Compute total error on training data • e t = Sum { D t (x i ) ½(1 – y i h t ( x i ))} – Set  t = ½ ln ( e t /(1 – e t )) – For i = 1… N • set D t +1 ( x i ) = D t ( x i ) exp(-  t y i h t ( x i )) – Normalize D t +1 to make it a distribution • The final classifier is – H ( x ) = sign( S t  t h t ( x )) 11755/18979 66

  28. The E1 “Stump” Classifier based on E1: F E H A G B C D if ( sign*wt(E1) > thresh) > 0) face = true -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 sign = +1 or -1 threshold Sign = +1, error = 3/8 Sign = -1, error = 5/8 ID E1 E2. Class Weight A 0.3 -0.6 +1 1/8 B 0.5 -0.5 +1 1/8 C 0.7 -0.1 +1 1/8 D 0.6 -0.4 +1 1/8 E 0.2 0.4 -1 1/8 F -0.8 -0.1 -1 1/8 G 0.4 -0.9 -1 1/8 H 0.2 0.5 -1 1/8 11755/18979 67

  29. The E1 “Stump” Classifier based on E1: F E H A G B C D if ( sign*wt(E1) > thresh) > 0) face = true -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 sign = +1 or -1 threshold Sign = +1, error = 3/8 Sign = -1, error = 5/8 ID E1 E2. Class Weight A 0.3 -0.6 +1 1/8 B 0.5 -0.5 +1 1/8 C 0.7 -0.1 +1 1/8 D 0.6 -0.4 +1 1/8 E 0.2 0.4 -1 1/8 F -0.8 -0.1 -1 1/8 G 0.4 -0.9 -1 1/8 H 0.2 0.5 -1 1/8 11755/18979 68

  30. The E1 “Stump” Classifier based on E1: F E H A G B C D if ( sign*wt(E1) > thresh) > 0) face = true -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 sign = +1 or -1 threshold Sign = +1, error = 3/8 Sign = -1, error = 5/8 ID E1 E2. Class Weight A 0.3 -0.6 +1 1/8 B 0.5 -0.5 +1 1/8 C 0.7 -0.1 +1 1/8 D 0.6 -0.4 +1 1/8 E 0.2 0.4 -1 1/8 F -0.8 -0.1 -1 1/8 G 0.4 -0.9 -1 1/8 H 0.2 0.5 -1 1/8 11755/18979 69

  31. The E1 “Stump” Classifier based on E1: F E H A G B C D if ( sign*wt(E1) > thresh) > 0) face = true -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 sign = +1 or -1 threshold Sign = +1, error = 3/8 Sign = -1, error = 5/8 ID E1 E2. Class Weight A 0.3 -0.6 +1 1/8 B 0.5 -0.5 +1 1/8 C 0.7 -0.1 +1 1/8 D 0.6 -0.4 +1 1/8 E 0.2 0.4 -1 1/8 F -0.8 -0.1 -1 1/8 G 0.4 -0.9 -1 1/8 H 0.2 0.5 -1 1/8 11755/18979 70

  32. The E1 “Stump” Classifier based on E1: if ( sign*wt(E1) > thresh) > 0) F E H A G B C D face = true -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 sign = +1 or -1 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 threshold Sign = +1, error = 3/8 Sign = -1, error = 5/8 ID E1 E2. Class Weight A 0.3 -0.6 +1 1/8 B 0.5 -0.5 +1 1/8 C 0.7 -0.1 +1 1/8 D 0.6 -0.4 +1 1/8 E 0.2 0.4 -1 1/8 F -0.8 -0.1 -1 1/8 G 0.4 -0.9 -1 1/8 H 0.2 0.5 -1 1/8 11755/18979 71

  33. The E1 “Stump” Classifier based on E1: if ( sign*wt(E1) > thresh) > 0) F E H A G B C D face = true -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 sign = +1 or -1 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 threshold Sign = +1, error = 2/8 Sign = -1, error = 6/8 ID E1 E2. Class Weight A 0.3 -0.6 +1 1/8 B 0.5 -0.5 +1 1/8 C 0.7 -0.1 +1 1/8 D 0.6 -0.4 +1 1/8 E 0.2 0.4 -1 1/8 F -0.8 -0.1 -1 1/8 G 0.4 -0.9 -1 1/8 H 0.2 0.5 -1 1/8 11755/18979 72

  34. The E1 “Stump” Classifier based on E1: if ( sign*wt(E1) > thresh) > 0) F E H A G B C D face = true -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 sign = +1 or -1 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 threshold Sign = +1, error = 1/8 Sign = -1, error = 7/8 ID E1 E2. Class Weight A 0.3 -0.6 +1 1/8 B 0.5 -0.5 +1 1/8 C 0.7 -0.1 +1 1/8 D 0.6 -0.4 +1 1/8 E 0.2 0.4 -1 1/8 F -0.8 -0.1 -1 1/8 G 0.4 -0.9 -1 1/8 H 0.2 0.5 -1 1/8 11755/18979 73

  35. The E1 “Stump” Classifier based on E1: if ( sign*wt(E1) > thresh) > 0) F E H A G B C D face = true -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 sign = +1 or -1 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 threshold Sign = +1, error = 2/8 Sign = -1, error = 6/8 ID E1 E2. Class Weight A 0.3 -0.6 +1 1/8 B 0.5 -0.5 +1 1/8 C 0.7 -0.1 +1 1/8 D 0.6 -0.4 +1 1/8 E 0.2 0.4 -1 1/8 F -0.8 -0.1 -1 1/8 G 0.4 -0.9 -1 1/8 H 0.2 0.5 -1 1/8 11755/18979 74

  36. The E1 “Stump” Classifier based on E1: if ( sign*wt(E1) > thresh) > 0) F E H A G B C D face = true -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 sign = +1 or -1 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 threshold Sign = +1, error = 1/8 Sign = -1, error = 7/8 ID E1 E2. Class Weight A 0.3 -0.6 +1 1/8 B 0.5 -0.5 +1 1/8 C 0.7 -0.1 +1 1/8 D 0.6 -0.4 +1 1/8 E 0.2 0.4 -1 1/8 F -0.8 -0.1 -1 1/8 G 0.4 -0.9 -1 1/8 H 0.2 0.5 -1 1/8 11755/18979 75

  37. The E1 “Stump” Classifier based on E1: if ( sign*wt(E1) > thresh) > 0) F E H A G B C D face = true -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 sign = +1 or -1 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 threshold Sign = +1, error = 2/8 Sign = -1, error = 6/8 ID E1 E2. Class Weight A 0.3 -0.6 +1 1/8 B 0.5 -0.5 +1 1/8 C 0.7 -0.1 +1 1/8 D 0.6 -0.4 +1 1/8 E 0.2 0.4 -1 1/8 F -0.8 -0.1 -1 1/8 G 0.4 -0.9 -1 1/8 H 0.2 0.5 -1 1/8 11755/18979 76

  38. The Best E1 “Stump” F E H A G B C D Classifier based on E1: if ( sign*wt(E1) > thresh) > 0) -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 face = true 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 Sign = +1 threshold Threshold = 0.45 Sign = +1, error = 1/8 ID E1 E2. Class Weight A 0.3 -0.6 +1 1/8 B 0.5 -0.5 +1 1/8 C 0.7 -0.1 +1 1/8 D 0.6 -0.4 +1 1/8 E 0.2 0.4 -1 1/8 F -0.8 -0.1 -1 1/8 G 0.4 -0.9 -1 1/8 H 0.2 0.5 -1 1/8 11755/18979 77

  39. The E2“Stump” Classifier based on E2: if ( sign*wt(E2) > thresh) > 0) Note order G A B D C F E H face = true -0.9 -0.6 -0.5 -0.4 -0.1 -0.1 0.4 0.5 sign = +1 or -1 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 threshold Sign = +1, error = 3/8 Sign = -1, error = 5/8 ID E1 E2. Class Weight A 0.3 -0.6 +1 1/8 B 0.5 -0.5 +1 1/8 C 0.7 -0.1 +1 1/8 D 0.6 -0.4 +1 1/8 E 0.2 0.4 -1 1/8 F -0.8 -0.1 -1 1/8 G 0.4 -0.9 -1 1/8 H 0.2 0.5 -1 1/8 11755/18979 78

  40. The Best E2“Stump” Classifier based on E2: if ( sign*wt(E2) > thresh) > 0) G A B D C F E H face = true -0.9 -0.6 -0.5 -0.4 -0.1 -0.1 0.4 0.5 sign = -1 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 Threshold = 0.15 threshold Sign = -1, error = 2/8 ID E1 E2. Class Weight A 0.3 -0.6 +1 1/8 B 0.5 -0.5 +1 1/8 C 0.7 -0.1 +1 1/8 D 0.6 -0.4 +1 1/8 E 0.2 0.4 -1 1/8 F -0.8 -0.1 -1 1/8 G 0.4 -0.9 -1 1/8 H 0.2 0.5 -1 1/8 11755/18979 79

  41. The Best “Stump” F E H A G B C D The Best overall classifier based on a single feature is -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 based on E1 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 If (wt(E1) > 0.45)  Face threshold Sign = +1, error = 1/8 ID E1 E2. Class Weight A 0.3 -0.6 +1 1/8 B 0.5 -0.5 +1 1/8 C 0.7 -0.1 +1 1/8 D 0.6 -0.4 +1 1/8 E 0.2 0.4 -1 1/8 F -0.8 -0.1 -1 1/8 G 0.4 -0.9 -1 1/8 H 0.2 0.5 -1 1/8 11755/18979 80

  42. The Best “Stump” 11755/18979 81

  43. The ADABoost Algorithm • Initialize D 1 ( x i ) = 1/ N • For t = 1, …, T – Train a weak classifier h t using distribution D t – Compute total error on training data • e t = Sum { D t (x i ) ½(1 – y i h t ( x i ))} – Set  t = ½ ln ( e t /(1 – e t )) – For i = 1… N – • set D t +1 ( x i ) = D t ( x i ) exp(-  t y i h t ( x i )) – Normalize D t +1 to make it a distribution • The final classifier is – H ( x ) = sign( S t  t h t ( x )) 11755/18979 82

  44. The Best “Stump” 11755/18979 83

  45. The Best Error F E H A G B C D The Error of the classifier is the sum of the weights of -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 the misclassified instances 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 threshold Sign = +1, error = 1/8 ID E1 E2. Class Weight A 0.3 -0.6 +1 1/8 B 0.5 -0.5 +1 1/8 C 0.7 -0.1 +1 1/8 D 0.6 -0.4 +1 1/8 E 0.2 0.4 -1 1/8 F -0.8 -0.1 -1 1/8 G 0.4 -0.9 -1 1/8 H 0.2 0.5 -1 1/8 NOTE: THE ERROR IS THE SUM OF THE WEIGHTS OF MISCLASSIFIED INSTANCES 11755/18979 84

  46. The ADABoost Algorithm • Initialize D 1 ( x i ) = 1/ N • For t = 1, …, T – Train a weak classifier h t using distribution D t – Compute total error on training data • e t = Sum { D t (x i ) ½(1 – y i h t ( x i ))} – Set  t = ½ ln ((1 – e t ) / e t ) – For i = 1… N • set D t +1 ( x i ) = D t ( x i ) exp(-  t y i h t ( x i )) – Normalize D t +1 to make it a distribution • The final classifier is – H ( x ) = sign( S t  t h t ( x )) 11755/18979 85

  47. Computing Alpha F E H A G B C D Alpha = 0.5ln((1-1/8) / (1/8)) -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 = 0.5 ln(7) = 0.97 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 threshold Sign = +1, error = 1/8 11755/18979 86

  48. The Boosted Classifier Thus Far F E H A G B C D Alpha = 0.5ln((1-1/8) / (1/8)) -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 = 0.5 ln(7) = 0.97 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 threshold Sign = +1, error = 1/8 h1(X) = wt(E1) > 0.45 ? +1 : -1 H(X) = sign(0.97 * h1(X)) It’s the same as h1(x) 11755/18979 87

  49. The ADABoost Algorithm • Initialize D 1 ( x i ) = 1/ N • For t = 1, …, T – Train a weak classifier h t using distribution D t – Compute total error on training data • e t = Average {½ (1 – y i h t ( x i ))} – Set  t = ½ ln ((1 – e t ) / e t ) – For i = 1… N • set D t +1 ( x i ) = D t ( x i ) exp(-  t y i h t ( x i )) – Normalize D t +1 to make it a distribution • The final classifier is – H ( x ) = sign( S t  t h t ( x )) 11755/18979 88

  50. The Best Error F E H A G B C D D t +1 ( x i ) = D t ( x i ) exp(-  t y i h t ( x i )) -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 exp(  t ) = exp(0.97) = 2.63 exp(-  t ) = exp(-0.97) = 0.38 threshold ID E1 E2. Class Weight Weight A 0.3 -0.6 +1 1/8 * 2.63 0.33 B 0.5 -0.5 +1 1/8 * 0.38 0.05 C 0.7 -0.1 +1 1/8 * 0.38 0.05 D 0.6 -0.4 +1 1/8 * 0.38 0.05 E 0.2 0.4 -1 1/8 * 0.38 0.05 F -0.8 0.1 -1 1/8 * 0.38 0.05 G 0.4 -0.9 -1 1/8 * 0.38 0.05 H 0.2 0.5 -1 1/8 * 0.38 0.05 Multiply the correctly classified instances by 0.38 Multiply incorrectly classified instances by 2.63 11755/18979 89

  51. AdaBoost 11755/18979 90

  52. AdaBoost 11755/18979 91

  53. The ADABoost Algorithm • Initialize D 1 ( x i ) = 1/ N • For t = 1, …, T – Train a weak classifier h t using distribution D t – Compute total error on training data • e t = Average {½ (1 – y i h t ( x i ))} – Set  t = ½ ln ((1 – e t ) / e t ) – For i = 1… N • set D t +1 ( x i ) = D t ( x i ) exp(-  t y i h t ( x i )) – Normalize D t +1 to make it a distribution • The final classifier is – H ( x ) = sign( S t  t h t ( x )) 11755/18979 92

  54. The Best Error F E H A G B C D D’ = D / sum(D) -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 threshold ID E1 E2. Class Weight Weight Weight A 0.3 -0.6 +1 1/8 * 2.63 0.33 0.48 B 0.5 -0.5 +1 1/8 * 0.38 0.05 0.074 C 0.7 -0.1 +1 1/8 * 0.38 0.05 0.074 D 0.6 -0.4 +1 1/8 * 0.38 0.05 0.074 E 0.2 0.4 -1 1/8 * 0.38 0.05 0.074 F -0.8 0.1 -1 1/8 * 0.38 0.05 0.074 G 0.4 -0.9 -1 1/8 * 0.38 0.05 0.074 H 0.2 0.5 -1 1/8 * 0.38 0.05 0.074 Multiply the correctly classified instances by 0.38 Multiply incorrectly classified instances by 2.63 Normalize to sum to 1.0 11755/18979 93

  55. The Best Error F E H A G B C D D’ = D / sum(D) -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 1/8 1/8 1/8 1/8 1/8 1/8 1/8 1/8 threshold ID E1 E2. Class Weight A 0.3 -0.6 +1 0.48 B 0.5 -0.5 +1 0.074 C 0.7 -0.1 +1 0.074 D 0.6 -0.4 +1 0.074 E 0.2 0.4 -1 0.074 F -0.8 0.1 -1 0.074 G 0.4 -0.9 -1 0.074 H 0.2 0.5 -1 0.074 Multiply the correctly classified instances by 0.38 Multiply incorrectly classified instances by 2.63 Normalize to sum to 1.0 11755/18979 94

  56. The ADABoost Algorithm • Initialize D 1 ( x i ) = 1/ N • For t = 1, …, T – Train a weak classifier h t using distribution D t – Compute total error on training data • e t = Average {½ (1 – y i h t ( x i ))} – Set  t = ½ ln ( e t /(1 – e t )) – For i = 1… N • set D t +1 ( x i ) = D t ( x i ) exp(-  t y i h t ( x i )) – Normalize D t +1 to make it a distribution • The final classifier is – H ( x ) = sign( S t  t h t ( x )) 11755/18979 95

  57. E1 classifier Classifier based on E1: if ( sign*wt(E1) > thresh) > 0) F E H A G B C D face = true -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 sign = +1 or -1 .074 .074 .074 .48 .074 .074 .074 .074 threshold Sign = +1, error = 0.222 Sign = -1, error = 0.778 ID E1 E2. Class Weight A 0.3 -0.6 +1 0.48 B 0.5 -0.5 +1 0.074 C 0.7 -0.1 +1 0.074 D 0.6 -0.4 +1 0.074 E 0.2 0.4 -1 0.074 F -0.8 0.1 -1 0.074 G 0.4 -0.9 -1 0.074 H 0.2 0.5 -1 0.074 11755/18979 96

  58. E1 classifier Classifier based on E1: if ( sign*wt(E1) > thresh) > 0) F E H A G B C D face = true -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 sign = +1 or -1 .074 .074 .074 .48 .074 .074 .074 .074 threshold Sign = +1, error = 0.148 Sign = -1, error = 0.852 ID E1 E2. Class Weight A 0.3 -0.6 +1 0.48 B 0.5 -0.5 +1 0.074 C 0.7 -0.1 +1 0.074 D 0.6 -0.4 +1 0.074 E 0.2 0.4 -1 0.074 F -0.8 0.1 -1 0.074 G 0.4 -0.9 -1 0.074 H 0.2 0.5 -1 0.074 11755/18979 97

  59. The Best E1 classifier Classifier based on E1: if ( sign*wt(E1) > thresh) > 0) F E H A G B C D face = true -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 sign = +1 or -1 .074 .074 .074 .48 .074 .074 .074 .074 threshold Sign = +1, error = 0.074 ID E1 E2. Class Weight A 0.3 -0.6 +1 0.48 B 0.5 -0.5 +1 0.074 C 0.7 -0.1 +1 0.074 D 0.6 -0.4 +1 0.074 E 0.2 0.4 -1 0.074 F -0.8 0.1 -1 0.074 G 0.4 -0.9 -1 0.074 H 0.2 0.5 -1 0.074 11755/18979 98

  60. The Best E2 classifier Classifier based on E2: if ( sign*wt(E2) > thresh) > 0) G A B D C F E H face = true -0.9 -0.6 -0.5 -0.4 -0.1 -0.1 0.4 0.5 sign = +1 or -1 .074 .48 .074 .074 .074 .074 .074 .074 threshold Sign = -1, error = 0.148 ID E1 E2. Class Weight A 0.3 -0.6 +1 0.48 B 0.5 -0.5 +1 0.074 C 0.7 -0.1 +1 0.074 D 0.6 -0.4 +1 0.074 E 0.2 0.4 -1 0.074 F -0.8 0.1 -1 0.074 G 0.4 -0.9 -1 0.074 H 0.2 0.5 -1 0.074 11755/18979 99

  61. The Best Classifier Classifier based on E1: F E H A G B C D if (wt(E1) > 0.45) face = true -0.8 0.2 0.2 0.3 0.4 0.5 0.6 0.7 .074 .074 .074 .48 .074 .074 .074 .074 Alpha = 0.5ln((1-0.074) / 0.074) threshold = 1.26 Sign = +1, error = 0.074 ID E1 E2. Class Weight A 0.3 -0.6 +1 0.48 B 0.5 -0.5 +1 0.074 C 0.7 -0.1 +1 0.074 D 0.6 -0.4 +1 0.074 E 0.2 0.4 -1 0.074 F -0.8 0.1 -1 0.074 G 0.4 -0.9 -1 0.074 H 0.2 0.5 -1 0.074 11755/18979 100

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

E9 205 Machine Learning for Signal Processing Introduction to Machine Learning of Sensory Signals

E9 205 Machine Learning for Signal Processing Introduction to Machine Learning of Sensory Signals

E9 205 Machine Learning for Signal Processing Introduction to Machine Learning of Sensory Signals 14-08-2017 Instructor - Sriram Ganapathy (sriram@ee.iisc.ernet.in) Teaching Assistant - Aravind Illa (aravindece77@gmail.com). Overview What

644 views • 30 slides
E9 205 Machine Learning for Signal Processing Introduction to Machine Learning of Sensory Signals

E9 205 Machine Learning for Signal Processing Introduction to Machine Learning of Sensory Signals

E9 205 Machine Learning for Signal Processing Introduction to Machine Learning of Sensory Signals 08-08-2018 Instructor - Sriram Ganapathy (sriramg@iisc.ac.in) Teaching Assistant - Akshara Soman (aksharas@iisc.ac.in). Feature Extraction

983 views • 28 slides
E9 205 Machine Learning for Signal Processing Introduction to Machine Learning of Sensory Signals

E9 205 Machine Learning for Signal Processing Introduction to Machine Learning of Sensory Signals

E9 205 Machine Learning for Signal Processing Introduction to Machine Learning of Sensory Signals 12-08-2019 Instructor - Sriram Ganapathy (sriramg@iisc.ac.in) Teaching Assistant - Prachi Singh (prachisingh@iisc.ac.in)

345 views • 15 slides
E9 205 Machine Learning for Signal Processing Introduction to Machine Learning of Sensory Signals

E9 205 Machine Learning for Signal Processing Introduction to Machine Learning of Sensory Signals

E9 205 Machine Learning for Signal Processing Introduction to Machine Learning of Sensory Signals 19-08-2019 Instructor - Sriram Ganapathy (sriramg@iisc.ac.in) Teaching Assistant - Prachi Singh (prachisingh@iisc.ac.in)

239 views • 21 slides
Machine Learning for Signal Processing Lecture 1: Signal Representations Class 1. 27 August

Machine Learning for Signal Processing Lecture 1: Signal Representations Class 1. 27 August

11-755/18-797 Machine Learning for Signal Processing Machine Learning for Signal Processing Lecture 1: Signal Representations Class 1. 27 August 2012 Instructor: Bhiksha Raj 27 Aug 2012 11-755/18-797 1 What is a signal A mechanism for

1.18k views • 96 slides
E9 205 Machine Learning for Signal Procesing Deep Learning for Audio and Vision 20-11-2019

E9 205 Machine Learning for Signal Procesing Deep Learning for Audio and Vision 20-11-2019

E9 205 Machine Learning for Signal Procesing Deep Learning for Audio and Vision 20-11-2019 Speech Recognition Noise Channel Automatic Speech Systems Courtesy Google Images Signal Modeling Short-term spectra integrated in mel frequency

555 views • 32 slides
Machine Learning for Signal Processing Lecture 1: Signal Representations Class 1. 29 August

Machine Learning for Signal Processing Lecture 1: Signal Representations Class 1. 29 August

Machine Learning for Signal Processing Lecture 1: Signal Representations Class 1. 29 August 2013 Instructor: Bhiksha Raj 29 Aug 2013 11-755/18-797 1 What is a signal A mechanism for conveying information Semaphores, gestures,

1.19k views • 97 slides
Machine Learning: Basic Principles Teaching demonstration Kalle Palomki Department of Signal

Machine Learning: Basic Principles Teaching demonstration Kalle Palomki Department of Signal

Machine Learning: Basic Principles Teaching demonstration Kalle Palomki Department of Signal Processing and Acoustics Aalto University Content 1. Goal 2. Machine learning: definition 3. Classification an important machine learning

1.27k views • 49 slides
E9 205 Machine Learning for Signal Processing Feature Extraction 08-08-2016 Recap Real-world

E9 205 Machine Learning for Signal Processing Feature Extraction 08-08-2016 Recap Real-world

E9 205 Machine Learning for Signal Processing Feature Extraction 08-08-2016 Recap Real-world signals Patterns in signal Learning - uncovering the underlying patterns Roadmap of the course Types of Learning Supervised Learning

482 views • 24 slides
Machine Learning for Signal Processing Lecture 1: Introduction Representing sound and images

Machine Learning for Signal Processing Lecture 1: Introduction Representing sound and images

Machine Learning for Signal Processing Lecture 1: Introduction Representing sound and images Class 1. 1 Sep 2015 Instructor: Bhiksha Raj 11-755/18-797 1 What is a signal A mechanism for conveying information Semaphores, gestures,

1.02k views • 78 slides
E9 205 Machine Learning for Signal Processing Linear Predictive Analysis 22-08-2016 Linear

E9 205 Machine Learning for Signal Processing Linear Predictive Analysis 22-08-2016 Linear

E9 205 Machine Learning for Signal Processing Linear Predictive Analysis 22-08-2016 Linear Prediction Current sample expressed as a linear combination of past samples n- 3 n- 2 n- 1 n a 1 a 2 a 3 Properties of LP Error signal (for the

489 views • 20 slides
Learning Word Embeddings from Speech NIPS Workshop on Machine Learning for Audio Signal Processing

Learning Word Embeddings from Speech NIPS Workshop on Machine Learning for Audio Signal Processing

Learning Word Embeddings from Speech NIPS Workshop on Machine Learning for Audio Signal Processing December 8, 2017 Yu-An Chung James Glass MIT Computer Science and Artificial Intelligence Laboratory Cambridge, MA Outline Motivation

1.29k views • 19 slides
E9 205 Machine Learning for Signal Processing Non-negative Matrix Factorization 16-09-2019 Audio

E9 205 Machine Learning for Signal Processing Non-negative Matrix Factorization 16-09-2019 Audio

E9 205 Machine Learning for Signal Processing Non-negative Matrix Factorization 16-09-2019 Audio Composition Audio signal generated by a number of individual components (example - different instruments).

287 views • 18 slides
PROMISE An End-To-End Design of a PROgrammable MIxed-Signal AccElerator for Machine Learning

PROMISE An End-To-End Design of a PROgrammable MIxed-Signal AccElerator for Machine Learning

PROMISE An End-To-End Design of a PROgrammable MIxed-Signal AccElerator for Machine Learning Algorithms Prakalp Srivastava *, Mingu Kang *, Sujan K. Gonugondla, Sungmin Lim, Jungwook Choi, Vikram Adve, Nam S. Kim, Naresh Shanbhag

659 views • 30 slides
E9 205 Machine Learning for Signal Processing Supervised-Dimensionality-Reduction. Decision

E9 205 Machine Learning for Signal Processing Supervised-Dimensionality-Reduction. Decision

E9 205 Machine Learning for Signal Processing Supervised-Dimensionality-Reduction. Decision Theory 26-08-2019 Probability Distributions Advantages and Disadvantages of PCA Simple linear transform Eigen decomposition of Data Covariance

487 views • 9 slides
E9 205 Machine Learning for Signal Processing 23-8-17 Outline Basics for Image Processing

E9 205 Machine Learning for Signal Processing 23-8-17 Outline Basics for Image Processing

E9 205 Machine Learning for Signal Processing 23-8-17 Outline Basics for Image Processing Filtering Smoothing Edge Detection Scale Invariant Feature Transform (SIFT) Reference: UCF, Computer Vision Course Link:

599 views • 42 slides
Problem and model selection and model selection Elisabeth Gnatowski Elisabeth Gnatowski

Problem and model selection and model selection Elisabeth Gnatowski Elisabeth Gnatowski

False discovery rate False discovery rate Problem and model selection and model selection Elisabeth Gnatowski Elisabeth Gnatowski Definition of the FDR Definition of the FDR Multiple Testing Multiple Testing FDR and pFDR FDR and pFDR

486 views • 5 slides
Stochastic Simulation Idea: probabilities samples Get probabilities from samples: X count X

Stochastic Simulation Idea: probabilities samples Get probabilities from samples: X count X

Stochastic Simulation Idea: probabilities samples Get probabilities from samples: X count X probability x 1 n 1 n 1 / m x 1 . . . . . . . . . . . . x k n k x k n k / m total m If we could sample from a variables

714 views • 18 slides
EDDIE: EM-Based Detection of Deviations in Program Execution Published at ISCA 2017 Alireza

EDDIE: EM-Based Detection of Deviations in Program Execution Published at ISCA 2017 Alireza

EDDIE: EM-Based Detection of Deviations in Program Execution Published at ISCA 2017 Alireza Nazari, Nader Sehatbakhsh, Monjur Alam, Alenka Zajic, Milos Prvulovic EECS 573 Presented by Janarthanan and Vivek 1 Goal Detect Malicious changes to

3.21k views • 28 slides
Understanding Quality Control: A Process Improvement Perspective Robert L. Schmidt MD, PhD, MBA

Understanding Quality Control: A Process Improvement Perspective Robert L. Schmidt MD, PhD, MBA

Understanding Quality Control: A Process Improvement Perspective Robert L. Schmidt MD, PhD, MBA Lauren N Pearson DO, MPH DISCLOSURE: Robert Schmidt In the past 12 months, I have not had any significant financial interest or other relationship

1.37k views • 81 slides
The Finite-Set Independence Criterion (FSIC) Zoltn Szab Arthur Gretton Wittawat Jitkrittum

The Finite-Set Independence Criterion (FSIC) Zoltn Szab Arthur Gretton Wittawat Jitkrittum

The Finite-Set Independence Criterion (FSIC) Zoltn Szab Arthur Gretton Wittawat Jitkrittum Gatsby Unit University College London wittawat@gatsby.ucl.ac.uk 3rd UCL Workshop on the Theory of Big Data 28 June 2017 1/10 What Is

1.09k views • 74 slides
Discrete Probabilistic Programming from First Principles Guy Van den Broeck The Fourth

Discrete Probabilistic Programming from First Principles Guy Van den Broeck The Fourth

Discrete Probabilistic Programming from First Principles Guy Van den Broeck The Fourth International Workshop on Declarative Learning Based Programming (DeLBP) Aug 11, 2019 What are probabilistic programs? What is the formal semantics? How

1.06k views • 63 slides
Implementing the LeybourneTaylor test for seasonal unit roots in Stata Christopher F Baum

Implementing the LeybourneTaylor test for seasonal unit roots in Stata Christopher F Baum

Implementing the LeybourneTaylor test for seasonal unit roots in Stata Christopher F Baum Jess Otero Stata London Users Meeting, September 2018 Baum, Otero (BC/DIW, U. del Rosario) LeybourneTaylor seasonal unit root tests London

943 views • 71 slides
Probabilistic Graphical Models Probabilistic Graphical Models Structure learning in Bayesian

Probabilistic Graphical Models Probabilistic Graphical Models Structure learning in Bayesian

Probabilistic Graphical Models Probabilistic Graphical Models Structure learning in Bayesian networks Siamak Ravanbakhsh Fall 2019 Learning objectives Learning objectives why structure learning is hard? two approaches to structure learning

1.04k views • 74 slides

More recommend