cs480 680 lecture 7 may 29 2019
play

CS480/680 Lecture 7: May 29, 2019 Classification with Mixture of - PowerPoint PPT Presentation

Feb 10, 2024 β€’372 likes β€’557 views

CS480/680 Lecture 7: May 29, 2019 Classification with Mixture of Gaussians [B] Sections 4.2, [M] Section 4.2 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Linear Models Probabilistic Generative Models Regression

  1. CS480/680 Lecture 7: May 29, 2019 Classification with Mixture of Gaussians [B] Sections 4.2, [M] Section 4.2 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1

  2. Linear Models β€’ Probabilistic Generative Models Regression Classification University of Waterloo CS480/680 Spring 2019 Pascal Poupart 2

  3. Probabilistic Generative Model β€’ Pr(𝐷) : prior probability of class 𝐷 β€’ Pr π’š 𝐷 : class conditional distribution of π’š β€’ Classification: compute posterior Pr(𝐷|π’š) according to Bayes’ theorem Pr π’š 𝐷 Pr 𝐷 Pr 𝐷 π’š = βˆ‘ ! Pr π’š 𝐷 Pr 𝐷 = 𝑙𝑄𝑠 π’š 𝐷 Pr(𝐷) University of Waterloo CS480/680 Spring 2019 Pascal Poupart 3

  4. Assumptions β€’ In classification, the number of classes is finite, so a natural prior Pr(𝐷) is the multinomial Pr 𝐷 = 𝑑 ! = 𝜌 ! β€’ When π’š ∈ β„œ " , then it is often OK to assume that Pr(π’š|𝐷) is Gaussian. β€’ Furthermore, assume that the same covariance matrix 𝚻 is used for each class. Pr π’š 𝑑 ! ∝ 𝑓 #$ % π’š#𝝂 𝒍 " 𝚻 #$ π’š#𝝂 𝒍 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 4

  5. Posterior Distribution )𝚻#$ π’š#𝝂𝒍 " " # #$ % π’š#𝝂𝒍 Pr 𝑑 ! π’š = )𝚻#$ π’š#𝝂𝒍 βˆ‘ " " " # #$ % π’š#𝝂𝒍 " " # #$ )𝚻#$+,𝝂" )𝚻#$𝝂" % π’š)𝚻#$π’š#%𝝂" = βˆ‘ " " " # #$ )𝚻#$π’š,𝝂" )𝚻#$𝒗" % π’š)𝚻#$π’š#%𝝂" Consider two classes 𝑑 ! and 𝑑 % & = )𝚻#$π’š#$ )𝚻#$𝝂/ ./0𝝂/ %𝝂/ &' )𝚻#$π’š#$ )𝚻#$𝝂" ."0𝝂" %𝝂" University of Waterloo CS480/680 Spring 2019 Pascal Poupart 5

  6. Posterior Distribution $ = "𝚻#$𝝂'#,-.& " 𝚻#$π’š*$ "𝚻#$𝝂&#$ "#𝝂' # 𝝂& +𝝂& +𝝂' .' $)* $ = $)* # 𝒙"π’š*01 where 𝒙 = 𝚻 #$ (𝝂 ! βˆ’ 𝝂 + ) $ $ . & - 𝚻 #$ 𝝂 + + ln - 𝚻 #$ 𝝂 ! + and π‘₯ , = βˆ’ % 𝝂 ! % 𝝂 + . ' University of Waterloo CS480/680 Spring 2019 Pascal Poupart 6

  7. Logistic Sigmoid $ β€’ Let 𝜏 𝑏 = $)* #2 Logistic sigmoid β€’ Then Pr 𝑑 ! π’š = 𝜏(𝒙 - π’š + π‘₯ , ) β€’ Picture: University of Waterloo CS480/680 Spring 2019 Pascal Poupart 7

  8. Logistic Sigmoid posterior class conditionals University of Waterloo CS480/680 Spring 2019 Pascal Poupart 8

  9. Prediction 𝑐𝑓𝑑𝑒 π‘‘π‘šπ‘π‘‘π‘‘ = 𝑏𝑠𝑕𝑛𝑏𝑦 ! Pr 𝑑 ! π’š 𝜏 𝒙 - π’š + π‘₯ , β‰₯ 0.5 = A𝑑 $ 𝑑 % otherwise - M Class boundary: 𝜏 𝒙 ! π’š = 0.5 $ ⟹ π’š = 0.5 "3 $)* # 𝒙& - M ⟹ 𝒙 ! π’š = 0 ∴ linear separator University of Waterloo CS480/680 Spring 2019 Pascal Poupart 9

  10. Multi-class Problems β€’ Consider Gaussian conditional distributions with identical Ξ£ 23 4 ! 23 π’š 𝑑 1 Pr 𝑑 1 π’š = βˆ‘ " 23 4 " 23 π’š 𝑑 6 (𝚻#$ π’š#𝝂𝒍 7 ! 8 #$ % π’š#𝝂! = ( βˆ‘ " 7 " 8 #$ 𝚻#$ π’š#𝝂" % π’š#𝝂" 7 ! 8 #$ (𝚻#$π’š+𝝂! (𝚻#$𝝂! % #%𝝂! = βˆ‘ " 7 " 8 #$ (𝚻#$π’š+𝝂" (𝚻#$𝝂" % #%𝝂" (𝚻#$π’š#$ (𝚻#$𝒗!+-. /! (1 8 𝝂! %𝝂! 8 𝒙! π’š = (𝚻#$𝝂"+-. /" = π’š ⟹ softmax (𝚻#$π’š#$ (1 𝒙" 𝝂" %𝝂" βˆ‘ " 8 βˆ‘ " 8 3 = (βˆ’ 4 3 𝚻 64 𝝂 2 + ln 𝜌 2 , 𝝂 2 3 𝚻 64 ) where 𝒙 2 5 𝝂 2 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 10

  11. Softmax β€’ When there are several classes, the posterior is a softmax (generalization of the sigmoid) 8 !" π’š β€’ Softmax distribution: Pr 𝑑 1 π’š = βˆ‘ $ 8 !$ π’š β€’ Argmax distribution: Pr 𝑑 1 π’š = -1 if 𝑙 = 𝑏𝑠𝑕𝑛𝑏𝑦 6 𝑔 6 (𝑦) 0 otherwise 9:;8 !" % = lim βˆ‘ $ 9:;8 !$ % 9:;8β†’= 8 !" % β‰ˆ (softmax approximation) βˆ‘ $ 8 !$(%) University of Waterloo CS480/680 Spring 2019 Pascal Poupart 11

  12. Softmax posterior class conditionals University of Waterloo CS480/680 Spring 2019 Pascal Poupart 12

  13. Parameter Estimation β€’ Where do Pr(𝑑 ! ) and Pr(π’š|𝑑 ! ) come from? β€’ Parameters: 𝜌 , 𝝂 𝟐 , 𝝂 πŸ‘ , 𝚻 Pr π’š 𝑑 ! = 𝑙 𝚻 𝑓 # ! " π’š#𝝂 𝟐 $ 𝚻 %! π’š#𝝂 ! Pr 𝑑 ! = 𝜌 , Pr π’š 𝑑 & = 𝑙 𝚻 𝑓 # ! " π’š#𝝂 πŸ‘ $ 𝚻 %! π’š#𝝂 " Pr 𝑑 & = 1 βˆ’ 𝜌, where 𝑙 𝚻 is the normalization constant that depends on 𝚻 β€’ Estimate parameters by – Maximum likelihood – Maximum a posteriori – Bayesian learning University of Waterloo CS480/680 Spring 2019 Pascal Poupart 13

  14. Maximum Likelihood Solution β€’ Likelihood: 𝑧 > ∈ {0,1} L 𝐘, 𝐳 = Pr 𝒀, 𝒛 𝜌, 𝝂 ' , 𝝂 ( , 𝚻 = * ! '+* ! 6 πœŒπ‘‚ π’š ) 𝝂 ' , 𝚻 1 βˆ’ 𝜌 𝑂 π’š ) 𝝂 ( , 𝚻 ) β€’ ML hypothesis: βˆ— , 𝚻 βˆ— > = βˆ— , 𝝂 ( < 𝜌 βˆ— , 𝝂 ' ( π’š ) βˆ’ 𝝂 ' 1 𝚻 +' π’š ) βˆ’ 𝝂 ' 𝑏𝑠𝑕𝑛𝑏𝑦 -,𝝂 " ,𝝂 " ,𝚻 βˆ‘ ) 𝑧 ) ln 𝜌 + ln 𝑙 𝚻 βˆ’ ' ( π’š ) βˆ’ 𝝂 ( 1 𝚻 +' π’š ) βˆ’ 𝝂 ( ln 1 βˆ’ 𝜌 + ln 𝑙 𝚻 βˆ’ ' + 1 βˆ’ 𝑧 ) University of Waterloo CS480/680 Spring 2019 Pascal Poupart 14

  15. Maximum Likelihood Solution β€’ Set derivative to 0 0 = G HI J 𝒀,𝒛 G" & & ⟹ 0 = βˆ‘ K 𝑧 K " + 1 βˆ’ 𝑧 K βˆ’ &L" ⟹ 0 = βˆ‘ K 𝑧 K 1 βˆ’ 𝜌 + (1 βˆ’ 𝑧 K )(βˆ’πœŒ) ⟹ βˆ‘ K 𝑧 K = 𝜌 βˆ‘ K 𝑧 K + βˆ‘ K 1 βˆ’ 𝑧 K ⟹ βˆ‘ K 𝑧 K = πœŒπ‘‚ (where 𝑂 is the # of training points) βˆ‘ ? M ? ∴ = 𝜌 N University of Waterloo CS480/680 Spring 2019 Pascal Poupart 15

  16. Maximum Likelihood Solution 0 = πœ– ln 𝑀 𝒀, 𝒛 /πœ–π‚ ! ⟹ 0 = βˆ‘ " 𝑧 " [βˆ’πš» #! π’š " βˆ’ 𝝂 ! ] ⟹ βˆ‘ " 𝑧 " π’š " = βˆ‘ " 𝑧 " 𝝂 ! ⟹ βˆ‘ " 𝑧 " π’š " = 𝑂 ! 𝝂 ! βˆ‘ ( % ( π’š ( βˆ‘ ( (!#% ( )π’š ( ∴ = 𝝂 ! Similarly: = 𝝂 * ' ) ' * where 𝑂 $ is the # of data points in class 1 𝑂 % is the # of data points in class 2 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 16

  17. Maximum Likelihood + ,- . 𝒀,𝒛 = 0 +𝚻 ⟹ β‹― ' ) ' * ⟹ Ξ£ = ' 𝑻 ! + ' 𝑻 * ! π’š " βˆ’ 𝝂 ! 5 ' ) βˆ‘ "∈4 ) π’š " βˆ’ 𝝂 ! where 𝑻 ! = ! π’š " βˆ’ 𝝂 * 5 ' * βˆ‘ "∈4 * π’š " βˆ’ 𝝂 * 𝑻 * = ( 𝑻 6 is the empirical covariance matrix of class 𝑙 ) University of Waterloo CS480/680 Spring 2019 Pascal Poupart 17

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

CS480/680 Lecture 2: May 8 th , 2019 Nearest Neighbour [RN] Sec. 18.8.1, [HTF] Sec. 2.3.2, [D]

CS480/680 Lecture 2: May 8 th , 2019 Nearest Neighbour [RN] Sec. 18.8.1, [HTF] Sec. 2.3.2, [D]

CS480/680 Lecture 2: May 8 th , 2019 Nearest Neighbour [RN] Sec. 18.8.1, [HTF] Sec. 2.3.2, [D] Chapt. 3, [B] Sec. 2.5.2, [M] Sec. 1.4.2 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Inductive Learning (recap) Induction

1.2k views β€’ 25 slides
CS480/680 Lecture 4: May 15, 2019 Statistical Learning [RN]: Sec 20.1, 20.2, [M]: Sec. 2.2, 3.2

CS480/680 Lecture 4: May 15, 2019 Statistical Learning [RN]: Sec 20.1, 20.2, [M]: Sec. 2.2, 3.2

CS480/680 Lecture 4: May 15, 2019 Statistical Learning [RN]: Sec 20.1, 20.2, [M]: Sec. 2.2, 3.2 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Statistical Learning View: we have uncertain knowledge of the world Idea:

1.43k views β€’ 25 slides
CS480/680 Lecture 8: June 3, 2019 Classification by Logistic Regression, Generalized linear

CS480/680 Lecture 8: June 3, 2019 Classification by Logistic Regression, Generalized linear

CS480/680 Lecture 8: June 3, 2019 Classification by Logistic Regression, Generalized linear models [RN] Sec 18.6.4, [B] Sec. 4.3, [M] Chapt. 8, [HTF] Sec. 4.4 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Beyond Mixtures of

756 views β€’ 18 slides
CS480/680 Lecture 18: July 8, 2019 Recurrent and Recursive Neural Networks [GBC] Chap. 10

CS480/680 Lecture 18: July 8, 2019 Recurrent and Recursive Neural Networks [GBC] Chap. 10

CS480/680 Lecture 18: July 8, 2019 Recurrent and Recursive Neural Networks [GBC] Chap. 10 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Variable length data Traditional feed forward neural networks can only handle fixed

609 views β€’ 18 slides
CS480/680 Lecture 14: June 24, 2019 Support Vector Machines (continued) [B] Sec. 7.1 [D] Sec.

CS480/680 Lecture 14: June 24, 2019 Support Vector Machines (continued) [B] Sec. 7.1 [D] Sec.

CS480/680 Lecture 14: June 24, 2019 Support Vector Machines (continued) [B] Sec. 7.1 [D] Sec. 11.5-11.6 [HTF] Chap. 12 [M] Sec. 14.5 [RN] 18.9 [MRT] Chap. 4 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Overlapping Class

496 views β€’ 14 slides
CS480/680 Lecture 12: June 17, 2019 Gaussian Processes [B] Section 6.4 [M] Chap. 15 [HTF] Sec.

CS480/680 Lecture 12: June 17, 2019 Gaussian Processes [B] Section 6.4 [M] Chap. 15 [HTF] Sec.

CS480/680 Lecture 12: June 17, 2019 Gaussian Processes [B] Section 6.4 [M] Chap. 15 [HTF] Sec. 8.3 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Gaussian Process Regression Idea: distribution over functions University of

526 views β€’ 23 slides
CS480/680 Lecture 19: July 10, 2019 Attention and Transformer Networks [Vaswani et al.,

CS480/680 Lecture 19: July 10, 2019 Attention and Transformer Networks [Vaswani et al.,

CS480/680 Lecture 19: July 10, 2019 Attention and Transformer Networks [Vaswani et al., Attention is All You Need, NeurIPS , 2017] University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Attention Attention in Computer Vision

708 views β€’ 13 slides
CS480/680 Machine Learning Lecture 3: May 13, 2019 Linear Regression [RN] Sec. 18.6.1, [HTF]

CS480/680 Machine Learning Lecture 3: May 13, 2019 Linear Regression [RN] Sec. 18.6.1, [HTF]

CS480/680 Machine Learning Lecture 3: May 13, 2019 Linear Regression [RN] Sec. 18.6.1, [HTF] Sec. 2.3.1, [D] Sec. 7.6, [B] Sec. 3.1, [M] Sec. 1.4.5 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Linear model for regression

582 views β€’ 14 slides
CS480/680 Lecture 22: July 22, 2019 Ensemble Learning [RN] Sec. 18.10, [M] Sec. 16.2.5, [B]

CS480/680 Lecture 22: July 22, 2019 Ensemble Learning [RN] Sec. 18.10, [M] Sec. 16.2.5, [B]

CS480/680 Lecture 22: July 22, 2019 Ensemble Learning [RN] Sec. 18.10, [M] Sec. 16.2.5, [B] Chap. 14, [HTF] Chap 15-16, [D] Chap. 11 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Outline Ensemble Learning Bagging

1.72k views β€’ 22 slides
CS480/680 Lecture 11: June 12, 2019 Kernel methods [D] Chap. 11 [B] Sec. 6.1, 6.2 [M] Sec.

CS480/680 Lecture 11: June 12, 2019 Kernel methods [D] Chap. 11 [B] Sec. 6.1, 6.2 [M] Sec.

CS480/680 Lecture 11: June 12, 2019 Kernel methods [D] Chap. 11 [B] Sec. 6.1, 6.2 [M] Sec. 14.1, 14.2 [HTF] Chap. 6 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Non-linear Models Recap Generalized linear models:

611 views β€’ 15 slides
CS480/680 Lecture 15: June 26, 2019 Deep Neural Networks [GBC] Chap. 6, 7, 8 University of

CS480/680 Lecture 15: June 26, 2019 Deep Neural Networks [GBC] Chap. 6, 7, 8 University of

CS480/680 Lecture 15: June 26, 2019 Deep Neural Networks [GBC] Chap. 6, 7, 8 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Outline Deep Neural Networks Gradient Vanishing Rectified linear units Overfitting

941 views β€’ 25 slides
CS480/680 Lecture 9: June 5, 2019 Perceptrons, Neural Networks [D] Chapt. 4, [HTF] Chapt. 11,

CS480/680 Lecture 9: June 5, 2019 Perceptrons, Neural Networks [D] Chapt. 4, [HTF] Chapt. 11,

CS480/680 Lecture 9: June 5, 2019 Perceptrons, Neural Networks [D] Chapt. 4, [HTF] Chapt. 11, [B] Sec. 4.1.7, 5.1, [M] Sec. 8.5.4, [RN] Sec. 18.7 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Outline Neural networks

885 views β€’ 29 slides
CS480/680 Lecture 24: July 29, 2019 Gradient Boosting, Bagging, Decision Forest [RN] Sec. 18.10,

CS480/680 Lecture 24: July 29, 2019 Gradient Boosting, Bagging, Decision Forest [RN] Sec. 18.10,

CS480/680 Lecture 24: July 29, 2019 Gradient Boosting, Bagging, Decision Forest [RN] Sec. 18.10, [M] Sec. 16.2.5, 16.4.5 [B] Chap. 14, [HTF] Chap 10, 15-16, [D] Chap. 13 University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Gradient

1.45k views β€’ 20 slides
CS480/680 Machine Learning Lecture 1: May 6 th , 2019 Course Introduction Pascal Poupart

CS480/680 Machine Learning Lecture 1: May 6 th , 2019 Course Introduction Pascal Poupart

CS480/680 Machine Learning Lecture 1: May 6 th , 2019 Course Introduction Pascal Poupart University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Outline Introduction to Machine Learning Course website and logistics University of

527 views β€’ 24 slides
CS480/680 Machine Learning Lecture 12: February 13 th , 2020 Expectation-Maximization Zahra

CS480/680 Machine Learning Lecture 12: February 13 th , 2020 Expectation-Maximization Zahra

CS480/680 Machine Learning Lecture 12: February 13 th , 2020 Expectation-Maximization Zahra Sheikhbahaee University of Waterloo CS480/680 Winter 2020 Zahra Sheikhbahaee 1 Outline -mean Clustering Gaussian Mixture model EM for

535 views β€’ 30 slides
CS480/680 Machine Learning Lecture 20: Convolutional Neural Network Zahra Sheikhbahaee March 29,

CS480/680 Machine Learning Lecture 20: Convolutional Neural Network Zahra Sheikhbahaee March 29,

CS480/680 Machine Learning Lecture 20: Convolutional Neural Network Zahra Sheikhbahaee March 29, 2020 University of Waterloo CS480/680 Winter 2020 Zahra Sheikhbahaee 1 Outline Convolution Zero Padding Stride Weight Sharing Pooling

1.24k views β€’ 22 slides
Supervised Learning: Linear Methods (1/2) Applied Multivariate Statistics Spring 2012 Overview

Supervised Learning: Linear Methods (1/2) Applied Multivariate Statistics Spring 2012 Overview

Supervised Learning: Linear Methods (1/2) Applied Multivariate Statistics Spring 2012 Overview Review: Conditional Probability LDA / QDA: Theory Fishers Discriminant Analysis LDA: Example Quality control: Testset and

739 views β€’ 15 slides
Test Case Software

Test Case Software

Test Case Software to be tested Output

452 views β€’ 7 slides
ADVANCED MACHINE LEARNING Caveats and Techniques to Deal with Imbalanced Datasets (Adapted from

ADVANCED MACHINE LEARNING Caveats and Techniques to Deal with Imbalanced Datasets (Adapted from

ADVANCED MACHINE LEARNING ADVANCED MACHINE LEARNING Caveats and Techniques to Deal with Imbalanced Datasets (Adapted from H. He and E. A. Garcia, Learning from Imbalanced Data, IEEE Trans. Knowledge and Data Engineering, vol. 21, issue 9,

285 views β€’ 24 slides
INFO 4300 / CS4300 Information Retrieval slides adapted from Hinrich Sch utzes, linked from

INFO 4300 / CS4300 Information Retrieval slides adapted from Hinrich Sch utzes, linked from

INFO 4300 / CS4300 Information Retrieval slides adapted from Hinrich Sch utzes, linked from http://informationretrieval.org/ IR 20/26: Linear Classifiers and Flat clustering Paul Ginsparg Cornell University, Ithaca, NY 10 Nov 2009 1 /

1.09k views β€’ 92 slides
E9 205 Machine Learning for Signal Procesing Support Vector Machines 9-10-2019 Linear

E9 205 Machine Learning for Signal Procesing Support Vector Machines 9-10-2019 Linear

E9 205 Machine Learning for Signal Procesing Support Vector Machines 9-10-2019 Linear Classifiers x f y est w x + b&gt;0 0 = denotes +1 b + x denotes -1 w How would you classify this data? w x + b&lt;0 SVM and applications,

496 views β€’ 22 slides
Lecture 9: Logistic Regression Discriminative vs. Generative Classification Aykut Erdem

Lecture 9: Logistic Regression Discriminative vs. Generative Classification Aykut Erdem

Lecture 9: Logistic Regression Discriminative vs. Generative Classification Aykut Erdem March 2016 Hacettepe University Administrative Assignment 2 is out! It is due March 18 (i.e. in 2 weeks) You will implement Naive Bayes

415 views β€’ 30 slides
Kindergarten students to another location). Overflow does become an added cost to the district

Kindergarten students to another location). Overflow does become an added cost to the district

Fishkill ES can continue to function/operate as is and overflow students to another school when reaching capacity. (Please note since we moved into a K-6 configuration and reduced class sizes, we have yet had to overflow Fishkill ES

172 views β€’ 4 slides
Class 15: Calculation of natural frequency Class 15: Calculation of natural frequency Old Slide

Class 15: Calculation of natural frequency Class 15: Calculation of natural frequency Old Slide

Class 15: Calculation of natural frequency Class 15: Calculation of natural frequency Old Slide Origin of simple harmonic motion U(x) = kx 2 Total energy Total energy F = - kx x V=0 a is max V=0, a is max. V=0 a is max V=0, a is max. a=0, v

156 views β€’ 4 slides

More recommend