Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick CS 7616 Pattern Recognition Bayesian Decision Theory Aaron Bobick School of Interactive Computing
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick Outline for “today” Simple Tuberculosis reminder of Bayes rule and how that relates to decision making Some basic discussions of what it means to make a good decision and the relation to Bayes Basic Bayesian decision making Minimum loss Application to Normal distributions Origins of linear classifiers? Why normals? Obvious and less obvious
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick Special thanks… Professor Srihari in Buffalo… posted lots of slides…
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick So you go to the doctor… Assume you go to the doctor because it’s that time of year… He tells you that you’re overdue for your Tuberculosis test You take the TB test ( 𝑌 ) and it’s positive !!! ( 𝑌 + ) But then he tells you not to worry because: The detection rate is 100% 𝑄 𝑌 + 𝑈 + ) = 1 Collectively exhaustive But the false alarm rate is 5% 𝑄 𝑌 + 𝑈 − ) = 0.05 Mutually exclusive The incident rate in Atlanta of TB is 0.1% 𝑄 𝑈 + = 0.0 01 Therefore the odds that you have TB given the test are: 𝑄 𝑌 + 𝑈 + 𝑄 𝑈 + 𝑄 𝑌 + 𝑈 + 𝑄 𝑈 + 𝑄 𝑈 + 𝑌 + ) = = 𝑄 𝑌 + | 𝑈 + 𝑄 𝑈 + +𝑄 𝑌 + 𝑈 − 𝑄 ( 𝑈 − ) 𝑄 𝑌 + 1 . 0∗0 . 001 = 1 . 0∗0 . 001+0 . 05∗0 . 999 = 0.0196 (ie 20 times what it was before the test) Bayes rule
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick So… Q1: if you had to decide right then whether you have TB or not, what would you decide? Q2: would you go get a chest X-ray? Why can’t you really answer that question? Cost of the X-ray? Cost of having TB and not finding out? (Prostate cancer treatments….) So to make the “right” decisions we needed to know: Prior probabilities 𝑄 𝑈 + Likelihoods 𝑄 𝑌 + 𝑈 + and 𝑄 𝑌 + 𝑈 − Cost (loss) functions
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick Bayes decision theory Bayesian theory is fundamental to decision theory and pattern recognition. Basically the mechanisms by which one can evaluate the probability of being right (and thus wrong). Allows one to compute an expectation of cost/reward (assuming some very non-ICBM – no infinities - types of loss) But… It presumes that that a variety of probabilities are known – or at least known about how much they are unknown (Bayes meets Rumsfeld???) We’ll ignore this concern for now…
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick Bayes 1: Priors We have states of nature 𝜕 𝑗 that are mutually exclusive and collectively exhaustive: i P ω = ∑ ( ) 1 i Decision rule if only two classes and based only on prior: if P 𝜕 1 > P 𝜕 2 choose class 𝜕 1 otherwise 𝜕 2 .
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick Bayes 2: Class conditional probabilities Need to know the probability of our data (measurements) given the possible states of nature: 𝑞 ( 𝑦 | 𝜕 𝑗 ) These are probability densities as opposed to distribution on the priors. I will definitely confuse this is class.
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick Bayes rule to get data conditioned probability ω ω ( | ) ( ) p x P ω = ( | ) j j P x j ( ) p x = ∑ ω ω ( ) ( | ) ( ) p x p x P where “evidence” j j j Read “posterior is the likelihood times the prior divided by the evidence”. And since the “evidence” 𝑞 𝑦 is fixed we can usually ignore that.
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick The posteriors from the division…
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick Bayesian decision rule If 𝑄 𝜕 1 𝑦 > 𝑄 𝜕 2 𝑦 then choose 𝜕 1 since the true state of nature is more likely 𝜕 1 …. Assuming there is no significant difference between being wrong in one direction or the other. What is probability of making an error? 𝑄 error 𝑦 = 𝑄 𝜕 1 | 𝑦 when we deicded 𝜕 2 and 𝑄 error 𝑦 = 𝑄 ω 2|x when we decided 𝜕 1 . So P error 𝑦 = min [ 𝑄 ( 𝜕 1 | 𝑦 ), 𝑄 𝜕 2 | 𝑦 ] (Bayes error)
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick Obvious generalizations: Feature is a vector (no real difference) More than two classes (as long as ME and CE no problem) Introduce a general loss function which is more general than just making an error … we’ll do this in a minute… And you can refuse to give an answer “ I don’t know”. We’ll talk more about that another time.
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick Loss functions and minimum risk Let 𝜕 𝑗 be the possible states of nature. Let { 𝛽 𝑘 } be the possible actions taken (usually announcing the class so as many actions as classes). Let 𝜇 𝛽 𝑘 𝜕 𝑗 be the “loss” incurred for taking action j when actual state of nature is i. Then the expected loss of taking action i when measurement 𝑦 : α = ∑ λ α ω ω ( | ) ( | ) ( | ) R x P x i i j j j So: select 𝛽 𝑗 with minimum expected loss. That’s what you’re “ risking ”. Bayes risk is the best you can do.
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick LRT – likelihood ratio test Action 𝛽 𝑗 is to choose i. Cost 𝜇 𝑗𝑘 is cost of choosing i when reality is j. α = λ ω + λ ω ( | ) ( | ) ( | ) R x P x P x Two risks: 1 11 1 12 2 α = λ ω + λ ω ( | ) ( | ) ( | ) R x P x P x 2 21 1 22 2 Choose 𝛽 1 is it’s risk is lower: λ − λ ω ω > λ − λ ω ω ( ) ( | ) ( ) ( ) ( | ) ( ) p x P p x P 21 11 1 1 12 22 2 2 Which gives a ratio test based on cost and priors: Choose 𝛽 1 if ω λ − λ ω ( | ) ( ) p x P > = 1 12 22 2 T ω λ − λ ω ( | ) ( ) p x P 2 21 11 1
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick A special loss function Cost 𝜇 𝑗𝑘 is 0 is 𝑗 = 𝑘 , 1 otherwise. Called zero-one loss funciton (duh). Which gives a ratio test: choose 𝛽 1 if ω ω ( | ) ( ) p x P > 1 2 ω ω ( | ) ( ) p x P 2 1 i.e. choose whichever class is more likely given the data. Which really means you combine likelihoods and priors, and you never separate them. That is, you just have a decision boundary on 𝒚 . That is you just discriminate based upon 𝒚 …
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick Introduction to discriminant functions Let 𝑗 � = −𝑆 ( 𝛽 _ 𝑗 | 𝑦 ) (So “max” discriminant function is min risk.) For minimum error rate (zero one loss): 𝑗 � 𝑦 = 𝑄 𝜕 𝑗 𝑦 (max discrimination is max posterior) Using Bayes rule : 𝑗 � 𝑦 ∝ 𝑞 𝑦 𝜕 𝑗 𝑄 𝜕 1 Finally and then monotonicity of ln let: 𝑗 𝑦 = ln 𝑞 𝑦 𝜕 𝑗 + ln ( 𝑄 𝜕 1 )
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick Two class discrimination Let 𝑦 = 1 𝑦 − 2 𝑦 Decide class 𝜕 1 if 𝑦 > 0 otherwise decide 𝜕 2
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick Next time… Linear discriminants applied to normal distributions.
Bayesian Decision Theory and more Introduction CS7616 Pattern Recognition – A. Bobick CS7616 Pattern Recognition – A. Bobick Remember your first assignment! Due next Tuesday, Jan 14. Find an available data set that corresponds to “modest” number of features and “small” number of classes Modest – plausible to try all or many possible subsets of features Small - maybe less than 5. 2 is ideal. 30 would be too many. Submit a one page description of the data, how we would get it within a week. (Are you making it? That’s OK)
Recommend
More recommend
