Imprecise Gaussian Discriminant Classification 11th International - PowerPoint PPT Presentation

Imprecise Gaussian Discriminant Classification 11th International Symposium on Imprecise Probabilities: Theories and Applications CARRANZA-ALARCON Yonatan-Carlos Ph.D. Candidate in Computer Science DESTERCKE Sébastien Ph.D Director 03 Jul 2018 to 09 Jul 2019

Classification Imprecise Classification Evaluation Conclusions and Perspectives Références Overview Classification ● ❍ Motivation ❍ Precise Decision ❍ Discriminant Analysis Imprecise Classification ● ❍ Imprecise Gaussian discriminant analysis ❍ Cautious Decision Evaluation ● ❍ Cautious accuracy measure and Datasets ❍ Experimental results Conclusions and Perspectives ● 3 11th International Symposium on Imprecise Probabilities Theories and Applications

Classification Imprecise Classification Evaluation Conclusions and Perspectives Références Classification - Outline (Example) i = 0 ⊆ R p × K ☞ Data training D = { x i , y i } N → → Objective Given training data D = { x i , y i } N i = 0 : ➊ learning a classification rule : ϕ : X → K . ➋ predicting new instances � ϕ ( x ∗ ) 4 11th International Symposium on Imprecise Probabilities Theories and Applications

Classification Imprecise Classification Evaluation Conclusions and Perspectives Références Motivation Precise Decision Discriminant Analysis Motivation What is the bigger problem in (precise) Classification? ⋆ � Group B ⋆ ⋆ Group A ⋆ • Precise models can produce ⋆ ⋆ many mistakes for hard to predict ⋆ • ? P (ˆ y ∗ | X = x ∗ ) ≈ 0 . 5 unlabeled instances. ⋆ � � � � � � ⋆ � Group B • One way to recognize such ⋆ ⋆ Group A ⋆ ⋆ ⋆ instances and avoid making such ⋆ mistakes too often → Making a y ∗ ⊆ { A, B } • ? ˆ ⋆ � cautious decision. � � � � � 5 11th International Symposium on Imprecise Probabilities Theories and Applications

Classification Imprecise Classification Evaluation Conclusions and Perspectives Références Motivation Precise Decision Discriminant Analysis Precise Classification Step ➊ Learning the conditional probability distribution P Y | x ∗ . Step ➋ Predicting the “optimal” label amongst K = { m 1 ,..., m K } , under L 0 / 1 loss function, for a new instance x ∗ : ⇒ P ( y = m i K | x ∗ ) > .... > P ( y = m i 1 | x ∗ ) m i K ≻ m i K − 1 ≻ .... ≻ m i 1 ⇐ ☞ Pick out the most preferable label m i K ⇒ maximal probability plausible P ( y = m i K | x ∗ ) ⇐ 6 11th International Symposium on Imprecise Probabilities Theories and Applications

Classification Imprecise Classification Evaluation Conclusions and Perspectives Références Motivation Precise Decision Discriminant Analysis (Precise) Gaussian Discriminant Analysis Applying Baye’s rules to P ( Y = m a | X = x ∗ ) : P ( X = x ∗ | y = m k ) P ( y = m k ) P ( y = m k | X = x ∗ ) = � m l ∈ K P ( X = x ∗ | y = m l ) P ( y = m l ) Normality P X | Y = m k ∼ N ( µ m k , Σ m k ) and precise marginal π m k : = P Y = m k . ⋆ ⋆ � Σ m a ⋆ � Group B • ⋆ � ⋆ Group A µ m a ⋆ ⋆ ⋆ � � � � • � µ m b � Precise estimations � Σ m b � � 7 11th International Symposium on Imprecise Probabilities Theories and Applications

Classification Imprecise Classification Evaluation Conclusions and Perspectives Références Overview Classification ● ❍ Motivation ❍ Precise Decision ❍ Discriminant Analysis Imprecise Classification ● ❍ Imprecise Gaussian discriminant analysis ❍ Cautious Decision Evaluation ● ❍ Cautious accuracy measure and Datasets ❍ Experimental results Conclusions and Perspectives ● 8 11th International Symposium on Imprecise Probabilities Theories and Applications

Classification Imprecise Classification Evaluation Conclusions and Perspectives Références Imprecise Gaussian discriminant analysis Cautious Decision Imprecise Gaussian Discriminant Analysis (IGDA) Objective : Making imprecise the parameter mean µ k of each Gaussian distribution family G k : = P X | Y = m k ∼ N ( µ k , � Σ m k ) Proposition : Using a set of posterior distribution P ([4, eq 17]). ⋆ ⋆ ⋆ � µ m a ⋆ � Group B ⋆ � Σ m a • ⋆ ⋆ � Group B ⋆ Group A • ⋆ ⋆ ⋆ Group A � µ m a ⋆ Set-box posterior � ⋆ estimators � µ ∗ ⋆ ⋆ ⋆ � � � � � � � • µ m b � � µ m b � � • Precise estimations � Σ m b � � � Precise estimations � � 9 11th International Symposium on Imprecise Probabilities Theories and Applications

Classification Imprecise Classification Evaluation Conclusions and Perspectives Références Imprecise Gaussian discriminant analysis Cautious Decision Decision Making in Imprecise Probabilities Definition (Partial Ordering by Maximality [1]) Under L 0 / 1 loss function and let P Y | x ∗ a set of probabilities then m a is preferred to m b if and only if P Y | x ∗ ∈ P Y | x ∗ P ( Y = m a | x ∗ ) − P ( Y = m b | x ∗ ) > 0 (1) inf ☞ This definition give us a partial order ≻ M ☞ The maximal element of partial order is the cautious decision : Y M = { m a ∈ K | � ∃ m b : m a ≻ M m b } 10 11th International Symposium on Imprecise Probabilities Theories and Applications

Classification Imprecise Classification Evaluation Conclusions and Perspectives Références Imprecise Gaussian discriminant analysis Cautious Decision Decision Making in IGDA ● Using the Bayes’ rule on the criterion of maximality : P Y | x ∗ ∈ P Y | x ∗ P ( Y = m a | x ∗ ) − P ( Y = m b | x ∗ ) > 0 (2) inf ● We can reduce it to solving two different optimization problems : − 1 2 ( x ∗ − µ m b ) T � Σ − 1 m b ( x ∗ − µ m b ) (BQP) P ( x ∗ | y = m b ) ⇐ sup ⇒ µ m b = argmax P ∈ P X | m b µ m b ∈ P µ m b − 1 2 ( x ∗ − µ m a ) T � Σ − 1 m a ( x ∗ − µ m a ) (NBQP) P ( x ∗ | y = m a ) ⇐ inf ⇒ µ m a = argmin P ∈ P X | m a µ m a ∈ P µ m a ☞ First problem box-constrained quadratic problem (BQP). ☞ Second problem non-convex BQP → solved through Branch and Bound method. 11 11th International Symposium on Imprecise Probabilities Theories and Applications

Classification Imprecise Classification Evaluation Conclusions and Perspectives Références Imprecise Gaussian discriminant analysis Cautious Decision Decision Making in IGDA ● Using the Bayes’ rule on the criterion of maximality : P Y | x ∗ ∈ P Y | x ∗ P ( Y = m a | x ∗ ) − P ( Y = m b | x ∗ ) > 0 (2) inf ● We can reduce it to solving two different optimization problems : − 1 2 ( x ∗ − µ m b ) T � Σ − 1 m b ( x ∗ − µ m b ) (BQP) P ( x ∗ | y = m b ) ⇐ sup ⇒ µ m b = argmax P ∈ P X | m b µ m b ∈ P µ m b − 1 2 ( x ∗ − µ m a ) T � Σ − 1 m a ( x ∗ − µ m a ) (NBQP) P ( x ∗ | y = m a ) ⇐ inf ⇒ µ m a = argmin P ∈ P X | m a µ m a ∈ P µ m a ☞ First problem box-constrained quadratic problem (BQP). ☞ Second problem non-convex BQP → solved through Branch and Bound method. 11 11th International Symposium on Imprecise Probabilities Theories and Applications

Classification Imprecise Classification Evaluation Conclusions and Perspectives Références Imprecise Gaussian discriminant analysis Cautious Decision Cautious decision zone (example with 2 class) ⋆ { ⋆ } then: m a ≻ M m b ⋆ � Group B ⋆ ⋆ ⋆ Group A • ⋆ New observation • ⋆ µ a � ⋆ � Lower/Upper estimations ⋆ � � � µ b � • � • � Set-box posterior estimators � µ ∗ � � ☞ Note the non-linearity boundary decision!! 12 11th International Symposium on Imprecise Probabilities Theories and Applications

Classification Imprecise Classification Evaluation Conclusions and Perspectives Références Overview Classification ● ❍ Motivation ❍ Precise Decision ❍ Discriminant Analysis Imprecise Classification ● ❍ Imprecise Gaussian discriminant analysis ❍ Cautious Decision Evaluation ● ❍ Cautious accuracy measure and Datasets ❍ Experimental results Conclusions and Perspectives ● 13 11th International Symposium on Imprecise Probabilities Theories and Applications

Classification Imprecise Classification Evaluation Conclusions and Perspectives Références Cautious accuracy measure and Datasets Experimental results Datasets and experimental setting ☞ 9 data sets issued from UCI repository [2]. ☞ 10 × 10-fold cross-validation procedure. ☞ Utility-discounted accuracy measure proposed to Zaffalon et al on [3]. # name # instances # features # labels iris 150 4 3 a � 0 if y ∉ � wine 178 13 3 Y M b u ( y , � Y M ) = forest 198 27 4 c Y M | − 1 − α else α seeds 210 7 3 Y M | 2 | � | � d e dermatology 385 34 6 Goal : reward cautiousness to some vehicle 846 18 4 f vowel 990 10 11 g degree α : h wine-quality 1599 11 6 ➠ α = 1 : cautiousness = randonness i wall-following 5456 24 4 ➠ α → ∞ : best classifier vacuous T ABLE – Data sets used in the experiments 14 11th International Symposium on Imprecise Probabilities Theories and Applications