An Empirical Study on Lazy Multilabel Classification Algorithms - - PowerPoint PPT Presentation

▶
an empirical study on lazy multilabel
SMART_READER_LITE
LIVE PREVIEW

An Empirical Study on Lazy Multilabel Classification Algorithms - - PowerPoint PPT Presentation

Oct 25, 2022 22 likes •258 views

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions An Empirical Study on Lazy Multilabel Classification Algorithms Eleftherios Spyromitros, Grigorios Tsoumakas and Ioannis Vlahavas Machine Learning

slide-1
SLIDE 1

An Empirical Study on Lazy Multilabel Classification Algorithms

Eleftherios Spyromitros, Grigorios Tsoumakas and Ioannis Vlahavas

Machine Learning & Knowledge Discovery Group Department of Informatics Aristotle University of Thessaloniki Greece

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions

slide-2
SLIDE 2

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Multilabel Classification Multilabel Classification Methods

What is Multilabel Classification?

  • Single-label Classification
  • Results are associated with a single label from a set of

disjoint labels

  • If , binary classification
  • If , multi-class classification
  • Multilabel Classification
  • Results are associated with a set of labels

 L

| | 2 L 

| | 2 L 

Y L 

slide-3
SLIDE 3

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Multilabel Classification Multilabel Classification Methods

Data With Multilabel Nature

  • Traditional
  • Text Classification
  • A web article concerning the Antikythera Mechanism Research Project can

be categorized into both categorys { Science_Technology, History_Culture }

  • Medical Diagnosis
  • Multiple diseases for a patient { Obesity, Hypertension}
  • Modern
  • Gene Function Classification
  • A gene usually has multiple functions { Protein Synthesis, Cellular

Biogenesis, Cellular Transport}

  • Classification of Music into Emotions
  • A song can make you feel { Sad_Lonely, Quiet_Still}
  • Semantic Scene Analysis
  • { Mountain, Trees, Lake }
slide-4
SLIDE 4

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Multilabel Classification Multilabel Classification Methods

Types of Multilabel Classification Methods

  • Problem transformation methods
  • They transform the learning problem into one (LP) or more (BR)

single-label classification or label ranking problems

  • Algorithm independent
  • Algorithm adaptation methods
  • They extend specific algorithms to handle multi-label data
  • SVM, decision tree, neural network, lazy, Bayesian, boosting
slide-5
SLIDE 5

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Multilabel Classification Multilabel Classification Methods

The Binary Relevance (BR) Method

  • How it works
  • Learns one binary classifier for each

different label

  • The original dataset is transformed into datasets
  • contains all examples of labeled as if they are

associated with and as otherwise

  • Criticism
  • Label correlations are not considered

: { , } h X

    L  

| | L

D D

D

 

slide-6
SLIDE 6

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Multilabel Classification Multilabel Classification Methods

The Label Powerset (LP) Method

  • How it works
  • Considers its different subset of as a single label
  • It learns one single-label classifier
  • Criticism
  • Large number of label subsets ( )
  • Most of these are associated with very few examples

L

: ( ) h X P L 

| |

2 L

slide-7
SLIDE 7

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions The BRkNN Algorithm The Problem of BRkNN Extensions of BRkNN MLkNN and LPkNN

The BRkNN Algorithm

  • Origin
  • Equivalent to using the BR method in conjunction with the

kNN algorithm

  • Refinement
  • times faster than BR + kNN in prediction
  • Avoids the redundant calculations of k nearest neighbors

in each one of the transformed datasets

  • A single k nearest neighbors search is followed by

independent predictions for each label

  • Benefit
  • Applies better in domains with large number of labels and

examples, requiring low response times

| | L

D

slide-8
SLIDE 8

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions The BRkNN Algorithm The Problem of BRkNN Extensions of BRkNN MLkNN and LPkNN

How it works

  • Confidence scores
  • BrKNN is based on the calculation of confidence scores for

each label

  • Confidence is obtained considering the percentage of the k

nearest neighbors that include each label

  • A label is included in the label-set when the percentage is

higher than or equal to 50% L  

c

slide-9
SLIDE 9

0% 5% 10% 15% 20% 25% 30% 35% 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 Percenage of instances, where the enpty set is output Nearest Neighbors scene yeast emotions

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions The BRkNN Algorithm The Problem of BRkNN Extensions of BRkNN MLkNN and LPkNN

Independent Predictions…

  • The weakness
  • The empty set is a possible overall output
  • Arises when none of the labels has a confidence higher than

50%

  • The reason
  • Independent predictions for each label, a general

disadvantage of the BR method

  • Is this common in BrkNN?
slide-10
SLIDE 10

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions The BRkNN Algorithm The Problem of BRkNN Extensions of BRkNN MLkNN and LPkNN

The Proposed Extensions

  • Trying to dissolve the aforementioned problem
  • BRkNN-a
  • Checks if BRkNN outputs the empty set
  • In that case outputs the label with the highest confidence
  • BRkNN-b
  • 1st step: Calculates the average size of the label sets of the k

nearest neighbors ( )

  • 2nd step: outputs the (nearest integer of s) labels with the

highest confidence

s

1

1 | |

k j j

s Y k

 

[ ] s

slide-11
SLIDE 11

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions The BRkNN Algorithm The Problem of BRkNN Extensions of BRkNN MLkNN and LPkNN

The MLkNN and LPkNN Algorithms

  • Two more lazy multi-label classification methods
  • LPkNN
  • The pairing of LP problem transformation method with

the kNN algorithm

  • A little discussed in the past
  • MLkNN
  • An adaptation of kNN for multi-label data
  • Main difference with BRkNN: prior and posterior

probabilities estimated from the training set

  • Extended with an option for min-max normalization
slide-12
SLIDE 12

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Evaluation Measures Datasets Evaluation Methodology

Evaluation Measures

  • Example-based
  • Calculate the difference between the actual and predicted label

sets for each example

  • Average the results over all examples of the test set
  • Label-based
  • Calculate a binary evaluation measure separately for each label
  • Micro/Macro averaging operations over all labels
slide-13
SLIDE 13

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Evaluation Measures Datasets Evaluation Methodology

Example Based Measures

  • Notation
  • Let be a multi-label example,
  • Let be a multi-label classifier
  • Let be the set of labels predicted by h for
  • Hamming Loss
  • , where is the symmetric difference of two sets
  • Classification Accuracy or Subset Accuracy
  • 1, if
  • 0, if
  • IR-inspired measures
  • Precision , Recall , F-measure

2| | | | | | Y Z Z Y 

( , ) x Y

h

( ) Z h x 

( , ) x Y

| | | | Y Z L

Y Z 

Y Z 

| | | | Y Z Z | | | | Y Z Y 2| | | | | | Y Z Z Y 

slide-14
SLIDE 14

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Evaluation Measures Datasets Evaluation Methodology

Label Based Measures

  • Any binary evaluation measure can be used
  • Accuracy, area under ROC curve, precision, recall, etc
  • Operations for averaging across all labels
  • Macro-averaging
  • Micro-averaging

| | 1

1 ( , , , ) | |

L macro

M M tp fp tn fn L

    

| | | | | | | | 1 1 1 1

, , ,

L L L L micro

M M tp fp tn fn

           

      

   

slide-15
SLIDE 15

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Evaluation Measures Datasets Evaluation Methodology

Datasets

  • Datasets
  • Scene: semantic indexing of still images
  • Emotions: classification of songs into 6 classes of emotion
  • Yeast: gene function classification
  • Multi-label Statistics
  • Distinct Subsets is the number of different label sets
  • Label Cardinality is the average number of labels per example
  • Label Density is equal to Label Cardinality divided by

| | L

Dataset Examples Attributes Numeric Discrete Labels Distinct Subsets Label Cardinality Label Density Scene 2,407 294 6 15 1,074 0.179 Emotions 593 72 6 27 1,868 0.311 Yeast 2,417 103 14 198 4,327 0.302

slide-16
SLIDE 16

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Evaluation Measures Datasets Evaluation Methodology

Evaluation Methodology

  • Multi-label algorithms evaluated
  • BRkNN
  • BRkNN-a / BRkNN-b
  • MLkNN
  • LPkNN
  • Varying number of nearest neighbors
  • k ranged from 1 to 30
  • Distance function: Normalized Euclidean
  • Evaluation
  • Example-based: hamming loss, accuracy, F-measure, subset accuracy
  • Label-based : micro and macro version of F-measure
  • 10-fold cross-validation
slide-17
SLIDE 17

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Do the Proposed Extensions Improve BRkNN? Comparison of BRkNN, LPkNN and MLkNN

Do the Proposed Extensions Improve BRkNN?

  • BRkNN against its extensions BRkNN-a and BRknn-b
  • Average performance across all 30 values of k

metric scene base ext-a ext-b emotions base ext-a ext-b yeast base ext-a ext-b Hamming loss

0,0950 0,0938 0,0941 0,1976 0,1982 0,2175 0,1974 0,1975 0,2082

Accuracy

0,6256 0,7226 0,7218 0,5215 0,5441 0,5430 0,5062 0,5080 0,5346

F-measure

0,6495 0,7539 0,7538 0,6275 0,6576 0,6590 0,5777 0,5795 0,6652

Subset accuracy

0,6281 0,7251 0,7230 0,2895 0,2971 0,2759 0,1958 0,1959 0,1766

micro F-measure

0,6386 0,7392 0,7381 0,6499 0,6577 0,6509 0,6374 0.6380 0.6567

macro F-measure

0,5993 0,6889 0,6886 0,6224 0,6303 0,6294 0.3926 0.3931 0.4261

#wins (#better)

6 (6) 0 (6) 1 4 (5) 1 (4) 1 1 (5) 4 (4)

slide-18
SLIDE 18

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Do the Proposed Extensions Improve BRkNN? Comparison of BRkNN, LPkNN and MLkNN

Do the Proposed Extensions Improve BRkNN?

  • Remarks
  • Both extensions outperform the base algorithm in more than

half of the metrics in all datasets

  • Performance pattern correlates with dataset cardinality
  • BRkNN-a dominates in scene and emotions (1.074, 1.868)
  • Increased probability for BRkNN to output the empty set
  • BRkNN-b dominates in yeast (4.237)
  • A mechanism to predict the number of labels
slide-19
SLIDE 19

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Do the Proposed Extensions Improve BRkNN? Comparison of BRkNN, LPkNN and MLkNN

Comparison of BRkNN, LPkNN and MLkNN

  • Best extension of BRkNN against LPknn and MLknn
  • Average performance across all 30 values of k

Metric scene ext-a LPkNN MLkNN emotions ext-a LPkNN MLkNN yeast ext-b LPkNN MLkNN Hamming loss

0,0938 0,0955 0,0884 0,1982 0,2094 0,2003 0,2082 0,2143 0,1950

Accuracy

0,7226 0,7181 0,6720 0,5441 0,5600 0,5233 0,5346 0,5280 0,5105

F-measure

0,7392 0,7343 0,6944 0,6576 0,6662 0,6352 0,6652 0,6375 0,5823

Subset accuracy

0,6889 0,6854 0,6272 0,2971 0,3287 0,2780 0,1766 0,2452 0,1780

micro F-measure

0,7296 0,7249 0,7316 0,6577 0,6649 0,6509 0,6567 0,6415 0,6422

macro F-measure

0,7363 0,7323 0,7341 0,6303 0,6505 0,6110 0,4261 0,4322 0,3701

#wins

4 2 1 5 3 2 1

slide-20
SLIDE 20

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Do the Proposed Extensions Improve BRkNN? Comparison of BRkNN, LPkNN and MLkNN

Comparison of BRkNN, LPkNN and MLkNN

  • Remarks
  • BRkNN-a dominates in scene
  • LPkNN dominates in emotions
  • BRkNN-b performs slightly better in yeast
  • Possible correlation between LPkNN performance and label density
slide-21
SLIDE 21

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Summary and Future Work Resources The End

Summary and Future Work

  • Use of kNN for multi-label classification
  • BRkNN an efficient implementation of BR plus kNN
  • Extensions that enhance BRkNN’s performance
  • Additional comparative experiments with LPkNN and MLkNN
  • Main contribution
  • Which method is most suitable for a dataset depending on

certain dataset characteristics.

  • Future work
  • Additional lazy multi-label classification approaches
  • Experiments with additional multi-label datasets
slide-22
SLIDE 22

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Summary and Future Work Resources The End

http://mlkd.csd.auth.gr/multilabel.html

  • The MUlti-LAbel classificatioN (MULAN)library
  • Open source software for multi-label classification
  • Several problem transformation and algorithm adaptation methods
  • Example/label/ranking based measures
  • Multi-label statistics
  • Built on top of Weka
  • Also hosted by Sourceforge (integrated with SVN)
  • Multi-label classification datasets (.arff format)
  • delicious, emotions, genbase, mediamill, rcv1v2, scene, tmc2007, yeast
  • Active multi-label classification bibliography
slide-23
SLIDE 23

Eleftherios Spyromitros, Griogorios Tsoumakas and Ioannis Vlahavas An Empirical Study on Lazy Multilabel Classification Algorithms

Introduction Lazy Multilabel Algorithms Experimental Setup Experimental Results Conclusions Summary and Future Work Resources The End

End of Presentation