Drawing Parallels between Multi-label Classification and - PowerPoint PPT Presentation

Drawing Parallels between Multi-label Classification and Multi-target Regression Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, and Ioannis Vlahavas Machine Learning and Knowledge Discovery (MLKD) group Department of Informatics, Aristotle University of Thessaloniki, Greece International Workshop on Multi-Target Prediction Drawing Parallels between Multi-label Classification and Multi-target Regression Nancy, France, September 15th, 2014 Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, and Ioannis Vlahavas

Multi-label Classification & Multi-target Regression • Two instances of multi-target prediction Multi-Label Classification (MLC) … … X 1 X 2 X n Y 1 Y 2 Y m … … 0.12 1 12 0 1 1 training … … 2.34 9 -5 0 1 0 examples … … 1.22 3 40 1 0 0 … … 2.18 2 8 ? ? ? unknown instances … … 1.76 7 23 ? ? ? m binary targets n inputs 2 International Workshop on Multi-Target Prediction Drawing Parallels between Multi-label Classification and Multi-target Regression Nancy, France, September 15th, 2014 Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, and Ioannis Vlahavas

Multi-label Classification & Multi-target Regression • Two instances of multi-target prediction Multi-Target (multivariate) Regression (MTR) … … X 1 X 2 X n Y 1 Y 2 Y m … … 0.12 1 12 0.14 10 -1.3 training … … 2.34 9 -5 4.15 12 -2.0 examples … … 1.22 3 40 1.01 28 -5.3 … … 2.18 2 8 ? ? ? unknown instances … … 1.76 7 23 ? ? ? m continuous targets n inputs 3 International Workshop on Multi-Target Prediction Drawing Parallels between Multi-label Classification and Multi-target Regression Nancy, France, September 15th, 2014 Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, and Ioannis Vlahavas

MLC and MTR Applications • MLC • Multimedia annotation/retrieval • Text categorization • Gene function prediction • …many more • MTR • Ecological modeling (e.g. water quality prediction) • Price prediction (stocks, airline tickets, etc.) • Power (solar/wind) generation forecasting • …and many recent Kaggle competitions 4 International Workshop on Multi-Target Prediction Drawing Parallels between Multi-label Classification and Multi-target Regression Nancy, France, September 15th, 2014 Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, and Ioannis Vlahavas

Motivation • Similar problems • Same baseline approach (an independent model for each target) • Shared challenges: • Scaling to large numbers of targets / Exploiting target dependencies • MLC is a more popular research topic • At least 4 MLC papers in ECML/PKDD 2014 (with MLC in title) • A multitude of new MLC methods • Questions: • Can one field benefit from the other? • Are there successful MLC methods that can be used in MTR? transfer of ideas 1 multi-label multi-target classification regression 5 International Workshop on Multi-Target Prediction Drawing Parallels between Multi-label Classification and Multi-target Regression Nancy, France, September 15th, 2014 Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, and Ioannis Vlahavas

Categorization of MLC Methods and Applicability on MTR • Problem transformation methods • Modelling single-labels: multiple binary classification problems • E.g. Binary Relevance, Multi-label Stacking 2,3 , Classifier Chains 4,5 • Almost directly applicable! • Modelling pairs: one-versus-one decomposition paradigm • E.g. Calibrated Label Ranking 6 • Approach not applicable! • Modelling sets: multi-class problems where distinct label subsets represent different class values • E.g. Label Powerset, RAkEL 7 , Pruned Sets 8 • Approach seems not applicable! • Algorithm adaptation methods • Applicability depends on ability to handle regression data Easy for decision-tree-based methods (e.g. PCT 9 framework) • 6 International Workshop on Multi-Target Prediction Drawing Parallels between Multi-label Classification and Multi-target Regression Nancy, France, September 15th, 2014 Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, and Ioannis Vlahavas

Single-target Decomposition Techniques 𝐶𝑆 ℎ 𝑗 𝒚 → 𝑧 𝑗 , 𝑗 = 1, . . . , 𝑛 • The simplest one is Binary Relevance: ℎ 𝒚 → 𝒛 … … X 1 X 2 X n Y 1 Y 2 Y m … … 0.12 1 12 0 1 1 … … 2.34 9 -5 0 1 0 … … 1.22 3 40 1 0 0 … … 2.18 2 8 ? ? ? … … 1.76 7 23 ? ? ? 7 International Workshop on Multi-Target Prediction Drawing Parallels between Multi-label Classification and Multi-target Regression Nancy, France, September 15th, 2014 Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, and Ioannis Vlahavas

Single-target Decomposition Techniques 𝐶𝑆 ℎ 𝑗 𝒚 → 𝑧 𝑗 , 𝑗 = 1, . . . , 𝑛 • The simplest one is Binary Relevance: ℎ 𝒚 → 𝒛 ℎ 1 … … X 1 X 2 X n Y 1 Y 2 Y m … … 0.12 1 12 0 1 1 … … 2.34 9 -5 0 1 0 … … 1.22 3 40 1 0 0 … … 2.18 2 8 ? ? ? … … 1.76 7 23 ? ? ? 8 International Workshop on Multi-Target Prediction Drawing Parallels between Multi-label Classification and Multi-target Regression Nancy, France, September 15th, 2014 Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, and Ioannis Vlahavas

Single-target Decomposition Techniques 𝐶𝑆 ℎ 𝑗 𝒚 → 𝑧 𝑗 , 𝑗 = 1, . . . , 𝑛 • The simplest one is Binary Relevance: ℎ 𝒚 → 𝒛 ℎ 2 … … X 1 X 2 X n Y 1 Y 2 Y m … … 0.12 1 12 0 1 1 … … 2.34 9 -5 0 1 0 … … 1.22 3 40 1 0 0 … … 2.18 2 8 ? ? ? … … 1.76 7 23 ? ? ? 9 International Workshop on Multi-Target Prediction Drawing Parallels between Multi-label Classification and Multi-target Regression Nancy, France, September 15th, 2014 Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, and Ioannis Vlahavas

Single-target Decomposition Techniques 𝐶𝑆 ℎ 𝑗 𝒚 → 𝑧 𝑗 , 𝑗 = 1, . . . , 𝑛 • The simplest one is Binary Relevance: ℎ 𝒚 → 𝒛 ℎ 𝑛 … … X 1 X 2 X n Y 1 Y 2 Y m … … 0.12 1 12 0 1 1 … … 2.34 9 -5 0 1 0 … … 1.22 3 40 1 0 0 … … 2.18 2 8 ? ? ? … … 1.76 7 23 ? ? ? 10 International Workshop on Multi-Target Prediction Drawing Parallels between Multi-label Classification and Multi-target Regression Nancy, France, September 15th, 2014 Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, and Ioannis Vlahavas

Single-target Decomposition Techniques 𝐶𝑆 ℎ 𝑗 𝒚 → 𝑧 𝑗 , 𝑗 = 1, . . . , 𝑛 • The simplest one is Binary Relevance: ℎ 𝒚 → 𝒛 • Better ones (considering label dependencies): • Classifier Chains • Stacking … … X 1 X 2 X n Y 1 Y 2 Y m … … 0.12 1 12 0 1 1 … … 2.34 9 -5 0 1 0 … … 1.22 3 40 1 0 0 … … 2.18 2 8 ? ? ? … … 1.76 7 23 ? ? ? 11 International Workshop on Multi-Target Prediction Drawing Parallels between Multi-label Classification and Multi-target Regression Nancy, France, September 15th, 2014 Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, and Ioannis Vlahavas

Single-target Decomposition Techniques 𝐶𝑆 ℎ 𝑗 𝒚 → 𝑧 𝑗 , 𝑗 = 1, . . . , 𝑛 • The simplest one is Binary Relevance: ℎ 𝒚 → 𝒛 • Better ones (considering label dependencies): 𝐷𝐷 ℎ 1 (𝒚) → 𝑧 1 , ℎ 2 (𝒚𝑧 1 ) → 𝑧 2 , … , ℎ 𝑛 (𝒚𝑧 1 … 𝑧 𝑛−1 ) → 𝑧 𝑛 • Classifier Chains: ℎ 𝒚 → 𝒛 • Stacking ℎ 1 … … X 1 X 2 X n Y 1 Y 2 Y m … … 0.12 1 12 0 1 1 … … 2.34 9 -5 0 1 0 … … 1.22 3 40 1 0 0 … … 2.18 2 8 ? ? ? … … 1.76 7 23 ? ? ? 12 International Workshop on Multi-Target Prediction Drawing Parallels between Multi-label Classification and Multi-target Regression Nancy, France, September 15th, 2014 Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, and Ioannis Vlahavas

Single-target Decomposition Techniques 𝐶𝑆 ℎ 𝑗 𝒚 → 𝑧 𝑗 , 𝑗 = 1, . . . , 𝑛 • The simplest one is Binary Relevance: ℎ 𝒚 → 𝒛 • Better ones (considering label dependencies): 𝐷𝐷 ℎ 1 (𝒚) → 𝑧 1 , ℎ 2 (𝒚𝑧 1 ) → 𝑧 2 , … , ℎ 𝑛 (𝒚𝑧 1 … 𝑧 𝑛−1 ) → 𝑧 𝑛 • Classifier Chains: ℎ 𝒚 → 𝒛 • Stacking ℎ 2 … … X 1 X 2 X n Y 1 Y 2 Y m … … 0.12 1 12 0 1 1 … … 2.34 9 -5 0 1 0 … … 1.22 3 40 1 0 0 … … 2.18 2 8 ? ? ? … … 1.76 7 23 ? ? ? 13 International Workshop on Multi-Target Prediction Drawing Parallels between Multi-label Classification and Multi-target Regression Nancy, France, September 15th, 2014 Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, and Ioannis Vlahavas