CS 4803 / 7643: Deep Learning Topics: Moving beyond supervised - PowerPoint PPT Presentation

CS 4803 / 7643: Deep Learning Topics: – Moving beyond supervised learning Zsolt Kira Georgia Tech

Administrativia • Projects! – Due April 30 th – Template online – Can use MS Word but follow the organization/rubric! • No posters/presentations (C) Zsolt Kira 2

Project Note • Important note: – Your project should include doing something beyond just downloading open-source code, fine-tuning, and showing the result – This can include: • implementation of additional approaches (even if leveraging open- source code), • a thorough analysis/investigation of some phenomena or hypothesis • theoretical analysis, or • When using external resources, provide references to anything you used in the write-up! (C) Zsolt Kira 3

Supervised Learning Supervised Learning ● ML has been focused largely on this ● Lots of other problem settings are now coming up: What if we have unlabeled data? ○ What if we have many datasets? ○ 4 What if we only have one example per (new) class? ○

But wait, there’s more! • Transfer Learning • Semi-supervised learning • One/Few-shot learning • Un/Self-Supervised Learning • Domain adaptation Setting Source Target Shift Type • Meta-Learning Semi-supervised Single Single None • Zero-shot learning labeled unlabeled • Continual / Lifelong-learning Domain Single Single Non- Adaptation labeled unlabeled semantic • Multi-modal learning Domain Multiple Unknown Non- • Multi-task learning Generalization labeled semantic Cross-Task Single Single Semantic • Active learning Transfer labeled unlabeled • … Few-Shot Single Single few- Semantic Learning labeled labeled Un/Self- Single Many labeled Both/Task Supervised unlabeled (C) Zsolt Kira 5

An Entire Class on this! • Deep Unsupervised Learning class (UC Berkeley) • Link: – https://sites.google.com/view/berkeley-cs294-158-sp20/home (C) Zsolt Kira 6

But wait, there’s more! • Transfer Learning • Semi-supervised learning • One/Few-shot learning • Un/Self-Supervised Learning • Domain adaptation • Meta-Learning • Zero-shot learning • Continual / Lifelong-learning • Multi-modal learning • Multi-task learning • Active learning • … (C) Zsolt Kira 7

What is Semi-Supervised Learning? Supervised Learning Semi-Supervised Learning 8 Slide Credit: Pieter Abbeel et al., CS294-158, UC Berkeley

What is Semi-Supervised Learning? Supervised Learning Semi-Supervised Learning 9 Slide Credit: Pieter Abbeel et al., CS294-158, UC Berkeley

Semi-Supervised Learning ● Classification: Fully Supervised ○ Training data: (image, label), predict label for new images. ● What if we have a few labeled samples and many unlabeled samples? Labeling is generally time-consuming and expensive in certain domains. ● Semi-Supervised Learning ○ Training data: Labeled data (image, label) and Unlabeled data (image) ○ Goal: Use the unlabeled data to make supervised learning better ○ Note: If we have lots of labeled data, this goal is much harder 10 Slide Credit: Pieter Abbeel et al., CS294-158, UC Berkeley

Why Semi-Supervised Learning? Slide: Thang Luong ● My take: Reality might be in-between: 11 ● Might be able to improve upon high-labeled data regime but with exponentially increasing unlabeled data (of the proper type) ● See Slide Credit: Pieter Abbeel et al., CS294-158, UC Berkeley

Agenda ■ Core concepts Confidence vs Entropy ■ Pseudo Labeling ■ Entropy minimization ■ Virtual Adversarial Training ■ Label Consistency ■ Make sure augmentations of the sample have the same class ■ Pi-Model, Temporal Ensembling, Mean Teacher ■ Regularization ■ Weight decay ■ Dropout ■ Data-Augmentation (MixUp, CutOut) ■ Unsupervised Data Augmentation (UDA), MixMatch ■ Co-Training / Self-Training / Pseudo Labeling (Noisy Student) ■ 12 Slide Credit: Pieter Abbeel et al., CS294-158, UC Berkeley

Pseudo Labeling ● Simple idea: ○ Train on labeled data ○ Make predictions on unlabeled data ○ Add confident predictions to training data ○ Can do these both end-to-end (no need to separate stages) 13 Slide Credit: Pieter Abbeel et al., CS294-158, UC Berkeley

Issue: Confidences on New Data ● Predictions on unlabeled data may be too flat (high entropy) ● Solution: Entropy minimization ● Several ways to achieve this ○ Explicit loss ○ Sharpening function (e.g. temperature scaling) 14 Image Credit: Figure modified from MixMatch paper

Label Consistency with Data Augmentation 15

Label Consistency with Data Augmentation Could be Unlabeled or Labeled 16

Label Consistency with Data Augmentation 17

Label Consistency with Data Augmentation Make sure that the logits are similar 18

More Data Augmentation -> Regularization 19

Realistic Evaluation of Semi-Supervised Learning 20

Outline ■ Realistic Evaluation of Semi-Supervised Learning pi-model ■ Temporal Ensembling ■ Mean Teacher ■ Virtual Adversarial Training ■ 21

pi-Model Temporal Ensembling for Semi-Supervised Learning 22

pi-Model Temporal Ensembling for Semi-Supervised Learning 23

Comparison 24

Comparison 25

Varying number of labels 26

Class Distribution Mismatch 27

MixMatch 28

MixMatch 29

MixMatch MixUp 30

MixMatch 31

MixMatch 32

MixMatch 33

MixMatch 34

FixMatch 35

FixMatch - Results 36

But wait, there’s more! • Transfer Learning • Semi-supervised learning • One/Few-shot learning • Un/Self-Supervised Learning • Domain adaptation • Meta-Learning • Zero-shot learning • Continual / Lifelong-learning • Multi-modal learning • Multi-task learning • Active learning • … (C) Zsolt Kira 37

Few-Shot Learning (C) Zsolt Kira 38 Slide Credit: Hugo Larochelle

Few-Shot Learning (C) Zsolt Kira 39 Slide Credit: Hugo Larochelle

Normal Approach • Do what we always do: Fine-tuning – Train classifier on base classes – Freeze features – Learn classifier weights for new classes using few amounts of labeled data (during “ query” time!) A Closer Look at Few-shot Classification, Wei-Yu Chen, Yen-Cheng Liu, (C) Zsolt Kira 40 Zsolt Kira, Yu-Chiang Frank Wang, Jia-Bin Huang

Cons of Normal Approach • The training we do on the base classes does not factor the task into account • No notion that we will be performing a bunch of N- way tests • Idea: simulate what we will see during test time (C) Zsolt Kira 41

Meta-Training Approach • Set up a set of smaller tasks during training which simulates what we will be doing during testing https://www.borealisai.com/en/blog/tutorial-2-few-shot-learning-and-meta-learning-i/ – Can optionally pre-train features on held-out base classes (not typical) • Testing stage is now the same, but with new classes (C) Zsolt Kira 42

Meta-Learning Approaches • Learning a model conditioned on support set (C) Zsolt Kira 43

More Sophisticated Meta-Learning Approaches • Learn gradient descent: – Parameter initialization and update rules – Output: • Parameter initialization • Meta-learner that decides how to update parameters • Learn just an initialization and use normal gradient descent (MAML) – Output: • Just parameter initialization! • We are using SGD (C) Dhruv Batra & Zsolt Kira 44

Meta-Learner • How to parametrize learning algorithms? • Two approaches to defining a meta-learner – Take inspiration from a known learning algorithm • kNN/kernel machine: Matching networks (Vinyals et al. 2016) • Gaussian classifier: Prototypical Networks (Snell et al. 2017) • Gradient Descent: Meta-Learner LSTM (Ravi & Larochelle, 2017) , MAML (Finn et al. 2017) – Derive it from a black box neural network • MANN (Santoro et al. 2016) • SNAIL (Mishra et al. 2018) (C) Zsolt Kira 45 Slide Credit: Hugo Larochelle

More Sophisticated Meta-Learning Approaches • Learn gradient descent: – Parameter initialization and update rules – Output: • Parameter initialization • Meta-learner that decides how to update parameters • Learn just an initialization and use normal gradient descent (MAML) – Output: • Just parameter initialization! • We are using SGD (C) Zsolt Kira 46

Meta-Learner LSTM (C) Zsolt Kira 47 Slide Credit: Hugo Larochelle

Meta-Learner LSTM (C) Zsolt Kira 48 Slide Credit: Hugo Larochelle

Meta-Learner LSTM (C) Zsolt Kira 49 Slide Credit: Hugo Larochelle

Meta-Learner LSTM (C) Zsolt Kira 50 Slide Credit: Hugo Larochelle

Model-Agnostic Meta-Learning (MAML) (C) Zsolt Kira 53 Slide Credit: Hugo Larochelle

Model-Agnostic Meta-Learning (MAML) (C) Zsolt Kira 55 Slide Credit: Sergey Levine

Model-Agnostic Meta-Learning (MAML) (C) Zsolt Kira 56 Slide Credit: Sergey Levine

Comparison (C) Zsolt Kira 57 Slide Credit: Sergey Levine