Semi-Supervised Learning Jia-Bin Huang Virginia Tech Spring 2019 - PowerPoint PPT Presentation

Semi-Supervised Learning Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / CS-5824

Administrative

Advanced Topics • Semi-supervised learning • Ensemble learning • Generative models • Sequence prediction models • Deep reinforcement learning

Semi-supervised Learning Problem Formulation • Labeled data 𝑦 1 , 𝑧 1 , 𝑦 2 , 𝑧 2 , ⋯ , 𝑦 𝑛 𝑚 , 𝑧 𝑛 𝑚 𝑇 𝑚 = • Unlabeled data 𝑇 𝑣 = 𝑦 1 , 𝑦 2 , ⋯ , 𝑦 𝑛 𝑣 • Goal: Learn a hypothesis ℎ 𝜄 (e.g., a classifier) that has small error

Deep Semi-supervised Learning

Semi-supervised Learning • Motivation • Problem formulation • Consistency regularization • Entropy-based method

Stochastic Perturbations/ Π -Model • Realistic perturbations 𝑦 → ො 𝑦 of data points 𝑦 ∈ 𝐸 𝑉𝑀 should not significantly change the output of h 𝜄 (𝑦)

Temporal Ensembling

Mean Teacher

Virtual Adversarial Training

Semi-supervised Learning • Motivation • Problem formulation • Consistency regularization • Entropy-based method

Entropy minimization • Encourages more confident predictions on unlabeled data. • EntMin • Pseudo-labeling • Add confidently predicted samples into the training set

Comparison

Varying number of labels

Class mismatch in Labeled/Unlabeled datasets hurts the performance

Lessons • Standardized architecture + equal budget for tuning hyperparameters • Unlabeled data from a different class distribution not that useful • Most methods don’t work well in the very low labeled -data regime • Transferring Pre-Trained Imagenet produces lower error rate • Conclusions based on small datasets though

Ensemble methods • Bagging • Gradient boosting • AdaBoosting Following slides are from Alex Ihler