Semi-Supervised Learning Jia-Bin Huang Virginia Tech Spring 2019 - - PowerPoint PPT Presentation

Semi-Supervised Learning

Jia-Bin Huang Virginia Tech

Spring 2019

ECE-5424G / CS-5824

Administrative

• HW 4 due April 10

Recommender Systems

• Motivation
• Problem formulation
• Content-based recommendations
• Collaborative filtering
• Mean normalization

Problem motivation

Movie Alice (1) Bob (2) Carol (3) Dave (4) 𝑦1 (romance) 𝑦2 (action) Love at last

5 5 0.9

Romance forever

5 ? ? 1.0 0.01

Cute puppies

• f love

? 4 ? 0.99

Nonstop car chases

5 4 0.1 1.0

Swords vs. karate

5 ? 0.9

Problem motivation

Movie Alice (1) Bob (2) Carol (3) Dave (4) 𝑦1 (romance) 𝑦2 (action) Love at last

5 5 ? ?

Romance forever

5 ? ? ? ?

Cute puppies

• f love

? 4 ? ? ?

Nonstop car chases

5 4 ? ?

Swords vs. karate

5 ? ? ?

𝜄 1 = 5 𝜄 2 = 5 𝜄 3 = 5 𝜄 4 = 5 𝑦 1 = ? ? ?

Optimization algorithm

• Given 𝜄 1 , 𝜄 2 , ⋯ , 𝜄 𝑜𝑣 , to learn 𝑦(𝑗):

min

𝑦(𝑗)

1 2 ෍

𝑘:𝑠 𝑗,𝑘 =1

(𝜄 𝑘 )⊤ 𝑦 𝑗 − 𝑧 𝑗,𝑘

2 + 𝜇

2 ෍

𝑙=1 𝑜

𝑦𝑙

(𝑗) 2

• Given 𝜄 1 , 𝜄 2 , ⋯ , 𝜄 𝑜𝑣 , to learn 𝑦(1), 𝑦(2), ⋯ , 𝑦(𝑜𝑛):

min

𝑦(1),𝑦(2),⋯,𝑦(𝑜𝑛)

1 2 ෍

𝑗=1 𝑜𝑛

෍

𝑘:𝑠 𝑗,𝑘 =1

(𝜄 𝑘 )⊤ 𝑦 𝑗 − 𝑧 𝑗,𝑘

2 + 𝜇

2 ෍

𝑗=1 𝑜𝑛

෍

𝑙=1 𝑜

𝑦𝑙

(𝑗) 2

Collaborative filtering

• Given 𝑦 1 , 𝑦 2 , ⋯ , 𝑦 𝑜𝑛 (and movie ratings),

Can estimate 𝜄 1 , 𝜄 2 , ⋯ , 𝜄 𝑜𝑣

• Given 𝜄 1 , 𝜄 2 , ⋯ , 𝜄 𝑜𝑣

Can estimate 𝑦 1 , 𝑦 2 , ⋯ , 𝑦 𝑜𝑛

Collaborative filtering optimization

• bjective
• Given 𝑦 1 , 𝑦 2 , ⋯ , 𝑦 𝑜𝑛 , estimate 𝜄 1 , 𝜄 2 , ⋯ , 𝜄 𝑜𝑣

min

𝜄 1 ,𝜄 2 ,⋯,𝜄 𝑜𝑣

1 2 ෍

𝑘=1 𝑜𝑣

෍

𝑗:𝑠 𝑗,𝑘 =1

(𝜄 𝑘 )⊤ 𝑦 𝑗 − 𝑧 𝑗,𝑘

2 + 𝜇

2 ෍

𝑘=1 𝑜𝑣

෍

𝑙=1 𝑜

𝜄𝑙

𝑘 2

• Given 𝜄 1 , 𝜄 2 , ⋯ , 𝜄 𝑜𝑣 , estimate 𝑦 1 , 𝑦 2 , ⋯ , 𝑦 𝑜𝑛

min

𝑦(1),𝑦(2),⋯,𝑦(𝑜𝑛)

1 2 ෍

𝑗=1 𝑜𝑛

෍

𝑘:𝑠 𝑗,𝑘 =1

(𝜄 𝑘 )⊤ 𝑦 𝑗 − 𝑧 𝑗,𝑘

2 + 𝜇

2 ෍

𝑗=1 𝑜𝑛

෍

𝑙=1 𝑜

𝑦𝑙

(𝑗) 2

Collaborative filtering optimization objective

• Given 𝑦 1 , 𝑦 2 , ⋯ , 𝑦 𝑜𝑛 , estimate 𝜄 1 , 𝜄 2 , ⋯ , 𝜄 𝑜𝑣

min

𝜄 1 ,𝜄 2 ,⋯,𝜄 𝑜𝑣

1 2 ෍

𝑘=1 𝑜𝑣

෍

𝑗:𝑠 𝑗,𝑘 =1

(𝜄 𝑘 )⊤ 𝑦 𝑗 − 𝑧 𝑗,𝑘

2 + 𝜇

2 ෍

𝑘=1 𝑜𝑣

෍

𝑙=1 𝑜

𝜄𝑙

𝑘 2

• Given 𝜄 1 , 𝜄 2 , ⋯ , 𝜄 𝑜𝑣 , estimate 𝑦 1 , 𝑦 2 , ⋯ , 𝑦 𝑜𝑛

min

𝑦(1),𝑦(2),⋯,𝑦(𝑜𝑛)

1 2 ෍

𝑗=1 𝑜𝑛

෍

𝑘:𝑠 𝑗,𝑘 =1

(𝜄 𝑘 )⊤ 𝑦 𝑗 − 𝑧 𝑗,𝑘

2 + 𝜇

2 ෍

𝑗=1 𝑜𝑛

෍

𝑙=1 𝑜

𝑦𝑙

(𝑗) 2

• Minimize 𝑦 1 , 𝑦 2 , ⋯ , 𝑦 𝑜𝑛 and 𝜄 1 , 𝜄 2 , ⋯ , 𝜄 𝑜𝑣 simultaneously

𝐾 = 1 2 ෍

𝑠 𝑗,𝑘 =1

(𝜄 𝑘 )⊤ 𝑦 𝑗 − 𝑧 𝑗,𝑘

2 + 𝜇

2 ෍

𝑘=1 𝑜𝑣

෍

𝑙=1 𝑜

𝜄𝑙

𝑘 2

+ 𝜇 2 ෍

𝑗=1 𝑜𝑛

෍

𝑙=1 𝑜

𝑦𝑙

(𝑗) 2

Collaborative filtering optimization objective

𝐾(𝑦 1 , 𝑦 2 , ⋯ , 𝑦 𝑜𝑛 , 𝜄 1 , 𝜄 2 , ⋯ , 𝜄 𝑜𝑣 ) = 1 2 ෍

𝑠 𝑗,𝑘 =1

(𝜄 𝑘 )⊤ 𝑦 𝑗 − 𝑧 𝑗,𝑘

2 + 𝜇

2 ෍

𝑘=1 𝑜𝑣

෍

𝑙=1 𝑜

𝜄𝑙

𝑘 2

+ 𝜇 2 ෍

𝑗=1 𝑜𝑛

෍

𝑙=1 𝑜

𝑦𝑙

(𝑗) 2

Collaborative filtering algorithm

• Initialize 𝑦 1 , 𝑦 2 , ⋯ , 𝑦 𝑜𝑛 , 𝜄 1 , 𝜄 2 , ⋯ , 𝜄 𝑜𝑣 to small random values
• Minimize 𝐾(𝑦 1 , 𝑦 2 , ⋯ , 𝑦 𝑜𝑛 , 𝜄 1 , 𝜄 2 , ⋯ , 𝜄 𝑜𝑣 ) using gradient

descent (or an advanced optimization algorithm). For every 𝑘 = 1 ⋯ 𝑜𝑣, 𝑗 = 1, ⋯ , 𝑜𝑛: 𝑦𝑙

𝑘 ≔ 𝑦𝑙 𝑘 − 𝛽

෍

𝑘:𝑠 𝑗,𝑘 =1

( 𝜄 𝑘

⊤ 𝑦 𝑗

− 𝑧 𝑗,𝑘 ) 𝜄𝑙

𝑗 + 𝜇 𝑦𝑙 (𝑗)

𝜄𝑙

𝑘 ≔ 𝜄𝑙 𝑘 − 𝛽

෍

𝑗:𝑠 𝑗,𝑘 =1

( 𝜄 𝑘

⊤ 𝑦 𝑗

− 𝑧 𝑗,𝑘 ) 𝑦𝑙

𝑗 + 𝜇 𝜄𝑙 (𝑘)

• For a user with parameter 𝜄 and movie with (learned) feature 𝑦, predict

a star rating of 𝜄⊤𝑦

Collaborative filtering

Movie Alice (1) Bob (2) Carol (3) Dave (4) Love at last

5 5

Romance forever

5 ? ?

Cute puppies of love

? 4 ?

Nonstop car chases

5 4

Swords vs. karate

5 ?

Collaborative filtering

• Predicted ratings:

𝑌 = − 𝑦 1

⊤ −

− 𝑦 2

⊤ −

⋮ − 𝑦 𝑜𝑛

⊤ −

Θ = − 𝜄 1

⊤ −

− 𝜄 2

⊤ −

⋮ − 𝜄 𝑜𝑣

⊤ −

Y = XΘ⊤

Low-rank matrix factorization

Finding related movies/products

• For each product 𝑗, we learn a feature vector 𝑦(𝑗) ∈ 𝑆𝑜

𝑦1: romance, 𝑦2: action, 𝑦3: comedy, …

• How to find movie 𝑘 relate to movie 𝑗?

Small 𝑦(𝑗) − 𝑦(𝑘) movie j and I are “similar”

Recommender Systems

• Motivation
• Problem formulation
• Content-based recommendations
• Collaborative filtering
• Mean normalization

Users who have not rated any movies

1 2 ෍

𝑠 𝑗,𝑘 =1

(𝜄 𝑘 )⊤ 𝑦 𝑗 − 𝑧 𝑗,𝑘

2 + 𝜇

2 ෍

𝑘=1 𝑜𝑣

෍

𝑙=1 𝑜

𝜄𝑙

𝑘 2

+ 𝜇 2 ෍

𝑗=1 𝑜𝑛

෍

𝑙=1 𝑜

𝑦𝑙

(𝑗) 2

𝜄(5) = 0

Movie Alice (1) Bob (2) Carol (3) Dave (4) Eve (5) Love at last

5 5 ?

Romance forever

5 ? ? ?

Cute puppies

• f love

? 4 ? ?

Nonstop car chases

5 4 ?

Swords vs. karate

5 ? ?

Users who have not rated any movies

1 2 ෍

𝑠 𝑗,𝑘 =1

(𝜄 𝑘 )⊤ 𝑦 𝑗 − 𝑧 𝑗,𝑘

2 + 𝜇

2 ෍

𝑘=1 𝑜𝑣

෍

𝑙=1 𝑜

𝜄𝑙

𝑘 2

+ 𝜇 2 ෍

𝑗=1 𝑜𝑛

෍

𝑙=1 𝑜

𝑦𝑙

(𝑗) 2

𝜄(5) = 0

Movie Alice (1) Bob (2) Carol (3) Dave (4) Eve (5) Love at last

5 5

Romance forever

5 ? ?

Cute puppies

• f love

? 4 ?

Nonstop car chases

5 4

Swords vs. karate

5 ?

Mean normalization

For user 𝑘, on movie 𝑗 predict: 𝜄 𝑘

⊤ 𝑦(𝑗) + 𝜈𝑗

User 5 (Eve): 𝜄 5 = 0 𝜄 5

⊤ 𝑦(𝑗) + 𝜈𝑗

Learn 𝜄(𝑘), 𝑦(𝑗)

Recommender Systems

• Motivation
• Problem formulation
• Content-based recommendations
• Collaborative filtering
• Mean normalization

Review: Supervised Learning

• K nearest neighbor
• Linear Regression
• Naïve Bayes
• Logistic Regression
• Support Vector Machines
• Neural Networks

Review: Unsupervised Learning

• Clustering, K-Mean
• Expectation maximization
• Dimensionality reduction
• Anomaly detection
• Recommendation system

Advanced Topics

• Semi-supervised learning
• Probabilistic graphical models
• Generative models
• Sequence prediction models
• Deep reinforcement learning

Semi-supervised Learning

• Motivation
• Problem formulation
• Consistency regularization
• Entropy-based method
• Pseudo-labeling

Semi-supervised Learning

• Motivation
• Problem formulation
• Consistency regularization
• Entropy-based method
• Pseudo-labeling

Classic Paradigm Insufficient Nowadays

• Modern applications: massive amounts of raw data.
• Only a tiny fraction can be annotated by human experts

Protein sequences Billions of webpages Images

Semi-supervised Learning

Active Learning

Semi-supervised Learning

• Motivation
• Problem formulation
• Consistency regularization
• Entropy-based method
• Pseudo-labeling

Semi-supervised Learning Problem Formulation

• Labeled data

𝑇𝑚 = 𝑦 1 , 𝑧 1 , 𝑦 2 , 𝑧 2 , ⋯ , 𝑦 𝑛𝑚 , 𝑧 𝑛𝑚

• Unlabeled data

𝑇𝑣 = 𝑦 1 , 𝑧 1 , 𝑦 2 , 𝑧 2 , ⋯ , 𝑦 𝑛𝑣 , 𝑧 𝑛𝑣

• Goal: Learn a hypothesis ℎ𝜄 (e.g., a classifier) that has small error

Combining labeled and unlabeled data

• Classical methods
• Transductive SVM [Joachims ’99]
• Co-training [Blum and Mitchell ’98]
• Graph-based methods [Blum and Chawla ‘01] [Zhu, Ghahramani,

Lafferty ‘03]

Transductive SVM

• The separator goes through low density regions of the space /

large margin

SVM

Inputs: 𝑦l

(𝑗), 𝑧l (𝑗)

min

𝜄 1 2 σ𝑘=1 𝑜

𝜄

𝑘 2

• s. t. 𝑧l

(𝑗)𝜄⊤𝑦𝑚 𝑗 ≥ 1

Transductive SVM

Inputs: 𝑦l

(𝑗), 𝑧l (𝑗)

, 𝑦u

(𝑗), 𝑧𝑣 (𝑗)

min

𝜄 1 2 σ𝑘=1 𝑜

𝜄

𝑘 2

• s. t.

𝑧l

(𝑗)𝜄⊤𝑦𝑚 𝑗 ≥ 1

෢ 𝑧u

(𝑗)𝜄⊤𝑦 𝑗 ≥ 1

෢ 𝑧u

𝑗 ∈ {−1, 1}

Transductive SVMs

• First maximize margin over the labeled points
• Use this to give initial labels to unlabeled points based on this

separator.

• Try flipping labels of unlabeled points to see if doing so can increase

margin

Deep Semi-supervised Learning

Semi-supervised Learning

• Motivation
• Problem formulation
• Consistency regularization
• Entropy-based method
• Pseudo-labeling

Stochastic Perturbations/Π-Model

• Realistic perturbations 𝑦 → ො

𝑦 of data points 𝑦 ∈ 𝐸𝑉𝑀 should not significantly change the output of h𝜄(𝑦)

Temporal Ensembling

Mean Teacher

Virtual Adversarial Training

Semi-supervised Learning

• Motivation
• Problem formulation
• Consistency regularization
• Entropy-based method
• Pseudo-labeling

EntMin

• Encourages more confident predictions on unlabeled data.

Semi-supervised Learning

• Motivation
• Problem formulation
• Consistency regularization
• Entropy-based method
• Pseudo-labeling

Comparison

Varying number of labels

Class mismatch in Labeled/Unlabeled datasets hurts the performance

Lessons

• Standardized architecture + equal budget for tuning hyperparameters
• Unlabeled data from a different class distribution not that useful
• Most methods don’t work well in the very low labeled-data regime
• Transferring Pre-Trained Imagenet produces lower error rate
• Conclusions based on small datasets though