SLIDE 1
Variational Sequential Labelers for Semi-Supervised Learning
Mingda Chen, Qingming Tang, Karen Livescu, Kevin Gimpel
SLIDE 2 Sequence Labeling
Part-of-Speech (POS) Tagging This item is a small one and easily missed .
determiner noun verb determiner adjective noun coordinating adverb verb punctuation conjunction
Named Entity Recognition (NER) EU rejects German call to boycott British lamb .
B-ORG O B-MISC O O O B-MISC O O
SLIDE 3
Overview
❖ Latent-variable generative models for sequence labeling ❖ 0.8 ~ 1% absolute improvements over 8 datasets without structured inference ❖ 0.1 ~ 0.3% absolute improvements from adding unlabeled data
SLIDE 4
Why latent-variable models?
❖ Natural way to incorporate unlabeled data ❖ Ability to disentangle representations via the configuration of latent variables ❖ Allow us to use neural variational methods
SLIDE 5 Variational Autoencoder (VAE)
[Kingma and Welling, ICLR’14; Rezende and Mohamed, ICML’15]
Observation Latent variable
SLIDE 6 Variational Autoencoder (VAE)
Evidence Lower Bound (ELBO)
[Kingma and Welling, ICLR’14; Rezende and Mohamed, ICML’15]
Observation Latent variable
SLIDE 7 Conditional Variational Autoencoder
Given context Observation Latent variable
SLIDE 8 Conditional Variational Autoencoder
Given context Observation Latent variable
SLIDE 9
The input words other than the word at position
SLIDE 10
The input words other than the word at position This item is a small one and easily missed .
SLIDE 11
The input words other than the word at position This item is a small one and easily missed .
SLIDE 12 Variational Sequential Labeler (VSL)
Observation Latent variable Given context
SLIDE 13 Variational Sequential Labeler (VSL)
ELBO Observation Latent variable Given context
SLIDE 14
Variational Sequential Labeler (VSL)
SLIDE 15
Variational Sequential Labeler (VSL)
SLIDE 16
Variational Sequential Labeler (VSL) Classification loss (CL)
SLIDE 17
Variational Sequential Labeler (VSL) Classification loss (CL)
SLIDE 18
VSL: Training and Testing
❖ Maximize where is a hyperparameter ❖ Use one sample from Gaussian distribution using reparameterization trick
Training Testing
❖ Use the mean of Gaussian distribution
SLIDE 19 Variants of VSL
VSL-G Position of classifier
SLIDE 20 Variants of VSL
VSL-G Stands for “Gaussian” Position of classifier
SLIDE 21 Variants of VSL
VSL-G Stands for “Gaussian” VSL-GG-Flat Position of classifier
SLIDE 22 Variants of VSL
VSL-G Stands for “Gaussian” VSL-GG-Flat Position of classifier
SLIDE 23 Variants of VSL
VSL-G Stands for “Gaussian” VSL-GG-Flat Position of classifier
SLIDE 24 Variants of VSL
VSL-G Stands for “Gaussian” VSL-GG-Flat VSL-GG-Hier Position of classifier
SLIDE 25 Variants of VSL
VSL-G Stands for “Gaussian” VSL-GG-Flat VSL-GG-Hier Position of classifier
SLIDE 26 Experiments
❖ Twitter POS Dataset ➢ Subset of 56 million English tweets as unlabeled data ➢ 25 tags ❖ Universal Dependencies POS Datasets ➢ 20% of original training set as labeled data ➢ 50% of original training set as unlabeled data ➢ 6 languages ➢ 17 tags ❖ CoNLL 2003 English NER Dataset ➢ 10% of original training set as labeled data ➢ 50% of original training set as unlabeled data ➢ BIOES labeling scheme
SLIDE 27
Results
SLIDE 28
Universal Dependencies POS
SLIDE 29 t-SNE Visualization
❖ Each point represents a word token ❖ Color indicates gold standard POS tag in Twitter dev set
BiGRU baseline
SLIDE 30 t-SNE Visualization
VSL-GG-Hier VSL-GG-Flat
y (label) variable z variable
SLIDE 31 Effect of Position of Classification Loss
VSL-GG-Hier Position of classifier
SLIDE 32 Effect of Position of Classification Loss
VSL-GG-Hier VSL-GG-Hier with classifier on Position of classifier
SLIDE 33 Effect of Position of Classification Loss
VSL-GG-Hier VSL-GG-Hier-z VSL-GG-Hier with classifier on Position of classifier
SLIDE 34
Effect of Position of Classification Loss
SLIDE 35
Effect of Position of Classification Loss
Hierarchical structure is only helpful when classification loss and reconstruction loss are attached to different latent variables
SLIDE 36
Effect of Variational Regularization (VR)
Randomness in the latent space VR KL divergence between approximated posterior and prior
SLIDE 37
Effect of VR
SLIDE 38
Effect of Unlabeled data
❖ Evaluate VSL-GG-Hier on Twitter dataset ❖ Subsample unlabeled data from 56 million tweets ❖ Vary the number of unlabeled data
SLIDE 39
Effect of Unlabeled data
SLIDE 40
Summary
❖ We introduced VSLs for semi-supervised learning ❖ Best VSL uses multiple latent variable and arranged in hierarchical structure ❖ Hierarchical structure is only helpful when classification loss and reconstruction loss are attached to different latent variables ❖ VSLs show consistent improvements across 8 datasets over a strong baseline
SLIDE 41
Thank you!
