Strong Baselines for Neural Semi-supervised Learning under Domain - - PowerPoint PPT Presentation

Strong Baselines for Neural Semi-supervised Learning under Domain Shift

Sebastian Ruder Barbara Plank

2

Learning under Domain Shift

• State-of-the-art domain adaptation approaches

2

Learning under Domain Shift

• State-of-the-art domain adaptation approaches
• leverage task-specific features

2

Learning under Domain Shift

• State-of-the-art domain adaptation approaches
• leverage task-specific features
• evaluate on proprietary datasets or on a single

benchmark

2

Learning under Domain Shift

• State-of-the-art domain adaptation approaches
• leverage task-specific features
• evaluate on proprietary datasets or on a single

benchmark

• Only compare against weak baselines

2

Learning under Domain Shift

• State-of-the-art domain adaptation approaches
• leverage task-specific features
• evaluate on proprietary datasets or on a single

benchmark

• Only compare against weak baselines
• Almost none evaluate against approaches from the

extensive semi-supervised learning (SSL) literature

2

Learning under Domain Shift

3

Revisiting Semi-Supervised Learning Classics in a Neural World

• How do classics in SSL compare to recent advances?

3

Revisiting Semi-Supervised Learning Classics in a Neural World

• How do classics in SSL compare to recent advances?
• Can we combine the best of both worlds?

3

Revisiting Semi-Supervised Learning Classics in a Neural World

• How do classics in SSL compare to recent advances?
• Can we combine the best of both worlds?
• How well do these approaches work on out-of-distribution

data?

3

Revisiting Semi-Supervised Learning Classics in a Neural World

Bootstrapping algorithms

• Self-training

Bootstrapping algorithms

• Self-training
• (Co-training)

Bootstrapping algorithms

• Self-training
• (Co-training)
• Tri-training

Bootstrapping algorithms

• Self-training
• (Co-training)
• Tri-training
• Tri-training with disagreement

Bootstrapping algorithms

5

Self-training

• 1. Train model on labeled data.

5

Self-training

• 1. Train model on labeled data.
• 2. Use confident predictions on unlabeled data

as training examples. Repeat.

5

Self-training

• 1. Train model on labeled data.
• 2. Use confident predictions on unlabeled data

as training examples. Repeat.

5

Self-training

• E

r r

• r

a m p l i f i c a t i

• n

• 1. Train model on labeled data.
• 2. Use confident predictions on unlabeled data

as training examples. Repeat.

5

Self-training

• E

r r

• r

a m p l i f i c a t i

• n

6

Self-training variants

• Calibration

6

Self-training variants

• Calibration
• Output probabilities in neural networks are poorly

calibrated.

6

Self-training variants

• Calibration
• Output probabilities in neural networks are poorly

calibrated.

• Throttling (Abney, 2007), i.e. selecting the top n highest

confidence unlabeled examples works best.

6

Self-training variants

• Calibration
• Output probabilities in neural networks are poorly

calibrated.

• Throttling (Abney, 2007), i.e. selecting the top n highest

confidence unlabeled examples works best.

• Online learning

6

Self-training variants

• Calibration
• Output probabilities in neural networks are poorly

calibrated.

• Throttling (Abney, 2007), i.e. selecting the top n highest

confidence unlabeled examples works best.

• Online learning
• Training until convergence on labeled data and then on

unlabeled data works best.

6

Self-training variants

7

Tri-training

Tri-training

7

• 1. Train three models on bootstrapped samples.

Tri-training

Tri-training

7

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree.

Tri-training

Tri-training

7

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree.

x

Tri-training

Tri-training

7

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree.

y = 1

x

Tri-training

Tri-training

7

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree.

y = 1

x

y = 1

Tri-training

Tri-training

7

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree.

y = 1

x

y = 1 1

Tri-training

Tri-training

8

Tri-training

Tri-training

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree.
• 3. Final prediction: majority voting

Tri-training

8

Tri-training

Tri-training

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree.
• 3. Final prediction: majority voting

Tri-training

x

8

Tri-training

Tri-training

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree.
• 3. Final prediction: majority voting

Tri-training

y = 1

x

8

Tri-training

Tri-training

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree.
• 3. Final prediction: majority voting

Tri-training

y = 1 y = 0

x

8

Tri-training

Tri-training

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree.
• 3. Final prediction: majority voting

Tri-training

y = 1 y = 1 y = 0

x

8

Tri-training

Tri-training

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree.
• 3. Final prediction: majority voting

Tri-training

y = 1 y = 1 y = 0 1

x

Tri-training
 with disagreement

9

Tri-training with disagreement

Tri-training
 with disagreement

9

Tri-training with disagreement

• 1. Train three models on bootstrapped samples.

Tri-training
 with disagreement

9

Tri-training with disagreement

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree

and prediction differs.

Tri-training
 with disagreement

9

Tri-training with disagreement

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree

and prediction differs.

x

Tri-training
 with disagreement

9

Tri-training with disagreement

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree

and prediction differs.

y = 1

x

Tri-training
 with disagreement

9

Tri-training with disagreement

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree

and prediction differs.

y = 1

x

y = 1

Tri-training
 with disagreement

9

Tri-training with disagreement

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree

and prediction differs.

y = 1

x

y = 1 y = 0

Tri-training
 with disagreement

9

Tri-training with disagreement

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree

and prediction differs.

y = 1

x

y = 1 1 y = 0

Tri-training
 with disagreement

9

Tri-training with disagreement

• 1. Train three models on bootstrapped samples.
• 2. Use predictions on unlabeled data for third if two agree

and prediction differs.

y = 1

x

y = 1 1 y = 0

• 3

i n d e p e n d e n t m

• d

e l s

10

Tri-training hyper-parameters

• Sampling unlabeled data

10

Tri-training hyper-parameters

• Sampling unlabeled data
• Producing predictions for all unlabeled examples is

expensive

10

Tri-training hyper-parameters

• Sampling unlabeled data
• Producing predictions for all unlabeled examples is

expensive

• Sample number of unlabeled examples

10

Tri-training hyper-parameters

• Sampling unlabeled data
• Producing predictions for all unlabeled examples is

expensive

• Sample number of unlabeled examples
• Confidence thresholding

10

Tri-training hyper-parameters

• Sampling unlabeled data
• Producing predictions for all unlabeled examples is

expensive

• Sample number of unlabeled examples
• Confidence thresholding
• Not effective for classic approaches, but essential for
• ur method

10

Tri-training hyper-parameters

11

Multi-task tri-training

Multi-task
 Tri-training

11

Multi-task tri-training

• 1. Train one model with 3 objective functions.

Multi-task
 Tri-training

11

Multi-task tri-training

• 1. Train one model with 3 objective functions.
• 2. Use predictions on unlabeled data for third if two agree.

Multi-task
 Tri-training

11

x

Multi-task tri-training

• 1. Train one model with 3 objective functions.
• 2. Use predictions on unlabeled data for third if two agree.

Multi-task
 Tri-training

11

y = 1

x

Multi-task tri-training

• 1. Train one model with 3 objective functions.
• 2. Use predictions on unlabeled data for third if two agree.

Multi-task
 Tri-training

11

y = 1

x

y = 1

Multi-task tri-training

• 1. Train one model with 3 objective functions.
• 2. Use predictions on unlabeled data for third if two agree.

Multi-task
 Tri-training

11

y = 1

x

y = 1 1

Multi-task tri-training

• 1. Train one model with 3 objective functions.
• 2. Use predictions on unlabeled data for third if two agree.

Multi-task
 Tri-training

11

y = 1

x

y = 1 1

Multi-task tri-training

• 1. Train one model with 3 objective functions.
• 2. Use predictions on unlabeled data for third if two agree.

Multi-task
 Tri-training

• 3. Restrict final layers to 


use different 
 representations.

11

y = 1

x

y = 1 1

Multi-task tri-training

• 1. Train one model with 3 objective functions.
• 2. Use predictions on unlabeled data for third if two agree.

Multi-task
 Tri-training

• 3. Restrict final layers to 


use different 
 representations.

• 4. Train third objective 


function only on 
 pseudo labeled to 
 bridge domain shift.

12

Multi-task
 Tri-training

12

BiLSTM

Multi-task
 Tri-training

(Plank et al., 2016)

12

BiLSTM w2

char BiLSTM

Multi-task
 Tri-training

(Plank et al., 2016)

12

BiLSTM w2

char BiLSTM

BiLSTM w1

char BiLSTM

Multi-task
 Tri-training

(Plank et al., 2016)

12

BiLSTM w2

char BiLSTM

BiLSTM w1

char BiLSTM

BiLSTM w3

char BiLSTM

Multi-task
 Tri-training

(Plank et al., 2016)

12

BiLSTM w2

char BiLSTM

BiLSTM w1

char BiLSTM

BiLSTM w3

char BiLSTM

m1

Multi-task
 Tri-training

(Plank et al., 2016)

12

BiLSTM w2

char BiLSTM

BiLSTM w1

char BiLSTM

BiLSTM w3

char BiLSTM

m1 m2

Multi-task
 Tri-training

(Plank et al., 2016)

12

BiLSTM w2

char BiLSTM

BiLSTM w1

char BiLSTM

BiLSTM w3

char BiLSTM

m1 m2 m3

Multi-task
 Tri-training

(Plank et al., 2016)

12

BiLSTM w2

char BiLSTM

BiLSTM w1

char BiLSTM

BiLSTM w3

char BiLSTM

m1 m2 m3 m1 m2 m3

Multi-task
 Tri-training

(Plank et al., 2016)

12

BiLSTM w2

char BiLSTM

BiLSTM w1

char BiLSTM

BiLSTM w3

char BiLSTM

m1 m2 m3 m1 m2 m3 m1 m2 m3

Multi-task
 Tri-training

(Plank et al., 2016)

12

BiLSTM w2

char BiLSTM

BiLSTM w1

char BiLSTM

BiLSTM w3

char BiLSTM

m1 m2 m3 m1 m2 m3 m1 m2 m3

• rthogonality constraint (Bousmalis et al., 2016)

Multi-task
 Tri-training

Lorth = ∥W⊤

m1Wm2∥2 F

(Plank et al., 2016)

12

BiLSTM w2

char BiLSTM

BiLSTM w1

char BiLSTM

BiLSTM w3

char BiLSTM

m1 m2 m3 m1 m2 m3 m1 m2 m3

• rthogonality constraint (Bousmalis et al., 2016)

Multi-task
 Tri-training

Lorth = ∥W⊤

m1Wm2∥2 F

L(θ) = − ∑

i ∑ 1,..,n

log Pmi(y| ⃗ h ) + γLorth

Loss: (Plank et al., 2016)

13

Data & Tasks

13

Data & Tasks

Two tasks: Domains:

13

Data & Tasks

Two tasks: Domains:

Sentiment analysis on Amazon reviews dataset (Blitzer et al, 2006)

13

Data & Tasks

Two tasks: Domains:

Sentiment analysis on Amazon reviews dataset (Blitzer et al, 2006) POS tagging on SANCL 2012 dataset (Petrov and McDonald, 2012)

Sentiment Analysis Results

Accuracy 75 76.75 78.5 80.25 82 Avg over 4 target domains

VFAE* DANN* Asym* Source only Self-training Tri-training Tri-training-Disagr. MT-Tri

* result from Saito et al., (2017) 14

Sentiment Analysis Results

Accuracy 75 76.75 78.5 80.25 82 Avg over 4 target domains

VFAE* DANN* Asym* Source only Self-training Tri-training Tri-training-Disagr. MT-Tri

* result from Saito et al., (2017) 14

Sentiment Analysis Results

Accuracy 75 76.75 78.5 80.25 82 Avg over 4 target domains

VFAE* DANN* Asym* Source only Self-training Tri-training Tri-training-Disagr. MT-Tri

* result from Saito et al., (2017) 14

Sentiment Analysis Results

Accuracy 75 76.75 78.5 80.25 82 Avg over 4 target domains

VFAE* DANN* Asym* Source only Self-training Tri-training Tri-training-Disagr. MT-Tri

* result from Saito et al., (2017) 14

Sentiment Analysis Results

Accuracy 75 76.75 78.5 80.25 82 Avg over 4 target domains

VFAE* DANN* Asym* Source only Self-training Tri-training Tri-training-Disagr. MT-Tri

* result from Saito et al., (2017) 14

• Multi-task tri-training slightly outperforms tri-training, but

has higher variance.

15

POS Tagging Results

Trained on 10% labeled data (WSJ)

Accuracy 88.7 88.975 89.25 89.525 89.8 Avg over 5 target domains

Source (+embeds) Self-training Tri-training Tri-training-Disagr. MT-Tri

15

POS Tagging Results

Trained on 10% labeled data (WSJ)

Accuracy 88.7 88.975 89.25 89.525 89.8 Avg over 5 target domains

Source (+embeds) Self-training Tri-training Tri-training-Disagr. MT-Tri

15

POS Tagging Results

Trained on 10% labeled data (WSJ)

Accuracy 88.7 88.975 89.25 89.525 89.8 Avg over 5 target domains

Source (+embeds) Self-training Tri-training Tri-training-Disagr. MT-Tri

15

POS Tagging Results

Trained on 10% labeled data (WSJ)

Accuracy 88.7 88.975 89.25 89.525 89.8 Avg over 5 target domains

Source (+embeds) Self-training Tri-training Tri-training-Disagr. MT-Tri

15

POS Tagging Results

Trained on 10% labeled data (WSJ)

Accuracy 88.7 88.975 89.25 89.525 89.8 Avg over 5 target domains

Source (+embeds) Self-training Tri-training Tri-training-Disagr. MT-Tri

• Tri-training with disagreement works best with little data.

16

POS Tagging Results

* result from Schnabel & Schütze (2014)

Trained on full labeled data (WSJ)

Accuracy 89 89.75 90.5 91.25 92 Avg over 5 target domains

TnT Stanford* Source (+embeds) Tri-training Tri-training-Disagr. MT-Tri

16

POS Tagging Results

* result from Schnabel & Schütze (2014)

Trained on full labeled data (WSJ)

Accuracy 89 89.75 90.5 91.25 92 Avg over 5 target domains

TnT Stanford* Source (+embeds) Tri-training Tri-training-Disagr. MT-Tri

16

POS Tagging Results

* result from Schnabel & Schütze (2014)

Trained on full labeled data (WSJ)

Accuracy 89 89.75 90.5 91.25 92 Avg over 5 target domains

TnT Stanford* Source (+embeds) Tri-training Tri-training-Disagr. MT-Tri

16

POS Tagging Results

* result from Schnabel & Schütze (2014)

Trained on full labeled data (WSJ)

Accuracy 89 89.75 90.5 91.25 92 Avg over 5 target domains

TnT Stanford* Source (+embeds) Tri-training Tri-training-Disagr. MT-Tri

• Tri-training works best in the full data setting.

17

POS Tagging Analysis

Accuracy on out-of-vocabulary (OOV) tokens

Accuracy on OOV tokens 50 57.5 65 72.5 80 % OOV tokens 2.75 5.5 8.25 11 Answers Emails Newsgroups Reviews Weblogs

OOV tokens Src Tri MT-Tri

17

POS Tagging Analysis

Accuracy on out-of-vocabulary (OOV) tokens

Accuracy on OOV tokens 50 57.5 65 72.5 80 % OOV tokens 2.75 5.5 8.25 11 Answers Emails Newsgroups Reviews Weblogs

OOV tokens Src Tri MT-Tri

17

POS Tagging Analysis

Accuracy on out-of-vocabulary (OOV) tokens

Accuracy on OOV tokens 50 57.5 65 72.5 80 % OOV tokens 2.75 5.5 8.25 11 Answers Emails Newsgroups Reviews Weblogs

OOV tokens Src Tri MT-Tri

17

POS Tagging Analysis

Accuracy on out-of-vocabulary (OOV) tokens

Accuracy on OOV tokens 50 57.5 65 72.5 80 % OOV tokens 2.75 5.5 8.25 11 Answers Emails Newsgroups Reviews Weblogs

OOV tokens Src Tri MT-Tri

• Classic tri-training works best on OOV tokens.

17

POS Tagging Analysis

Accuracy on out-of-vocabulary (OOV) tokens

Accuracy on OOV tokens 50 57.5 65 72.5 80 % OOV tokens 2.75 5.5 8.25 11 Answers Emails Newsgroups Reviews Weblogs

OOV tokens Src Tri MT-Tri

• Classic tri-training works best on OOV tokens.
• MT-Tri does worse than source-only baseline on OOV.

18

POS accuracy per binned log frequency

Accuracy delta vs. src-only baseline

• 0.005

0.005 0.009 0.014 0.018 Binned frequency 1 2 3 4 5 6 7 8 9 10 11 12 13 14

MT-Tri Tri

POS Tagging Analysis

18

POS accuracy per binned log frequency

Accuracy delta vs. src-only baseline

• 0.005

0.005 0.009 0.014 0.018 Binned frequency 1 2 3 4 5 6 7 8 9 10 11 12 13 14

MT-Tri Tri

POS Tagging Analysis

18

POS accuracy per binned log frequency

Accuracy delta vs. src-only baseline

• 0.005

0.005 0.009 0.014 0.018 Binned frequency 1 2 3 4 5 6 7 8 9 10 11 12 13 14

MT-Tri Tri

• Tri-training works best on low-frequency tokens (leftmost

bins).

POS Tagging Analysis

19

POS Tagging Analysis

Accuracy on unknown word-tag (UWT) tokens

Accuracy on UWT tokens 8 12.5 17 21.5 26 % UWT tokens 1 2 3 4 Answers Emails Newsgroups Reviews Weblogs

UWT rate Src Tri MT-Tri FLORS*

* result from Schnabel & Schütze (2014)

19

POS Tagging Analysis

Accuracy on unknown word-tag (UWT) tokens

Accuracy on UWT tokens 8 12.5 17 21.5 26 % UWT tokens 1 2 3 4 Answers Emails Newsgroups Reviews Weblogs

UWT rate Src Tri MT-Tri FLORS*

v e r y d i f fi c u l t c a s e s

* result from Schnabel & Schütze (2014)

19

POS Tagging Analysis

Accuracy on unknown word-tag (UWT) tokens

Accuracy on UWT tokens 8 12.5 17 21.5 26 % UWT tokens 1 2 3 4 Answers Emails Newsgroups Reviews Weblogs

UWT rate Src Tri MT-Tri FLORS*

v e r y d i f fi c u l t c a s e s

* result from Schnabel & Schütze (2014)

19

POS Tagging Analysis

Accuracy on unknown word-tag (UWT) tokens

Accuracy on UWT tokens 8 12.5 17 21.5 26 % UWT tokens 1 2 3 4 Answers Emails Newsgroups Reviews Weblogs

UWT rate Src Tri MT-Tri FLORS*

v e r y d i f fi c u l t c a s e s

* result from Schnabel & Schütze (2014)

19

POS Tagging Analysis

Accuracy on unknown word-tag (UWT) tokens

Accuracy on UWT tokens 8 12.5 17 21.5 26 % UWT tokens 1 2 3 4 Answers Emails Newsgroups Reviews Weblogs

UWT rate Src Tri MT-Tri FLORS*

• No bootstrapping method works well on unknown word-

tag combinations.

v e r y d i f fi c u l t c a s e s

* result from Schnabel & Schütze (2014)

19

POS Tagging Analysis

Accuracy on unknown word-tag (UWT) tokens

Accuracy on UWT tokens 8 12.5 17 21.5 26 % UWT tokens 1 2 3 4 Answers Emails Newsgroups Reviews Weblogs

UWT rate Src Tri MT-Tri FLORS*

• No bootstrapping method works well on unknown word-

tag combinations.

• Less lexicalized FLORS approach is superior.

v e r y d i f fi c u l t c a s e s

* result from Schnabel & Schütze (2014)

20

Takeaways

Tri-training

• Classic tri-training works best: outperforms recent

state-of-the-art methods for sentiment analysis.

20

Takeaways

Tri-training

• Classic tri-training works best: outperforms recent

state-of-the-art methods for sentiment analysis.

• We address the drawback of tri-training (space &

time complexity) via the proposed MT-Tri model

• MT-Tri works best on sentiment, but not for POS.

20

Takeaways

Tri-training

• Classic tri-training works best: outperforms recent

state-of-the-art methods for sentiment analysis.

• We address the drawback of tri-training (space &

time complexity) via the proposed MT-Tri model

• MT-Tri works best on sentiment, but not for POS.
• Importance of:
• Comparing neural methods to classics (strong

baselines)

• Evaluation on multiple tasks & domains

20

Takeaways

Tri-training