Meta-Learning with Shared Amortized Variational Inference Ekaterina - - PowerPoint PPT Presentation

Meta-Learning with Shared Amortized Variational Inference

Ekaterina Iakovleva Jakob Verbeek Karteek Alahari

ICML | 2020

Thirty-seventh International Conference

• n Machine Learning

Inria Facebook Inria

Standard classification task pipeline

ICML | 2020

2

ICML | 2020

Meta-learning classification task pipeline

Meta test data

3

Schmidhuber 1999, Ravi & Larochelle ICLR’17

ICML | 2020

Overview

• This work focuses on the empirical Bayes meta-learning approach.
• We propose a novel scheme for amortized variational inference.
• We demonstrate that earlier work based on Monte-Carlo approximation

underestimates model variance.

• We show the advantage of our approach on miniImageNet and FC100.

4

ICML | 2020

Meta-learning classification task definition

• K - shot N - way classification task
• Episodic training: each task t is sampled from a distribution over tasks 𝑞 𝛶
• Support data 𝐸! =

(𝑦",$

! , 𝑧",$ ! ) ",$%& ',(

• Query data *

𝐸! = (+ 𝑦),$

!

, + 𝑧),$

!

) ),$%&

*,(

5

ICML | 2020

Meta-learning approaches

• Distance-based classifiers

v Learned metric relies on the distance to individual samples or class prototypes. v E.g. Prototypical Networks [1], Matching Nets [2].

6

[1] – Snell et al. NeurIPS’17, [2] – Vinyals et al. NeurIPS’16

ICML | 2020

Meta-learning approaches

• Distance-based classifiers

v Learned metric relies on the distance to individual samples or class prototypes. v E.g. Prototypical Networks [1], Matching Nets [2].

• Optimization-based approaches

v Vanilla SGD approach is replaced by a trainable update mechanism. v E.g. MAML [3], Meta LSTM [4].

6

[1] – Snell et al. NeurIPS’17, [2] – Vinyals et al. NeurIPS’16, [3] – Finn et al. ICML’17, [4] – Ravi & Larochelle ICLR’17

ICML | 2020

Meta-learning approaches

• Distance-based classifiers

v Learned metric relies on the distance to individual samples or class prototypes. v E.g. Prototypical Networks [1], Matching Nets [2].

• Optimization-based approaches

v Vanilla SGD approach is replaced by a trainable update mechanism. v E.g. MAML [3], Meta LSTM [4].

• Latent variable models

v The model parameters are treated as latent variables. v Their variance is explicitly modeled in a Bayesian framework. v E.g. Neural Processes [5], VERSA [6].

6

[1] – Snell et al. NeurIPS’17, [2] – Vinyals et al. NeurIPS’16, [3] – Finn et al. ICML’17, [4] – Ravi & Larochelle ICLR’17, [5] – Garnelo et al. ICML’18, [6] – Gordon et al. ICLR’19

ICML | 2020

Multi-task generative model

• The multi-task graphical model includes:
• task-agnostic parameters 𝜄
• task-specific latent parameters {𝑥!}!%&

+

7

ICML | 2020

Multi-task generative model

• The multi-task graphical model includes:
• task-agnostic parameters 𝜄
• task-specific latent parameters {𝑥!}!%&

+

Marginal likelihood of the query labels 0 𝑍 = {0 𝑍!}!%&

+

given query samples 0 𝑌 = { 0 𝑌!}!%&

+

and the support sets 𝐸 = {𝐸!}!%&

+

= (𝑌!, 𝑍!) !

+

𝑞 0 𝑍| 0 𝑌, 𝐸, 𝜄 = 4

!%& +

5 𝑞 0 𝑍 0 𝑌, 𝑥! 𝑞, 𝑥! 𝐸!, 𝜄 𝑒𝑥! Intractable integral requires approximation for training and prediction.

7

ICML | 2020

Multi-task generative model

• The multi-task graphical model includes:
• task-agnostic parameters 𝜄
• task-specific latent parameters {𝑥!}!%&

+

Marginal likelihood of the query labels 0 𝑍 = {0 𝑍!}!%&

+

given query samples 0 𝑌 = { 0 𝑌!}!%&

+

and the support sets 𝐸 = {𝐸!}!%&

+

= (𝑌!, 𝑍!) !

+

𝑞 0 𝑍| 0 𝑌, 𝐸, 𝜄 = 4

!%& +

5 𝑞 0 𝑍 0 𝑌, 𝑥! 𝑞, 𝑥! 𝐸!, 𝜄 𝑒𝑥! Intractable integral requires approximation for training and prediction.

7

ICML | 2020

Multi-task generative model

• The multi-task graphical model includes:
• task-agnostic parameters 𝜄
• task-specific latent parameters {𝑥!}!%&

+

Marginal likelihood of the query labels 0 𝑍 = {0 𝑍!}!%&

+

given query samples 0 𝑌 = { 0 𝑌!}!%&

+

and the support sets 𝐸 = {𝐸!}!%&

+

= (𝑌!, 𝑍!) !

+

𝑞 0 𝑍| 0 𝑌, 𝐸, 𝜄 = 4

!%& +

5 𝑞 0 𝑍 0 𝑌, 𝑥! 𝑞, 𝑥! 𝐸!, 𝜄 𝑒𝑥! Intractable integral requires approximation for training and prediction.

7

ICML | 2020

Monte Carlo approximation

• Monte Carlo approximation of the marginal log-likelihood using 𝑥-

!~𝑞, 𝑥! 𝐸!, 𝜄 :

log 𝑞 0 𝑍! 0 𝑌!, 𝐸!, 𝜄 ≈ 1 𝑈𝑁 ?

!%& +

?

)%& *

log 1 𝑀 ?

• %&

.

𝑞 + 𝑧)

! +

𝑦)

! , 𝑥- !

.

• This objective function has been used in VERSA [1].
• Our experiments show that this approach learns degenerate prior 𝑞, 𝑥! 𝐸!, 𝜄 .

8

[1] – Gordon et al. ICLR’19

ICML | 2020

Amortized variational inference

• Variational evidence lower bound (ELBO) with the amortized approximate posterior [1]

parameterized by 𝜔: log 𝑞 0 𝑍!| 0 𝑌!, 𝐸!, 𝜄 ≥ 𝔽/! log 𝑞 0 𝑍! 0 𝑌!, 𝑥! − 𝛾𝒠'. 𝑟0 𝑥! 0 𝑍!, 0 𝑌!, 𝐸!, 𝜄 ||𝑞, 𝑥! 𝐸!, 𝜄

9

[1] – Kingma & Welling ICLR’14

ICML | 2020

Amortized variational inference

• Variational evidence lower bound (ELBO) with the amortized approximate posterior [1]

parameterized by 𝜔: log 𝑞 0 𝑍!| 0 𝑌!, 𝐸!, 𝜄 ≥ 𝔽/! log 𝑞 0 𝑍! 0 𝑌!, 𝑥! − 𝛾𝒠'. 𝑟0 𝑥! 0 𝑍!, 0 𝑌!, 𝐸!, 𝜄 ||𝑞, 𝑥! 𝐸!, 𝜄

9

[1] – Kingma & Welling ICLR’14 Reconstruction loss

ICML | 2020

Amortized variational inference

• Variational evidence lower bound (ELBO) with the amortized approximate posterior [1]

parameterized by 𝜔: log 𝑞 0 𝑍!| 0 𝑌!, 𝐸!, 𝜄 ≥ 𝔽/! log 𝑞 0 𝑍! 0 𝑌!, 𝑥! − 𝛾𝒠'. 𝑟0 𝑥! 0 𝑍!, 0 𝑌!, 𝐸!, 𝜄 ||𝑞, 𝑥! 𝐸!, 𝜄

9

[1] – Kingma & Welling ICLR’14 Regularization

ICML | 2020

Amortized variational inference

• Variational evidence lower bound (ELBO) with the amortized approximate posterior [1]

parameterized by 𝜔: log 𝑞 0 𝑍!| 0 𝑌!, 𝐸!, 𝜄 ≥ 𝔽/! log 𝑞 0 𝑍! 0 𝑌!, 𝑥! − 𝛾𝒠'. 𝑟0 𝑥! 0 𝑍!, 0 𝑌!, 𝐸!, 𝜄 ||𝑞, 𝑥! 𝐸!, 𝜄

• We use regularization coefficient 𝛾 [2] to weight KL term.

9

[1] – Kingma & Welling ICLR’14, [2] – Higgins et al. ICLR’17

ICML | 2020

Amortized variational inference

• Variational evidence lower bound (ELBO) with the amortized approximate posterior [1]

parameterized by 𝜔: log 𝑞 0 𝑍!| 0 𝑌!, 𝐸!, 𝜄 ≥ 𝔽/! log 𝑞 0 𝑍! 0 𝑌!, 𝑥! − 𝛾𝒠'. 𝑟0 𝑥! 0 𝑍!, 0 𝑌!, 𝐸!, 𝜄 ||𝑞, 𝑥! 𝐸!, 𝜄

• We use regularization coefficient 𝛾 [2] to weight KL term.
• Predictions are made via Monte Carlo sampling from the learned prior:

𝑞 + 𝑧)

! +

𝑦)

! , 𝐸!, 𝜄 ≈ 1

𝑀 ?

• %&

.

𝑞 + 𝑧)

! +

𝑦)

! , 𝑥- ! ,

where 𝑥-

!~𝑞, 𝑥! 𝐸!, 𝜄 .

9

[1] – Kingma & Welling ICLR’14, [2] – Higgins et al. ICLR’17

ICML | 2020

Shared amortized variational inference: SAMOVAR

• Both prior and posterior are conditioned on labeled sets.
• The inference network can be shared between prior and posterior.

log 𝑞 0 𝑍!| 0 𝑌!, 𝐸!, 𝜄 ≥ 𝔽/" log 𝑞 0 𝑍! 0 𝑌!, 𝑥! − 𝛾𝒠'. 𝑟0 𝑥! 0 𝑍!, 0 𝑌!, 𝐸!, 𝜄 ||𝑞, 𝑥! 𝐸!, 𝜄

10

ICML | 2020

Shared amortized variational inference: SAMOVAR

• Both prior and posterior are conditioned on labeled sets.
• The inference network can be shared between prior and posterior.

log 𝑞 0 𝑍!| 0 𝑌!, 𝐸!, 𝜄 ≥ 𝔽/" log 𝑞 0 𝑍! 0 𝑌!, 𝑥! − 𝛾𝒠'. 𝑟, 𝑥! 0 𝑍!, 0 𝑌!, 𝐸!, 𝜄 ||𝑞, 𝑥! 𝐸!, 𝜄

10

ICML | 2020

Shared amortized variational inference: SAMOVAR

• Both prior and posterior are conditioned on labeled sets.
• The inference network can be shared between prior and posterior.

log 𝑞 0 𝑍!| 0 𝑌!, 𝐸!, 𝜄 ≥ 𝔽/" log 𝑞 0 𝑍! 0 𝑌!, 𝑥! − 𝛾𝒠'. 𝑟, 𝑥! 0 𝑍!, 0 𝑌!, 𝐸!, 𝜄 ||𝑞, 𝑥! 𝐸!, 𝜄

• Sharing reduces memory footprint, and encourages learning non-degenerate prior.

10

ICML | 2020

SAMOVAR design based on VERSA

• Task-agnostic feature extractor 𝑔

1 produces embeddings of the input images 𝑦.

• Task-specific linear classifier 𝑥! predicts labels for query samples +

𝑦: 𝑞 + 𝑧)

! +

𝑦)

! , 𝑥! = softmax 𝑥!𝑔 1 +

𝑦)

!

.

• Shared amortized inference network 𝑕, returns the parameters {𝜈$

! , 𝜏$ !} of a

Gaussian over weight vector 𝑥$

! for each class 𝑜:

𝑞 𝑥$

! 𝐸!, 𝜄 = 𝒪 𝜈$ ! , diag(𝜏$ !) ,

where 2#

$

3#

$

= 𝑕,

& ' ∑"%& '

𝑔

1(𝑦",$ ! )

11

VERSA – Gordon et al. ICLR’19

ICML | 2020

Improved architectural design based on TADAM

• Scaled cosine similarity (-SC).

The linear classifier is replaced with the cosine similarity classifier scaled with α.

• Task encoding network (-TEN).

TEN provides task-conditioned batch norm parameters for feature maps in 𝑔

1.

• Auxiliary co-training (-AT).

𝑔

1 is shared with an auxiliary classification task across all meta-train classes.

12

TADAM – Oreshkin et al. NeurIPS’18

ICML | 2020

Experiments: synthetic task

• Hierarchical generative process: 𝑞 𝑥! = 𝒪 0, 1 , and 𝑞 𝑧! 𝑥! = 𝒪(𝑥!, 𝜏4

5).

• 𝑈 = 250 sampled tasks, 𝐿 = 5 support observations 𝐸! = 𝑧"

! "%& '

and 𝑁 = 15 query

• bservations *

𝐸! = + 𝑧)

! "%& *

.

• Posterior over latent variable 𝑟,(𝑥!| 𝐸!) = 𝒪(𝜈/, 𝜏/

5).

• Parameters of the posterior are obtained via inference network with parameters 𝜚:

𝜈/ log 𝜏/

5

= 𝜚& ?

"%& '

𝑧"

! + 𝜚5

13

ICML | 2020

Results: synthetic task

• Exact marginal log-likelihood log 𝑞(*

𝐸!|𝐸!).

• Monte Carlo estimation of log 𝑞(*

𝐸!|𝐸!) with L samples from the prior.

• Variational inference for log 𝑞(*

𝐸!|𝐸!) with L samples from the posteror.

• Monte Carlo requires large sample sets compared to variational inference.

(b) 𝜏! = 0.5

14

estimated / true

ICML | 2020

Experimental setup for real data

• 5-shot and 1-shot, 5-way classification tasks.
• Test data contains 15 query samples per class.
• Evaluation is performed on 5,000 randomly sampled tasks.
• We report the mean accuracy over these tasks, and 95% confidence intervals.

15

ICML | 2020

Comparison with VERSA on miniImageNet

• SAMOVAR-base and VERSA train the same meta-learning model.
• SAMOVAR with separate prior and posterior is inferior to other models.
• SAMOVAR is comparable with VERSA on 1-shot task, and outperforms it on 5-shot task.

16

VERSA – Gordon et al. ICLR’19

ICML | 2020

Comparison with VERSA on miniImageNet

• SAMOVAR-base and VERSA train the same meta-learning model.
• SAMOVAR with separate prior and posterior is inferior to other models.
• SAMOVAR is comparable with VERSA on 1-shot task, and outperforms it on 5-shot task.

16

VERSA – Gordon et al. ICLR’19

ICML | 2020

Comparison with VERSA on miniImageNet

• SAMOVAR-base and VERSA train the same meta-learning model.
• SAMOVAR with separate prior and posterior is inferior to other models.
• SAMOVAR is comparable with VERSA on 1-shot task, and outperforms it on 5-shot task.

16

VERSA – Gordon et al. ICLR’19

ICML | 2020

Comparison with TADAM on miniImageNet

• Both models are trained with auxiliary co-training.
• SAMOVAR consistently improves TADAM across all ablations.

TADAM – Oreshkin et al. NeurIPS’18

17

𝛽: cosine scaling, AT: auxiliary co-training, TEN: task embedding network Additional ablations can be found in the paper.

ICML | 2020

Comparison with TADAM on miniImageNet

• Both models are trained with auxiliary co-training.
• SAMOVAR consistently improves TADAM across all ablations.

Comparison with TADAM on miniImageNet

• Both models are trained with auxiliary co-training.
• SAMOVAR consistently improves TADAM across all ablations.

𝛽: cosine scaling, AT: auxiliary co-training, TEN: task embedding network Additional ablations can be found in the paper.

17

TADAM – Oreshkin et al. NeurIPS’18

ICML | 2020

Comparison with state of the art on miniImageNet

• SAMOVAR demonstrates competitive results with and without data augmentation.
• SAMOVAR is complementary to approaches like CTM [Li et al. CVPR’19] or [Gidaris et al.

ICCV’19]

†: Transductive methods.

18

ICML | 2020

Comparison with state of the art on miniImageNet

• SAMOVAR demonstrates competitive results with and without data augmentation.
• SAMOVAR is complementary to approaches like CTM [Li et al. CVPR’19] or [Gidaris et al.

ICCV’19]

†: Transductive methods.

18

ICML | 2020

Comparison with state of the art on miniImageNet

• SAMOVAR demonstrates competitive results with and without data augmentation.
• SAMOVAR is complementary to approaches like CTM [Li et al. CVPR’19] or [Gidaris et al.

ICCV’19]

†: Transductive methods.

18

ICML | 2020

Summary

• Monte Carlo approximation underestimatea the variance in model parameters.
• We propose SAMOVAR, a meta-learning model based on shared amortized variational

inference.

• Task on synthetic data shows that VI approach preserves stochasticity.
• SAMOVAR combined with TADAM shows competitive results on miniImageNet, FC100.

19

Thank you!

ICML | 2020

Thirty-seventh International Conference

• n Machine Learning