Geometric losses for distributional learning Arthur Mensch (1) , - - PowerPoint PPT Presentation

Geometric losses for distributional learning

Arthur Mensch(1), Mathieu Blondel(2), Gabriel Peyr´ e(1)

(1) ´

Ecole Normale Sup´ erieure, DMA Centre National pour la Recherche Scientifique Paris, France

(2) NTT Communication Science Laboratories

Kyoto, Japan

June 12, 2019

1

Introduction

2

Fenchel-Young losses for distribution spaces

3

Geometric softmax from Sinkhorn negentropies

4

Applications

Introduction: Predicting distributions

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 1 / 16

Introduction: Predicting distributions

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 1 / 16

Introduction: Predicting distributions

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 1 / 16

Introduction: Predicting distributions

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 1 / 16

Introduction: Predicting distributions

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 1 / 16

Introduction: Predicting distributions

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 1 / 16

Contribution: losses and links for continuous metrized output

Handling output geometry Link and loss with cost between classes C : Y × Y → R Output distribution over continuous space Y

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 2 / 16

Contribution: losses and links for continuous metrized output

Handling output geometry Link and loss with cost between classes C : Y × Y → R Output distribution over continuous space Y New geometric losses and associated link functions: 1 Construction from duality between distributions and scores

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 2 / 16

Contribution: losses and links for continuous metrized output

Handling output geometry Link and loss with cost between classes C : Y × Y → R Output distribution over continuous space Y New geometric losses and associated link functions: 1 Construction from duality between distributions and scores 2 Need: Convex functional on distribution space Provided by regularized optimal transport

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 2 / 16

Background: learning with a cost over outputs Y

Cost augmentation of losses1, 2: Convex cost-aware loss Lc : [1, d] × Rd → R

!

△ Undefined link functions: Rd → △d: what to predict at test time ?

1Ioannis Tsochantaridis et al. “Large margin methods for structured and interdependent output variables”. In: JMLR (2005). 2Kevin Gimpel and Noah A Smith. “Softmax-margin CRFs: Training log-linear models with cost functions”. In: NAACL. 2010.

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 3 / 16

Background: learning with a cost over outputs Y

Cost augmentation of losses1, 2: Convex cost-aware loss Lc : [1, d] × Rd → R

!

△ Undefined link functions: Rd → △d: what to predict at test time ?

Use a Wasserstein distance between output distributions3: Ground metric C defines a distance WC between distributions Prediction with a softmax link ℓ(α, f) WC(softmax(f), α))

!

△ Non-convex loss and costly to compute

1Ioannis Tsochantaridis et al. “Large margin methods for structured and interdependent output variables”. In: JMLR (2005). 2Kevin Gimpel and Noah A Smith. “Softmax-margin CRFs: Training log-linear models with cost functions”. In: NAACL. 2010. 3Charlie Frogner et al. “Learning with a Wasserstein loss”. In: NIPS. 2015.

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 3 / 16

1

Introduction

2

Fenchel-Young losses for distribution spaces

3

Geometric softmax from Sinkhorn negentropies

4

Applications

Predicting distributions from topological duality

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 4 / 16

Predicting distributions from topological duality

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 4 / 16

Predicting distributions from topological duality

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 4 / 16

Predicting distributions from topological duality

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 4 / 16

Predicting distributions from topological duality

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 4 / 16

Predicting distributions from topological duality

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 4 / 16

All you need is a convex functional

Fenchel-Young losses4 5: Convex function Ω : △d → R

4John C. Duchi et al. “Multiclass Classification, Information, Divergence, and Surrogate Risk”. In: Annals of Statistics (2018). 5Mathieu Blondel et al. “Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms”. In: AISTATS. 2019.

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 5 / 16

All you need is a convex functional

Fenchel-Young losses4 5: Convex function Ω : △d → R and conjugate Ω⋆(f) = min

α∈△d Ω(α) − α, f

4John C. Duchi et al. “Multiclass Classification, Information, Divergence, and Surrogate Risk”. In: Annals of Statistics (2018). 5Mathieu Blondel et al. “Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms”. In: AISTATS. 2019.

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 5 / 16

All you need is a convex functional

Fenchel-Young losses4 5: Convex function Ω : △d → R and conjugate Ω⋆(f) = min

α∈△d Ω(α) − α, f

ℓΩ(α, f) = Ω(α) + Ω⋆(f) − α, f 0

4John C. Duchi et al. “Multiclass Classification, Information, Divergence, and Surrogate Risk”. In: Annals of Statistics (2018). 5Mathieu Blondel et al. “Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms”. In: AISTATS. 2019.

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 5 / 16

All you need is a convex functional

Fenchel-Young losses4 5: Convex function Ω : △d → R and conjugate Ω⋆(f) = min

α∈△d Ω(α) − α, f

ℓΩ(α, f) = Ω(α) + Ω⋆(f) − α, f 0 Define link functions between dual and primal ∇Ω(α) = argmin

f∈Rd

ℓΩ(α, f) ∇Ω⋆(f) = argmin

α∈△d ℓΩ(α, f)

4John C. Duchi et al. “Multiclass Classification, Information, Divergence, and Surrogate Risk”. In: Annals of Statistics (2018). 5Mathieu Blondel et al. “Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms”. In: AISTATS. 2019.

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 5 / 16

Discrete canonical example: Shannon entropy

Ω(α) = −H(α) =

d

• i=1

αi log αi Ω∗(f) = logsumexp(f)

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 6 / 16

Discrete canonical example: Shannon entropy

Ω(α) = −H(α) =

d

• i=1

αi log αi Ω∗(f) = logsumexp(f)

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 6 / 16

Discrete canonical example: Shannon entropy

Ω(α) = −H(α) =

d

• i=1

αi log αi Ω∗(f) = logsumexp(f)

Not defined on continuous distributions, cost-agnostic

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 6 / 16

1

Introduction

2

Fenchel-Young losses for distribution spaces

3

Geometric softmax from Sinkhorn negentropies

4

Applications

Sinkhorn entropies from regularized optimal transport

Self regularized optimal transportation distance: ΩC(α) = −1 2OTC,ε=2(α, α) = − max

f∈C(Y)α, f − logα ⊗ α, exp(f ⊕ f − C

2 )

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 7 / 16

Sinkhorn entropies from regularized optimal transport

Self regularized optimal transportation distance: ΩC(α) = −1 2OTC,ε=2(α, α) = − max

f∈C(Y)α, f − logα ⊗ α, exp(f ⊕ f − C

2 )

Continuous convex

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 7 / 16

Sinkhorn entropies from regularized optimal transport

Self regularized optimal transportation distance: ΩC(α) = −1 2OTC,ε=2(α, α) = − max

f∈C(Y)α, f − logα ⊗ α, exp(f ⊕ f − C

2 )

Continuous convex Special cases ε → ∞: MMD autocorrelation

C =

    

∞ . . . ∞ ... . . . ... ...

    

Shannon entropy Gini index

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 7 / 16

Dual mapping from Sinkhorn negentropy

Sinkhorn entropy: Ω(α) = − max

f∈C(Y)α, f−logα⊗α, e

f⊕f−C 2

• Arthur Mensch, Mathieu Blondel, Gabriel Peyr´

e Geometric losses for distributional learning 8 / 16

Returning to primal: geometric softmax

Ω∗ = g-logsumexp : f → − log min

α∈M+

1 (Y)α ⊗ α, exp(−f ⊕ f + C

2 ) ∇Ω∗ = geometric-softmax. Minimizes a simple quadratic.

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 9 / 16

Geomtric loss construction and computation

Training with the geometric logistic loss:

ℓC(α, f) = geometric-LSEC(f) + sinkhorn-negentropyC(α) − α, f

Tractable discrete Y: use mirror descent/L-BFGS 10× as costly as a softmax Backpropagation: ∇Ω⋆ = geometric-softmax Continuous case: Frank-Wolfe scheme, adding one Dirac at each iteration

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 10 / 16

Properties of the geometric-softmax

−2 2

f1 − f2

0.0 0.2 0.4 0.6 0.8 1.0

∇Ω⋆(f)1 C = γ γ

• 0.00

0.25 0.50 0.75 1.00

α

0.0 0.2 0.4 0.6

−Ω([α, 1 − α])

γ = ∞ γ = 0.1 γ = 2

∇Ω⋆ returns from Sinkhorn potentials: ∇Ω⋆ ◦ ∇Ω = Id

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 11 / 16

Properties of the geometric-softmax

−2 2

f1 − f2

0.0 0.2 0.4 0.6 0.8 1.0

∇Ω⋆(f)1 C = γ γ

• 0.00

0.25 0.50 0.75 1.00

α

0.0 0.2 0.4 0.6

−Ω([α, 1 − α])

γ = ∞ γ = 0.1 γ = 2

∇Ω⋆ returns from Sinkhorn potentials: ∇Ω⋆ ◦ ∇Ω = Id ∇Ω ◦ ∇Ω∗ projects f onto F the set of symmetric Sinkhorn potentials Sparse ∇Ω∗(f) ← minimization on the simplex

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 11 / 16

Properties of the geometric-softmax ε → 0 Mode finding ε → ∞ Positive deconvolution

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 12 / 16

Consistent learning with the geometric logistic loss Bregman divergence from Sinkhorn negentropy: Dg(α|β) = Ω(α) − Ω(β) − ∇Ω(α), α − β.

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 13 / 16

Consistent learning with the geometric logistic loss Bregman divergence from Sinkhorn negentropy: Dg(α|β) = Ω(α) − Ω(β) − ∇Ω(α), α − β. Sample distribution (xi, αi)i ∈ X × M+

1 (Y).

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 13 / 16

Consistent learning with the geometric logistic loss Bregman divergence from Sinkhorn negentropy: Dg(α|β) = Ω(α) − Ω(β) − ∇Ω(α), α − β. Sample distribution (xi, αi)i ∈ X × M+

1 (Y).

Fisher consistency: min

β:X→M+

1 (Y) E [Dg (α, β (x))] =

min

g:X→C(Y) E [ℓg (α, ∇Ω∗ (g (x)))]

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 13 / 16

1

Introduction

2

Fenchel-Young losses for distribution spaces

3

Geometric softmax from Sinkhorn negentropies

4

Applications

Applications: variational auto-encoder

Goal: Generate nearly 1D distribution in 2D images Dataset: Google Quickdraw Traditional sigmoid activation layer ← replaced by geometric softmax

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 14 / 16

Applications: variational auto-encoder

Goal: Generate nearly 1D distribution in 2D images Dataset: Google Quickdraw Traditional sigmoid activation layer ← replaced by geometric softmax

Deconvolutional effect Cost-informed non-linearity

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 14 / 16

Applications: variational auto-encoders

softmax g-softmax Reconstruction Generation

Better defined generated images

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 15 / 16

Conclusion

Geometric softmax: New loss and projector onto output probabilities Discrete/continuous, aware of a cost between outputs Fenchel duality in Banach spaces + regularized optimal transport Application in VAE and ordinal regression

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 16 / 16

Conclusion

Geometric softmax: New loss and projector onto output probabilities Discrete/continuous, aware of a cost between outputs Fenchel duality in Banach spaces + regularized optimal transport Application in VAE and ordinal regression Future directions: How to improve computation methods (continuous FW) Geometric logisitic loss in super resolution6

6Nicholas Boyd et al. “DeepLoco: Fast 3D localization microscopy using neural networks”. In: BioRxiv (2018), p. 267096.

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 16 / 16

Mathieu Blondel Gabriel Peyr´ e

Poster # 179

Arthur Mensch, Mathieu Blondel, Gabriel Peyr´ e Geometric losses for distributional learning 16 / 16