FRACTIONAL UNDERDAMPED LANGEVIN DYNAMICS: Umut im ekli LTCI, Tlcom - - PowerPoint PPT Presentation

Umut Şimşekli

LTCI, Télécom Paris, Institut Polytechnique de Paris

FRACTIONAL UNDERDAMPED LANGEVIN DYNAMICS: RETARGETING SGD WITH MOMENTUM UNDER HEAVY-TAILED GRADIENT NOISE

Umut Şimşekli*, Lingjiong Zhu*, Yee Whye Teh, Mert Gürbüzbalaban

(Florida State University) (University of Oxford) (Rutgers University) *equal contribution

International Conference on Machine Learning, 2020

§ Deep learning (in general) § Optimization Algorithm – Stochastic Gradiend Descent with momentum

DEEP LEARNING & SGD-MOMENTUM

2

network weights non-convex cost function data points minibatch minibatch size

˜ vk+1 =˜ γ˜ vk ˜ ηr ˜ fk+1(xk) xk+1 =xk + ˜ vk+1

step-size (learning rate) stochastic gradient

x? = arg min

x∈Rd

n f(x) , 1 n

n

X

i=1

f (i)(x)

• <latexit sha1_base64="fF9Fa6Oi0eSLx1J4zadUKP9lCRI=">AENnicfZLfbtMwFMbdlcEIsHVwyY1FhbRxUTUDATeTprFJTBpi/Ok2qW4rx3Fa7YTbGe0s/w4vAOvwg1cIW5BJwuZV2KsBTp+PzOd+wcf1HGmTbt9rfaUv3G8s1bK7eDO3fvra41u8f6zRXhHZIylN1GmFNOZO0Y5jh9DRTFIuI05Po7FXBT86p0iyVH80koz2Bh5IljGDjU4PGBI2jPtIGK7gNEVZDwSTiTDCjB9YziJiESGlq+rGDaJcNObIw2fBoEyKjGJZDTj9BlChMQit9jc7F3w5sO3R9CZO+3WCbrlT5Jgq5QaPZbrWnCy4GYRk0QbmOButLX1CcklxQaQjHWnfDdmZ6FivDCKcuQLmGSZneEi7PpRYUN2z0xk5+NhnYpikyn/SwGl2XmGx0HoiIl8psBnpKiuS/2Ld3CQve5bJLDdUksuDkpxDk8Ji4DBmihLDJz7ARDF/V0hG2E/L+GcJkKSfSoElnExbmdRcUKU2LFz1+H5HDyvwos5eFGFxevNcGTfV/H+Fduvsr2p1CKCOTxdgJn0cFw9TcRlQyVsXNXsiu4W4WHc/DQ9RHm2QhXag7iqz89qDbQWM9u+8Fdymc7FwRoj3rfKPrGq9mVGTqie2NL3zPhqiIvhfHR6XdT4IrlmkuJU64LY4dVGy8Gx1ut8Glr692z5s7z0uIr4CF4BDZACF6AHfAaHIEOIOBHbm2Wlurf61/r/+s/7osXaqVmgfg2qr/gOeTHUc</latexit>

r ˜ fk(x) , 1 b X

i∈Ωk

f (i)(x)

velocity (momentum) Momentum decay

UNDERSTANDING SGD-M

§ Theory: better-established for convex problems still in early phase for non-convex problems § Useful approach for analysis à Stochastic Differential Equations (SDE) gradient noise: § If we assume and invoke CLT: § SGD-m à Euler-Maruyama Discretization of the SDE:

3

Uk(x) , r ˜ fk(x) rf(x)

EkUk(x)k2 < 1

Uk ∼ Gaussian

Brownian Motion

dvt = (γvt + rf(xt))dt + r2γ β dBt dxt = vtdt

Inverse temperature friction Underdamped (a.k.a. Kinetic) Langevin Dynamics

Dalalyan&Riou-Durand’18 Gao et al.’18

UNDERDAMPED LANGEVIN DYNAMICS

§ Favorable properties: Targets Boltzmann-Gibbs measure à marginal density Minima of ⇔ Maxima of the target measure As , target concentrates on the global minimum of § Problem: gradient noise à non-Gaussian heavy-tailed in deep nets

4

dvt = (γvt + rf(xt))dt + r2γ β dBt dxt = vtdt

∝ exp(−βf(x))

f

β → ∞

f

• 10
• 5

5 10

x

10-3 10-2 10-1

p(x)

α=1.5 α=1.7 α=2.0

Gaussian when α=2 Infinite variance when α≠2 (Simsekli et al.’19)

DYNAMICS WITH HEAVY-TAILED NOISE

§ Gaussian increments à Brownian motion α-stable increments à α-stable Lévy motion § Problem: doesn’t target the Gibbs measure! The modes are shifted à bias Analytical results for bias when

5

dvt = (γvt− + rf(xt))dt + r2γ β dLα

t

dxt = vtdt

(Sliusarenko et al.’13) (Capała & Dybiec’19)

γ → ∞

500 1000 1500 2000 2500 3000

t

• 60
• 40
• 20

20 40

Lα

t

α=1.5 α=1.7 α=2.0

MAIN QUESTION & GOALS

§ In other words: “retarget SGD-m towards the Gibbs measure” § Make sure à the modes match the minima of Make sure à the dynamics is computationally tractable

Overdamped case à retargeting possible but not tractable (Simsekli’17)

§ Relation to gradient clipping and differential geometric approaches § Asymptotic analysis of the discretization scheme

6

Can we design an SDE that targets the Gibbs measure under α–stable gradient noise?

f

PROPOSED DYNAMICS

§ : fractional Riesz derivative à non-local à often no analytical expression hard to approximate (Simsekli’17) § General recipe: specify the Kinetic energy g à imposes a drift c § Two choices of g: 1) Gaussian 2) α-stable à analytical Riesz derivatives!

7

(c(v, α))i := Dα−2

vi

(ψ(v)∂vig(v)) ψ(v) , ψ(v) := e−g(v)

Theorem 1 (Fractional Underdamped Langevin Dynamics – FULD)

dvt = (γc(vt−, α) + rf(xt))dt + ⇣γ β ⌘1/α dLα

t

dxt = rg(vt)dt

Consider the dynamics with Then, the Boltzmann-Gibbs measure is an invariant measure of this SDE.

Kinetic Energy

Dγ

v

Drift

Gaussian

GAUSSIAN KINETIC ENERGY

§ α=2, recovers standard ULD § Non-Lipschitz, explosive § Uniqueness not guaranteed § Doesn’t have much practical value

8

Gamma fn. Kummer confluent hypergeometric fn.

Let . Then, Theorem 2 (FULD with Gaussian Kinetic energy)

(c(v, α))i = 2

α 2 vi

√π Γ ✓α + 1 2 ◆

1F 1

✓2 − α 2 ; 3 2; v2

i

2 ◆

g(v) = 1 2kvk2

ALPHA-STABLE KINETIC ENERGY

§ Resulting SDE: § Proposition: is Lipschitz for all α. § Uniqueness: with standard conditions on

9

Theorem 3 (FULD with α-Stable Kinetic energy) Let be the pdf of the α-stable distribution. Define the kinetic energy: with . Then, .

e−gα(v)

ψ(v) = e−Gα(v)

(c(v, α))i = vi

Gα(v) = Xd

i=1 gα(vi)

dvt = γvt−dt rf(xt)dt + γ1/αdLα

t ,

dxt = rGα(vt)dt

rGα

f

DISCRETIZATION

§ Analysis: weak convergence of the Euler-Maruyama discretization § Our interest: via

10

vk+1 = ˜ γkvk ηkrf(xk) + ⇣ηkγ β ⌘1/α sk+1 xk+1 = xk + ηkrGα(vk+1)

˜ γk = 1 − γηk

Standard α-stable r.v.

π(h) := EX∼π[h(X)]

¯ πK(h) := PK

k=1 ηkh(xk)

PK

k=1 ηk

Corollary 1 (Weak Convergence of Sample Averages) Assume: , Assume: is Lipschitz + linear growth + standard Lyapunov condition Then, almost surely, as .

limk→∞ ηk = 0

limk→∞ Xk

i=1 ηi = ∞

rf

¯ πK(h) → π(h)

K → ∞

CONNECTIONS

§ Gradient Clipping: used when “exploding gradients” Heavy-tails à exploding grads à clipping improves convergence § Naturally arises in our scheme:

11

§ Natural gradient § Precondition by Fisher Information § The “Composite” gradient: (α=1) with § Can be seen as a (biased) estimator

• f the diagonal of FIM

E[rf(x)rf(x)>]

rG1(r ˜ fk(x)) = Mk(x)−1r ˜ fk(x)

mii = ((r ˜ fk(x))2

i + 1)/2

(Amari’98) (Pascanu et al.’13) (Zhang et al.’19)

EXPERIMENTS – SYNTHETIC

§ f: “Double-well” (4-th order polynomial)

12

α=1.0 α=1.9 α=1.0 α=1.9

EXPERIMENTS – NEURAL NETWORKS

§ Fully connected networks – ReLU activations – Cross Entropy loss § No additional noise – true gradient noise

13

MNIST CIFAR10

CONCLUSION

§ Gradient noise in deep networks can be heavy-tailed § Heavy-tails à might shift the modes § Proposed dynamics à targets the Gibbs measure à no shift § Brings theoretical understanding for gradient clipping

14

THANK YOU FOR YOU ATTENTION!