On the Noisy Gradient Descent that Generalizes as SGD Jingfeng Wu , - - PowerPoint PPT Presentation

On the Noisy Gradient Descent that Generalizes as SGD

Jingfeng Wu, Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, Zhanxing Zhu

Johns Hopkins University, Missouri University of Science and Technology, Baidu Research, Styling AI, Peking University

Stochastic gradient descent (SGD)

L(✓) = 1 n

n

X

i=1

`(xi; ✓)

1 b X

i∈Bt

rθ `(xi; ✓t)

Loss function

GD

SGD

(unbiased) gradient noise

vsgd(θt)

θt+1 = θt η z }| { ˜ g(θt) = θt η rθ L(θt) | {z } η (˜ g(θt) rθ L(θt)) | {z }

Noise matters!

CIFAR-10, ResNet-18, w/o weight decay, w/o data augmentation

SGD >> GD

• How?

<= Still open…

• Which?

<= This work!

Which noise matters?

• 1. Magnitude

<= YES! (e.g., Jastrzkebski et al. 2017)

• 2. Covariance structure <= YES! (e.g., Zhu et al. 2018)
• 3. Distribution class

<= ? No!!! (this work) Bernoulli? Gaussian? Levy?...

vsgd(θ) = ˜ g(θ) rθ L(θ)

Intuition

• nly depends on the first two moments of 𝜄!, which only

depend on the first two moments of 𝑤(𝜄).

θt+1 = θt η rθ L(θt) | {z } ηv(θt)

Linear

x,θT [`(x; ✓T ) − `(x; ✓∗)]

For quadratic loss, the generalization error

Noise matters! But noise class does not!!!

A closer look at the noise of SGD

rθ L(θ) · 1 n

rθ L(θ) · Wsgd

Sampling noise

• Gradient matrix
• Sampling vector
• Sampling noise

rθ L(✓) = (rθ `(x1; ✓), . . . , rθ `(xn, ✓))

Wsgd : #1 b = b, #0 = n − b

Vsgd = Wsgd − 1 n

vsgd(θ) | {z } = ˜ g(θ) |{z} rθ L(θ) | {z } = rθ L(θ) · Vsgd |{z}

Gradient noise

Gradient noise vs. sampling noise

Gradient noise

• State-dependent
• Noise of gradient

Gradient matrix

• State-dependent
• Deterministic

Sampling noise

• State-independent
• Noise of mini-batch

sampling

v(θ) |{z} = rθ L(θ) | {z } · V |{z}

Noisy gradient descent

Option 1: use gradient noise v(θ)

v(θ) = rθ L(θ) · V

Option 2: use sampling noise

• in the same magnitude/covariance
• from different classes

L J θt+1 = θt η rθ L(θt) | {z } η v(θ) |{z}

GD noise with

Multiplicative SGD (MSGD)

θt+1 = θt η rθ L(θt) · W, W = 1 n + V

Algorithm:

• 1. Generate sampling vector
• 2. Compute randomized loss
• 3. Compute stochastic gradient
• 4. Update parameters

˜ L(θ) = L(θ) · W

rθ ˜ L(θ)

θ θ η rθ ˜ L(θ)

W = 1/n + V

Injecting noise by MSGD

• 1. SGD class
• 2. Gaussian class
• 3. “Bernoulli” class
• 4. “Sparse Gaussian” class

Wsgd : #1 b = b, #0 = n − b

WG ∼ N (1/n, Var [Wsgd])

✓ W(i)

B = 1

b ◆ = b n, ⇣ W(i)

B = 0

⌘ = 1 − b n

Mini-batch + Gaussian noise

θt+1 = θt η rθ L(θt) · W, W = 1 n + V

Experiments

Small SVHN. More results are available in the paper!

Gaussian noise Sparse Gaussian noise Bernoulli noise SGD noise

Take Home

• 1. Noise class is not crucial
• 2. Multiplicative SGD algorithm
• 3. Sampling noise perspective

Get the paper!

Join our poster session for more details!