Stochastic Gradient Descent Many slides attributable to: Prof. - - PowerPoint PPT Presentation

Stochastic Gradient Descent

2

Tufts COMP 135: Introduction to Machine Learning https://www.cs.tufts.edu/comp/135/2020f/

Many slides attributable to: Erik Sudderth (UCI), Emily Fox (UW), Finale Doshi-Velez (Harvard) James, Witten, Hastie, Tibshirani (ISL/ESL books)

• Prof. Mike Hughes

Objectives Today (day 12) Stochastic Gradient Descent

• Review: Gradient Descent
• Repeatedly step downhill until converged
• Review: Training Neural Nets with Backprop
• Backprop = chain rule plus dynamic programming
• L-BFGS : How to step in better direction?
• Stochastic Gradient Descent : How to go fast?

3

Mike Hughes - Tufts COMP 135 - Fall 2020

Review: Gradient Descent in 1D

4

Mike Hughes - Tufts COMP 135 - Fall 2020

input: initial θ ∈ R input: step size α ∈ R+ while not converged: θ ← θ − α d dθJ(θ)

Q: Which direction to step? A: Straight downhill (steepest descent at current location) Q: How far to step in that direction? A: Step size parameter picked in advance, unaware of current location

α ·

• ∂

∂θJ

• <latexit sha1_base64="QD3y6jOAWeQ5sqrhpmn6rqR5jJ8=">ACN3icbVBNSyNBEO3xY9Wsu0Y9emkMC3sKM+7CehS9iAdRMCqkQ6jp1GQaez7orhHCmH/lxb/hTS8eFPHqP7AnieLHPmjq8V4V1fXCXCtLvn/jTU3PzH6bm1+ofV/8XOpvrxybLPCSGzJTGfmNASLWqXYIkUaT3ODkIQaT8Kznco/OUdjVZYe0SDHTgL9VEVKAjmpW98XoPMYuJC9jLjQGNEFf60iMiBLkYMhBXr4xrigGAmGfI8Lo/px1Tqu3XrDb/oj8K8kmJAGm+CgW78WvUwWCaYkNVjbDvycOmW1R2oc1kRhMQd5Bn1sO5pCgrZTju4e8l9O6fEoM+6lxEfq+4kSEmsHSeg6E6DYfvYq8X9eu6Bos1OqNC8IUzleFBWaU8arEHlPGZSkB46ANMr9lcsYXFjkoq65EILPJ38lxvN4E9z4/BvY2t7Esc8W2Pr7DcL2D+2xXbZAWsxyS7ZLbtnD96Vd+c9ek/j1ilvMrPKPsB7fgFLZq1r</latexit>

input: initial θ ∈ R input: step size α ∈ R+ while not converged: θ ← θ − α d dθJ(θ)

D

Review: Gradient Descent in 2D+

5

Mike Hughes - Tufts COMP 135 - Fall 2020

Q: Which direction to step? A: Straight downhill (steepest descent at current location) Q: How far to step in that direction? A: Step size parameter picked in advance, unaware of current location gradient = vector of partial derivatives

θ θ αrθJ(θ)

rθJ(θ) = 

∂ ∂θ0 J ∂ ∂θ1 J

• <latexit sha1_base64="N3+JepYHD0brMFSlWy+mt6ByhW4=">ACfnichVHLatwFJXdV+q+ps2yG7VDStqSqZ0Umk0hNJuQVQqdJDAy5lpzPSMiy0a6DgzGn9Ef6y7f0k01M6bkUegFweHcx86N6+1chTHV0F47/6Dh482HkdPnj57/mLw8tWpqxorcSwrXdnzHBxqZXBMijSe1xahzDWe5ReHy/zZJVqnKvODFjWmJcyMKpQE8lQ2+CkM5BqyVtAcCTp+vL1G7/nXSGgsaBKJHGfKtGAtLpWdpEoLMhW1GBJge7+Ir4uzWLfJhKC/1eYrIRopn3zSFg1m1OaDYbxKF4FvwuSHgxZHyfZ4JeYVrIp0ZDU4NwkiWtK2+U0qdH3bRzWIC9ghMPDZTo0nZlX8e3PDPlRWX9M8RX7PWKFkrnFmXulSXQ3N3OLcl/5SYNFftpq0zdEBq5HlQ0mlPFl7fgU2VRkl54ANIqvyuXc/CWkb9Y5E1Ibn/5LjdHSV7o93vn4cH3o7Nthr9pZts4R9YQfsiJ2wMZPsd/Am+B8DFn4LtwJP62lYdDXbLIbEe7/AYrCwhU=</latexit>

α · ||rθJ(θ)||

Even in one dimension, tough to select step size. Recommendations

• Try multiple values
• Might need different sizes at different locations

Review: Step size matters

6

Mike Hughes - Tufts COMP 135 - Fall 2020 𝑔(𝒚) 𝒚 𝑔(𝒚) 𝒚 𝒚 𝒚

Review: Neural Net as computational graph

2 directions of propagation

Forward: compute loss Backward: compute grad

7

Mike Hughes - Tufts COMP 135 - Fall 2020

Review: Training Neural Nets

8

Mike Hughes - Tufts COMP 135 - Fall 2020

min

w N

X

n=1

E(yn, ˆ y(xn, w))

Training Objective: Gradient Descent Algorithm:

w = initialize_weights_at_random_guess(random_state=0) while not converged: total_grad_wrt_w = zeros_like(w) for n in 1, 2, … N: loss[n], grad_wrt_w[n] = forward_and_backward_prop(x[n], y[n], w) total_grad_wrt_w += grad_wrt_w[n] w = w – alpha * total_grad_wrt_w

How to pick step size reliably? How to go fast on big datasets?

Step size strategy: Slow decay

9

Mike Hughes - Tufts COMP 135 - Fall 2020

input: initial w 2 R input: initial step size s0 2 R+ while not converged: w w strwL(w) st decay(s0, t) t t + 1

Linear decay Exponential decay

Often helpful, requires tuning and hard to get right!

t : number of steps

θ

θ

θ

θ θ αrθJ(θ)

αt

s0 kt αt

s0e−kt αt

αt

αt

αt

Q: How far to step? A: Line search

Find good step size for current location

Search for the best scalar s >= 0, such that:

10

Mike Hughes - Tufts COMP 135 - Fall 2020

min

x f(x)

x

∆x = rxf(x) s∗ = arg min

s≥0 f(x + s∆x) ∆x

Possible step lengths

Step Direction: Goal:

In Python code: scipy.optimize.line_search s = 0.5 s = 1.3 s = 5.1

Can be expensive, but often worth it

11

Mike Hughes - Tufts COMP 135 - Fall 2020

Q: Better direction to step than straight downhill? A: Yes. Modify direction using second-order derivative.

min

θ

J(θ)

∆θ = −J0(θ)

∆θ = − 1 J00(θ)J0(θ)

∆θ = rθJ(θ)

1-D 2-D+ 1st order only decent direction Using 2nd order Newton descent direction

∆θ = H(θ)−1rθJ(θ)

Hessian matrix for J H is a D x D matrix

All second-order partial derivatives

∆θ

Approximate second order method

• Computes first-order gradient vector exactly on provided training dataset
• Computes efficient approximation of Hessian via recent history of steps

L-BFGS: Smarter Gradient Descent

12

Mike Hughes - Tufts COMP 135 - Fall 2020

L-BFGS : Limited Memory Broyden–Fletcher–Goldfarb–Shanno (BFGS)

Q: Which direction to step? A: Downhill, adjusted by curvature at current location Q: How far to step in that direction? A: Efficient line search Step size adjusted to current location (as implemented in SciPy)

Objectives Today (day 12) Stochastic Gradient Descent

13

Mike Hughes - Tufts COMP 135 - Fall 2020

• Review: Gradient Descent
• Repeatedly step downhill until converged
• Review: Training Neural Nets with Backprop
• Backprop = chain rule plus dynamic programming
• L-BFGS : How to step in better direction?
• Stochastic Gradient Descent : How to go fast?

Stochastic Estimate of Loss Function

• Standard “full-dataset” objective
• Rewrite as an “expected value”

14

Mike Hughes - Tufts COMP 135 - Fall 2020

L(w) = 1 N

N

X

n=1

Ln(xn, yn, w)

L(w) = Exi,yi∼Unif({xn,yn}N

n=1) [Li(xi, yi, w)] Empirical distribution over

• ur N training examples

Each index i selected with probability 1/N

Stochastic Estimate of Loss Function

• Standard “full-dataset” objective
• Rewrite as an “expected value”
• Approximate with one randomly-drawn sample

15

Mike Hughes - Tufts COMP 135 - Fall 2020

L(w) = 1 N

N

X

n=1

Ln(xn, yn, w)

L(w) = Exi,yi∼Unif({xn,yn}N

n=1) [Li(xi, yi, w)]

L(w) ≈ Li(xi, yi, w) xi, yi ∼ Unif({xn, yn}N

n=1)

Each index i selected with probability 1/N

Stochastic Estimate of Gradient

• Standard “full-dataset” gradient
• Approximate with one randomly-drawn sample

16

Mike Hughes - Tufts COMP 135 - Fall 2020 Each index i selected with probability 1/N

rwL(w) = 1 N

N

X

n=1

rwLn(xn, yn, w)

rwL(w) ⇡ rwLi(xi, yi, w) xi, yi ⇠ Unif({xn, yn}N

n=1)

Gradient Descent using Noisy Estimates of the “True” Gradient

17

Mike Hughes - Tufts COMP 135 - Fall 2020

Intuition As long as each noisy step takes us in a direction that is correct on average, we will

• ver many steps make progress

in minimizing the loss. Formal guarantees Our Monte Carlo estimate of gradient is unbiased, so its expected value is exactly equal to the true whole-dataset gradient

18

Mike Hughes - Tufts COMP 135 - Fall 2020

Stochastic gradient descent (SGD)

using one example at a time

input: initial w 2 R input: step size s 2 R+ while not converged: {xi, yi} ⇠ Unif({xn, yn}N

n=1)

w w srwL(xi, yi, w)

Should we only use one example i to estimate gradient?

α

α

19

Mike Hughes - Tufts COMP 135 - Fall 2020

SGD with minibatches of size B

input: initial w 2 R input: step size s 2 R+ while not converged: {xi, yi} ⇠ Unif({xn, yn}N

n=1)

w w srwL(xi, yi, w)

B = 1 recovers previous slide. B = N recovers standard GD. In between: trade off quality of estimate with cost of estimate

α

α

{xb, yb}B

b=1

Unif({xn, yn}N

n=1,

size = B, replace = False)

α · 1 B

B

X

i=1

rwL(xi, yi, w)

Objectives Today (day 12) Stochastic Gradient Descent

20

Mike Hughes - Tufts COMP 135 - Fall 2020

• Review: Gradient Descent
• Repeatedly step downhill until converged
• Review: Training Neural Nets with Backprop
• Backprop = chain rule plus dynamic programming
• Line Search: How to take step of good size?
• L-BFGS : How to step in better direction?
• Stochastic Gradient Descent : How to go fast?

Breakout to Lab

Warning: Notation can be confusing

• Alpha in these slides refers to step size (aka

learning rate)

• In sklearn’s MLPClassifier, alpha refers to a

different hyperparameter: the scalar strength

• f a small L2 penalty on the magnitudes of the

weights

• To set step size in sklearn:

learning_rate_init=0.5

21

Mike Hughes - Tufts COMP 135 - Fall 2020