Reconciling Traditional Optimization Analyses and Modern Deep - - PowerPoint PPT Presentation
Reconciling Traditional Optimization Analyses and Modern Deep Learning: the Intrinsic Learning Rate Kaifeng Lyu * Sanjeev Arora Zhiyuan Li * Tsinghua University Princeton University & IAS Princeton University NeurIPS Dec, 2020 *
NeurIPS Dec, 2020
*οΌ these authors contribute equally
Scale invariance: π π₯!; πͺπ = π ππ₯#; πͺπ
arising from SGD in normalized nets. Our analysis show ππ½, i.e. WD*LR, is a better measure for the speed of learning, and thus we call ππ½ the intrinsic Learning Rate.
πͺπ: batch of training data at iteration t π₯": weights of neural net π: Weight Decay (WD) factor, aka β# regularization π": Learning Rate (LR) at iteration t
βπ π₯!; πͺπ = πβπ ππ₯#; πͺπ ; β$π π₯!; πͺπ = π$β$π ππ₯#; πͺπ , βπ > π
Wisdom 2: To achieve the best generalization, LR must be large initially for quite a few epochs. Wisdom 1: As LR β π,
converges to a deterministic path (Gradient Flow) along which training loss strictly decreases. Wisdom 3: Modeling SGD via SDE with a fixed Gaussian
process that mixes to some Gibbs-like distribution.
Figure from blog post by Holden Lee and Andrej Risteski
Resnet with BN trained by full-batch GD + WD on subsampled CIFAR10 Resnet with BN trained by SGD with diff. LR schedules on CIFAR10
Wisdom 2: To achieve the best generalization, LR must be large initially for quite a few epochs. Wisdom 1: As LR β 0,
converges to a deterministic path (Gradient Flow) along which training loss strictly decreases. Wisdom 3: Modeling SGD via SDE with a fixed Gaussian
process that mixes to some Gibbs-like distribution.
Figure from blog post by Holden Lee and Andrej Risteski
Resnet with BN trained by full-batch GD + WD on subsampled CIFAR10 Resnet with BN trained by SGD with diff. LR schedules on CIFAR10
Wisdom 2: To achieve the best generalization, LR must be large initially for quite a few epochs. Wisdom 1: As LR β 0,
converges to a deterministic path (Gradient Flow) along which training loss strictly decreases. Wisdom 3: Modeling SGD via SDE with a fixed Gaussian
process that mixes to some Gibbs-like distribution.
Figure from blog post by Holden Lee and Andrej Risteski
Resnet with BN trained by full-batch GD + WD on subsampled CIFAR10 Resnet with BN trained by SGD with diff. LR schedules on CIFAR10
(Against Wisdom 2) Small LR can generalize equally well as large LR (Against Wisdom 3) Random walk/SDE view of SGD is way
mixing as traditionally understood, at least within normal training times. Loss β as hessian β when π₯ β 0
Stochastic Weight Averaging improves acc. ( Izmailov et al., 2018)
SWA rules out that SGD as a diffusion process is mixing to a unique global equilibrium.
(Against Wisdom 1): Full batch gradient descent β gradient flow.
Resnet with BN trained by full-batch GD + WD on subsampled CIFAR10. Resnet with BN trained by SGD with diff. LR schedules on CIFAR10
Same setting, longer training! Same setting, longer training!
(Against Wisdom 2) Small LR can generalize equally well as large LR (Against Wisdom 1): Full batch gradient descent β gradient flow. (Against Wisdom 3) Random walk/SDE view of SGD is way
mixing as traditionally understood, at least within normal training times. Loss β as hessian β when π₯ β 0
Stochastic Weight Averaging improves acc. ( Izmailov et al., 2018)
SWA rules out that SGD as a diffusion process is mixing to a unique global equilibrium.
Resnet with BN trained by full-batch GD + WD on subsampled CIFAR10. Resnet with BN trained by SGD with diff. LR schedules on CIFAR10
Same setting, longer training! Same setting, longer training!
(Against Wisdom 2) Small LR can generalize equally well as large LR (Against Wisdom 3) Random walk/SDE view of SGD is way
mixing as traditionally understood, at least within normal training times. Loss β as hessian β when π₯ β 0
Stochastic Weight Averaging improves acc. ( Izmailov et al., 2018)
SWA rules out that SGD as a diffusion process is mixing to a unique global equilibrium.
(Against Wisdom 1): Full batch gradient descent β gradient flow.
(Against Wisdom 2) Small LR can generalize equally well as large LR (Against Wisdom 1): Full batch gradient descent β gradient flow. (Against Wisdom 3) Random walk/SDE view of SGD is way
mixing as traditionally understood, at least within normal training times.
Loss β as hessian β when π₯ β 0
Stochastic Weight Averaging improves acc. ( Izmailov et al., 2018)
SWA rules out that SGD as a diffusion process is mixing to a unique global equilibrium.
Main Theorem:
Effective Learning rate
Intrinsic Learning Rate
πΏ" = |π
"|$
π#
Three implications of main theorem:
"π.π = π½ πΏπ π.
2. ππ alone determines the evolution of the system
) *#) time (steps)
dynamics of norm dynamics of direction
/+).
/+) steps, i.e.,
any input data, but itβs possible to distinguish them via looking at their parameters (e.g., via Stochastic Weight Averaging).
Figure from βWide Resnetβ paper (Zagoruyko & Komodakis, 2017)
Reason: Effective LR first divided by 10,but its stationary value only decrease by roughly ππ.