An Exponential LR Schedule for Deep Learning An Exponential LR - - PowerPoint PPT Presentation

An Exponential LR Schedule for Deep Learning

(Strange Behavior of Normalization Layers)

Sanjeev Arora

Princeton & IAS

Zhiyuan Li

Princeton

An Exponential LR Schedule for Deep Learning

(Strange Behavior of Normalization Layers)

Sanjeev Arora

Princeton & IAS

Zhiyuan Li

Princeton

ICLR April, 2019

𝑥 !"# ← 𝑥 ! − 𝜃 ⋅ ∇𝑀 𝑥 ! Traditional: Start with some LR; decay over time.

(extensive literature in optimization justifying this)

Confusingly, exotic LR schedules also reported to work: triangular[Smith,2015 ], cosine [Loshchilov&Hutter, 2016] etc.; no justification in theory.

Lea Learning g rate e in traditional optimi mization

Result 1 (empirical): Possible to train today’s deep architectures, while growing LR exponentially (i.e., at each iteration multiply by (1 + 𝑑) for some 𝑑 > 0) Result 2 (theory): Mathematical proof that exponential LR schedule can yield (in function space*) every net produced by existing training schedules.

(* In all nets that use batch norm [Ioffe-Szegedy’13] or any other normalization scheme.)

Raises questions for theory; highlights importance of trajectory analysis ( vs landscape)

Th This is wo work: Ex Exponential LR LR sch chedules

PreResNet32 on CIFAR10 with fixed LR converted to Exp LR with 𝜃! = 0.1×1.481!,

Learning Rate

“General training algorithm; fixed LR”

Momentum ℓ! regularizer (aka Weight decay)

Thm: For nets w/ batch norm or layer norm, following is equivalent to above : weight decay 0, momentum 𝛿, and LR schedule 𝜃! = 𝜃𝛽$%!$# where 𝛽 (𝛽 < 1) is nonzero root of

x2 − (1 + γ − λη)x + γ = 0,

(proof shows nets in the new trajectory are equivalent in function space to nets in original trajectory)

Ma Main Th Thm (original sch chedule has fixed LR)

Stochastic Loss at round t

Original schedule(Step Decay): K phases. In phase 𝐽, iteration [𝑈&, 𝑈&"# − 1], use LR 𝜃&

∗.

Mo More e gen ener eral (ori riginal schedule has varyi ying LR)

Thm: Step Decay can be realized with tapered exponential LR schedule

Tapered Exponential LR schedule (TEXP):

Exponential growing LR in each phase + when entering a new phase:

• Switching to a slower exponential growing rate;
• Dividing the current LR by some constant.

(See details in the paper)

Observation: Batch norm + fixed final layer è Loss is scale invariant (True for Feed-forward nets, Resnets, DenseNets, etc.; see our appendix) 𝑀 𝑑 ⋅ 𝜄 = 𝑀 𝜄 for every vector 𝜄 of net parameters and 𝑑 > 0

Ke Key conce cept in proof: Sc Scale-in invar arian iant tr traini ning ng loss loss

𝜾𝒖 −𝛂𝐌(𝜾𝒖) 𝒅𝜾𝒖 −𝛂𝐌(𝐝𝜾𝒖) Lemma: Gradient for such losses satisfies Scale-invariant loss fn sufficient for state of the art deep learning![Hoffer et al,18]

𝜾𝒖

#, 𝜽′ → (𝒅𝜾𝒖, 𝒅𝟑𝜽)

𝜽 𝜾𝒖

# = 𝒅𝜾𝒖

𝛂𝐌(𝐝𝜾𝒖) = 𝐝"𝟐 𝛂𝐌(𝜾𝒖)

𝜾𝒖 −𝛂𝐌(𝜾𝒖) 𝜽# = c𝟑𝜽 −𝐝𝟑𝜽𝛂𝐌(𝐝𝜾𝒖) −𝜽𝛂𝐌(𝐝𝜾𝒖) 𝜽 𝜾𝒖%𝟐

#

, 𝜽′ → (𝒅𝜾𝒖%𝟐, 𝒅𝟑𝜽) 𝜾𝒖%𝟐

#

= 𝜾𝒖

# − 𝜽′𝛂𝐌(𝜾𝒖 #)

𝜾𝒖%𝟐 = 𝜾𝒖%𝟐 − 𝜽𝛂𝐌(𝜾𝒖) 𝜽# = c𝟑𝜽 𝜾𝒖%𝟐

#

𝜾𝒖%𝟐

Proof sketch ch when momentum=0

𝜾𝒖

0, 𝜽′ → (𝒅𝜾𝒖, 𝒅𝟑𝜽)

𝜾𝒖"𝟐 , 𝜽′ → (𝒅𝜾𝒖"𝟐, 𝒅𝟑𝜽) 𝜾𝒖"𝟐 = 𝜾𝒖

0 − 𝜽′𝛂𝐌(𝜾𝒖 0)

𝜾𝒖"𝟐 = 𝜾𝒖"𝟐 − 𝜽𝛂𝐌(𝜾𝒖)

𝜾𝒖

", 𝜽′

𝜾𝒖, 𝜽 𝜾𝒖#𝟐, 𝜽 𝜾𝒖#𝟐

"

, 𝜽′

Wa Warm-up: up: Equivalence ce of

• f mo

momen mentum-fr free ca case

State =

Equivalent scaling: 𝚸𝟐

𝐝 ∘ 𝚸𝟑 𝒅𝟑

𝜾𝒖

", 𝜽′

𝜾𝒖, 𝜽 𝜾𝒖#𝟐, 𝜽 𝜾𝒖#𝟐

"

, 𝜽′

𝐇𝐄𝐮 𝐇𝐄𝐮 𝚸𝟐

𝐝 ∘ 𝚸𝟑 𝒅𝟑

Wa Warm-up: up: Equivalence ce of

• f mo

momen mentum-fr free ca case

(𝝇 = 𝟐 − 𝝁𝜽) 𝜾, 𝜽 𝜾, 𝝇$𝟐𝜽 𝚸𝟑

𝝇"𝟐

𝜾 − 𝝇$𝟐𝜽𝛂𝐌𝐮(𝜾), 𝝇$𝟐𝜽 𝐇𝐄𝐮 𝚸𝟑

𝝇 ∘ 𝚸𝟐 𝝇

𝝇$𝟐𝜾 − 𝜽𝛂𝐌𝐮(𝜾), 𝜽

Wa Warm-up: up: Equivalence ce of

• f mo

momen mentum-fr free ca case

(𝝇 = 𝟐 − 𝝁𝜽) = 𝚸𝟑

𝝇"𝟐

𝐇𝐄𝐮 𝚸𝟑

𝝇 ∘ 𝚸𝟐 𝝇

Theorem: GD + WD + constant LR= GD + Exp LR.

𝚸𝟐

𝝇"𝒖 ∘ 𝚸𝟑 𝝇"𝟑𝒖 ∘ 𝑯𝑬𝒖$𝟐 𝝇

∘ ⋯ ∘ 𝑯𝑬𝟏

𝝇 = 𝚸𝟑 𝝇"𝟐 ∘ 𝐇𝐄𝐮$𝟐 ∘ 𝚸𝟑 𝝇"𝟑 ∘ ⋯ ∘ 𝐇𝐄𝟐 ∘ 𝚸𝟑 𝝇"𝟑𝐇𝐄𝟏 ∘ 𝚸𝟑 𝝇"𝟐

𝜾𝟐 𝜾𝟏 = 𝜾𝟏

2

𝜾𝟑 𝜾𝟒 𝜾𝟐

2

𝜾𝟑

2

𝜾𝟒

2

Proof:

𝚸𝟐

𝝇"𝟐

∘ 𝚸𝟑

𝝇"𝟑

𝚸𝟑

𝝇"𝟐

𝚸𝟑

𝝇"𝟐

=

Con Conclusion

• ns
• Scale-Invariance (provided by BN) makes the training procedure

incredibly robust to LR schedules, even to exponentially growing schedules.

• Space of good LR schedules in current architectures is vast (hopeless

to search for best schedule??)

• Current ways of thinking about training/optimization should be

rethought;

• should focus on trajectory, not landscape.