Characterization of Convex Objective Functions and Optimal Expected - PowerPoint PPT Presentation

Characterization of Convex Objective Functions and Optimal Expected Convergence Rates of SGD Phuong Ha Nguyen 1 Marten van Dijk 1 , Lam M. Nguyen 2 and Dzung T. Phan 2 Marten Lam P. Ha Dzung 1. Secure Computation Laboratory, ECE, University of Connecticut 2. IBM Research, Thomas J. Watson Research Center International Conference on Machine Learning (ICML) Long Beach, California, 2019

Problem Setting § Solve $∈& ' {𝐺(𝑥) = 𝐹 𝜊 [𝑔(𝑥; 𝜊)]} min § Assumptions ­ Convex: 𝑔 𝑥; 𝜊 − 𝑔 𝑥 6 ; 𝜊 ≥ 𝛼𝑔 𝑥 6 ; 𝜊 , 𝑥 − 𝑥 6 ­ Smooth: ||𝛼𝑔 𝑥; 𝜊 − 𝛼𝑔 𝑥 6 ; 𝜊 || ≤ 𝑀||𝑥 − 𝑥 6 || § Find a 𝑥 = close to 𝑋 ∗ = {𝑥 ∗ ∈ 𝑆 A ∶ ∀ $∈& ' , 𝐺 𝑥 ≥ 𝐺 𝑥 ∗ } § Problem: Characterize Expected Convergence Rates E ∗ ∈F ∗ ||w H − w ∗ || I 𝐹 inf and 𝐹[𝐺(𝑥 = ) − 𝐺(𝑥 ∗ )] 2

Problem Setting § Solve $∈& ' {𝐺(𝑥) = 𝐹 𝜊 [𝑔(𝑥; 𝜊)]} min § Assumptions ­ Convex: 𝑔 𝑥; 𝜊 − 𝑔 𝑥 6 ; 𝜊 ≥ 𝛼𝑔 𝑥 6 ; 𝜊 , 𝑥 − 𝑥 6 ­ Smooth: ||𝛼𝑔 𝑥; 𝜊 − 𝛼𝑔 𝑥 6 ; 𝜊 || ≤ 𝑀||𝑥 − 𝑥 6 || § Find a 𝑥 = close to 𝑋 ∗ = {𝑥 ∗ ∈ 𝑆 A ∶ ∀ $∈& ' , 𝐺 𝑥 ≥ 𝐺 𝑥 ∗ } § Problem: Characterize Expected Convergence Rates E ∗ ∈F ∗ ||w H − w ∗ || I 𝐹 inf and 𝐹[𝐺(𝑥 = ) − 𝐺(𝑥 ∗ )] 3

Problem Setting § Solve $∈& ' {𝐺(𝑥) = 𝐹 𝜊 [𝑔(𝑥; 𝜊)]} min § Assumptions ­ Convex: 𝑔 𝑥; 𝜊 − 𝑔 𝑥 6 ; 𝜊 ≥ 𝛼𝑔 𝑥 6 ; 𝜊 , 𝑥 − 𝑥 6 ­ Smooth: ||𝛼𝑔 𝑥; 𝜊 − 𝛼𝑔 𝑥 6 ; 𝜊 || ≤ 𝑀||𝑥 − 𝑥 6 || § Find a 𝑥 = close to 𝑋 ∗ = {𝑥 ∗ ∈ 𝑆 A ∶ ∀ $∈& ' , 𝐺 𝑥 ≥ 𝐺 𝑥 ∗ } § Problem: Characterize Expected Convergence Rates E ∗ ∈F ∗ ||w H − w ∗ || I 𝐹 inf and 𝐹[𝐺(𝑥 = ) − 𝐺(𝑥 ∗ )] 4

Problem Setting § Solve $∈& ' {𝐺(𝑥) = 𝐹 𝜊 [𝑔(𝑥; 𝜊)]} min § Assumptions Stochastic Gradient Descend (SGD): ­ Convex: 𝑔 𝑥; 𝜊 − 𝑔 𝑥 6 ; 𝜊 ≥ 𝛼𝑔 𝑥 6 ; 𝜊 , 𝑥 − 𝑥 6 Initialize : 𝑥 J ­ Smooth: ||𝛼𝑔 𝑥; 𝜊 − 𝛼𝑔 𝑥 6 ; 𝜊 || ≤ 𝑀||𝑥 − 𝑥 6 || Iterate : for 𝑢 = 0, 1, 2, … , do § Find a 𝑥 = close to Choose 𝜃 = > 0 𝑋 ∗ = {𝑥 ∗ ∈ 𝑆 A ∶ ∀ $∈& ' , 𝐺 𝑥 ≥ 𝐺 𝑥 ∗ } Generate random 𝜊 = Compute 𝛼𝑔 𝑥 = ; 𝜊 = § Problem: Characterize Expected Convergence Rates Update 𝑥 =RS = 𝑥 = − 𝜃 = 𝛼𝑔 𝑥 = ; 𝜊 = E ∗ ∈F ∗ ||w H − w ∗ || I 𝐹 inf and 𝐹[𝐺(𝑥 = ) − 𝐺(𝑥 ∗ )] end for 5

Problem Setting § Solve $∈& ' {𝐺(𝑥) = 𝐹 𝜊 [𝑔(𝑥; 𝜊)]} min § Assumptions Stochastic Gradient Descend (SGD): ­ Convex: 𝑔 𝑥; 𝜊 − 𝑔 𝑥 6 ; 𝜊 ≥ 𝛼𝑔 𝑥 6 ; 𝜊 , 𝑥 − 𝑥 6 Initialize : 𝑥 J ­ Smooth: ||𝛼𝑔 𝑥; 𝜊 − 𝛼𝑔 𝑥 6 ; 𝜊 || ≤ 𝑀||𝑥 − 𝑥 6 || Iterate : for 𝑢 = 0, 1, 2, … , do § Find a 𝑥 = close to Choose 𝜃 = > 0 𝑋 ∗ = {𝑥 ∗ ∈ 𝑆 A ∶ ∀ $∈& ' , 𝐺 𝑥 ≥ 𝐺 𝑥 ∗ } Generate random 𝜊 = Compute 𝛼𝑔 𝑥 = ; 𝜊 = § Problem: Characterize Expected Convergence Rates Update 𝑥 =RS = 𝑥 = − 𝜃 = 𝛼𝑔 𝑥 = ; 𝜊 = E ∗ ∈F ∗ ||w H − w ∗ || I 𝐹 inf and 𝐹[𝐺(𝑥 = ) − 𝐺(𝑥 ∗ )] end for 6

Beyond convex and strongly convex functions Strongly Convex Plain Convex T I ||𝑥 − 𝑥 ∗ || I 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ 0 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥

𝜕 -Convexity 𝜕 − Convex Strongly Convex Plain Convex $ ∗ ∈F ∗ ||𝑥 − 𝑥 ∗ || I , 𝜕 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ inf T I ||𝑥 − 𝑥 ∗ || I 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ 0 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ 𝜕 6 > 0, 𝜕 66 < 0 ,

𝜕 -Convexity with curvature ℎ ∈ [0,1] 𝜕 − Convex Strongly Convex Plain Convex $ ∗ ∈F ∗ ||𝑥 − 𝑥 ∗ || I , 𝜕 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ inf T I ||𝑥 − 𝑥 ∗ || I 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ 0 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ 𝜕 6 > 0, 𝜕 66 < 0 , ] ≥ 𝛽 $ ∗ ∈F ∗ ||𝑥 − 𝑥 ∗ || I 𝐺 𝑥 − 𝐺 𝑥 ∗ inf ℎ = 0 ℎ ∈ (0,1) ℎ = 1

𝜕 -Convexity with curvature ℎ ∈ [0,1] 𝜕 − Convex Strongly Convex Plain Convex $ ∗ ∈F ∗ ||𝑥 − 𝑥 ∗ || I , 𝜕 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ inf T I ||𝑥 − 𝑥 ∗ || I 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ 0 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ 𝜕 6 > 0, 𝜕 66 < 0 , ] ≥ 𝛽 $ ∗ ∈F ∗ ||𝑥 − 𝑥 ∗ || I 𝐺 𝑥 − 𝐺 𝑥 ∗ inf ℎ = 0 ℎ ∈ (0,1) ℎ = 1 ] ≥ 𝛽 $ ∗ ∈F ∗ ||𝑥 − 𝑥 ∗ || I , where ℎ ∈ 0,2 . HEB (Holderian Error Bound): 𝐺 𝑥 − 𝐺 𝑥 ∗ inf HEB and 𝜕 -convexity are not subclasses of one another but they do intersection for ℎ ∈ 0,1 . [Bolte, J., Nguyen, T. P., Peypouquet, J., and Suter, B. W. From error bounds to the complexity of first order descent methods for convex functions. Mathematical Programming, 165(2):471–507, Oct 2017]

Close to optimal stepsize 𝜕 − Convex Strongly Convex Plain Convex $ ∗ ∈F ∗ ||𝑥 − 𝑥 ∗ || I , 𝜕 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ inf T I ||𝑥 − 𝑥 ∗ || I 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ 0 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ 𝜕 6 > 0, 𝜕 66 < 0 , ℎ = 1 ℎ = 0 ] ≥ 𝛽 $ ∗ ∈F ∗ ||𝑥 − 𝑥 ∗ || I 𝐺 𝑥 − 𝐺 𝑥 ∗ inf ℎ ∈ (0,1) ` 𝜃 = = SGD HRa b/ def 𝐷𝑚𝑝𝑡𝑓 𝑢𝑝 𝑝𝑞𝑢𝑗𝑛𝑏𝑚 𝑡𝑢𝑓𝑞𝑡𝑗𝑨𝑓

Convergence Rate of SGD 𝜕 − Convex Strongly Convex Plain Convex $ ∗ ∈F ∗ ||𝑥 − 𝑥 ∗ || I , 𝜕 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ inf T I ||𝑥 − 𝑥 ∗ || I 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ 0 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ 𝜕 6 > 0, 𝜕 66 < 0 , ℎ = 1 ℎ = 0 ] ≥ 𝛽 $ ∗ ∈F ∗ ||𝑥 − 𝑥 ∗ || I 𝐺 𝑥 − 𝐺 𝑥 ∗ inf ℎ ∈ (0,1) E ∗ ∈F ∗ ||w H − w ∗ || I = 𝑃 𝑢 r]/(Ir]) 𝐹 inf ` 𝜃 = = SGD HRa b/ def I= 1 𝐷𝑚𝑝𝑡𝑓 𝑢𝑝 𝑝𝑞𝑢𝑗𝑛𝑏𝑚 𝑡𝑢𝑓𝑞𝑡𝑗𝑨𝑓 = 𝑃(𝑢 rS/(Ir]) ) 𝑢 s 𝐹 𝐺 𝑥 t − 𝐺 𝑥 ∗ tu=RS 12

Convergence Rate of SGD 𝜕 − Convex Strongly Convex Plain Convex $ ∗ ∈F ∗ ||𝑥 − 𝑥 ∗ || I , 𝜕 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ inf T I ||𝑥 − 𝑥 ∗ || I 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ 0 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ 𝜕 6 > 0, 𝜕 66 < 0 , ℎ = 1 ℎ = 0 ] ≥ 𝛽 $ ∗ ∈F ∗ ||𝑥 − 𝑥 ∗ || I 𝐺 𝑥 − 𝐺 𝑥 ∗ inf ℎ ∈ (0,1) E ∗ ∈F ∗ ||w H − w ∗ || I = 𝑃 𝑢 r]/(Ir]) 𝐹 inf [Useful,1] [Useless,0] 0 ← ℎ → 1 I= 1 [Useful,1] = 𝑃(𝑢 rS/(Ir]) ) 𝑢 s 𝐹 𝐺 𝑥 t − 𝐺 𝑥 ∗ [Useful,0] tu=RS 13

Convergence Rate of SGD 𝜕 − Convex Strongly Convex Plain Convex $ ∗ ∈F ∗ ||𝑥 − 𝑥 ∗ || I , 𝜕 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ inf T I ||𝑥 − 𝑥 ∗ || I 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ 0 𝐺 𝑥 − 𝐺 𝑥 ∗ ≥ 𝜕 6 > 0, 𝜕 66 < 0 , ℎ = 1 ℎ = 0 ] ≥ 𝛽 $ ∗ ∈F ∗ ||𝑥 − 𝑥 ∗ || I 𝐺 𝑥 − 𝐺 𝑥 ∗ inf ℎ ∈ (0,1) E ∗ ∈F ∗ ||w H − w ∗ || I = 𝑃 𝑢 r]/(Ir]) h= ½ 𝐹 inf 𝐺 𝑥 = 𝐼 𝑥 + 𝜇𝐻 𝑥 , 𝐼 𝑥 − 𝑑𝑝𝑜𝑤𝑓𝑦 I= 1 A = 𝑃(𝑢 rS/(Ir]) ) 𝑢 s 𝐹 𝐺 𝑥 t − 𝐺 𝑥 ∗ [𝑓 $ € +𝑓 r$ € − 2 − 𝑥 t I ] 𝐻 𝑥 = s tu=RS 14 tuS

Experiment Curvature 0 (convex) Curvature unknown † 𝑥 = 𝑔 … 𝑥)) 𝑔 t 𝑥 + 𝜇 𝑥 𝑔 t 𝑥 = log(1 + exp(−𝑧 t 𝑦 t t Curvature ½ Curvature 1 (strongly convex) t 𝑥 + 𝜇 † 𝑥 = 𝑔 I ` 𝑥 = 𝑔 𝑔 t 𝑥 + 𝜇𝐻 𝑥 𝑔 𝑥 t t 2 A [𝑓 $ € +𝑓 r$ € − 2 − 𝑥 t I ] 𝐻 𝑥 = s tuS 15

Conclusion § 𝜕 - convexity notion: plain convex, strongly convex and something in between § SGD with 𝜕-convex objective functions Thank you for your attention! J https://arxiv.org/abs/1810.04100 Poster Number: #193 – Pacific Ballroom. – 06:30—09:00PM – 06/11 16