Semi-Cyclic SGD Hubert Eichner Tomer Koren Brendan McMahan Kunal - PowerPoint PPT Presentation

Semi-Cyclic SGD Hubert Eichner Tomer Koren Brendan McMahan Kunal Talwar Google Google Google Google Nati Srebro

𝑥 𝑢+1 ← 𝑥 𝑢 − 𝜃∇𝑔 𝑥 𝑢 , 𝑨 𝑢 𝑥 𝑈 ෝ SGD is great ……

𝑥 𝑢+1 ← 𝑥 𝑢 − 𝜃∇𝑔 𝑥 𝑢 , 𝑨 𝑢 A b A g A z B e B o B y C h C l C o C o C u D a D e D i D o D r D y E d E f 𝑥 𝑈 ෝ E l E n E p E r E s E t E x F a F i F l F o F r F u G e G i SGD is great …… G l G m G r H a H i H o if you run on iid (randomly shuffled) data

𝑥 𝑢+1 ← 𝑥 𝑢 − 𝜃∇𝑔 𝑥 𝑢 , 𝑨 𝑢 Samples in block 𝑗 = 1. . 𝑛 are sampled from as 𝑨 𝑢 ∼ 𝒠 𝑗 1 𝑛 σ 𝑗 𝒠 𝑗 overall distribution: 𝒠 = 𝑥 𝑈 ෝ SGD is great …… if you run on iid (randomly shuffled) data Cyclically varying (not fully shuffled) data

𝑥 𝑢+1 ← 𝑥 𝑢 − 𝜃∇𝑔 𝑥 𝑢 , 𝑨 𝑢 Samples in block 𝑗 = 1. . 𝑛 are sampled from as 𝑨 𝑢 ∼ 𝒠 𝑗 1 𝑛 σ 𝑗 𝒠 𝑗 overall distribution: 𝒠 = 𝑥 𝑈 ෝ SGD is great …… if you run on iid (randomly shuffled) data Cyclically varying (not fully shuffled) data, e.g. in Federated Learning • Train model by executing SGD steps on user devices when device available (plugged in, idle, on WiFi) • Diurnal variations (e.g. Day vs night available devices; US vs UK vs India)

𝑥 𝑢+1 ← 𝑥 𝑢 − 𝜃∇𝑔 𝑥 𝑢 , 𝑨 𝑢 Samples in block 𝑗 = 1. . 𝑛 are sampled from as 𝑨 𝑢 ∼ 𝒠 𝑗 Samples in block 𝑗 = 1. . 𝑛 are sampled from as 𝑨 𝑢 ∼ 𝒠 𝑗 1 𝑛 σ 𝑗 𝒠 𝑗 overall distribution: 𝒠 = 𝑥 𝑈 ෝ • Train ෝ 𝑥 𝑈 by running block-cyclic SGD ➔ could be MUCH slower, by an arbitrary large factor

𝑥 𝑢+1 ← 𝑥 𝑢 − 𝜃∇𝑔 𝑥 𝑢 , 𝑨 𝑢 Samples in block 𝑗 = 1. . 𝑛 are sampled from as 𝑨 𝑢 ∼ 𝒠 𝑗 𝑥 1 ෝ 𝑥 2 ෝ • Train ෝ 𝑥 𝑈 by running block-cyclic SGD ➔ could be MUCH slower, by an arbitrary large factor 𝑥 𝑗 for each block 𝑗 = 1. . 𝑛 Pluralistic approach: learn different ෝ 𝑥 𝑗 separately on data from that block (across all cycles) • Train each ෝ ➔ could be slower/less efficient by a factor of 𝑛

𝑥 𝑢+1 ← 𝑥 𝑢 − 𝜃∇𝑔 𝑥 𝑢 , 𝑨 𝑢 Samples in block 𝑗 = 1. . 𝑛 are sampled from as 𝑨 𝑢 ∼ 𝒠 𝑗 𝑥 1 ෥ 𝑥 2 ෥ • Train ෝ 𝑥 𝑈 by running block-cyclic SGD ➔ could be MUCH slower, by an arbitrary large factor 𝑥 𝑗 for each block 𝑗 = 1. . 𝑛 Pluralistic approach: learn different ෝ 𝑥 𝑗 separately on data from that block (across all cycles) • Train each ෝ ➔ could be slower/less efficient by a factor of 𝑛 𝑥 𝑗 using single SGD chain+ “ pluralistic averaging ” • Our solution: train ෥ ➔ exactly same guarantee as if using random shuffling (no degradation) ➔ no extra comp. cost, no assumptions about 𝓔 𝒋 nor relatedness