Semi-Cyclic SGD Hubert Eichner Tomer Koren Brendan McMahan Kunal - - PowerPoint PPT Presentation

Semi-Cyclic SGD

Hubert Eichner

Google

Tomer Koren

Google

Brendan McMahan

Google

Kunal Talwar

Google

Nati Srebro

𝑥𝑢+1 ← 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢, 𝑨𝑢

SGD is great……

ෝ 𝑥𝑈

A b A z B

• C

u D e D

• E

n E r E t F r G e G l A g B e D a D i E p E s F u G i C h C

• D

y E f F a F l G r H i B y C l C

• D

r E d E l E x F i F

• G

m H a H

• 𝑥𝑢+1 ← 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢, 𝑨𝑢

SGD is great…… if you run on iid (randomly shuffled) data

ෝ 𝑥𝑈

𝑥𝑢+1 ← 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢, 𝑨𝑢

SGD is great…… if you run on iid (randomly shuffled) data Cyclically varying (not fully shuffled) data

ෝ 𝑥𝑈

Samples in block 𝑗 = 1. . 𝑛 are sampled from as 𝑨𝑢 ∼ 𝒠𝑗

• verall distribution: 𝒠 =

1 𝑛 σ𝑗 𝒠𝑗

𝑥𝑢+1 ← 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢, 𝑨𝑢

SGD is great…… if you run on iid (randomly shuffled) data Cyclically varying (not fully shuffled) data, e.g. in Federated Learning

• Train model by executing SGD steps on user devices

when device available (plugged in, idle, on WiFi)

• Diurnal variations (e.g. Day vs night available devices; US vs UK vs India)

ෝ 𝑥𝑈

Samples in block 𝑗 = 1. . 𝑛 are sampled from as 𝑨𝑢 ∼ 𝒠𝑗

• verall distribution: 𝒠 =

1 𝑛 σ𝑗 𝒠𝑗

𝑥𝑢+1 ← 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢, 𝑨𝑢 Samples in block 𝑗 = 1. . 𝑛 are sampled from as 𝑨𝑢 ∼ 𝒠𝑗

• Train ෝ

𝑥𝑈 by running block-cyclic SGD ➔ could be MUCH slower, by an arbitrary large factor

ෝ 𝑥𝑈

Samples in block 𝑗 = 1. . 𝑛 are sampled from as 𝑨𝑢 ∼ 𝒠𝑗

• verall distribution: 𝒠 =

1 𝑛 σ𝑗 𝒠𝑗

𝑥𝑢+1 ← 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢, 𝑨𝑢 Samples in block 𝑗 = 1. . 𝑛 are sampled from as 𝑨𝑢 ∼ 𝒠𝑗

• Train ෝ

𝑥𝑈 by running block-cyclic SGD ➔ could be MUCH slower, by an arbitrary large factor Pluralistic approach: learn different ෝ 𝑥𝑗 for each block 𝑗 = 1. . 𝑛

• Train each ෝ

𝑥𝑗 separately on data from that block (across all cycles) ➔ could be slower/less efficient by a factor of 𝑛

ෝ 𝑥2 ෝ 𝑥1

𝑥𝑢+1 ← 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢, 𝑨𝑢 Samples in block 𝑗 = 1. . 𝑛 are sampled from as 𝑨𝑢 ∼ 𝒠𝑗

• Train ෝ

𝑥𝑈 by running block-cyclic SGD ➔ could be MUCH slower, by an arbitrary large factor Pluralistic approach: learn different ෝ 𝑥𝑗 for each block 𝑗 = 1. . 𝑛

• Train each ෝ

𝑥𝑗 separately on data from that block (across all cycles) ➔ could be slower/less efficient by a factor of 𝑛

• Our solution: train ෥

𝑥𝑗 using single SGD chain+“pluralistic averaging” ➔ exactly same guarantee as if using random shuffling (no degradation) ➔ no extra comp. cost, no assumptions about 𝓔𝒋 nor relatedness

෥ 𝑥2 ෥ 𝑥1