From Local SGD to Local Fixed-Point Methods for Federated Learning - - PowerPoint PPT Presentation

Fixed-Point Methods with Local Steps

KAUST

/ 22

From Local SGD to Local Fixed-Point Methods for Federated Learning

Peter 
 Richtárik Dmitry
 Kovalev Elnur 
 Gasanov Grigory
 Malinovsky Laurent Condat King Abdullah University of Science and Technology (KAUST),
 Thuwal, Saudi Arabia

1

Fixed-Point Methods with Local Steps

KAUST

/ 22

Distributed Algorithms

2

Master Node 1 Node 2 Node M ...

Fixed-Point Methods with Local Steps

KAUST

/ 22 3

Master

• p. T1

• p. T2

• p. TM

Distributed Algorithms

Fixed-Point Methods with Local Steps

KAUST

/ 22

Distributed fixed-point problem

4

T : x 2 Rd 7! 1 M

M

X

i=1

Ti(x).

We define the average operator

Fixed-Point Methods with Local Steps

KAUST

/ 22

Distributed fixed-point problem

4

T : x 2 Rd 7! 1 M

M

X

i=1

Ti(x).

T (x?) = x?.

Our goal is to find x? ∈ Rd such that

We define the average operator

Fixed-Point Methods with Local Steps

KAUST

/ 22

Distributed fixed-point problem

4

T : x 2 Rd 7! 1 M

M

X

i=1

Ti(x).

T (x?) = x?.

Our goal is to find x? ∈ Rd such that

We define the average operator A fixed-point algorithm iterates:

xk+1 = T (xk)

Fixed-Point Methods with Local Steps

KAUST

/ 22 5

Optimization algorithms

Fixed-point algorithms:

* Find a minimizer of a function

xk+1 = xk γrF(xk)

Gradient descent: Proximal point algorithm: xk+1 = arg min

x

F(x) +

1 2γ kx xkk2

Fixed-Point Methods with Local Steps

KAUST

/ 22 6

Fixed-point algorithms:

* Find a minimizer of a function * Proximal splitting algorithms * Primal-dual algorithms * Cyclic or shuffled GD * (Block-)coordinate methods * Methods with inertia, momentum... * Conjugate gradient methods * Higher-order methods * ...

Optimization algorithms

Fixed-Point Methods with Local Steps

KAUST

/ 22 7

Fixed-point algorithms:

* Find a minimizer of a function * Find a saddle point of a convex-concave function * Find a solution of a PDE * Find an eigenvector * Solve a monotone inclusion or variational inequality * ...

Fixed-point methods

Fixed-Point Methods with Local Steps

KAUST

/ 22 8

* Stich, S. U. Local SGD Converges Fast and Communicates Little. In

International Conference on Learning Representations, 2019.

* Khaled, A., Mishchenko, K., and Richtárik, P. First analysis of local GD

• n heterogeneous data. In NeurIPS Workshop on Federated Learning for

Data Privacy and Confidentiality, 2019.

* Khaled, A., Mishchenko, K., and Richtárik, P. Tighter theory for local SGD

• n identical and heterogeneous data. In The 23rd International

Conference on Artificial Intelligence and Statistics (AISTATS 2020), 2020.

* Ma, C., Konecny, J., Jaggi, M., Smith, V., Jordan, M. I., Richtárik,P.,and

Takác, M. Distributed optimization with arbitrary local solvers. Optimization Methods and Software, 32(4):813–848, 2017.

* Haddadpour, F. and Mahdavi, M. On the convergence of local descent

methods in federated learning. arXiv preprint arXiv:1910.14425, 2019.

Prior work: local gradient descent

Fixed-Point Methods with Local Steps

KAUST

/ 22

Algorithm 1

9

Algorithm 1 Local distributed fixed-point method Input: Initial estimate ˆ x0 ∈ Rd, stepsize λ > 0, sequence of synchronization times 0 = t0 < t1 < ... Initialize: x0

i := ˆ

x0, for i = 1, ... , M for k = 0, 1, ... do for i = 1, 2, ... , M in parallel do hk+1

i

:= (1 − λ)xk

i + λTi(xk i )

if k + 1 = tn, for some n, then Communicate hk+1

i

to master node else xk+1

i

:= hk+1

i

end if end for if k + 1 = tn, for some n, then At master node: ˆ xk+1 := 1

M

PM

i=1 hk+1 i

Broadcast: xk+1

i

:= ˆ xk+1 for all i = 1, ... , M end if end for

Fixed-Point Methods with Local Steps

KAUST

/ 22

Algorithm 1

9

Algorithm 1 Local distributed fixed-point method Input: Initial estimate ˆ x0 ∈ Rd, stepsize λ > 0, sequence of synchronization times 0 = t0 < t1 < ... Initialize: x0

i := ˆ

x0, for i = 1, ... , M for k = 0, 1, ... do for i = 1, 2, ... , M in parallel do hk+1

i

:= (1 − λ)xk

i + λTi(xk i )

if k + 1 = tn, for some n, then Communicate hk+1

i

to master node else xk+1

i

:= hk+1

i

end if end for if k + 1 = tn, for some n, then At master node: ˆ xk+1 := 1

M

PM

i=1 hk+1 i

Broadcast: xk+1

i

:= ˆ xk+1 for all i = 1, ... , M end if end for

n-th epoch: sequence

• f iterations

k + 1 = tn−1 + 1, ... , tn

Fixed-Point Methods with Local Steps

KAUST

/ 22 10

Communication times

Nb of iterations in each epoch supposed bounded: Assumption: 1 ≤ tn − tn−1 ≤ H, for every n ≥ 1.

Fixed-Point Methods with Local Steps

KAUST

/ 22 10

Communication times

Nb of iterations in each epoch supposed bounded: Assumption: 1 ≤ tn − tn−1 ≤ H, for every n ≥ 1.

Example: tn = nH

Fixed-Point Methods with Local Steps

KAUST

/ 22 11

Analysis in the contractive case

tn = nH

All Ti are χ-contractive, for χ ∈ [0, 1)

• kTi(x) Ti(y)k  χkx yk, 8x, y

i.e.

Fixed-Point Methods with Local Steps

KAUST

/ 22 12

Analysis in the contractive case

tn = nH

All Ti are χ-contractive, for χ ∈ [0, 1)

• We define the operator

e T = 1 M

M

X

i=1

• λTi + (1 − λ)Id

H

ˆ x(n+1)H = 1 M

M

X

i=1

h(n+1)H

i

= e T (ˆ xnH)

Then

Fixed-Point Methods with Local Steps

KAUST

/ 22 13

Analysis in the contractive case

Theorem 2.11 (linear convergence) The fixed point x† of e T exists and is unique, and ˆ xnH converges linearly to x†. More precisely, (i) e T is ξH-contractive, with ξ = max

• λχ + (1 λ), λ(1 + χ) 1
• .

(ii) 8n 2 N, kˆ x(n+1)H x†k  ξHkˆ xnH x†k. (iii) 8n 2 N, kˆ xnH x†k  ξnHkˆ x0 x†k.

Fixed-Point Methods with Local Steps

KAUST

/ 22 13

Analysis in the contractive case

Theorem 2.11 (linear convergence) The fixed point x† of e T exists and is unique, and ˆ xnH converges linearly to x†. More precisely, (i) e T is ξH-contractive, with ξ = max

• λχ + (1 λ), λ(1 + χ) 1
• .

(ii) 8n 2 N, kˆ x(n+1)H x†k  ξHkˆ xnH x†k. (iii) 8n 2 N, kˆ xnH x†k  ξnHkˆ x0 x†k.

Note: Without further knowledge, set λ = 1.

Fixed-Point Methods with Local Steps

KAUST

/ 22 14

Analysis in the contractive case

Theorem 2.14 (size of the neighborhood) Suppose that λ = 1. So, ξ = χ. Then kx† x?k  S, where S = ξ 1 ξ 1 ξH−1 1 ξH 1 M

M

X

i=1

kTi(x?) x?k.

Fixed-Point Methods with Local Steps

KAUST

/ 22 14

Analysis in the contractive case

Theorem 2.14 (size of the neighborhood) Suppose that λ = 1. So, ξ = χ. Then kx† x?k  S, where S = ξ 1 ξ 1 ξH−1 1 ξH 1 M

M

X

i=1

kTi(x?) x?k.

Note 1: S = 0 if H = 1, or M = 1, or Ti = T , or ξ = 0.

Fixed-Point Methods with Local Steps

KAUST

/ 22 14

Analysis in the contractive case

Theorem 2.14 (size of the neighborhood) Suppose that λ = 1. So, ξ = χ. Then kx† x?k  S, where S = ξ 1 ξ 1 ξH−1 1 ξH 1 M

M

X

i=1

kTi(x?) x?k.

Note 2: If H : 1 % +1, S : 0 % S∞ with

S∞ = ξ 1 ξ 1 M

M

X

i=1

kTi(x?) x?k.

Note 1: S = 0 if H = 1, or M = 1, or Ti = T , or ξ = 0.

Fixed-Point Methods with Local Steps

KAUST

/ 22 15

Analysis in the contractive case

Theorem 2.14 (size of the neighborhood) Suppose that λ = 1. So, ξ = χ. Then kx† x?k  S, where S = ξ 1 ξ 1 ξH−1 1 ξH 1 M

M

X

i=1

kTi(x?) x?k.

Corollary: For every n ∈ N,

kˆ xnH x?k  ξnHkˆ x0 x†k + S  ξnH(kˆ x0 x?k + S) + S.

Fixed-Point Methods with Local Steps

KAUST

/ 22 16

Results: logistic regression

Fixed-Point Methods with Local Steps

KAUST

/ 22

x?

x†

Epsilon-accuracy

17

Fixed-Point Methods with Local Steps

KAUST

/ 22

x?

x†

Epsilon-accuracy

Local GD: O L

µ 1 H log( 1 ✏)

• <latexit sha1_base64="po9pHWqPS3XoJXswkNrKCzuPSgA=">AFZ3iclVTLbtNAFL1NGyjh0RQkhMQmEVqURXFgChVhRQBiy5ALRJpK9VRZTuTZBTbY/lRCJbZ8h18DWIHS/6A8hWcuXZKaJVCHcW+c+49Z+5jbDtwZRS3Wj/mSvML5UuXF69Url67fmOpunxzN1J6IiOo1wV7tWJFzpi04sY1fsB6GwPNsVe/bohfbvHYkwksp/G48D0fWsgS/70rFiQIfVzW3TloMVsx9aTvoqS0vyfKFkaVbmemqiRNrUwSRdJWfrWrS6mG13mq2+KqdNYzCqLeNj582v87/2lHLpQqZ1CNFDiXkSCfYtguWRThd0AGtSgA1qUWAhLsl9QRpqbIEogwgI6wn2AVQpbIUYhPqMaNeBXiAxha+Ua+xNW0ehsHQs56TzGeNqFlgc0piHQf/Emkf/LOwAaU5+ecrUSvQgY0V1xCpWEO6Azr01VFUMhAKbtHvwhbIeZk57WmBNx7bqPFvt/cqRG9dopYhM6PifLBvKMWEUr9ugZJtSkDeRcQdZDYC4d4R6fOyHNHvCEJHfFu2C05I7MZky6HYOlz0JURIdco6B3fAY8jvNRQpfn+emuzsGInitT4muJMUOQ9SlVRozd41YbXrys2c93akuc/uneHkOkvP9c6JT4HqCI6lx7mF0LTZ0rNYpzU89T/Xecl8l1mCXhd7bBcnxuIehfSAe2Cxlse7ZsU7NwC+doJdVDHnSX5+4Pc2R23EvOdJnfZn/B3Z0NeTk6/GWP3YdN41Hz8xqi3n1N+LdJduk8r+GqsU5u2aIc6yPMzfaFv9H3huLxUvl2+k4eW5grOLfrKt/7DT6xIeU=</latexit>

but Note: H = O(1 + ✏)

O L

µ log( 1 ✏)

• <latexit sha1_base64="u4C8PNrXV5hIGp34PSy0RZd8E1Q=">AFXHiclVTLbtNAFL1NEygpSlIbNhERJFaVEU2IEqFkCJgwQLUIpG2UhNVtjNJRrE9lh+FYJktf8DXsAWJT2DGvgKzlw7JbRKoRPFvj73DP3MbYduDKDePbQmxXLl0elKdfnqyrXV2tr1vUgloSM6jnJVeGBbkXClLzqxjF1xEITC8mxX7Nvjp9q/fyzCSCr/dTwJRM+zhr4cSMeKAR3VjJ2uLYfr3UFoOemLO16SdZ1RQxgYgkq7ysw3N3DiqNYyWwat+1jALo9E23949HXx165aK1WpS31S5FBCHgnyKYbtkURfodkEBsB6lwEJYkv2CMtKxCVgCDAvoGNchnlLYChwFfkZ1asKvwAxha+U6+xNW0eh8HQs56TwmuNuFlgc0phHQf8VNmf8bdwg0pgE95GolehEworviFCoJd0BnXp+pKoZCAEzbfhD2A5HTnta5iIa9d9tNj/g5ka1c9OwU3o5zlZNpFnxCpasU+PMaEWbSPnKrIeAXPpGNf43Anp6CFPSHJXvAuyJXdkfsS02zGi9FmICnbINQp6w2fAY56PKlL4Bjw3d0JEMHP+pToSlLsMEJdWqU5d9eI1WYnP3/Ws53qcezgVFyeg+R8/5zoFLie4Jhr6XNuITRtvQstmgTd/3PdZ5xvMtRgl4We+wUJ8biHoV0h3tgsZbHu2bFOzcEvnmCXVQxj5N8f8fvbY7a4LzlSZ32Z/wd2dbrwclX46yxd7dl3mvdf2U2k8oX0t0i27TOr4aW9Sm57RLHeT5kT7RZ/pS/l4pV5YrKzm1tFDE3KC/VuXmb6u3HRY=</latexit>

17

Fixed-Point Methods with Local Steps

KAUST

/ 22 18

Analysis in the non-contractive case

tn = nH

• convergence to x†, a fixed point of

e T = 1

M

PM

i=1

• λTi + (1 − λ)Id

H

sublinear rates on kˆ x(n+1)H ˆ xnHk2 or kˆ xk T (ˆ xk)k2

• tn = nH

convergence w.r.t. nb. epochs 1 to H times faster

Fixed-Point Methods with Local Steps

KAUST

/ 22 19

Algorithm 1 Randomized distributed fixed-point method Input: Initial estimate ˆ x0 ∈ Rd, stepsize λ > 0, communication probability 0 < p ≤ 1 Initialize: x0

i = ˆ

x0, for all i = 1, ... , M for k = 1, 2, ... do for i = 1, 2, ... , M in parallel do hk+1

i

:= (1 − λ)xk

i + λTi(xk i )

end for Flip a coin and with probability p do Communicate hk+1

i

to master, for i = 1, ... , M At master node: ˆ xk+1 := 1

M

PM

i=1 hk+1 i

Broadcast: xk+1

i

:= ˆ xk+1, for all i = 1, ... , M else, with probability 1 − p, do xk+1

i

:= hk+1

i

, for all i = 1, ... , M end for

2

Algorithm 2

Fixed-Point Methods with Local Steps

KAUST

/ 22 20

Analysis of Algorithm 2

Ψk := kˆ xk x?k2 + 5λ p 1 M

M

X

i=1

• xk

i ˆ

xk

• 2

Lyapunov function:

Assumption 3.1 (1 + ρ)kTi(x) Ti(y)k2  kx yk2 kx Ti(x) y + Ti(y)k2 for some ρ > 0

Fixed-Point Methods with Local Steps

KAUST

/ 22 20

Analysis of Algorithm 2

Ψk := kˆ xk x?k2 + 5λ p 1 M

M

X

i=1

• xk

i ˆ

xk

• 2

Lyapunov function:

Assumption 3.1 (1 + ρ)kTi(x) Ti(y)k2  kx yk2 kx Ti(x) y + Ti(y)k2 for some ρ > 0

For λ small enough:

E[Ψk]  ✓ 1 min ✓ λρ 1 + ρ, p 5 ◆◆k Ψ0 + 150 min ⇣

⇢ 1+⇢, p 5

⌘ p2 λ3 M

M

X

i=1

kx? Ti(x?)k2

Theorem 3.2

Fixed-Point Methods with Local Steps

KAUST

/ 22 22

Local steps: good to achieve a medium-accuracy solution faster, if communication is the bottleneck

Conclusion