Poster 158 1 / 4 Poster 158 Security in Distributed ML Zeno: - - PowerPoint PPT Presentation

Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance

Cong Xie, Oluwasanmi Koyejo, Indranil Gupta

June 12, 2019

Poster 158

1 / 4

Poster 158

Security in Distributed ML Zeno: distributed synchronous SGD that

• tolerates an arbitrary number of malicious workers
• provides convergence guarantees for non-convex problems

Goal: converge under attacks/failures, regardless of false negative Prev. Ours Tolerates a majority of malicious workers No Yes Considers the progress of optimization No Yes Tolerates stealth adversary (empirically) No Yes

2 / 4

Poster 158

Byzantine-tolerant SGD

m workers, distributed SGD:

4: Aggregation 2: Gradient Computation

Server

Honest Worker Honest Worker Byzantine Worker

3: Push 1: Pull

C

• r

r e c t G r a d i e n t B y z a n t i n e G r a d i e n t

ps_byz.drawio https://www.draw.io/#Wb778d378ebdb14ee/B778D378EBDB14EE!33942 1 of 1 5/8/2019, 12:27 PM

3 / 4

Poster 158

Main Idea & Results

⋆ Sort gi(x), i ∈ [m] by the Stochastic descent score: Definition

Stochastic descent score of any update u: Scoreγ,ρ(u, x) = fr(x) − fr(x − γu) − ρu2, fr(x): unbiased estimator of the loss F(x), for validation.

⋆ Zeno: filter the b “worst” gradients

1 m−b

m−b

i=1 ˜

v(i), b > q. ⋆ Convergence after T iterations:

T−1

t=0 E∇F(xt)2

T ≤ O 1 √ T

• + O

(b − q + 1)(m − q) (m − b)2

• .

4 / 4