Poster 158 1 / 4 Poster 158 Security in Distributed ML Zeno: - - PowerPoint PPT Presentation

poster 158
SMART_READER_LITE
LIVE PREVIEW

Poster 158 1 / 4 Poster 158 Security in Distributed ML Zeno: - - PowerPoint PPT Presentation

Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance Cong Xie, Oluwasanmi Koyejo, Indranil Gupta June 12, 2019 Poster 158 1 / 4 Poster 158 Security in Distributed ML Zeno: distributed synchronous SGD that


slide-1
SLIDE 1

Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance

Cong Xie, Oluwasanmi Koyejo, Indranil Gupta

June 12, 2019

Poster 158

1 / 4

slide-2
SLIDE 2

Poster 158

Security in Distributed ML Zeno: distributed synchronous SGD that

  • tolerates an arbitrary number of malicious workers
  • provides convergence guarantees for non-convex problems

Goal: converge under attacks/failures, regardless of false negative Prev. Ours Tolerates a majority of malicious workers No Yes Considers the progress of optimization No Yes Tolerates stealth adversary (empirically) No Yes

2 / 4

slide-3
SLIDE 3

Poster 158

Byzantine-tolerant SGD

m workers, distributed SGD:

4: Aggregation 2: Gradient Computation

Server

Honest Worker Honest Worker Byzantine Worker

3: Push 1: Pull

C

  • r

r e c t G r a d i e n t B y z a n t i n e G r a d i e n t

ps_byz.drawio https://www.draw.io/#Wb778d378ebdb14ee/B778D378EBDB14EE!33942 1 of 1 5/8/2019, 12:27 PM

3 / 4

slide-4
SLIDE 4

Poster 158

Main Idea & Results

⋆ Sort gi(x), i ∈ [m] by the Stochastic descent score: Definition

Stochastic descent score of any update u: Scoreγ,ρ(u, x) = fr(x) − fr(x − γu) − ρu2, fr(x): unbiased estimator of the loss F(x), for validation.

⋆ Zeno: filter the b “worst” gradients

1 m−b

m−b

i=1 ˜

v(i), b > q. ⋆ Convergence after T iterations:

T−1

t=0 E∇F(xt)2

T ≤ O 1 √ T

  • + O

(b − q + 1)(m − q) (m − b)2

  • .

4 / 4