Distributed Machine Learning with a Serverless Architecture Hao Wang - - PowerPoint PPT Presentation

Distributed Machine Learning with a Serverless Architecture

Hao Wang1, Di Niu2, Baochun Li1

1University of Toronto, 2University of Alberta

INFOCOM’19, Paris, FR

What is machine learning?

Deep Learning

Machine Learning

5

Gradients Numerical optimization

Objective Data

ML Workflow

6

Model Design Model Tuning Training & Evaluation

😮 😈

Datasets … Convergence Loss rate …

Resource Reservation

Objective Budget

Our Key Insights

• Most current ML training jobs are data parallel
• Model quality and resource investment have a nonlinear relation
• ML training is inevitably a trial-and-error process

7

IaaS PaaS Pricing Per hour Per hour Maintanance By users By providers Examples AWS EC2, Google Cloud Compute … Azure ML Studio, Google Cloud ML Engine …

Distributed ML Infrastructure

8

…

Serverless?

9

IaaS PaaS Serverless Pricing Per hour Per hour Per call Maintanance By users By providers By providers Examples AWS EC2 Azure ML Studio AWS Lambda

Serverless Computing?

Stateless function

• Only input and output, no intermediate states

10

Go Serverless?

Pro:

• 1. Flexible concurrency
• 2. Instant response
• 3. Easy to deploy
• 4. Cheap? Runtime * MemSize

Con:

• 1. Execution model is too simple
• 2. Runtime limitations (~15min)
• 3. Communication overhead

11

λ

ML Training on Serverless?

• MapReduce on Serverless Cloud (PyWren, [SoCC’17])
• Video processing on Serverless Cloud (Sprocket [SoCC’18])

Stochastic Gradient Descent (SGD)

Input Samples

θ j = θ j +α(yi − h

θ(xi))xi j

13

Mini-batch SGD

Input Samples

θ j = θ j + a b

k=i i+b−1

∑ (yk − h

θ(xk ))x j k

14

Parameter Server

• Model replicas on workers
• Servers update parameters

Li, Mu, et al. Scaling distributed machine learning with the parameter server. OSDI'14

SGD on Lambda

…

θ j = θ j + a b

k=i i+b−1

∑ (yk − h

θ(xk ))x j k

θ j = θ j + a b

k=i i+b−1

∑ (yk − h

θ(xk ))x j k

θ j = θ j + a b

k=i i+b−1

∑ (yk − h

θ(xk ))x j k

Input Samples

• r

KV Storage

16

Function Function Function

• r

ML Training on Lambda

…

Input Samples KV Storage

17

Func. Func. Func.

Toy Example

• AWS Lambda
• 20 functions
• 150 functions
• X functions (dynamic # of func.)
• S3 storage
• EC2 c5.2xlarge
• 8 CPUs, 16GB mem
• Local storage
• Workload
• A logistic regression model

Toy Example

• Loss value v.s. training time
• Loss value v.s. monetary cost

20 functions 8-core EC2 150 functions X functions Loss value 0.010 0.015 Time (s) 100 200 300 Cost ($) 0.005 0.010 0.015 0.020 0.025 0.030

Toy Example

Slowest, no cheap Fastest, expensive Fast, cheap

• The first epoch: 120 functions
• The last epoch: 10 functions
• Intermediate epochs: 20 functions

X functions:

Challenges

• Functions on Serverless
• Limitation on performance and deployment
• Dynamic Resource Provisioning
• Speed v.s. cost (given a budget, how fast could be?)

Siren

• Hybrid Synchronous Parallel (HSP)
• Experience-Driven Resource Scheduler

Cloud

Architecture

User- Defined Model API Libs

DRL Agent Function Manager Local Client Scheduler

resource scheme

Step 1

function status

Step 3

action states code package

Stateless Functions

Enforce Parallelism on Siren

Synchronous or Asynchronous

Synchronous training Asynchronous training

Hybrid Synchronous Parallel (HSP)

… epoch t epoch t+1 time Fetching input Computing Updating parameters mini-batch …

ft,i ft,i+1 ft+1,i

Experience-Driven Scheduler

Toy Example - Find the X

Slowest, no cheap Fastest, expensive Fast, cheap

Deep Reinforcement Learning

Environment Agent

Stateless Functions

Policy parameters Features

θ

Action

at

Reward

rt

State

st−1

Policy

π(at|st−1, θ)

State

st = (t, ℓt, Pt, PF

t , PC t , PU t , ut, wt, bt)

Action

Approximating with Gaussian distribution

at = (nt, mt) nt, mt ∈ ℤ+

π(a|s, θ) = 1 σ(s, θ) 2π exp( − (a − μ(s, θ)2 2σ(s, θ)2 )

nt × mt choices ~ 138,000 actions on AWS

Policy

π(at|st−1, θ)

Reward

At each epoch ,

t

rt = − βPt, t = 1,…, T − 1

At the final epoch ,

T

regularizer a constant as the final reward/penalty reach the expected loss value,

• r use up all budget

Training

Policy gradient:

max

T

∑

t=1

γtrt, γ ∈ (0,1]

Maximize cumulative discounted reward:

∇θ𝔽π[

T

∑

t=1

γtrt] = 𝔽π[∇θln π(a|s, θ)qπ(s, a)]

policy function expected reward with and a s discount factor

DRL

Environment Agent

Stateless Functions

Policy parameters Features

θ

Action

at

Reward

rt

State

st−1

Policy

π(at|st−1, θ)

Cloud

Workflow

User- Defined Model API Libs

DRL Agent Function Manager Local Client Scheduler

resource scheme

Step 1

function status

Step 3 Step 5

action

Step 4

states code package

Stateless Functions

Step 2

Evaluation

• Simulation: OpenAI Gym
• Testbed: AWS Lambda + AWS EC2
• 44.3% ⬇ on job completion time

Simulation - overview

• Workload: mini-batched SGD algorithms
• Goal: DRL agent v.s. Grid search (# of functions)

Simulation - grid search

GS-300 GS-200 GS-100 GS-50 Total rewards −100 100 Number of functions 1000 2000 GS Siren 10.03% 12.87% 36% Time (s) 2000 3000 4000 5000 Budget ($) 100 200 300

Simulation

Simulation - DRL training

Siren-300 Siren-200 Siren-100 Number of functions 500 1000 Epoch of ML training 100 200 300 Siren-300 Total rewards −200 −100 100 Iteration 100 200 300

Testbed

• Siren on AWS Lambda v.s. MXNet on EC2
• m4.large: 2 vCPU, 8GB memory, $0.1/hr
• m4.xlarge: 4 vCPU, 16GB memory, $0.2/hr
• m4.2xlarge: 8 vCPU, 32GB memory, $0.4/hr
• Workload
• LeNet on MNIST
• CNN on movie review
• Linear Classification on click-through prediction dataset

Testbed - Siren and EC2 on LeNet

m4.large m4.xlarge m4.2xlarge Siren Cost($) 0.02 0.04 0.06 0.08 Time (s) 50 100 150 200 250 Number of EC2 instances 2 4 6 8 Siren

Testbed - DRL training

# of functions Memory # of functions 500 1000 Memory (MB) 200 400 600 800 Training epoch of LeNet 2 4 6 8 Total rewards −100 100 Iteration 100 200 300

Testbed - time v.s. cost

EC2 Siren Time (s) 100 200 300 400 Cost ($) 0.02 0.04 0.06 0.08

Testbed - given the same cost

50 100 LeNet 1000 2000 CNN 1000 2000 3000 Linear Classfication m4.2xlarge Siren Time (s)

Conclusion

• Siren: Distributed Machine Learning with a Serverless Architecture
• Hybrid Synchronous Parallel (HSP)
• Experience-Driven Resource Scheduler
• Evaluation
• Simulation & Testbed
• 44.3% ⬇ on job completion time

Q&A Thank You