CS 744: CLIPPER Shivaram Venkataraman Fall 2020 ADMINISTRIVIA - - PowerPoint PPT Presentation

CS 744: CLIPPER

Shivaram Venkataraman Fall 2020

Good

morning !

ADMINISTRIVIA

Course Project Proposals

• Due on Friday!
• See Piazza for template
• Submission instructions soon

Midterm details

• Open book, open notes
• Held in class time 9.30-10.45am Central Time
• Type / Upload photos (extra 15 mins)

section

/

dangle

midterms

→ at the

→

upto

ML

• n Pinera

MACHINE LEARNING: INFERENCE

÷:

:

÷:

÷

O

O

GOALS

• Interactive latencies (tail latency < 100ms)
• High throughput to handle load
• Improved prediction accuracy
• Generality (?)

(

Fw!!

ggg percentile

• r

99.9

"

percentile

→

how

latency

[ - - -iger

→

many

many

requests

that

need

to be

made

↳

ML

specific

II t \

handle

many mi

models /

frameworks

qwitpj.AM?.wxe..m

possible

ARCHITECHTURE

Requests

HTTP

inform

• ver

^

\

• n*iJL÷¥÷÷

.

t

• Improve

accuracy eager

'

• r

eight L

• → Replicates

It [

• ↳ zseikit

/

• ↳ a spark

rabbi

fair

herd

←

X

D Deel

go

←

L

t

MODEL CONTAINERS

Run using Docker containers Can be replicated across machines

tint #dell

people

API

÷dciM

↳

Interface

• nce

per

framework

'ate TF shim

]µ initiative

• 'r

model

instantiate

TF

Mim

frameworks

are →

.pe?f

1¥

win rent

• so

MODEL ABSTRACTION LAYER

Caching

• Improve performance for frequent queries
• LRU eviction policy
• Important for feedback

datapoint

Predict

, * good

gpeoloinieddder

¥kiEm;;

to

predict

→

• es

• 101

• r

spark -50

Tt

Her

:

• T

feedback

high

dir ith

¢

I

↳

www.tfiidrYDTE.jo/iv.::.:Eiesf&dback

and

Predictions

• update

BATCHING, QUEUING

Goals, Insight

• Increase latency (within SLO)

for improved throughput

• Reduce RPC overheads
• GPU / BLAS acceleration

Approach

• Per container queues.
• Why?

To

do

RPC

that

+

put while

↳ fixed

lost

eating SLO

• (

both

• verhead
• → amortize

batches

we lead

hasmdddisswt

;÷%

:L

→ hardware

• pium

could vary

f-

each

model !

✓

parellism

Gpu

Cpu

ADAPTIVE BATCHING

1 2 3 4 5 2 4 6 8 10 Batch Size Time

AIMD: Additive Inc Multiplicative Dec Why ? Delayed: Wait until batch exists Why?

• bserve

latency

• LAND

)

write

late

SLO Increase

batch

in

carefully

inc .

batch

4

& ed-domrmaffiHE.7.su !

FL

f

• ①

2

Collect

examples

Gang

upto

certainties

then

dinette

Elias

→link) should

wait ?

↳Der

few

should Iad ?

MODEL SELECTION

→

Improve

Accuracy

ensembles

÷

→

• y

SINGLE MODEL SELECTION

Multi-Arm Bandit formulation

• Explore vs Exploit
• Regret: Loss by not

picking optimal action

• Goal: Minimize regret

Clipper

• Exp3 algorithm
• Single evaluation
• Scales to more models

a n÷÷÷

• →

atta "¥a-get

with each

• ption
• update

weights

based

¥y

• n feedback

④ model I

Omodd2

MULTI MODELS

Ensemble

• Combine output from models (weighted average)
• How do we get the weights ?

Robust Prediction

• React to model changes
• Output confidence score

→

ensembles predict

movies

it 5¥

\

ft

Este

Apart linear

Combination tf

=L y.at/32i

• se

t

t

Binary

classifier

• cat

Expo

update

d &

B

0.25

• .

CZ

• 4

• 6

↳

combine

&

threshold

dog

STRAGGLER MITIGATION

Why do stragglers occur? Approach

• 9TH

wait

for

N

model

containers

to

totem

reply ,

some

• f

them

might

be

slow ?

↳

replicas

locating ?

↳

Approx

result

based

• n

1¥

whatever

has

finished

→ Better

approx

them

late !

specific

SUMMARY

• Clipper: ML inference Workloads + Requirements
• Layered architecture provides generality
• Caching, Batching, Replication to improve latency, throughput
• Multi-Arm bandits to improve accuracy

DISCUSSION

https://forms.gle/FCVhPURqz7HSbDtg6

Consider a scenario where you run a model serving service that hosts a number of different applications. The traffic for some applications is sporadic (e.g. only a few hours where they are used). What are some advantages / disadvantages of using Clipper for such a service?

Advantages

Disadvantages

Rade might

contented

→

Adaptive

batching

delayed

→

multiple

replicas

• hooted

.net?fashim ?

elasticity

roti

frequent

greeted

→ Containerization

inlet

applications

• ppc

T

• we. provided

pt⇒

Effie

bing.g.ms

.

homie :O

:L,

smug

)

different

things ?

judith

?

O

D

O

O -

D

Ao

:

↳ µ

ensembles

Treasonable

accurate

tetany inflation

very

low