Serverless Platforms Diptanu Choudhury QCon New York @diptanu 1 2 - - PowerPoint PPT Presentation

serverless platforms
SMART_READER_LITE
LIVE PREVIEW

Serverless Platforms Diptanu Choudhury QCon New York @diptanu 1 2 - - PowerPoint PPT Presentation

Serverless Platforms Diptanu Choudhury QCon New York @diptanu 1 2 The GOES Mission A lineage of geostationary satellites Current mission is called GOES-R ABI views earth with 16 different spectral bands GOES-R Products Atmospheric


slide-1
SLIDE 1

Serverless Platforms

Diptanu Choudhury @diptanu

1

QCon New York

slide-2
SLIDE 2

2

slide-3
SLIDE 3

The GOES Mission

A lineage of geostationary satellites Current mission is called GOES-R ABI views earth with 16 different spectral bands

slide-4
SLIDE 4

GOES-R Products

  • Atmospheric events
  • Fire control
  • Vegetation Monitoring
  • NASA makes most of the data available to

research community

slide-5
SLIDE 5

5

Product Development Workflow

slide-6
SLIDE 6

6

Product Development Workflow

Iterative Experimentation
 based Model Development Deployment of Models 
 in Production and 
 Image Processing Data serving 
 and 
 Collaboration APIs

slide-7
SLIDE 7

7

The organization

Science Engineering

  • Plans mission objectives
  • Develop models and products
  • Conducts research and drives collaboration
  • Operates infrastructure to run models
  • Design systems to scale to petabytes of data and 10s

  • f 1000s of cores.

  • Design real time services for services to consume data
  • Dev-Infra to improve productivity of researchers
slide-8
SLIDE 8

Infrastructure

  • Cluster Schedulers that can scale to 10s of 1000s of cores
  • Storage services that can scale to petabytes of data
  • Interactive batch processing systems
  • Services to stream real time information
  • APIs to access data from various transformation jobs
slide-9
SLIDE 9

9

Job Management System Authorization/ Authentication Service Identity Management Telemetry System Compute Node Compute Node Object Store

Jobs

slide-10
SLIDE 10

10

slide-11
SLIDE 11

Infrastructure Stack of Circa 2017

Locking and Co-ordination Management Cluster Schedulers Block Storage Service Discovery Telemetry Services Object Store Microservices Batch Processing LBaaS Identity Management Tracing Services Applications

The Platform

slide-12
SLIDE 12

Platforms as a Service

Locking and Co-ordination Management Cluster Schedulers Block Storage Service Discovery Telemetry Services Object Store Microservices Batch Processing LBaaS Identity Management Tracing Services Applications Release Management Build Infrastructure Configuration Management

slide-13
SLIDE 13

13

Serverless Architecture

slide-14
SLIDE 14

Serverless Architecture

Function Controller Requests Application Logic Application Container

Payload

slide-15
SLIDE 15

Serverless Architecture

  • Incremental evolution over PAAS
  • Serverless from the perspective of users
  • Clear separation of concerns between

infrastructure engineering and application developers

slide-16
SLIDE 16

Serverless Platform

  • Optimistically concurrent Cluster Scheduler
  • Application Container
  • Function Controller
  • RPC Middleware
  • Function Invocation shims in data sources
  • Packaging and distribution service
slide-17
SLIDE 17

Cluster Scheduler

  • Low latency and high throughput scheduling
  • No head of line blocking
  • Optimistically concurrent
  • Scale up to a large number of containers
slide-18
SLIDE 18

18

slide-19
SLIDE 19

19

THE PATH TO PRODUCTION FOR A SCHEDULER IS FILLED WITH PAIN

CHOOSE AN EXISTING SCHEDULER IF YOU CAN

slide-20
SLIDE 20

Event Driven Scheduler

Evaluations ~= State Change Event

slide-21
SLIDE 21

Data Model

slide-22
SLIDE 22

22

External Event Evaluation Creation Evaluation Queuing Evaluation Processing Optimistic Coordination State Updates

slide-23
SLIDE 23

Templated Jobs

  • Can be used for low throughput use cases
  • Expensive to start new processes
  • Bounded by network operations of scheduler
slide-24
SLIDE 24

Variability

  • Sharing resources
  • Daemons
  • Global resources
  • Maintainance activities
  • Queuing
  • Power Limits
slide-25
SLIDE 25

25

Source: Tale at Scale

slide-26
SLIDE 26

Reduce Variability

  • Differentiate service classes
  • Break down long running functions into small ones.
  • Minimize background activity
slide-27
SLIDE 27

Application Containers

  • Application containers range from language sandboxes

to full blown containers

  • Reduced surface area helps in making the UX better
  • Application containers have function invocation

middleware which are packaged during build

slide-28
SLIDE 28

Function Controller

  • Maintains the network topology of functions
  • Optimizes for low latency and not cost
  • Works with the scheduler to scale containers
  • Works with telemetry system to scale underlying

clusters

slide-29
SLIDE 29

Software Load Balancer

  • Service discovery aware LB
  • Reactive Socket protocol for delivery of request payloads to

function containers

  • Doesn't effect container life cycles
  • Drops requests which doesn't have any containers associated
slide-30
SLIDE 30

Reactive Sockets

  • RSocket is a means for asynchronous communication between functions and

data sources

  • Messages are multiplexed over a single connection
  • Flexible interaction models
  • Provides mechanisms for back pressure and request cancellation
slide-31
SLIDE 31

RPC Middleware

  • RPC implementation moved into the platform
  • Hedged requests to reduce tail latency
  • Tied requests to reduce mean and tail latency
  • Heavy use of back pressure
slide-32
SLIDE 32

Cluster Topology

  • Scheduler chooses the cluster topology for containers
  • Function Controller can change the topology based on

real time latency

  • Functions which are chained needs to be placed as

close as possible

slide-33
SLIDE 33

Caching

  • Various forms of caching
  • Depends on life cycle management of functions
  • External caches like Redis, Memcached works best
  • Hard to use in-memory clustered caches like

Groupcache

slide-34
SLIDE 34

External Data Sources

  • Functions can read from external data sources just like

traditional applications

  • Avoid polling external data sources
  • Data Sources like Kafka could co-operate with functions by a

shim which calls functions with data mutations

slide-35
SLIDE 35

Build and Distribution

  • Packaging is part of the server less experience
  • Packaging pipeline should transform a function to an

application container.

  • Unit Testing should be much easier with Functions than

they were for traditional applications

  • Performance of the platform can be improved by caching

containers on compute nodes

slide-36
SLIDE 36

Operations in the world of Serverless

  • Reduced amount of operations for developers
  • Metrics related to various concerns of the platform should

be very targetted.

slide-37
SLIDE 37

Application metrics

  • Latency at the API Gateway level
  • Throughput of events processed
  • Latency distribution and planning.
slide-38
SLIDE 38

Platform Metrics

  • Latency of function invocation
  • Throughput of events dispatched to the

platform

  • Number of active functions