Beyond Microbenchmarks The SPEC-RG Vision for A Comprehensive - - PowerPoint PPT Presentation

▶
beyond microbenchmarks
SMART_READER_LITE
LIVE PREVIEW

Beyond Microbenchmarks The SPEC-RG Vision for A Comprehensive - - PowerPoint PPT Presentation

Sep 27, 2022 44 likes •265 views

Beyond Microbenchmarks The SPEC-RG Vision for A Comprehensive Serverless Benchmark Erwin van Eyk Joel Scheuner Simon Eismann Cristina L. Abad Alexandru Iosup HotCloudPerf @ ICPE 2020 SPEC RG CLOUD Serverless Activity Exploring

slide-1
SLIDE 1

Erwin van Eyk Joel Scheuner Simon Eismann Cristina L. Abad Alexandru Iosup

Beyond Microbenchmarks

The SPEC-RG Vision for A Comprehensive Serverless Benchmark

HotCloudPerf @ ICPE 2020

slide-2
SLIDE 2

SPEC RG CLOUD Serverless Activity

Exploring community-wide (performance) challenges in serverless computing.

https://research.spec.org/working-groups/rg-cloud.html

2

slide-3
SLIDE 3
  • 1. An introduction to serverless computing.
  • 2. Their current state and challenges.
  • 3. Motivations for a (new) serverless benchmark.
  • 4. Goals and design of the benchmark.
  • 5. Roadmap

Outline

3

Serverless computing is in a desperate need of more benchmarks, which require your expertise.

slide-4
SLIDE 4

Decomposing cloud applications

Cloud Application

Operational Logic

Business Logic

Cloud users: reduce (complex) operational logic Cloud providers: improve resource utilization Logic related to QoS

  • f the application

Logic directly related to use case

  • f the user

4

slide-5
SLIDE 5

Serverless and Function-as-a-Service

Serverless Computing 1. Minimal operational logic 2. Granularly billed, by usage 3. Event-driven executions Function-as-a-Service (FaaS)

  • A form of serverless computing
  • User provides a function (source)
  • cloud provider deploys and

manages the function

5

Van Eyk, Erwin, et al. "The SPEC cloud group's research vision on FaaS and serverless architectures." Proceedings of the 2nd International Workshop on Serverless Computing. 2017.

slide-6
SLIDE 6

Function-as-a-Service Market estimated to be worth $7.72 Billion by 2021

https://www.marketsandmarkets.com/Market-Reports/function-as-a-service-market-127202409.html http://get.cloudability.com/ebook-state-of-cloud-2018.html

2017

Adoption of FaaS (AWS Lambda) Gartner 2017 Function-as-a-Service is one

  • f the top trends in cloud computing.

6

slide-7
SLIDE 7

7

slide-8
SLIDE 8

8

Conceptually

Key problem: how to compare FaaS platforms?

Azure Functions

  • r

Functionally Architecturally Performance-wise

slide-9
SLIDE 9

Challenges of benchmarking FaaS platforms

  • 1. Performance requirements
  • 2. Opaqueness of the systems
  • 3. System heterogeneity
  • 4. Complex ecosystems

9

slide-10
SLIDE 10

10

Scheuner, Joel, and Philipp Leitner. "The State of Research on Function-as-a-Service Performance Evaluation: A Multivocal Literature Review." arXiv preprint arXiv:2004.03276 (2020).

The need for yet another serverless benchmark

Prior performance evaluation studies exist, but tend to use microbenchmarks to focus

  • nly on...

1. Hardware-resource performance 2. Start-up latency 3. Concurrency and elasticity

slide-11
SLIDE 11

11

Scope of the serverless benchmark

Van Eyk, Erwin, et al. "The SPEC-RG Reference Architecture for FaaS: From Microservices and Containers to Serverless Platforms." IEEE Internet Computing (2019)

slide-12
SLIDE 12

Focus on cost Support open-source platforms Realistic applications Evaluate external service integrations

12

Scope of the serverless benchmark (2)

Eismann, Simon, et al. "A Review of Serverless Use Cases and their Characteristics." (work-in-progress)

slide-13
SLIDE 13

13

Serverless benchmark design

slide-14
SLIDE 14

14

Roadmap and future work

First version of benchmark GDPR-compliant workloads Data-intensive workloads Other serverless models Add more (OSS) platforms Realistic applications Explore platform aspects

slide-15
SLIDE 15

Serverless computing is (still) in a desperate need of more benchmarks, which require your expertise.

Take-away message

15

slide-16
SLIDE 16

Interested in serverless computing? Join us!

https://research.spec.org/working-groups/rg-cloud.html @erwinvaneyk e.vaneyk@atlarge-research.com

16

slide-17
SLIDE 17

Additional Slides

17

slide-18
SLIDE 18

Challenges of benchmarking FaaS platforms (2)

18

Motivating experiments: performance and cost differences are non-negligible.

slide-19
SLIDE 19

Evolution of cloud application architectures

Infrastructure Operational Logic Application Monoliths Infrastructure OL OL OL OL μs μs μs μs (Micro)services Infrastructure Operational Logic Function Function Function Function Function Function FaaS

19

slide-20
SLIDE 20

20

High-level goals (and non-goals) of the benchmark

  • three pillars
  • metrics

Add ref arch figure

Van Eyk, Erwin, et al. "The SPEC-RG Reference Architecture for FaaS: From Microservices and Containers to Serverless Platforms." IEEE Internet Computing (2019)

Scope of the serverless benchmark (2)

Event propagation Software flow Function runtime Cold starts

slide-21
SLIDE 21

The goal of SPEC RG CLOUD - Serverless

Contributing to a deeper understanding of serverless and FaaS architectures, with a focus on performance (evaluation).

2018 - Further exploration of serverless performance [2] 2019 - Reference architecture for FaaS platforms [3] 2020 - Use case survey and serverless benchmark

21

2017 - Initial exploration of serverless concepts [1]

slide-22
SLIDE 22

Further reading

[1] The SPEC Cloud Group's Research Vision on FaaS and Serverless Architectures Second International Workshop on Serverless Computing (WoSC@Middleware 2017) [2] A SPEC RG Cloud Group's Vision on the Performance Challenges of FaaS Cloud Architectures 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018) [3] The SPEC-RG Reference Architecture for FaaS: From Microservices and Containers to Serverless Platforms IEEE Internet Computing (Nov/Dec issue, 2019)

22