Performance Benchmarking with Cloud Workbench (CWB) Presenters - - PowerPoint PPT Presentation

▶
performance benchmarking with cloud workbench cwb
SMART_READER_LITE
LIVE PREVIEW

Performance Benchmarking with Cloud Workbench (CWB) Presenters - - PowerPoint PPT Presentation

Jan 24, 2023 291 likes •822 views

Performance Benchmarking with Cloud Workbench (CWB) Presenters Joel Scheuner Philipp Leitner https://icet-lab.eu @IcetLab Chalmers 2 Performance Matters Chalmers 3 Benchmarking IaaS Clouds Chalmers 4 Capacity Planning in the

slide-1
SLIDE 1

Performance Benchmarking with Cloud Workbench (CWB)

Presenters Joel Scheuner Philipp Leitner

slide-2
SLIDE 2

Chalmers 2

https://icet-lab.eu @IcetLab

slide-3
SLIDE 3

Chalmers 3

Performance Matters

slide-4
SLIDE 4

Chalmers 4

Benchmarking IaaS Clouds

slide-5
SLIDE 5

Chalmers 5

Capacity Planning in the Cloud is hard

20 40 60 80 100 120 140 160 180 2 6 2 7 2 8 2 9 2 1 2 1 1 2 1 2 2 1 3 2 1 4 2 1 5 2 1 6 2 1 7 2 1 8 Number of Instance Type

t2.nano 0.05-1 vCPU 0.5 GB RAM $0.006/h x1e.32xlarge 128 vCPUs 3904 GB RAM $26.688 hourly

à Impractical to Test all Instance Types

Source: https://aws.amazon.com/blogs/aws/ec2-instance-history/

slide-6
SLIDE 6

Chalmers 6

Capacity Planning in the Cloud is hard

“The instance type itself is a very major tunable parameter”

@brendangregg re:Invent’17 https://youtu.be/89fYOo1V2pA?t=5m4s

slide-7
SLIDE 7

Chalmers 7

What cloud provider should I choose? Should I go for many small or few large instances? General-purpose or *-optimized? Pay for better IOPS or not? ……………

➡ Need for Benchmarking

slide-8
SLIDE 8

Chalmers 8

Basic Cloud Benchmarking Approach

Benchmark Manager Provider API

results provision

Instance

start benchmark destroy

slide-9
SLIDE 9

Chalmers 9

Basic Cloud Benchmarking Approach

CCGrid 2017 “An Approach and Case Study of Cloud Instance Type Selection for Multi-Tier Web Applications”

CWB Server Chef Server Vagrant

Scheduler

Provider API

IaaS Provider

JMeter Master SUT AcmeAir Webapplication MongoDB request DRIVER response Test Plan results

JMeter Slave

provision provision provision provision acquire start-up

CWB Client

Chef Client Chef Client JMeter Slave Chef Client Chef Client JMeter Slave Chef Client Chef Client
slide-10
SLIDE 10

Chalmers 10

Benchmark Types

Generic Artificial Resource-specific Specific Real-World Resource- heterogeneous

Micro Benchmarks

CPU Memory I/O Network Overall performance (e.g., response time)

Application Benchmarks

slide-11
SLIDE 11

Chalmers 11

Micro Benchmark Examples

1) Prepare

I/O

2) Run 3) Extract Result File I/O: 4k random read 4) Cleanup

3.5793 MiB/sec

Network

Bandwidth Server Client Result

972 Mbits/sec

Micro Benchmarks

CPU Memory I/O Network

slide-12
SLIDE 12

Chalmers 12

Application Benchmark Examples

Molecular Dynamics Simulation (MDSim) WordPress Benchmark (WPBench)

Multiple short blogging session scenarios (read, search, comment)

20 40 60 80 100 00:00 01:00 02:00 03:00 04:00 05:00 06:00 07:00 08:00 Elapsed Time [min] Number of Concurrent Threads

Overall performance (e.g., response time)

Application Benchmarks

slide-13
SLIDE 13

Chalmers 13

Cloud Workbench

CloudCom 2014 “Cloud Work Bench - Infrastructure-as-Code Based Cloud Benchmarking”

Tool for scheduling cloud experiments

Code: https://github.com/sealuzh/cloud-workbench Demo: https://www.youtube.com/watch? v=0yGFGvHvobk

slide-14
SLIDE 14

Chalmers 14

Planned Schedule

My First CWB Benchmark [~ 30 mins] CWB Architecture and Selected Previous Results [~ 30 mins] ~ Coffee Break ~ 🎊 Building and Running a Benchmark from Ground Up [~ 90 mins] Wrap-Up and Outlook [5 mins]

slide-15
SLIDE 15

Chalmers 15

My First CWB Benchmark

Interactive Session

slide-16
SLIDE 16

Chalmers 16

Online Material


http://bit.ly/cwb-tutorial

slide-17
SLIDE 17

Chalmers 17

Benchmarking with CWB

  • 1. Write benchmark config to setup environment


(optional for simple benchmarks)

  • 2. Declare IaaS resources and parametrize benchmark config
  • 3. Trigger execution or define periodic schedule
  • 4. Download metrics as CSV file
  • 5. Analyze results
slide-18
SLIDE 18

Chalmers 18

  • 1. Write benchmark config

Setup environment Write CWB execution hook

slide-19
SLIDE 19

Chalmers 19

  • 2. Declare IaaS resources and parametrize BM config
slide-20
SLIDE 20

Chalmers 20

  • 3. Trigger or schedule execution
slide-21
SLIDE 21

Chalmers 21

  • 4. Download metrics as CSV file
slide-22
SLIDE 22

Chalmers 22

CWB Architecture

slide-23
SLIDE 23

Chalmers 23

Provisioning Service Upload Configuration REST Experimenter

Configurations

slide-24
SLIDE 24

Chalmers 24

CWB Server Web Interface Provisioning Service REST Upload Configuration Access Web Interface REST Business Logic Scheduler

Relational Database

Experimenter

Configurations

slide-25
SLIDE 25

Chalmers 25

IaaS Provider IaaS Provider IaaS Providers CWB Server Web Interface Provisioning Service REST Upload Configuration Access Web Interface Provider API Manage VMs REST Business Logic Scheduler

Relational Database

REST Provider Plugin Experimenter

Configurations

slide-26
SLIDE 26

Chalmers 26

IaaS Provider IaaS Provider IaaS Providers CWB Server Web Interface Provisioning Service REST Upload Configuration Access Web Interface Provider API Manage VMs Provision VMs + Execute Commands REST REST Business Logic Scheduler

Relational Database

REST Provider Plugin Experimenter

Configurations

Cloud VM

Benchmark Execution Environment

Cloud VMs SSH Fetch Configuration

slide-27
SLIDE 27

Chalmers 27

IaaS Provider IaaS Provider IaaS Providers CWB Server Web Interface Provisioning Service REST REST Upload Configuration Access Web Interface Provider API Manage VMs Provision VMs + Execute Commands Notify State + Submit Metrics REST REST Business Logic Scheduler

Relational Database

REST Provider Plugin Experimenter

Configurations

Cloud VM CWB Client Library

Benchmark Execution Environment

Cloud VMs SSH Fetch Configuration

slide-28
SLIDE 28

Chalmers 28

IaaS Provider IaaS Provider IaaS Providers CWB Server Web Interface Provisioning Service REST REST Upload Configuration Access Web Interface Provider API Manage VMs Provision VMs + Execute Commands Notify State + Submit Metrics REST REST Business Logic Scheduler

Relational Database

REST Provider Plugin Experimenter

Configurations

Cloud VM CWB Client Library

Benchmark Execution Environment

Cloud VMs SSH Fetch Configuration

Ruby DSL for defining infrastructure (mostly VMs) Ruby DSL for configuring machines

slide-29
SLIDE 29

Chalmers 29

Experimenter / Scheduler CWB Server

Trigger Execution

Provider API

Acquire Resources

Cloud VM

Provision VM

Provisioning Service

Fetch VM Configurations Apply VM Configurations Start Benchmark Run Run Benchmark Notify Benchmark Completed Postprocess Results Notify Postprocessing Completed Release Resources Submit Metric(s)

Benchmark Execution Lifecycle

slide-30
SLIDE 30

Chalmers 30

Selected Previous Results

slide-31
SLIDE 31

Chalmers 31

Example Study 1 - Performance Testing of the Cloud

Study setup Benchmarked 22 cloud configurations using 5 benchmarks Two types of experiments Isolated: 300 - 500 repetitions Continuous: 15 repetitions per configuration

TOIT 2016 “Patterns in the Chaos - A Study of Performance Variation and Predictability in Public IaaS Clouds”

slide-32
SLIDE 32

Chalmers 32

slide-33
SLIDE 33

Chalmers 33

Results Summary

TOIT 2016 “Patterns in the Chaos

  • A Study of Performance

Variation and Predictability in Public IaaS Clouds”

slide-34
SLIDE 34

Chalmers 34

Observed CPU Models

(for m1.small and Azure Small in North America)

TOIT 2016 “Patterns in the Chaos - A Study of Performance Variation and Predictability in Public IaaS Clouds”

slide-35
SLIDE 35

Chalmers 35

Impact of Different Days / Times

(for m3.large in Europe)

TOIT 2016 “Patterns in the Chaos - A Study of Performance Variation and Predictability in Public IaaS Clouds”

10 20 30 40 50 00:00 04:00 08:00 12:00 16:00 20:00 Time of the Day IO Bandwidth [Mb/s] Mon Tue Wed Thu Fri Sat Sun Day of the Week 10 20 30 40 50 00:00 04:00 08:00 12:00 16:00 20:00 Time of the Day IO Bandwidth [Mb/s] Mon Tue Wed Thu Fri Sat Sun Day of the Week 10 20 30 40 50 00:00 04:00 08:00 12:00 16:00 20:00 Time of the Day IO Bandwidth [Mb/s] Mon Tue Wed Thu Fri Sat Sun 10 20 30 40 50 00:00 04:00 08:00 12:00 16:00 20:00 Time of the Day IO Bandwidth [Mb/s] Mon Tue Wed Thu Fri Sat Sun Day of the Week 10 20 30 40 50 00:00 04:00 08:00 12:00 16:00 20:00 Time of the Day IO Bandwidth [Mb/s] Mon Tue Wed Thu Fri Sat Sun Day of the Week 10 20 30 40 50 00:00 04:00 08:00 12:00 16:00 20:00 Time of the Day IO Bandwidth [Mb/s] Mon Tue Wed Thu Fri Sat Sun Day of the Week 10 20 30 40 50 00:00 04:00 08:00 12:00 16:00 20:00 Time of the Day IO Bandwidth [Mb/s] Mon Tue Wed Thu Fri Sat Sun Day of the Week 10 20 30 40 50 00:00 04:00 08:00 12:00 16:00 20:00 Time of the Day IO Bandwidth [Mb/s] Mon Tue Wed Thu Fri Sat Sun Day of the Week 10 20 30 40 50 00:00 04:00 08:00 12:00 16:00 20:00 Time of the Day IO Bandwidth [Mb/s] 10 20 30 40 50 00:00 04:00 08:00 12:00 16:00 20:00 Time of the Day IO Bandwidth [Mb/s] Mon Tue Wed Thu Fri Sat Sun Day of the Week 10 20 30 40 50 00:00 04:00 08:00 12:00 16:00 20:00 Time of the Day IO Bandwidth [Mb/s] Mon Tue Wed Thu Fri Sat Sun Day of the Week 20 40 60 00:00 04:00 08:00 12:00 16:00 20:00 Time of the Day IO Bandwidth [Mb/s] Mon Tue Wed Thu Fri Sat Sun Day of the Week 10 20 30 40 50 00:00 04:00 08:00 12:00 16:00 20:00 Time of the Day IO Bandwidth [Mb/s] Mon Tue Wed Thu Fri Sat Sun Day of the Week

slide-36
SLIDE 36

Chalmers 36

Recent Results

(unpublished data)

(Feb 2019) 2015

slide-37
SLIDE 37

Chalmers 37

Instance Runtime

(unpublished data)

(Feb 2019) 2015

3.5 4.0 4.5 5.0 5.5 20 40 60

Benchmark Runtime [h] Benchmark Value

Continuous io azure D2s

slide-38
SLIDE 38

Chalmers 38

Example Study 2 - Estimating Application Performance from Microbenchmarks

Research question: How accurately can we predict application performance from system-level microbenchmarks? Study setup: 2 applications (Wordpress, Molecular Dynamics Simulation) 23 microbenchmarks Study executed in AWS (11 instance types) Linear regression for prediction

Joel Scheuner, Philipp Leitner (2018). Estimating Cloud Application Performance Based on Micro-Benchmark Profiling. IEEE CLOUD.

slide-39
SLIDE 39

Chalmers 39

Joel Scheuner, Philipp Leitner (2018). Estimating Cloud Application Performance Based on Micro-Benchmark Profiling. IEEE CLOUD. CPU

  • sysbench/cpu-single-thread
  • sysbench/cpu-multi-thread
  • stressng/cpu-callfunc
  • stressng/cpu-double
  • stressng/cpu-euler
  • stressng/cpu-ftt
  • stressng/cpu-fibonacci
  • stressng/cpu-int64
  • stressng/cpu-loop
  • stressng/cpu-matrixprod

Memory

  • sysbench/memory-4k-block-size
  • sysbench/memory-1m-block-size

Broad resource coverage and specific resource testing

Micro Benchmarks

Micro Benchmarks

CPU Memory I/O Network

I/O

  • [file I/O] sysbench/fileio-1m-seq-write
  • [file I/O] sysbench/fileio-4k-rand-read
  • [disk I/O] fio/4k-seq-write
  • [disk I/O] fio/8k-rand-read

Network

  • iperf/single-thread-bandwidth
  • iperf/multi-thread-bandwidth
  • stressng/network-epoll
  • stressng/network-icmp
  • stressng/network-sockfd
  • stressng/network-udp
slide-40
SLIDE 40

Chalmers 40

Joel Scheuner, Philipp Leitner (2018). Estimating Cloud Application Performance Based on Micro-Benchmark Profiling. IEEE CLOUD.

1000 2000 25 50 75 100

Sysbench − CPU Multi Thread Duration [s] WPBench Read − Response Time [ms] Instance Type

m1.small m3.medium (pv) m3.medium (hvm) m1.medium m3.large m1.large c3.large m4.large c4.large c3.xlarge c4.xlarge c1.xlarge

Group

test train

Example Results: Predicting Wordpress

slide-41
SLIDE 41

Chalmers 41

Example Study 3 - Software Performance Testing in the Cloud

Research question: Executed 19 software performance tests in different environments How small performance regressions can we find? Study setup: 4 open source projects in Java and Go Study executed in AWS, Azure, Google Baseline: baremetal server in Softlayer / Bluemix

Christoph Laaber, Joel Scheuner, Philipp Leitner (2019). Software Microbenchmarking in the Cloud. How Bad is it Really? Empirical Software Engineering (EMSE). To appear.

slide-42
SLIDE 42

Chalmers 42

Background - JMH Microbenchmarks

MSR’18. An Evaluation of Open- Source Software Microbenchmark Suites for Continuous Performance Assessment.

slide-43
SLIDE 43

Chalmers 43

Summary - Variability of Software Benchmark Results

Christoph Laaber, Joel Scheuner, Philipp Leitner (2019). Software Microbenchmarking in the Cloud. How Bad is it Really? Empirical Software Engineering (EMSE). To appear.

slide-44
SLIDE 44

Chalmers 44

Sources of Variability

2 4 6

RSD

AWS CPU / etcd−2

25 50 75 100

RSD

Azure Std / etcd−2

25 50 75

RSD

AWS CPU / log4j2−5

10 20 30

RSD

GCE Mem / etcd−4

Per Trial Per Instance Total

Christoph Laaber, Joel Scheuner, Philipp Leitner (2019). Software Microbenchmarking in the Cloud. How Bad is it Really? Empirical Software Engineering (EMSE). To appear.

slide-45
SLIDE 45

Chalmers 45

Example Study 4 - Credit-based Bursting Instance Types

Research question: How do t2 instance types perform in terms of CPU and IO speed in comparison to other instances?
 When are t2 bursting instance types more cost-efficient than other instance types?
 How do t2 instance types perform in comparison to the previous generation (t1) types? Study setup: Sysbench CPU and IO benchmarks Study executed in AWS
 3 bursting vs non-bursting instance types

Philipp Leitner, Joel Scheuner (2015). Bursting With Possibilities – an Empirical Study of Credit-Based Bursting Cloud Instance Types (UCC).

slide-46
SLIDE 46

Chalmers 46

Example Study 4 - Credit-based Bursting Instance Types

  • 10x
100 200 300 18:10 18:20 18:30 18:40 18:50 19:00 19:10 19:20 19:30 19:40 19:50 20:00 20:10 20:20 20:30

Experiment Duration Execution Time (s)

25 50 75 100 18:10 18:20 18:30 18:40 18:50 19:00 19:10 19:20 19:30 19:40 19:50 20:00 20:10 20:20 20:30

Experiment Duration CPU Time (%)

CPU Time user steal idle Peak Baseline

Peak Baseline

t2.micro 1 2 3 4 t2.micro − Peak t2.micro − Base

Medium−Instance Equivalents

t1.micro t1.micro − Peak

Burstable T2.* highly predictable unlike previous generation T1.*

slide-47
SLIDE 47

Chalmers 47

Building and Running a Benchmark from Ground Up

Interactive Session

http://bit.ly/cwb-tutorial

slide-48
SLIDE 48

Chalmers 48

Wrap-Up and Outlook

slide-49
SLIDE 49

Chalmers 49

Three Steps Towards a Benchmark

(Optional) Step 1: Write Chef Cookbook

slide-50
SLIDE 50

Chalmers 50

Three Steps Towards a Benchmark

Step 1I: Define IaaS config and schedule

slide-51
SLIDE 51

Chalmers 51

Three Steps Towards a Benchmark

Step III: Execute and download result CSV

slide-52
SLIDE 52

Chalmers 52

Cloud Workbench

CloudCom 2014 “Cloud Work Bench - Infrastructure-as-Code Based Cloud Benchmarking”

Tool for scheduling cloud experiments

Code: https://github.com/sealuzh/cloud-workbench Demo: https://www.youtube.com/watch? v=0yGFGvHvobk