eSulabSolutions Integrating Automated Scalability - - PowerPoint PPT Presentation

▶
esulabsolutions integrating automated scalability
SMART_READER_LITE
LIVE PREVIEW

eSulabSolutions Integrating Automated Scalability - - PowerPoint PPT Presentation

Oct 03, 2022 429 likes •753 views

eSulabSolutions Integrating Automated Scalability Assessment into DevOps Alberto Avritzer et al. The Eighth International Workshop on Load Testing and Benchmarking of Software Systems (LTB 2020) Our

slide-1
SLIDE 1


 
 
 
 
 Integrating Automated Scalability Assessment into DevOps 
 
 


Alberto Avritzer et al.

The Eighth International Workshop on Load Testing and Benchmarking of Software Systems (LTB 2020)

eSulabSolutions

slide-2
SLIDE 2

Avritzer et al.: Automated Scalability Assessment in DevOps 2

Our Team

‘

eSulabSolutions, 


near Princeton University

Alberto Avritzer André van Hoorn Henning Schulz Vincenzo Ferme Barbara Russo Andrea Janes Daniel Menasche Vilc Rufino

slide-3
SLIDE 3

Avritzer et al.: Automated Scalability Assessment in DevOps

  • Senior Researcher at Siemens and AT&T Bell Labs for about 24 years
  • Published over 70 papers in journals, refereed conference proceedings, and book

chapters in those areas: (http://dblp.uni-trier.de/pers/hd/a/Avritzer:Alberto)

  • Resilience Assessment and Evaluation of Computing Systems. Springer 2012
  • Handbook of Software Aging and Rejuvenation, World Scientific, May 2020, World Scientific

(https://worldscientific.com/worldscibooks/10.1142/11673)

  • Siemens/MTA: The NY Subway PA/CIS Scalability Assessment
  • Load Testing and Performance Analysis of AT&T Operations support systems:
  • Monitoring for Software aging and Rejuvenation (1993)
  • Performance testing using Markov chain (1995)
  • Founder of eSulabSolutions (https://esulabsolutions.godaddysites.com/):
  • Automated scalability assessment in DevOps and micro service architecture

3

eSulabSolutions About Alberto Avritzer

slide-4
SLIDE 4

Motivation: Recent Scalability Related Disasters

4

NYS Labor website crashes Americans crash the Canadian immigration

slide-5
SLIDE 5

Avritzer et al.: Automated Scalability Assessment in DevOpsps

Motivation: Influence of Poor Performance on the Success of Businesses

5

slide-6
SLIDE 6

Avritzer et al.: Automated Scalability Assessment in DevOps 6

Examples of Operational Profile Representations

Observed load situations

Time Load Level

Empirical distribution of load situations

Load level

  • Rel. Freq.
slide-7
SLIDE 7

Avritzer et al.: Automated Scalability Assessment in DevOps

»DevOps is ..

SECURE RELIABLE RESILIENT FAST COST- EFFICIENT ELASTIC … a set of
 intended to
 reduce the time between committing a change to a system and the change being placed into normal production, 
 
 while ensuring .« HIGH QUALITY

Practices Chaos Engineering Live Experi- mentation Automation CD Pipelines Cultural Change Micro-
 services M

  • n

i t

  • r

i n g Polyglot Agile Fast Feedback Docker *

  • a

s

  • C
  • d

e

7

slide-8
SLIDE 8

Avritzer and van Hoorn: Operational Profile Data for Continuous Dependability Assessment in DevOps

  • A. van Hoorn (U Stuttgart) – Continuous Engineering of Dependable Distributed Software Systems


SECURE RELIABLE RESILIENT FAST COST- EFFICIENT ELASTIC HIGH QUALITY

Chaos Engineering Live Experi- mentation Automation CD Pipelines Cultural Change Micro-
 services M

  • n

i t

  • r

i n g Polyglot Agile Fast Feedback Docker *

  • a

s

  • C
  • d

e

My Questions of Interest

8

How can 
 DevOps practices support 
 quality assurance? How to integrate „established“ quality assurance with DevOps practices?

slide-9
SLIDE 9
  • PPTAM: Production and Performance Testing Based Application

Monitoring

Avritzer et al.: Automated Scalability Assessment in DevOps

slide-10
SLIDE 10

10

Scalability Requirement Automated Measurement

Avritzer et al.: Automated Scalability Assessment in DevOps

slide-11
SLIDE 11

Avritzer et al.: Automated Scalability Assessment in DevOps

… and Classic Problems The Classic Load Testing Approach

There are no suitable load tests Complex analysis of performance regressions High manual effort for maintaining load tests Load tests need much time to execute Workload Specification Test Results Evaluation Expert Load Driver Expert/Basic Rules

SUT

LOA D

11

slide-12
SLIDE 12

Avritzer et al.: Automated Scalability Assessment in DevOps

… How Problems Get Worse Load Testing in Continuous Delivery Pipelines

void main(String[] args) {
 int foo;
 
 // do something
 bar(foo);
 
 System.out.println(“Hi“);
 }

Implementation Build Performance Testing Functional Testing

void main(String[] args) {
 int foo;
 
 // do something
 bar(foo);
 
 System.out.println(“Hi“);
 } void main(String[] args) {
 int foo;
 
 // do something
 bar(foo);
 
 System.out.println(“Hi“);
 } main(Strin
 int foo;
 
 // do somet
 bar(foo);

vs. Pipeline automation Fast & frequent releases vs. Complex load tests for every release impossible vs. Complex analysis of performance regressions High manual effort for maintaining load tests Load tests need much time to execute Service-focus requires multiple tests vs. There are no suitable load tests

12

slide-13
SLIDE 13
  • PPTAM: Production and Performance Testing Based Application

Monitoring

Avritzer et al.: Automated Scalability Assessment in DevOps

slide-14
SLIDE 14
  • Challenge: assess performance of architectural deployment alternatives


(e.g., number of replicas, CPU/memory allocation, technology stack) under fuzzy requirements

  • Our approach
  • Use operational data to generate and weigh load tests
  • Measure baseline requirements
  • Metric allows quantitative comparison of deployment alternatives
  • Builds on previous work from telecommunication systems:


Avritzer, A., Weyuker, E.J.: The automatic generation of load test suites and the assessment of the resulting software. IEEE Trans. Softw. Eng. 21(9) (Sep 1995)

14

Quantitative Assessment of Deployment Alternatives

Alberto Avritzer, Vincenzo Ferme, Andrea Janes, Barbara Russo, Henning Schulz, and André van Hoorn: 


A Quantitative Approach for the Assessment of Microservice Architecture Deployment Alternatives by Automated Performance Testing. 


In Proceedings of the 12th European Conference on Software Architecture (ECSA). LNCS, Springer, 2018 (Accepted)

slide-15
SLIDE 15

Avritzer et al.: Automated Scalability Assessment in DevOps 15

Overview of Approach

Observed load situations

Time Load Level

Production Baseline test

Sampled load tests

Scalability criteria Deployment conf.

Empirical distribution of load situations

Load level

  • Rel. Freq.

1

Empirical Distribution of Load situations

Sampled load level

  • Aggr. Rel. Freq.

2

0.12 0.14 0.20 0.16 0.11

Test results

3

Domain Metric

0.73

4

slide-16
SLIDE 16

Avritzer et al.: Automated Scalability Assessment in DevOps 16

Experiments

Scal = avg + 3σ Production

12 microservices

Custom Op. Mix

Sampled Load Tests

Empirical Distribution of Load situations

6 Load Levels

50,100,150,200,250,300 Concurrent user sessions

1,2

Deployment Config.

10 configurations Replicas CPU

RAM

3

slide-17
SLIDE 17

Avritzer et al.: Automated Scalability Assessment in DevOps

17

Experiment Results: Computation of Domain Metric (1/2)

Custom Op. Mix Users

  • Aggr. Rel. Freq.

50 0.10582 100 0.18519 150 0.22222 200 0.22222 250 0.20370 300 0.06085

  • Aggr. Rel. Freq.
  • Contrib. to Domain Metric

Max: 0.20370 Actual: 0.13580 Deployment Configuration: 1 GB RAM, 0.25 CPU, 1 Replica API Scalability Criteria GET / PASS GET /cart PASS POST /item FAIL

slide-18
SLIDE 18

Avritzer et al.: Automated Scalability Assessment in DevOps 18

Experiment Results: Computation of Domain Metric (2/2)

Users Contribution 50 0.10582 100 0.18519 150 0.22222 200 0.07999 250 0.13580 300 0.04729

  • Contrib. to Domain Metric

Deployment Configuration: 1 GB RAM, 0.25 CPU, 1 Replica

Max: 1 Domain Metric

4

0.77631

Actual:

slide-19
SLIDE 19

Avritzer et al.: Automated Scalability Assessment in DevOps 19

Experiment Results: Single-Metric Comparison of Alternatives

slide-20
SLIDE 20

Avritzer et al.: Automated Scalability Assessment in DevOps 20

Experiment Results: Visual Comparison of Alternatives

  • Contrib. to Domain Metric

Sampled Load Tests

Max Contrib.

  • Depl. Conf.
slide-21
SLIDE 21
  • Production and Performance Testing Based Application Monitoring

PPTAM

Avritzer et al.: Automated Scalability Assessment in DevOps

slide-22
SLIDE 22

PPTAM

Mirai

Avritzer and Russo: Operational Profile Data for Continuous Dependability Assessment in DevOps

slide-23
SLIDE 23

23

  • Mirai is a malware that has been used to turn networked devices

running Linux into remotely controlled bots

  • We use it to attack the system. It can perform different types of attack
  • by now, we have explored http, syn, ack

Mirai BotNet

Avritzer and Russo: Operational Profile Data for Continuous Dependability Assessment in DevOps

slide-24
SLIDE 24

24

PPTAM, No attack and attack plots with 50 users

slide-25
SLIDE 25
  • Apply the approach to a large telecom system

Identify Scalability Impacting Architecture Components

Arch Cockpi t APM Mirai

slide-26
SLIDE 26

Counter_name Measurement Load Service_name Response Time_X Y L S Response Time_X Y L S Response Time_X Y L S Response Time_X Y L S

What is the problem with this log?

Identify Scalability Impacting Architecture Components from Performance Testing Log

26

Avritzer et al.: Automated Scalability Assessment in DevOps

slide-27
SLIDE 27

Counter_Name Low Load_ RT Std _RT Baseline_RT Response Time X 2.488602 3.513397 12.033352

Compute Scalability Baseline, Normalized Distance from Baseline, and Linear Regression Slope

27

slide-28
SLIDE 28

Counter_Name Low Load_ RT Std _RT Baseline_RT Response Time X 2.488602 3.513397 12.033352

Compute Scalability Baseline, Normalized Distance from Baseline, and Linear Regression Slope

28

slide-29
SLIDE 29

Multivariate analysis used to prioritize re-factoring using Slope and Normalized Distance Profile

29

slide-30
SLIDE 30

Summary

  • DevOps provides access to real-time

data, enables production and testing integration

  • Use cases
  • Performance, Scalability, Security
  • other illities: survivability, resilience
  • Provide detailed guidance to

architects and developers in real time:

  • Domain-metric based scalability

assessment dashboards

  • Prioritize component re-factoring

based on load testing results

slide-31
SLIDE 31
  • A. Avritzer, D. S. Menasché, V. Rufino, B. Russo, A. Janes, V.

Ferme, A. van Hoorn, H. Schulz, PPTAM: Production and Performance Testing

Based Application Monitoring. ICPE Companion 2019: 39-40

  • A. Avritzer, V. Ferme, A. Janes, B. Russo, H. Schulz, A. van

Hoorn, A Quantitative Approach for the Assessment of Microservice Architecture

Deployment Alternatives by Automated Performance Testing. ECSA 2018: 159-174

  • A. Avritzer, D. S. Menasché, V. Rufino, B. Russo, A. Janes, V.

Ferme, A. van Hoorn, H. Schulz, Scalability Assessment of Microservice

Architectural Configurations: A Domain-based Approach Leveraging Operational Profiles and Load Tests, Journal of Systems and Software, to appear 2020.

Publications