Visualization of Performance Anomalies with Kieker Bachelors Thesis - - PowerPoint PPT Presentation

▶
visualization of performance anomalies with kieker
SMART_READER_LITE
LIVE PREVIEW

Visualization of Performance Anomalies with Kieker Bachelors Thesis - - PowerPoint PPT Presentation

Sep 24, 2023 129 likes •698 views

Visualization of Performance Anomalies with Kieker Bachelors Thesis Sren Henning September 8, 2016 Sren Henning Visualization of Performance Anomalies September 8, 2016 1 / 41 Outline 1. Introduction 2. Foundations 3. Approach 4.

slide-1
SLIDE 1

Visualization of Performance Anomalies with Kieker

Bachelor’s Thesis Sören Henning September 8, 2016

Sören Henning Visualization of Performance Anomalies September 8, 2016 1 / 41

slide-2
SLIDE 2

Outline

  • 1. Introduction
  • 2. Foundations
  • 3. Approach
  • 4. Evaluation
  • 5. Conclusion and Future Work

Sören Henning Visualization of Performance Anomalies September 8, 2016 2 / 41

slide-3
SLIDE 3

Motivation

Introduction

◮ Performance is a significant quality characteristic

◮ e.g., Amazon: 100 ms delay → 1% decrease in sales (Huang 2011) ◮ e.g., Google: 500 ms delay → 20% drop in traffic (Huang 2011) Sören Henning Visualization of Performance Anomalies September 8, 2016 3 / 41

slide-4
SLIDE 4

Motivation

Introduction

◮ Performance is a significant quality characteristic

◮ e.g., Amazon: 100 ms delay → 1% decrease in sales (Huang 2011) ◮ e.g., Google: 500 ms delay → 20% drop in traffic (Huang 2011)

◮ Detect performance issues to react on them

◮ As soon as possible ◮ Use monitoring (e.g., measure execution times) ◮ Investigate these measurements for anomalies Sören Henning Visualization of Performance Anomalies September 8, 2016 3 / 41

slide-5
SLIDE 5

Motivation

Introduction

◮ Performance is a significant quality characteristic

◮ e.g., Amazon: 100 ms delay → 1% decrease in sales (Huang 2011) ◮ e.g., Google: 500 ms delay → 20% drop in traffic (Huang 2011)

◮ Detect performance issues to react on them

◮ As soon as possible ◮ Use monitoring (e.g., measure execution times) ◮ Investigate these measurements for anomalies

◮ Visualization can support interpretation of anomalies

Sören Henning Visualization of Performance Anomalies September 8, 2016 3 / 41

slide-6
SLIDE 6

ΘPAD’s Anomaly Detection Approach

Introduction

ΘPAD and ΘPADx

◮ Provide anomaly detection ◮ Part of Kieker ◮ Only R algorithms ◮ Problematic anomaly score ◮ No visualization ◮ More on this later

Sören Henning Visualization of Performance Anomalies September 8, 2016 4 / 41

slide-7
SLIDE 7

Thesis’ Goal

Introduction

Development of an approach to detect performance anomalies with Kieker and visualize them

Sören Henning Visualization of Performance Anomalies September 8, 2016 5 / 41

slide-8
SLIDE 8

Goals

Introduction

◮ G1: Migrate ΘPAD to a TeeTime configuration

Sören Henning Visualization of Performance Anomalies September 8, 2016 6 / 41

slide-9
SLIDE 9

Goals

Introduction

◮ G1: Migrate ΘPAD to a TeeTime configuration ◮ G2: Implement multiple forecast algorithms

Sören Henning Visualization of Performance Anomalies September 8, 2016 6 / 41

slide-10
SLIDE 10

Goals

Introduction

◮ G1: Migrate ΘPAD to a TeeTime configuration ◮ G2: Implement multiple forecast algorithms ◮ G3: Provide a visualization of measured time series and detected

anomalies

Sören Henning Visualization of Performance Anomalies September 8, 2016 6 / 41

slide-11
SLIDE 11

Goals

Introduction

◮ G1: Migrate ΘPAD to a TeeTime configuration ◮ G2: Implement multiple forecast algorithms ◮ G3: Provide a visualization of measured time series and detected

anomalies

◮ G4: Evaluate the Implementation

◮ G4.1. Feasibility evaluation ◮ G4.2. Scalability evaluation Sören Henning Visualization of Performance Anomalies September 8, 2016 6 / 41

slide-12
SLIDE 12

Outline

Foundations

  • 1. Introduction
  • 2. Foundations
  • 3. Approach
  • 4. Evaluation
  • 5. Conclusion and Future Work

Sören Henning Visualization of Performance Anomalies September 8, 2016 7 / 41

slide-13
SLIDE 13

Foundations

Foundations

◮ Performance Metrics (Koziolek 2008)

◮ Time behavior and resource efficiency

◮ Time Series (Mitsa 2010)

◮ Sequence of measurements at regular temporal intervals

◮ Anomaly Detection (Chandola, Banerjee, and Kumar 2009)

◮ Anomaly: Abnormal data patterns ◮ Detection: Compare measured values with reference model Sören Henning Visualization of Performance Anomalies September 8, 2016 8 / 41

slide-14
SLIDE 14

ΘPAD’s Anomaly Detection Approach

Time Series Normalization

Foundations 1 5 8 6 4 3 6 7 3 3 1

1

1 1 4 3 5 3 7 4 3 1 4

raw measurements normalized time series

aggrmean

2 3 4 5 6

Figure: Based on Bielefeld (2012) and Frotscher (2013)

Sören Henning Visualization of Performance Anomalies September 8, 2016 9 / 41

slide-15
SLIDE 15

ΘPAD’s Anomaly Detection Approach

Time Series Forecasting

Foundations

3 7 4 3 1 4 3

sliding window forecastingmean 1 2 3 4 5 6 7 Figure: Based on Bielefeld (2012) and Frotscher (2013)

Sören Henning Visualization of Performance Anomalies September 8, 2016 10 / 41

slide-16
SLIDE 16

ΘPAD’s Anomaly Detection Approach

Anomaly Score Calculation

Foundations

3

7

6

p a

A(a,p) = 0.33

anomaly score calculation anomaly decision

Anomaly

!

threshold t = 0.2 0.33 ≥ 0.2

Sören Henning Visualization of Performance Anomalies September 8, 2016 11 / 41

slide-17
SLIDE 17

Technologies

Foundations

◮ Microservices Architectural Pattern (Wolff 2015) ◮ Kieker Monitoring Framework (Hoorn et al. 2009) ◮ TeeTime Pipe and Filter Framework (Wulf, Ehmke, and

Hasselbring 2014)

◮ And further technologies (see next slides)

Sören Henning Visualization of Performance Anomalies September 8, 2016 12 / 41

slide-18
SLIDE 18

Outline

Approach

  • 1. Introduction
  • 2. Foundations
  • 3. Approach
  • 4. Evaluation
  • 5. Conclusion and Future Work

Sören Henning Visualization of Performance Anomalies September 8, 2016 13 / 41

slide-19
SLIDE 19

Graphical Overview of our Approach

Approach

Application Monitoring Anomaly Detection Anomaly Visualization

measurements time series

Sören Henning Visualization of Performance Anomalies September 8, 2016 14 / 41

slide-20
SLIDE 20

Architecture of our Approach

Approach

Server Client

Visualization Visualization Provider Analysis R-based Forecast Database Sören Henning Visualization of Performance Anomalies September 8, 2016 15 / 41

slide-21
SLIDE 21

Architecture of our Implementation

Approach

Server Client

Containerized with

Sören Henning Visualization of Performance Anomalies September 8, 2016 16 / 41

slide-22
SLIDE 22

Performance Anomaly Detection

TeeTime Configuration

Approach

...

Record Converter Record Reconstructor TCP Reader Anomaly Detector Flow Record Filter Record Distributor Record Converter Anomaly Detector One analysis branch per Filter Distribute by Filter

Filter:

◮ Operation signature ◮ Class signature ◮ Host name ◮ ...

Sören Henning Visualization of Performance Anomalies September 8, 2016 17 / 41

slide-23
SLIDE 23

Performance Anomaly Detection

TeeTime Configuration

Approach

...

Record Converter Record Reconstructor TCP Reader Anomaly Detector Flow Record Filter Record Distributor Record Converter Anomaly Detector

Distributor Filter Filter

Sören Henning Visualization of Performance Anomalies September 8, 2016 17 / 41

slide-24
SLIDE 24

Performance Anomaly Detection

TeeTime Configuration

Approach

...

Record Converter Record Reconstructor TCP Reader Anomaly Detector Flow Record Filter Record Distributor Record Converter Anomaly Detector

Sören Henning Visualization of Performance Anomalies September 8, 2016 17 / 41

slide-25
SLIDE 25

Time Series Analysis and Anomaly Detection

TeeTime Configuration

Approach

Threshold Filter Threshold Filter Anomaly Score Calculator Measurement Forecast Decorator Forecaster Time Series Loader Distributor Normalizer Storager Distributor Threshold Filter Sliding Window

In Memory

Database Adapter Interface

  • n startup

Anomaly Detection Stage Sören Henning Visualization of Performance Anomalies September 8, 2016 18 / 41

slide-26
SLIDE 26

Demo of Visualization

Approach Sören Henning Visualization of Performance Anomalies September 8, 2016 19 / 41

slide-27
SLIDE 27

Architecture of Visualization

Approach

update

Client (Web Browser)

DB

Server

CanvasPlot

  • chart plotting
  • interactive zooming

Anomaliz.js

  • chart controlling
  • value management
  • automatic scrolling

Data Updating UI Handling update request every x sec. send new data

◮ Usage of Arne Johanson’s CanvasPlot (Johanson 2016)

Sören Henning Visualization of Performance Anomalies September 8, 2016 20 / 41

slide-28
SLIDE 28

Outline

Evaluation

  • 1. Introduction
  • 2. Foundations
  • 3. Approach
  • 4. Evaluation
  • 5. Conclusion and Future Work

Sören Henning Visualization of Performance Anomalies September 8, 2016 21 / 41

slide-29
SLIDE 29

Feasibility Evaluation

Scenarios

Evaluation

100 200 10000 20000 30000 40000

Time in ms Response Time in ms

100 200 300 400 500 10000 20000 30000 40000

Time in ms Response Time in ms

100 200 300 10000 20000 30000 40000

Time in ms Response Time in ms

100 200 300 400 500 10000 20000 30000 40000

Time in ms Response Time in ms Sören Henning Visualization of Performance Anomalies September 8, 2016 22 / 41

slide-30
SLIDE 30

Feasibility Evaluation

Scenario: Seasonal with Anomaly

Evaluation

100 200 300 10000 20000 30000 40000

Time in ms Response Time in ms

−0.6 −0.4 −0.2 0.0 0.2 0.4 0.6 10000 20000 30000 40000

Time in ms Anomaly Score

ARIMAForecaster ExponentialWeightedForecaster LinearWeightedForecaster LogarithmicWeightedForecaster MeanForecaster RegressionForecaster

Sören Henning Visualization of Performance Anomalies September 8, 2016 23 / 41

slide-31
SLIDE 31

Scalability Evaluation

Configuration

Evaluation

◮ Take time for record processing in analysis

Sören Henning Visualization of Performance Anomalies September 8, 2016 24 / 41

slide-32
SLIDE 32

Scalability Evaluation

Configuration

Evaluation

◮ Take time for record processing in analysis ◮ Evaluate: Execution time ≤ measurement frequency ?

Sören Henning Visualization of Performance Anomalies September 8, 2016 24 / 41

slide-33
SLIDE 33

Scalability Evaluation

Configuration

Evaluation

◮ Take time for record processing in analysis ◮ Evaluate: Execution time ≤ measurement frequency ? ◮ For all parameter combinations:

Measurement frequencies 2 ms, 5 ms, 10 ms, 50 ms, 100 ms, 150 ms, 200 ms Sliding window 10,000 ms, 50,000 ms, 100,000 ms, 150,000 ms, 200,000 ms, 400,000 ms Normalization interval 10 ms, 20 ms, 100 ms, 200 ms, 500 ms, 1000 ms, 2000 ms Forecast algorithm ARIMAForecaster, RegressionForecaster Normalization algorithm MeanAggregator

Sören Henning Visualization of Performance Anomalies September 8, 2016 24 / 41

slide-34
SLIDE 34

Scalability Evaluation

Results

Evaluation

Some examples:

freq.

  • sld. window
  • norm. intvl.

forecaster ∅ exec. time 5 10,000 200 Regression 1.64 5 400,000 20 Regression 4.35 50 50,000 200 ARIMA 69.98 100 100,000 500 ARIMA 78.24 150 200,000 2,000 ARIMA 187.21 ...

all values in ms

Sören Henning Visualization of Performance Anomalies September 8, 2016 25 / 41

slide-35
SLIDE 35

Outline

Conclusion and Future Work

  • 1. Introduction
  • 2. Foundations
  • 3. Approach
  • 4. Evaluation
  • 5. Conclusion and Future Work

Sören Henning Visualization of Performance Anomalies September 8, 2016 26 / 41

slide-36
SLIDE 36

Conclusion

Conclusion and Future Work

◮ Further development of the ΘPAD Approach

◮ Proving infrastructure via Docker containers ◮ Immediately record processing Sören Henning Visualization of Performance Anomalies September 8, 2016 27 / 41

slide-37
SLIDE 37

Conclusion

Conclusion and Future Work

◮ Further development of the ΘPAD Approach

◮ Proving infrastructure via Docker containers ◮ Immediately record processing

◮ Native Java algorithms for anomaly detection

Sören Henning Visualization of Performance Anomalies September 8, 2016 27 / 41

slide-38
SLIDE 38

Conclusion

Conclusion and Future Work

◮ Further development of the ΘPAD Approach

◮ Proving infrastructure via Docker containers ◮ Immediately record processing

◮ Native Java algorithms for anomaly detection ◮ Providing an interactive, real time visualization

Sören Henning Visualization of Performance Anomalies September 8, 2016 27 / 41

slide-39
SLIDE 39

Conclusion

Conclusion and Future Work

◮ Further development of the ΘPAD Approach

◮ Proving infrastructure via Docker containers ◮ Immediately record processing

◮ Native Java algorithms for anomaly detection ◮ Providing an interactive, real time visualization ◮ All implementations available as open source:

github.com/SoerenHenning

Sören Henning Visualization of Performance Anomalies September 8, 2016 27 / 41

slide-40
SLIDE 40

Future Work

Conclusion and Future Work

◮ Handling of fast incoming measurements

◮ Aggregate before analysis (ΘPAD) ◮ Cache time series operations Sören Henning Visualization of Performance Anomalies September 8, 2016 28 / 41

slide-41
SLIDE 41

Future Work

Conclusion and Future Work

◮ Handling of fast incoming measurements

◮ Aggregate before analysis (ΘPAD) ◮ Cache time series operations

◮ Parallelized and distributed analysis

◮ Is or will be supported by TeeTime Sören Henning Visualization of Performance Anomalies September 8, 2016 28 / 41

slide-42
SLIDE 42

Future Work

Conclusion and Future Work

◮ Handling of fast incoming measurements

◮ Aggregate before analysis (ΘPAD) ◮ Cache time series operations

◮ Parallelized and distributed analysis

◮ Is or will be supported by TeeTime

◮ Take advantage of Cassandra’s features for data storage

Sören Henning Visualization of Performance Anomalies September 8, 2016 28 / 41

slide-43
SLIDE 43

Future Work

Conclusion and Future Work

◮ Handling of fast incoming measurements

◮ Aggregate before analysis (ΘPAD) ◮ Cache time series operations

◮ Parallelized and distributed analysis

◮ Is or will be supported by TeeTime

◮ Take advantage of Cassandra’s features for data storage ◮ Configuration via Rest/GUI

Sören Henning Visualization of Performance Anomalies September 8, 2016 28 / 41

slide-44
SLIDE 44

References I

References

Bielefeld, Tillmann Carlos (2012). “Online Performance Anomaly Detection for Large-Scale Software Systems”. Diploma Thesis. Germany: Kiel University. Chandola, Varun, Arindam Banerjee, and Vipin Kumar (2009). “Anomaly detection: A survey”. In: ACM computing surveys (CSUR) 41.3, p. 15. Frotscher, Tom (2013). “Architecture-Based Multivariate Anomaly Detection for Software Systems”. Master’s Thesis. Germany: Kiel University. Hoorn, André van et al. (2009). Continuous Monitoring of Software Services: Design and Application of the Kieker Framework.

  • Tech. rep. TR-0921. Department of Computer Science, Kiel

University, Germany.

Sören Henning Visualization of Performance Anomalies September 8, 2016 29 / 41

slide-45
SLIDE 45

References II

References

Huang, Cheng (2011). Public DNS System and Global Traffic

  • Management. Accessed: 2016-08-25. URL:

❤tt♣✿✴✴r❡s❡❛r❝❤✳♠✐❝r♦s♦❢t✳❝♦♠✴❡♥✲ ✉s✴✉♠✴♣❡♦♣❧❡✴❝❤❡♥❣❤✴s❧✐❞❡s✴♣✉❜❞♥s✶✶✳♣♣t①✳♣❞❢. Johanson, Arne (2016). CanvasPlot. Accessed: 2016-08-25. URL: ❤tt♣s✿✴✴❛✲❥♦❤❛♥s♦♥✳❣✐t❤✉❜✳✐♦✴❝❛♥✈❛s✲♣❧♦t. Koziolek, Heiko (2008). “Dependability Metrics”. In: ed. by Irene Eusgeld, Felix C. Freiling, and Ralf Reussner. Berlin, Heidelberg: Springer-Verlag. Chap. Introduction to Performance Metrics, pp. 199–203. ISBN: 3-540-68946-X, 978-3-540-68946-1. Mitsa, T. (2010). Temporal Data Mining. Chapman & Hall/CRC Data Mining and Knowledge Discovery Series. CRC Press. ISBN: 9781420089776.

Sören Henning Visualization of Performance Anomalies September 8, 2016 30 / 41

slide-46
SLIDE 46

References III

References

Wolff, E. (2015). Microservices: Grundlagen flexibler

  • Softwarearchitekturen. Dpunkt.Verlag GmbH. ISBN:

9783864903137. Wulf, Christian, Nils Christian Ehmke, and Wilhelm Hasselbring (2014). “Toward a Generic and Concurrency-Aware Pipes & Filters Framework”. In: Symposium on Software Performance 2014: Joint Descartes/Kieker/Palladio Days.

Sören Henning Visualization of Performance Anomalies September 8, 2016 31 / 41

slide-47
SLIDE 47

Feasibility Evaluation

Scenario: Constant with Anomaly

Feasibility Evaluation

100 200 10000 20000 30000 40000

Time in ms Response Time in ms

−0.4 −0.2 0.0 0.2 0.4 0.6 0.8 1.0 10000 20000 30000 40000

Time in ms Anomaly Score

ARIMAForecaster ExponentialWeightedForecaster LinearWeightedForecaster LogarithmicWeightedForecaster MeanForecaster RegressionForecaster

Sören Henning Visualization of Performance Anomalies September 8, 2016 32 / 41

slide-48
SLIDE 48

Feasibility Evaluation

Scenario: Constant with Anomaly - Detail

Feasibility Evaluation

100 200 29500 30000 30500

Time in ms Response Time in ms

−0.2 0.0 0.2 0.4 0.6 0.8 1.0 29500 30000 30500

Time in ms Anomaly Score

ARIMAForecaster ExponentialWeightedForecaster LinearWeightedForecaster LogarithmicWeightedForecaster MeanForecaster RegressionForecaster

Sören Henning Visualization of Performance Anomalies September 8, 2016 33 / 41

slide-49
SLIDE 49

Feasibility Evaluation

Scenario: Linearly Increasing with Anomaly

Feasibility Evaluation

100 200 300 400 500 10000 20000 30000 40000

Time in ms Response Time in ms

0.0 0.2 0.4 10000 20000 30000 40000

Time in ms Anomaly Score

ARIMAForecaster ExponentialWeightedForecaster LinearWeightedForecaster LogarithmicWeightedForecaster MeanForecaster RegressionForecaster

Sören Henning Visualization of Performance Anomalies September 8, 2016 34 / 41

slide-50
SLIDE 50

Feasibility Evaluation

Scenario: Linearly Increasing with Anomaly - Detail

Feasibility Evaluation

400 500 29000 29500 30000 30500

Time in ms Response Time in ms

0.0 0.2 0.4 29000 29500 30000 30500

Time in ms Anomaly Score

ARIMAForecaster ExponentialWeightedForecaster LinearWeightedForecaster LogarithmicWeightedForecaster MeanForecaster RegressionForecaster

Sören Henning Visualization of Performance Anomalies September 8, 2016 35 / 41

slide-51
SLIDE 51

Feasibility Evaluation

Scenario: Seasonal with Anomaly

Feasibility Evaluation

100 200 300 10000 20000 30000 40000

Time in ms Response Time in ms

−0.6 −0.4 −0.2 0.0 0.2 0.4 0.6 10000 20000 30000 40000

Time in ms Anomaly Score

ARIMAForecaster ExponentialWeightedForecaster LinearWeightedForecaster LogarithmicWeightedForecaster MeanForecaster RegressionForecaster

Sören Henning Visualization of Performance Anomalies September 8, 2016 36 / 41

slide-52
SLIDE 52

Feasibility Evaluation

Scenario: Seasonal with Anomaly - Detail

Feasibility Evaluation

100 200 29000 29500 30000 30500

Time in ms Response Time in ms

−0.4 −0.2 0.0 0.2 0.4 29000 29500 30000 30500

Time in ms Anomaly Score

ARIMAForecaster ExponentialWeightedForecaster LinearWeightedForecaster LogarithmicWeightedForecaster MeanForecaster RegressionForecaster

Sören Henning Visualization of Performance Anomalies September 8, 2016 37 / 41

slide-53
SLIDE 53

Feasibility Evaluation

Scenario: Exponential Increasing

Feasibility Evaluation

100 200 300 400 500 10000 20000 30000 40000

Time in ms Response Time in ms

−0.2 0.0 0.2 0.4 0.6 0.8 1.0 10000 20000 30000 40000

Time in ms Anomaly Score

ARIMAForecaster ExponentialWeightedForecaster LinearWeightedForecaster LogarithmicWeightedForecaster MeanForecaster RegressionForecaster

Sören Henning Visualization of Performance Anomalies September 8, 2016 38 / 41

slide-54
SLIDE 54

Feasibility Evaluation

Scenario: Exponential Increasing - Detail

Feasibility Evaluation

200 300 400 500 30000 32500 35000 37500 40000

Time in ms Response Time in ms

0.0 0.2 0.4 0.6 0.8 30000 32500 35000 37500 40000

Time in ms Anomaly Score

ARIMAForecaster ExponentialWeightedForecaster LinearWeightedForecaster LogarithmicWeightedForecaster MeanForecaster RegressionForecaster

Sören Henning Visualization of Performance Anomalies September 8, 2016 39 / 41

slide-55
SLIDE 55

Screenshots of Visualization

Feasibility Evaluation Sören Henning Visualization of Performance Anomalies September 8, 2016 40 / 41

slide-56
SLIDE 56

Screenshots of Visualization

Feasibility Evaluation Sören Henning Visualization of Performance Anomalies September 8, 2016 41 / 41