Sensitivity Analysis of Network Performance Models Final talk for - - PowerPoint PPT Presentation

▶
sensitivity analysis of network performance models
SMART_READER_LITE
LIVE PREVIEW

Sensitivity Analysis of Network Performance Models Final talk for - - PowerPoint PPT Presentation

Feb 03, 2023 228 likes •465 views

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Sensitivity Analysis of Network Performance Models Final talk for the Bachelors Thesis by Niklas Beck advised by Max Helm, Henning Stubbe

slide-1
SLIDE 1

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Sensitivity Analysis of Network Performance Models

Final talk for the Bachelor’s Thesis by

Niklas Beck

advised by Max Helm, Henning Stubbe Wednesday 17th June, 2020 Chair of Network Architectures and Services Department of Informatics Technical University of Munich

slide-2
SLIDE 2

Background

Sensitivity Analysis (SA)

  • determines sensitivity of parameters
  • effect of input parameters on the output of a model
  • purposes:
  • model validation
  • investigating model behavior
  • model optimization

Niklas Beck — SA 2

slide-3
SLIDE 3

Background

Sensitivity Analysis (SA) model-based formula-based local

  • systematically evaluating model
  • uses network formula
  • change one parameter after another
  • differentiation of network formula

global

  • simultaneous variation of multiple input parameters
  • full exploration of complete input space (to a feasible granularity)

Niklas Beck — SA 3

slide-4
SLIDE 4

Background

Network Calculus (NC) Model

Niklas Beck — SA 4

slide-5
SLIDE 5

Background

NC Analysis Methods

  • Total Flow Analysis (TFA)
  • Seperate Flow Analysis (SFA)
  • Pay Multiplexing Only Once Analysis (PMOOA)
  • Tandem Matching Analysis (TMA)

Niklas Beck — SA 5

slide-6
SLIDE 6

Related Work

Sensitivity Analysis for Map/Map/1 Queues [2]

  • queueing models (QM) are very similar to NC models
  • parameters (e.g. service times (QM) - service curve (NC))
  • performance measures (e.g. number of customers in the system (QM) - backlog bound (NC))
  • only local sensitivity analysis was performed
  • investigated only one specific queuing model

This thesis: local and global SA performed with 16 different network calculus models

Niklas Beck — SA 6

slide-7
SLIDE 7

Implementation

Model Creation

  • NC models are built with a deterministic network calculator (DiscoDNC [1])
  • restricted to feed-forward networks
  • tandem model
  • random model
  • generate an Internet-like random graph with a customized Barabási-Albert algorithm
  • custom extension of DiscoDNC needed to parse and build NC model

Niklas Beck — SA 7

slide-8
SLIDE 8

Random Internet-like Network Graph

Niklas Beck — SA 8

slide-9
SLIDE 9

Random Internet-like Network Graph

Niklas Beck — SA 9

slide-10
SLIDE 10

Random Internet-like Network Graph

Niklas Beck — SA 10

slide-11
SLIDE 11

Implementation

Local SA

  • model-based
  • runs model with altering input parameters
  • isolated evaluation
  • formula-based
  • differentiation with SymPy

Global SA

  • model independent open source Python library: SALib [3]
  • SALib implements varius methods for global SA
  • Sobol Sensitivity Analysis [5]
  • Fourier Amplitude Sensitivity Test (FAST) [4]
  • computes sensitivity indices from model outputs

All results can be automatically plotted with MatPlotLib

Niklas Beck — SA 11

slide-12
SLIDE 12

Evaluation

Model Parameters Parameter Description Abbreviation Number of Network Nodes n Service Curve Rate scr Service Curve Latency scl Usage of Maximum Service Curve umsc Maximum Service Curve Rate mscr Maximum Service Curve Latency mscl Arrival Curve Rate acr Arrival Curve Burst acb Number of Cross-Traffic Flows xf

Niklas Beck — SA 12

slide-13
SLIDE 13

Evaluation

Local Sensitivity Values: Tandem Network - Delay Bound

n scr scl mscr acr acb xf 2 4 6 8 10 12 Sensitivity Value (Delay Bound) 0.0164

  • 6.7e-09

8.519 0.0 5.2e-08 1.5e-06 0.022 0.01

  • 2.8e-09

6.5 0.0 2e-08 8.7e-07 0.0157 0.01

  • 1.4e-09

5.75 0.0 1e-08 3.7e-07 0.0073 0.01

  • 1.4e-09

5.75 0.0 1e-08 3.75e-07 0.0073 TFA SFA PMOOA TMA Niklas Beck — SA 13

slide-14
SLIDE 14

Evaluation

Local Sensitivity Analysis: Tandem Network - Delay Bound

5 10 15 20 25 30 Number of Network Nodes 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Delay Bound TFA SFA PMOOA TMA 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Service Curve Rate 1e8 0.06 0.08 0.10 0.12 0.14 0.16 Delay Bound TFA SFA PMOOA TMA 0.0 0.1 0.2 0.3 0.4 0.5 Service Curve Latency 1 2 3 4 Delay Bound TFA SFA PMOOA TMA 2 4 6 8 Number of Cross-Traffic Flows 0.2 0.4 0.6 0.8 Delay Bound TFA SFA PMOOA TMA

Niklas Beck — SA 14

slide-15
SLIDE 15

Evaluation

Local Sensitivity Analysis: Random Network - Delay Bound

0.0 0.1 0.2 0.3 0.4 0.5 Service Curve Latency 2 4 6 8 10 12 Delay Bound TFA SFA PMOOA TMA 0.0 0.2 0.4 0.6 0.8 1.0 Arrival Curve Burst 1e5 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Delay Bound TFA SFA PMOOA TMA

  • inaccuracy of curve computation in DiscoDNC
  • computational error of a single server node leads to flawed performance bounds
  • worst case: could lead to false results for local SA

Niklas Beck — SA 15

slide-16
SLIDE 16

Evaluation

Global Sensitivity Indices: Tandem Network - Delay Bound

n scr scl umsc mscr mscl acr acb xf 0.0 0.2 0.4 0.6 0.8 1.0

Sensitivity Index

0.1529 0.0338 0.3147 0.0 0.0 0.0 0.0107 0.0 0.0026 0.5109 0.3278 0.4864 0.0094 0.0113 0.0187 0.1659 0.0184 0.1818 First Total n scr scl umsc mscr mscl acr acb xf 0.0 0.2 0.4 0.6 0.8 1.0

Sensitivity Index

0.4179 0.006 0.3959 0.0 0.0 0.0 0.0022 0.0 0.0026 0.5773 0.037 0.5489 0.0077 0.0075 0.0014 0.0433 0.0032 0.045 First Total

Total Flow Analysis Seperate Flow Analysis

Niklas Beck — SA 16

slide-17
SLIDE 17

Evaluation

Global Sensitivity Indices: Tandem Network - Backlog Bound

n scr scl umsc mscr mscl acr acb xf 0.0 0.2 0.4 0.6 0.8 1.0 Sensitivity Index 0.2138 0.0 0.226 0.0 0.0 0.0 0.2094 0.0 0.0553 0.4004 0.0 0.4184 0.0 0.0 0.0 0.3916 0.0 0.1231 First Total

n-scr n-scl n-umsc n-mscr n-mscl n-acr n-acb n-xf scr-scl scr-umsc scr-mscr scr-mscl scr-acr scr-acb scr-xf scl-umsc scl-mscr scl-mscl scl-acr scl-acb scl-xf umsc-mscr umsc-mscl umsc-acr umsc-acb umsc-xf mscr-mscl mscr-acr mscr-acb mscr-xf mscl-acr mscl-acb mscl-xf acr-acb acr-xf acb-xf 0.0 0.1 0.2 0.3 0.4 0.5

Sensitivity Index

0.005721 0.072762 0.005715 0.005721 0.005735 0.074852 0.005728 0.022218 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.000956 0.00095 0.000959 0.067554 0.000947 0.021081 2.1e-05 2.2e-05 2.9e-05 2.1e-05 5e-06 0.0 0.0 0.0 0.0 4.5e-05 2.9e-05 2.2e-05

  • 0.00261

0.015113 5.4e-05

Second

TFA first- and total-order indices TFA second-order indices

Niklas Beck — SA 17

slide-18
SLIDE 18

Conclusion

  • sensitivity analysis performed
  • local (formula-based - model-based)
  • global (Sobol - FAST)
  • 16 different network calculus models investigated
  • performance measure (delay bound - backlog bound)
  • analysis methods (TFA, SFA, PMOOA, TMA)
  • real-world applicability through Internet-like network topologies
  • comprehensive sensitivity analysis Python framework provided

Niklas Beck — SA 18

slide-19
SLIDE 19

Bibliography

[1]

  • S. Bondorf and J. B. Schmitt.

The DiscoDNC v2 – a comprehensive tool for deterministic network calculus. In Proc. of the International Conference on Performance Evaluation Methodologies and Tools, ValueTools ’14, pages 44–49, December 2014. [2]

  • A. Heindl.

Sensitivity analysis for map/map/1 queues. pages 235–244, 01 2004. [3]

  • J. Herman and W. Usher.

Salib: An open-source python library for sensitivity analysis. Journal of Open Source Software, 2(9):97, 2017. [4]

  • A. Saltelli, S. Tarantola, and K. P

.-S. Chan. A quantitative model-independent method for global sensitivity analysis of model output. Technometrics, 41(1):39–56, 1999. [5]

  • I. Sobol.

Global sensitivity indices for nonlinear mathematical models and their monte carlo estimates. Mathematics and Computers in Simulation, 55(1):271 – 280, 2001. The Second IMACS Seminar on Monte Carlo Methods.

Niklas Beck — SA 19

slide-20
SLIDE 20

Appendix

Local Sensitivity Analysis: Tandem Network - Backlog Bound

5 10 15 20 25 30 Number of Network Nodes 1 2 3 4 Backlog Bound 1e5 TFA SFA PMOOA TMA 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Service Curve Rate 1e8 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 Backlog Bound 1e5 TFA SFA PMOOA TMA 0.0 0.1 0.2 0.3 0.4 0.5 Service Curve Latency 1 2 3 4 5 Backlog Bound 1e6 TFA SFA PMOOA TMA 2 4 6 8 Number of Cross-Traffic Flows 1 2 3 4 5 6 7 8 Backlog Bound 1e5 TFA SFA PMOOA TMA

Niklas Beck — SA 20

slide-21
SLIDE 21

Appendix

Local Sensitivity Analysis: Random Network - Backlog Bound

0.2 0.4 0.6 0.8 1.0 1.2 Service Curve Rate 1e8 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Backlog Bound 1e5 TFA SFA PMOOA TMA 0.0 0.1 0.2 0.3 0.4 0.5 Service Curve Latency 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Backlog Bound 1e7 TFA SFA PMOOA TMA 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Arrival Curve Rate 1e6 1 2 3 4 5 Backlog Bound 1e5 TFA SFA PMOOA TMA 0.0 0.2 0.4 0.6 0.8 1.0 Arrival Curve Burst 1e5 0.2 0.4 0.6 0.8 1.0 Backlog Bound 1e6 TFA SFA PMOOA TMA

Niklas Beck — SA 21

slide-22
SLIDE 22

Appendix

Global Sensitivity Indices: Random Network - Delay Bound

scr scl umsc mscr mscl acr acb 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Sensitivity Index 0.0408 0.9046 0.0 0.0 0.0 0.009 0.0 0.0833 0.9252 0.0 0.0 0.0 0.0387 0.0 First Total

scr-scl scr-umsc scr-mscr scr-mscl scr-acr scr-acb scl-umsc scl-mscr scl-mscl scl-acr scl-acb umsc-mscr umsc-mscl umsc-acr umsc-acb mscr-mscl mscr-acr mscr-acb mscl-acr mscl-acb acr-acb 0.0 0.1 0.2 0.3 0.4 0.5

Sensitivity Index

0.020407 0.003284 0.003278 0.003273 0.031051 0.003286

  • 0.000543
  • 0.00054
  • 0.00054

0.000625

  • 0.000515
  • 5e-06
  • 5e-06
  • 4e-06
  • 5e-06

0.0 0.0 0.0

  • 1e-06
  • 2e-06
  • 0.001688

Second

SFA first- and total-order indices SFA second-order indices

Niklas Beck — SA 22

slide-23
SLIDE 23

Appendix

Global Sensitivity Indices: Random Network - Backlog Bound

scr scl umsc mscr mscl acr acb 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Sensitivity Index 0.0029 0.4679 0.0 0.0 0.0 0.3917 0.0 0.0118 0.5981 0.0 0.0 0.0 0.5245 0.0 First Total

scr-scl scr-umsc scr-mscr scr-mscl scr-acr scr-acb scl-umsc scl-mscr scl-mscl scl-acr scl-acb umsc-mscr umsc-mscl umsc-acr umsc-acb mscr-mscl mscr-acr mscr-acb mscl-acr mscl-acb acr-acb 0.0 0.1 0.2 0.3 0.4 0.5

Sensitivity Index

0.003955 0.001776 0.001774 0.001765 0.008153 0.001754

  • 9.4e-05
  • 9.8e-05
  • 0.000102

0.127453

  • 8.7e-05
  • 1e-06

0.0

  • 1e-06
  • 1e-06

0.0 0.0 0.0 0.0 5e-06

  • 0.001476

Second

TFA first- and total-order indices TFA second-order indices

Niklas Beck — SA 23