Sensitivity Analysis of Network Performance Models Intermediate talk - - PowerPoint PPT Presentation

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Sensitivity Analysis of Network Performance Models

Intermediate talk for the Bachelor’s Thesis by

Niklas Beck

advised by Max Helm, Henning Stubbe Wednesday 11th March, 2020 Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Sensitivity Analysis

Basics

• determines sensitivity of parameters
• effect of input parameters on the output of a model
• purposes:
• model validation
• investigating model behavior
• model optimization

Niklas Beck — SA 2

Sensitivity Analysis

General Approaches parametric numeric local

• systematically evaluating model
• uses network formula
• One-at-a-time (OAT)
• differentiation of network formula
• change one parameter after another
• time and resource efficient
• fast and easy to implement
• fast and easy to implement

global

• simultaneous variation of input parameters
• full exploration of complete input space
• Sobol method [4]
• Fourier Amplitude Sensitivity Test [3]

Niklas Beck — SA 3

Sensitivity Analysis

Network Model

Niklas Beck — SA 4

Work Done

Framework - Local SA

• evaluate_model()

model dependent function returns output of a given model

• oat()

implements One-at-a-time approach change one parameter at a time, keeping others fixed based on parametric SA

• solv()

implements differentiation and evaluation of given formula function verified by comparison with Trivedi et al. [5] based on numerical SA

Niklas Beck — SA 5

Work Done

Framework - Global

• model independent open source Python library: SALib [2]
• SALib implements varius methods for global SA
• Sobol Sensitivity Analysis
• Fourier Amplitude Sensitivity Test (FAST)
• generates model sample inputs
• computes sensitivity indices from model outputs

Niklas Beck — SA 6

Niklas Beck — SA 7

Niklas Beck — SA 8

Niklas Beck — SA 9

Niklas Beck — SA 10

Next Steps

• done:
• implement OAT approach
• implement differentiation with Sympy
• make SALib usable for NC models and implement automated plotting
• build DNC models (using DiscoDNC [1]) from random generated graphs
• to be done:
• automatically generate data flows through the DNC network
• apply framework to various models
• analyze and evaluate results

Niklas Beck — SA 11

Bibliography

[1]

• S. Bondorf and J. B. Schmitt.

The DiscoDNC v2 – a comprehensive tool for deterministic network calculus. In Proc. of the International Conference on Performance Evaluation Methodologies and Tools, ValueTools ’14, pages 44–49, December 2014. [2]

• J. Herman and W. Usher.

Salib: An open-source python library for sensitivity analysis. Journal of Open Source Software, 2(9):97, 2017. [3]

• A. Saltelli, S. Tarantola, and K. P

.-S. Chan. A quantitative model-independent method for global sensitivity analysis of model output. Technometrics, 41(1):39–56, 1999. [4]

• I. Sobol.

Global sensitivity indices for nonlinear mathematical models and their monte carlo estimates. Mathematics and Computers in Simulation, 55(1):271 – 280, 2001. The Second IMACS Seminar on Monte Carlo Methods. [5]

• K. Trivedi, R. Mansharamani, D. Kim, M. Grottke, and M. Nambiar.

Recovery from failures due to mandelbugs in it systems. 12 2011.

Niklas Beck — SA 12

Appendix

Framework - Global Problem:

• SALib needs bounds for each parameter
• model is not allowed to give invalid output within bounds
• network performance models could return invalid output (e.g. "delay bound: Infinity")

Solution:

• Bound Inflation increments one parameter after another
• checks bound validity after every increase

Niklas Beck — SA 13

Appendix

Bound Inflation

Niklas Beck — SA 14