Sensitivity Analysis of Network Performance Models Intermediate talk - PowerPoint PPT Presentation

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Sensitivity Analysis of Network Performance Models Intermediate talk for the Bachelor’s Thesis by Niklas Beck advised by Max Helm, Henning Stubbe Wednesday 11 th March, 2020 Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Sensitivity Analysis Basics • determines sensitivity of parameters • effect of input parameters on the output of a model • purposes: • model validation • investigating model behavior • model optimization Niklas Beck — SA 2

Sensitivity Analysis General Approaches parametric numeric local • systematically evaluating model • uses network formula • One-at-a-time (OAT) • differentiation of network formula • change one parameter after another • time and resource efficient • fast and easy to implement • fast and easy to implement • simultaneous variation of input parameters global • full exploration of complete input space • Sobol method [4] • Fourier Amplitude Sensitivity Test [3] Niklas Beck — SA 3

Sensitivity Analysis Network Model Niklas Beck — SA 4

Work Done Framework - Local SA • evaluate_model() model dependent function returns output of a given model • oat() implements One-at-a-time approach change one parameter at a time, keeping others fixed based on parametric SA • solv() implements differentiation and evaluation of given formula function verified by comparison with Trivedi et al. [5] based on numerical SA Niklas Beck — SA 5

Work Done Framework - Global • model independent open source Python library: SALib [2] • SALib implements varius methods for global SA • Sobol Sensitivity Analysis • Fourier Amplitude Sensitivity Test (FAST) • generates model sample inputs • computes sensitivity indices from model outputs Niklas Beck — SA 6

Niklas Beck — SA 7

Niklas Beck — SA 8

Niklas Beck — SA 9

Niklas Beck — SA 10

Next Steps • done: • implement OAT approach • implement differentiation with Sympy • make SALib usable for NC models and implement automated plotting • build DNC models (using DiscoDNC [1]) from random generated graphs • to be done: • automatically generate data flows through the DNC network • apply framework to various models • analyze and evaluate results Niklas Beck — SA 11

Bibliography [1] S. Bondorf and J. B. Schmitt. The DiscoDNC v2 – a comprehensive tool for deterministic network calculus. In Proc. of the International Conference on Performance Evaluation Methodologies and Tools , ValueTools ’14, pages 44–49, December 2014. [2] J. Herman and W. Usher. Salib: An open-source python library for sensitivity analysis. Journal of Open Source Software , 2(9):97, 2017. [3] A. Saltelli, S. Tarantola, and K. P .-S. Chan. A quantitative model-independent method for global sensitivity analysis of model output. Technometrics , 41(1):39–56, 1999. [4] I. Sobol. Global sensitivity indices for nonlinear mathematical models and their monte carlo estimates. Mathematics and Computers in Simulation , 55(1):271 – 280, 2001. The Second IMACS Seminar on Monte Carlo Methods. [5] K. Trivedi, R. Mansharamani, D. Kim, M. Grottke, and M. Nambiar. Recovery from failures due to mandelbugs in it systems. 12 2011. Niklas Beck — SA 12

Appendix Framework - Global Problem: • SALib needs bounds for each parameter • model is not allowed to give invalid output within bounds • network performance models could return invalid output (e.g. "delay bound: Infinity") Solution: • Bound Inflation increments one parameter after another • checks bound validity after every increase Niklas Beck — SA 13

Appendix Bound Inflation Niklas Beck — SA 14