Systematic Evaluation of Complex Systems Acknowledgement: Parts of - PDF document

Systematic Evaluation of Complex Systems Acknowledgement: Parts of these slides are based on [Jai91] 1 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems Systematic Evaluation of Complex Systems Motivation: Analysis of TCP Congestion Control § 2 k - Factorial Designs § 2 k r - Factorial Designs with Replications § 2 k-p – Fractional Factorial Designs § One Factor Experiments § Two Factor Experiments § Two Factor Experiments with Replications § General Full Factorial Designs § 2 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

Quick Reminder: TCP Congestion Control Initial Reset CW congestion to 1, new threshold threshold = CW/2 20=40/2 Additive increase, congestion avoidance New slow start to new Slow Initial threshold, then congestion start linear increase window size 3 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems TCP Congestion Control Parameters Performance affected by § Initial Congestion Window • Initial Congestion Window Threshold • Timeout • Enable Duplicate Acknowledgements • Size of TCP-Buffers • Maximum Segment Size (MSS) • … • MacOS 10.7: sysctl -a | grep tcp | wc –l => 79 Parameters • MacOS 10.10: sysctl -a | grep tcp | wc –l => 116 Parameters • MacOS 10.11: sysctl -a | grep tcp | wc –l => 120 Parameters • Some Boolean other numeric • For different scenarios, e.g. § Internet, LAN, DSL, Satellite, Congestion etc. • For different runs § 4 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

Choosing Optimal Parameters Choosing good parameters is a multi-criteria optimization problem § Finding optimal: § 79 parameter (each assumed to have 2 values only) • 5 scenarios • 32 runs • Requires 2 79 x 5 x 32 = 9.67 x 10 25 simulations! • Combinatorial Explosion § Care must be taken in choice of examined settings! § → Analysis of Variance (ANOVA) 5 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems Some Terminology Response Variables: § Outcome of experiment with regards to a performance index • Primary Factors: § Freely chosen parameter that may influence the response variable • Secondary Factors: § Parameters whose influence is not of interest (e.g. fixed ones) • Levels: § Possible values of factors (e.g. true/false, 1Gb, 2Gb, 5s, …) • Replications: § Number of runs performed • 6 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

More Terminology Interaction: § Which factors influence each other? • A 1 A 2 B 1 3 5 B 2 6 8 A 1 A 2 B 1 3 5 B 2 6 9 7 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems Rules for the Planning of Experiments Identify & Control all important parameters § Isolate effects of different parameters § Do not changes too many parameters at once • Aggregate parameters changes reasonably § Do not check each parameter combination • Treat interactions § Check for parameters that influence each other (positive and negative!) • Regard variation of responses with fixed parameters § Perform multiple runs • Use confidence intervals • 8 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

Systematic Evaluation of Complex Systems Motivation: Analysis of TCP Congestion Control § 2 k - Factorial Designs § 2 k r - Factorial Designs with Replications § 2 k-p – Fractional Factorial Designs § One Factor Experiments § Two Factor Experiments § Two Factor Experiments with Replications § General Full Factorial Designs § 9 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems 2 k - Factorial Designs Examine effects of § k factors F • with two levels • No replications • Single scenario • Which of the factors has the largest impact? § To get a feeling of the factors in the beginning of a study • To find a point to start further optimization • To reduce number of primary factors • 10 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

2 k - Factorial Designs – Approximation of Responses Observation: Impact of factors often follows strictly monotone functions § Level 2 Level 1 Idea: Approximate effects with a linear function § 11 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems 2 k - Factorial Designs – Example for a 2 2 design (I) 2 Factors at two levels § TCP Throughput MSS = 100 Bytes MSS = 400 Bytes Buffer = 10Kb 15 45 Buffer = 20Kb 25 75 Response variable with § y = q 0 + q A x A + q B x B + q AB x A x B - Estimated average throughput (Buffer = 15Kb, MSS = 250 Bytes) • q 0 - Impact by changing MSS • q A - Impact of different buffer sizes • q B - Impact due to interaction of MSS and Buffers (0 iff no interaction) • q AB ( ( − 1 if MSS = 100 − 1 if Bu ff er = 10 x A = x B = 1 if MSS = 400 1 if Bu ff er = 20 12 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

2 k - Factorial Designs – Example for a 2 2 design (II) Calculation of by linear equation system: § q 0 , q A , q B , q AB 15 = y 0 = q 0 − q A − q B + q AB 45 = y 1 = q 0 + q A − q B − q AB 25 = y 2 = q 0 − q A + q B − q AB 75 = y 3 = q 0 + q A + q B + q AB y = 40 + 20 x A + 10 x B + 5 x A x B Influence of factor twice as high as § x A x B Positive correlation between both factors § 13 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems 2 k - Factorial Designs – Example for a 2 2 design (III) Sign Table Method § I A B AB y j +1 -1 -1 +1 15 +1 +1 -1 -1 45 +1 -1 +1 -1 25 +1 +1 +1 +1 75 160 80 40 20 Total 40 20 10 5 Total / 4 Column AB is calculated by Column A * Column B § Total is calculated sum of y j values with sign of row § 14 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

2 k - Factorial Designs – Estimating the Variation Variance may be a better indicator for variables with high impact § P 2 n i =1 ( y i − y ) 2 s 2 = 2 n − 1 Simplification: compare changes of variation (without normalizing) § 2 n X ( y i − y ) 2 SST = i =1 This is by definition ( , vectors x are orthogonal) § q 0 = y X X 2 k q 2 f = 2 k X q 2 SST = SS f = f f ∈ P ( F ) f ∈ P ( F ) f ∈ P ( F ) 15 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems 2 k - Factorial Designs – Example for a 2 2 design (IV) Total variation in the example: § SST = SS A + SS B + SS AB = 2 2 q 2 A + 2 2 q 2 B + 2 2 q 2 AB = 2 2 20 2 + 2 2 10 2 + 2 2 5 2 = 1600 + 400 + 100 = 2100 Quadratic influence of the factors: § influence of A = SS A SST = 1600 2100 = 76 . 2% influence of B = SS B SST = 400 2100 = 19 . 0% → Conclusion for the study: evaluate A further! 16 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

Exercise: Estimating the impact in a 2 3 design Which of the factors in the following measurements require further § analysis? A1 A2 C1 C2 C1 C2 B1 100 15 120 10 B2 40 30 20 50 17 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems Exercise: Solution A B C AB AC BC ABC y !1 !1 !1 1 1 1 !1 100 !1 !1 1 1 !1 !1 1 15 !1 1 !1 !1 1 !1 1 40 !1 1 1 !1 !1 1 !1 30 1 !1 !1 !1 !1 1 1 120 1 !1 1 !1 1 !1 !1 10 1 1 !1 1 !1 !1 !1 20 1 1 1 1 1 1 1 50 15 '105 '175 '15 15 215 65 Total 1.875 '13.125 '21.875 '1.875 1.875 26.875 8.125 Total4/48 3.515625 172.265625 478.515625 3.515625 3.515625 722.265625 66.015625 28.125 1378.125 3828.125 28.125 28.125 5778.125 528.125 11596.875 0.2% 11.9% 33.0% 0.2% 0.2% 49.8% 4.6% 18 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

Systematic Evaluation of Complex Systems Motivation: Analysis of TCP Congestion Control § 2 k - Factorial Designs § 2 k r - Factorial Designs with Replications § 2 k-p – Fractional Factorial Designs § One Factor Experiments § Two Factor Experiments § Two Factor Experiments with Replications § General Full Factorial Designs § 19 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems 2 k r - Factorial Designs with Replications Examine effects of § k factors • with two levels • r replications • Single scenario • How sure can we be that the factors are correctly estimated? § 20 Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems