IN5060 Performance in distributed systems autumn course What is - - PowerPoint PPT Presentation
IN5060 Performance in distributed systems autumn course What is performance? Stage performance Download performance by position World Opera Production Dec 2011 @ Troms HTTP Adaptive Streaming measured on Bygdy Ferry, 2011 Download
IN5060
Stage performance
World Opera Production – Dec 2011 @ Tromsø
Stage performance
Third Life Project@WUK – Oct 2015 @ Vienna
Download performance by position
HTTP Adaptive Streaming measured on Bygdøy Ferry, 2011
Download performance by operator & algorithm
HTTP Adaptive Streaming, MONROE nodes, 2018
IN5060
Stage performance
World Opera Production – Dec 2011 @ Tromsø
Stage performance
Third Life Project@WUK – Oct 2015 @ Vienna
Download performance by position
HTTP Adaptive Streaming measured on Bygdøy Ferry, 2011
Download performance by operator & algorithm
HTTP Adaptive Streaming, MONROE nodes, 2018
Users’ perception (Quality of Experience)
Asynchrony between audio and video, 2015
IN5060
Engineers and researchers solve quantifiable challenges
Idea Prototype Simulation Study Product Deployment Feedback
U.S. PatentEEN
XVwww.www.www.www.w IN5060
Engineers and researchers solve quantifiable challenges
Idea Prototype Simulation Study Product Deployment Feedback
U.S. PatentEEN
XVwww.www.www.www.wEach step requires a performance assessment
IN5060
Engineers and researchers solve quantifiable challenges
Idea Prototype Simulation Study Product Deployment Feedback
Analysis Simulation Emulation Monitoring and measurement User studies
IN5060
Engineers and researchers solve quantifiable challenges
Idea Prototype Simulation Study Product Deployment Feedback
Simulation Monitoring and measurement User studies
IN5060
Designing and conducting studies §
pre-considerations
§
avoiding bias
§
measurement points and methods
§
data reduction
§
drawing conclusions
Specific considerations §
simulation
§
monitoring and measurement
§
user studies
Presentation and reporting §
formulating a message
§
selecting relevant factors
§
extracting and interpreting statistics
§
dimension reduction
§
selecting presentation modes
IN5060
− confront you with the tradeoffs encountered when analysing real systems − confront you with the error sources and red herrings encountered when analysing real systems
IN5060
IN5060
The Best and Greatest system of All Time
superior to the competition
convince the reviewers that their research improves on the state- of-the-art
the system that is best suited for their purpose
favourite multinational corporation> is real, or fake news
IN5060
− By providing a comparison between their own system and “the competition” − The results need to be (or appear) convincing to the target audience − This comparison is made through proper system performance analysis
useful for solving performance problems
− IN5060 will focus on experimental design, simulation, measurement and analysis − For modelling try for instance: MAT-INF3100 - Linear Optimisation
IN5060
results
− Hardware differences − Non-deterministic systems − Unexpected workloads
− Mathematical analysis − Simulation − Emulation − Measurement − User studies − Measurement techniques (monitors) − Data analysis (statistics and presentation) − Experimental design
IN5060
IN5060
Which type of monitor (or “probe”, hardware or software) would be suitable for measuring each of the following:
IN5060
File Size Packets lost on Link A Packets lost on Link B 1000 5 10 1200 7 3 1300 3 50 1
IN5060
Design measurement and simulation experiments to provide the most information with the least effort
How many experiments are needed? How do you estimate the performance impact of each factor? The performance of a system depends on the following factors:
IN5060
meaningful information.
445 446 397 226 388 3445 188 1002 47762 432 54 12 98 345 2245 8839 77492 472 565 999 1 34 882 545 4022 827 572 597 364
“Impossible things usually don’t happen.”
Statistics helps us quantify “usually.”
IN5060
− One event does not affect the other − Knowing probability of one event does not change estimate
− A variable is called a random variable if it takes one of a specified set of values with a specified probability
IN5060
IN5060
The probability density function, pdf, as . The cumulative distribution function, cdf, as .
IN5060
IN5060
hist(x) mean median mode (a) hist(x) mean median (b) modes (c) hist(x) mean median no mode (d) hist(x) mode median mean (d) hist(x) mode median mean
IN5060
à need statement about variability
“Then there is the man who drowned crossing a stream with an average depth of six inches.” – W.I.E. Gates
Frequency mean Response Time Frequency mean Response Time
If two systems have same mean, tend to prefer one with less variability
IN5060
− Variance: Square of the distance between a set of values 𝑦! with relative frequency 𝑞! and the mean 𝜈
𝑦 − 𝜈 ! = ∑"#$
%
𝑞" 𝑦" − 𝜈 !
− or, if you have exactly 𝑜 samples 𝑦" … 𝑦#
𝑦 − 𝜈 ! =
$ % ∑"#$ %
𝑦" − 𝜈 !
− Standard deviation, s, is square root of variance − Coefficient of Variation (C.O.V. ): Ratio of standard deviation to mean: = s / µ
− The x value at which the cdf takes a value α is called the α- percentile and denoted xa, so F(xa) = a
IN5060
range (SIQR)
− Q1, Q2 and Q3
IN5060
− Ex: Disk I/O mean 13, variance 48. Ok. Perhaps more useful to say data is uniformly distributed between 1 and 25. − Plus, distribution useful for later simulation or analytic modeling
− Plot histogram
For more formal testing: statistical comparison of CDF (Komolgorov-Smirnov test ) or PDF (Chi-square test) The Art of Computer Systems Performance Analysis, pp. 460-465
IN5060
− Confidence intervals
“Statistics are like alienists – they will testify for either side.” – Fiorello La Guardia
IN5060
− mean µ and stddev s. − µ is population mean
− Sample {x1, x2, …, xn} has mean x, stddev s
− With many samples, x1 != x2 != …
− Instead, get estimate of µ from x1, x2, …
IN5060
− Prob {c1 < µ < c2} = 1-a
IN5060
Sample Includes µ? 1 yes 2 yes
3
No … ... Total yes >100(1-a) f(x)
µ
construct confidence interval for each sample, the interval would include the population mean in 90 cases.
IN5060
− (x-t[1-a/2;n-1]s/sqrt(n), x+t[1-a/2;n-1]s/sqrt(n))
IN5060
− Lottery ticket $1, pays $5 million − Chance of winning is 10-7 (1 in 10 million) − To win with 90% confidence, need 9 million tickets
− So, most people happy with 0.01% confidence
IN5060
Like a work of art, a successful evaluation cannot be produced mechanically Every evaluation requires an intimate knowledge of the system and a careful selection of methodology, workloads and tools. Example of the need for knowledge: know your tradeoffs
probing for bandwidth fill up a large FIFO queue leading to added delay for all flows traversing this bottleneck.
shorter queues are recommended.
IN5060
IN5060
System Workload 1 Workload 2 A 20 10 B 10 20 System Workload 1 Workload 2 Average A 20 10 15 B 10 20 15 System Workload 1 Workload 2 Average A 2 0.5 1.25 B 1 1 1 System Workload 1 Workload 2 Average A 1 1 1 B 0.5 2 1.25
Comparing the average throughput Throughput with respect to system B Throughput with respect to system A
IN5060
Investigation of router queue length development in DASH streaming for different TCP Congestion Control algorithms CUBIC Vegas
showing an independent parameter X (time) and a dependent parameter Y (queue length)
queue length for CUBIC, but no actual distribution
IN5060
BIEB KLUDCP Tribler TRDA 5 10 15 20 Memory Usage (MBytes) average mean peak
Investigation of memory requirements for several DASH streaming algorithms
when X-axis values have no metric relation (no measure of any distance between them)
metric relation!
memory use and average peak memory use (average of peaks of several simulation runs) – this does not scale to many questions
IN5060
25 50 75 100 8 6 4 2 0 20 40 Delay /ms Loss rate /% Trebuf /s 1 Mbit/s 5 Mbits/s 15 Mbits/s 100 Mbits/s
capacity), 1 dependent variable (rebuffering time)
Some HTTP adaptive video streaming strategies can fails when packet loss is high and network delay is high as well. How long are the cumulative waiting times?
IN5060
Investigation of sender’s congestion window size in the same study. Video segments have a duration of 2 seconds (top) and 10 seconds (bottom), the algorithm attempts to choose a quality that can be downloaded in 1 second.
independent parameter X (time) and a dependent Y (congestion window size)
incapable of maintaining its congestion window between 2-second DASH segments, but enters TCP slow start
CUBIC
IN5060
Investigation of the distribution of video quality in the same study. Segments 2-sec. (left in each column) and 10-sec. (right). Patterns indicate qualities (0 stall, 5 best). Shows the shares of qualities for entire film.
segment duration independent, Y dependent)
enough recognized
IN5060
30 1e-07 1e-06 1e-05 1e-04 0.001 0.01 0.1 1 5 10 15 20 25 30 Fraction of late packets Startup delay (sec) T/mu=1.2 T/mu=1.4 T/mu=1.6 T/mu=1.8 T/mu=2.0 T/mu=2.2 T/mu=2.4
(b)
Analytical performance study to discover a relation between streaming (video) over TCP and the likelihood of stalling
Analytical graph provides deterministic, repeatable results
conditions
expose differences at when very few packets are late
parameters
IN5060
Performance study to discover a relation between streaming (video) over TCP and the likelihood of stalling – model validated by ns-2 simulation (“experiment”)
Simulation
model and simulation
and error bars show the 95% confidence interval
from the result of 1000 simulation runs
Fraction of late pkts (measurement)
1e-05 0.0001 0.001 0.01 0.1 1 2 3 4 5 6 7 8 9 10 11 Fraction of late packets
Startup delay (sec) model experiment
(b) Stored-media streaming.
IN5060
Comparing the performance of the 3 implementations
Transform” (SIFT)
a set of very specific image pairs
point in expressing errors
condition that defines “match” (adopted from an independent study that developed a good comparison method)
IN5060
Development of traffic shares
A graph using percentages to express the share of application types on the Internet
percentages
allows easy recognition of types, as well as appearance of new types
IN5060
Development of absolute mobile traffic over time
A graph using absolute value to communicate the rapid growth of mobile traffic
text in graph
allows easy recognition of types, as well as appearance of new types
IN5060
Function (CDF) provides the percentage of measurement points up to a given X value
samples not identical
samples is quite large
Hypothesis: “Thin-stream” modifications to Linux’s implementation of TCP New Reno reduces latency.
IN5060
development over time
anecdotal value
changes
property)
price for this is nearly very low quality
Comparing the bandwidth efficiency and stability of several HTTP Adaptive Streaming methods
IN5060
Stovner to Oslo S
distance from Stovner
measurement run, the measurements have been collected, and the graph shows both the average bandwidth and 1 standard deviation
Documenting the repeatability of bandwidth measurement on a typical commuter path
IN5060
100 200 300 400 500 0.0 0.2 0.4 0.6 0.8 1.0 Accepted delay(ms) Delay score density
median = 39.00 75 percentile = 85.14 Empirical cumulative density Cumulative gamma distribution
14.10 39.00 85.14
97-182 51-90 26-40
Hypothesis: users can detect that they are experiencing hand-eye latency below 100ms
(CDF) provides the percentage of measurement points up to a given X value (using dots in this case)
cumulative gamma distribution)
distribution and validate generality
IN5060
Average RTT allows for a satisfactory user experience (in theory).
Highest
application-layer latency: 67 seconds!
both dimensions
provide more information than histograms or cumulative distribution functions
Application-layer behaviour of a popular MMORPG estimated from a single server-sided measurement probe
IN5060
NVidia Tegra K1 impact of frequency on throughput
expressiveness to one or more of the dimensions
IN5060
Window of temporal integration
audio lead audio lag
Hypothesis: poor video quality can mask asynchrony between audio and video streams (note: proven wrong)
in the presentation
plotted with error bars
percent of the study population
using linear regression (shown on next slide)
IN5060
Pre-study: perception of asynchrony for different content
in the presentation
percent of the study population
associate measurements with fitted curves
50 100 150 200 250 300 350 400 10 20 30 40 50 60 70 80 90 100 Audio lead → audio lag asynchrony (ms) Chess Drums Speech
199 ms
Window of temporal integration (FWHM)
274 ms 244 ms
Drums Chess Speech
IN5060
The influence of semantic relations between visual elements on human attention.
Heatmaps allow a presentation of 2D data accumulated over time.
base data.
IN5060
Sent and acknowledged data (KB/s) Duration of TCP connection (minutes)
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 100 200 300 400 500 600 700 800 900 1000 Average data throughput (HSPA ⊕ WLAN) Average data throughput (WLAN ⊕ HSPA) Range between minimum and maximum throughput Bandwidth of emulated WLAN Bandwidth of emulated HSPA
long-lived TCP flows whose packets are alternately sent over 2 very different paths
average throughput)
Linux TCP’s ability to recover from
IN5060
Average aggregation benefit [%] −90 −80 −70 −60 −50 −40 −30 −20 −10 RTT heterogeneity (ΔRTT = RTTpri − RTTsec) [ms] Bandwidth heterogeneity (ΔBW = BWpri − BWsec) [KB/s] −93 −94 −95 −95 −95 −95 −95 −96 −97 −97 −96 −78 −72 −70 −84 −87 −87 −89 −89 −90 −91 −92 −92 −93 −94 −93 −94 −95 −96 −95 −95 −95 −96 −96 −96 −95 −81 −78 −76 −77 −78 −79 −90 −90 −91 −92 −93 −93 −94 −94 −93 −94 −96 −96 −96 −96 −96 −96 −96 −95 −93 −81 −77 −75 −74 −76 −79 −90 −90 −91 −92 −93 −93 −93 −94 −94 −95 −95 −96 −96 −96 −95 −95 −95 −94 −91 −76 −71 −68 −81 −84 −89 −90 −91 −91 −91 −92 −92 −92 −93 −94 −95 −95 −96 −96 −95 −95 −94 −94 −93 −89 −73 −66 −65 −78 −82 −86 −88 −90 −90 −91 −91 −91 −92 −91 −94 −94 −95 −95 −95 −95 −94 −94 −92 −92 −88 −72 −66 −64 −77 −80 −85 −88 −90 −89 −89 −89 −90 −91 −90 −94 −94 −94 −94 −94 −94 −93 −93 −92 −91 −88 −66 −57 −55 −73 −79 −83 −86 −88 −87 −88 −87 −90 −90 −90 −93 −93 −93 −93 −93 −92 −89 −89 −87 −88 −88 −58 −48 −46 −74 −79 −81 −80 −84 −82 −86 −87 −89 −89 −90 −93 −92 −92 −93 −92 −92 −88 −88 −86 −88 −84 −48 −35 −35 −69 −75 −78 −76 −82 −81 −85 −85 −87 −87 −87 −93 −91 −91 −91 −91 −90 −83 −81 −75 −79 −73 −44 −31 −29 −56 −65 −71 −71 −77 −77 −84 −85 −88 −87 −84 −92 −90 −87 −85 −84 −84 −82 −82 −74 −76 −68 −43 −26 −20 −44 −57 −66 −68 −67 −70 −76 −81 −85 −85 −80 −91 −89 −82 −79 −76 −75 −69 −68 −61 −65 −58 −32 −18 −17 −45 −61 −67 −70 −67 −71 −77 −82 −84 −83 −80 −86 −85 −79 −72 −70 −66 −63 −59 −58 −62 −51 −17 −9 −20 −61 −72 −77 −76 −73 −74 −78 −83 −85 −85 −84 −79 −81 −78 −72 −67 −62 −57 −54 −50 −54 −43 −13 −11 −32 −73 −81 −84 −85 −88 −86 −89 −89 −91 −90 −91 −71 −78 −79 −72 −66 −55 −53 −49 −53 −56 −46 −20 −23 −43 −80 −85 −89 −90 −91 −91 −92 −94 −95 −95 −96 −66 −77 −80 −72 −65 −56 −55 −54 −53 −56 −49 −33 −37 −51 −83 −88 −90 −92 −93 −94 −95 −95 −96 −96 −96 −64 −76 −80 −72 −69 −61 −54 −54 −58 −67 −59 −35 −40 −53 −86 −90 −92 −93 −84 −84 −85 −96 −96 −96 −97 −69 −78 −81 −76 −75 −69 −61 −61 −61 −70 −59 −33 −38 −52 −89 −91 −93 −94 −85 −85 −86 −97 −97 −97 −97 −79 −83 −83 −82 −80 −76 −68 −68 −70 −76 −69 −41 −47 −57 −91 −92 −94 −95 −85 −86 −86 −97 −97 −97 −97 −87 −88 −87 −86 −85 −80 −74 −73 −73 −74 −70 −54 −61 −68 −92 −94 −95 −96 −96 −97 −97 −97 −97 −97 −97 −89 −89 −88 −86 −85 −85 −81 −80 −76 −76 −70 −62 −68 −78 −94 −95 −96 −96 −97 −97 −97 −97 −97 −97 −97 −88 −88 −87 −86 −87 −86 −83 −81 −78 −80 −73 −64 −68 −78 −94 −94 −96 −96 −97 −97 −97 −97 −97 −97 −97 −88 −87 −87 −87 −87 −86 −85 −84 −82 −81 −73 −60 −66 −76 −94 −95 −96 −96 −97 −97 −98 −98 −98 −98 −98 −88 −86 −86 −86 −87 −85 −84 −84 −83 −84 −80 −68 −71 −77 −95 −95 −96 −97 −97 −97 −98 −98 −98 −98 −98 −87 −84 −85 −84 −85 −83 −82 −83 −82 −83 −80 −71 −75 −81 −96 −96 −97 −97 −98 −98 −98 −98 −98 −98 −98 −240 −160 −80 80 160 240 1800 1200 600 −600 −1200 −1800more primary path bandwidth higher primary path RTT
Average Aggregation Benefit [%]
Average aggregation gain [%] −80 −60 −40 −20 20 40 60 RTT heterogeneity (∆RTT = RTTpri − RTTsec) [ms] Capacity heterogeneity (∆C = Cpri − Csec) [KB/s] −87 −91 −91 −87 −76 −64 −53 −51 −50 −41 −25 −3 6 17 16 13 8 −1 −2 −6 −3 −7 −8 −11 −8 −90 −91 −91 −85 −73 −57 −42 −37 −34 −28 −15 6 14 16 13 13 15 8 7 5 3 2 2 1 −92 −92 −90 −82 −66 −51 −32 −29 −24 −24 −13 10 21 23 16 13 16 12 10 8 6 7 9 7 −1 −93 −93 −90 −81 −64 −45 −23 −16 −13 −12 1 23 33 34 28 27 26 23 21 21 19 15 13 11 11 −95 −94 −88 −74 −56 −39 −24 −13 −9 −1 7 26 30 34 31 34 31 26 22 24 24 21 17 15 12 −95 −93 −88 −75 −60 −41 −28 −14 −6 11 20 35 32 38 39 45 39 33 31 33 28 23 20 22 25 −96 −93 −81 −70 −52 −37 −19 −10 17 24 37 33 40 41 42 39 32 33 35 32 30 27 28 23 −95 −85 −66 −52 −38 −27 −9 −5 5 19 32 43 43 50 45 41 38 33 33 31 30 31 30 28 19 −95 −87 −55 −34 −11 −3 12 12 16 25 40 48 51 53 49 45 41 35 32 35 39 40 38 34 30 −93 −80 −47 −25 −4 7 23 22 23 32 44 52 54 59 55 52 45 42 38 37 38 36 37 34 33 −91 −82 −51 −26 −4 9 23 31 35 42 50 54 56 61 62 61 53 48 44 40 41 38 41 36 36 −88 −62 −31 −8 9 18 32 41 49 54 58 64 62 65 63 62 59 54 45 39 34 41 45 36 13 −49 −28 −1 22 33 33 36 48 60 64 63 66 67 68 66 63 61 50 41 36 38 47 47 38 14 −11 11 32 44 55 49 53 53 63 64 63 66 63 63 57 55 53 44 39 39 40 39 33 24 3 30 39 50 53 56 49 52 53 62 61 58 56 53 53 45 44 42 37 36 37 37 29 20 13 9 44 46 53 50 54 46 57 52 61 56 58 56 54 50 39 34 30 33 35 36 30 16 5 −2 −5 44 42 48 46 54 46 51 50 58 56 53 52 47 40 31 29 31 32 26 21 14 12 2 −5 −9 47 39 42 47 53 46 45 43 49 49 47 48 43 36 26 23 23 21 15 10 8 4 −7 −18 −21 32 34 37 45 44 43 37 44 47 48 41 41 30 24 14 11 14 10 4 −2 −1 −10 −23 −33 28 35 40 44 40 40 38 40 41 40 39 37 27 20 12 4 −6 −7 −3 −3 −6 −18 −29 −37 20 27 33 33 34 35 34 36 37 38 37 35 22 14 3 −6 −12 −15 −14 −11 −14 −16 −26 −34 −43 23 31 38 38 43 42 35 31 28 27 27 29 20 14 −6 −13 −18 −17 −16 −18 −25 −34 −40 −44 19 23 26 27 30 30 21 22 19 17 17 20 12 3 −14 −16 −22 −24 −24 −27 −30 −37 −43 −47 −46 16 22 24 25 20 21 15 19 17 11 12 14 9 −4 −19 −21 −27 −30 −31 −33 −35 −42 −49 −51 −50 13 16 10 13 2 6 5 8 12 9 13 12 4 −9 −27 −33 −43 −43 −44 −44 −47 −51 −53 −54 −57 −240 −160 −80 80 160 240 1800 1200 600 −600 −1200 −1800New Reno Linux TCP’s “New Reno”
aggregation benefit (color) and detected reordering (number)
confidence intervals) gets lost
TCP’s ability to benefit from using the capacity
IN5060
No goals:
choices of techniques, tools, metrics, workloads. §
Without goals, modeling must be identical to reality
− imagine weather models or models of the universe without specific goals
§
There are no general-purpose models. Models are always simplifications of the real world, actively dropping detail.
− without goals, there is no simplification − without simplification, modeling is identical to building
§
Defining goals is difficult, especially in combination with bias
IN5060
No goals:
choices of techniques, tools, metrics, workloads. Biased goals:
to perform a fair evaluation of the systems that are compared.
Be aware of the risk of bias that is present in these interests! bias
(Webster’s dictionary) 1.c) deviation of the expected value of a statistical estimate from the quantity it estimates 1.d) systematic error introduced into sampling or testing by selecting or encouraging one outcome or answer over others
IN5060
Unsystematic approach:
workloads etc. Random choices will provide inaccurate answers. §
Identify a complete set of
− goals − system parameters − factors − metrics − workloads
§
then define a goal and select the appropriate subset
IN5060
Unsystematic approach:
workloads etc. Random choices will provide inaccurate answers. Analysis without understanding the Problem:
what is really the problem. This will improve the chances of success by a large factor. §
Identify the real problem
− this may require a lot of prior work − the answer of the preparation may diverge from expectations or common assumptions
§
This is not always easy
− e.g.: for decades, TCP has been improved for throughput
it was very hard to sell latency as a valid problem
IN5060
Incorrect performance metrics:
metrics. §
e.g.: “everybody knows” about TCP that acknowledgement for the same packet that arrives at the sender 3 times triggers a congestion event and a retransmission
− except that it doesn’t happen in Linux TCP
§
e.g.: Network performance measurement was all about throughput and
IN5060
Unrepresentative workload:
§
the usual simulation for TCP research looks like this, and “greedy” streams are sent through the bottleneck
§
ignoring that most flows in real networks are extremely short
IN5060
Criterion Analytical Modeling Simulation Measurement Stage any any Post-prototype Time required small medium varies Tools analysts programs instrumentation Accuracy low moderate varies Trade-off evaluation easy moderate difficult Cost low medium high Saleability low medium high Insight high medium low
The combination of two and more of these techniques add to sellability! Modelling gives you the best understanding of what's going
by one of the the other two
IN5060
Overlooking important parameters
characteristics that may affect the performance
between parameters to include in the study to allow completion
IN5060
Ignoring significant factors
which factors to use §
note that a factor is an input parameter
− there are factors that can usually be ignored because they are mostly constant − but these may have huge influence when they do vary – make a pre-study before removing
them
§
a new challenge has arrived with the prevalence of machine learning:
− failing to attempt to isolate and understand parameters − assuming that you created a machine learning network that will discover them by itself
IN5060
Inappropriate experimental design
§
Be careful when selecting the numbers of experiments to run and when
§
selecting parameter values.
§
If there are dependencies between the effects of some parameters and other parameters in the experiment, a full factorial experiment or fractional factorial experiment may improve the results.
§
design should be simple but not too simply
− but it looses detail − can you afford that?
IN5060
Inappropriate level of detail
§
When modelling, the formulation should not be too broad, nor too narrow.
§
No analysis
§
After collecting a huge pile of data, make sure to apply analytical skills to ease the new knowledge out of the raw data
§
very different: high-level model
§
compare details: detailed model
§
measurement campaigns can frequently end in this problem
− you have to conduct them when the opportunity arises − you have to collect whatever you can think of − you cannot go back and collect more
§
filtering the right parameters is a major challenges, tools PCA help only for independent Euclidian variables – so you may be in trouble
IN5060
Erroneous analysis
§
Be careful to avoid common mistakes when analysing the data
§
Be careful to not apply wishful thinking in the analysis No sensitivity analysis
§
The results may be sensitive to workload and system parameters.
§
Analyse the outcomes considering such sensitivity.
§
a very typical danger in analytical approaches is to forget the assumption that parameters are normally distributed before applying a statistical operation
§
a result may not be desirable even if it is best in an example, but it is highly unstable, meaning that performance results change strongly (to the negative) when one or more parameters change slightly
§
a result may not be trustworthy if a jhigh-impact parameter is assumed to be constant, but it isn’t in reality
IN5060
Ignoring errors in input
§
Often the parameters of interest cannot be measured and is estimated using another parameter.
§
In such cases, the analyst needs to adjust confidence of the output obtained from such data.
§
a recent example
− assumptions about the presence of an advanced queue management (AQMs)
strategy at the network level in a wireless system
− to design algorithms in wireless systems, it is important to know whether AQM are
deployed
− but time slicing at the link layer level can look like AQM and prevent its correct
detetion
IN5060
Improper treatment of outliers
included requires intimate knowledge of the system Assuming no change in the future
might need to be taken into consideration §
confidence intervals
§
but can they be ignored?
§
what is an outlier?
§
A hugely important question in crowdsourcing! è filtering based on assumptions
IN5060
§
this is a typical sight in paper today
§
time-based plots and average as the only applied statistical method
§
it makes it impossible to discover and expose instabilities from factors
§
it makes it really hard to understand variability in results
IN5060
§
simple questions may have a simple answer
§
I saw in a paper
− use of a Poisson-distribution for packet interarrival time, its average interarrival
time E given
− then, use of a machine learning model to detect average interarrival time − Why?
IN5060
Improper presentation of results
properties of the analysis fairly Ignoring social aspects
need to sell the analysis to decision makers.
maker(s) §
Even if bias was avoided in the study, it can still be in the presentation
IN5060
Omitting Assumptions and Limitations
analysis.
inappropriate scenarios (for instance as referenced work)
§
a study is always limited to some extent
§
be aware of your limitations and share them with your audience
§
even better, make your study repeatable by sharing code and data
IN5060
# What to check 1. Is the system correctly defined and the goals clearly stated? 2. Are the goals stated in an unbiased manner? 3. Have all the steps of the analysis followed systematically? 4. Is the problem clearly understood before analyzing it? 5. Are the performance metrics relevant for this problem? 6. Is the workload correct for this problem? 7. Is the evaluation technique appropriate? 8. Is the list of parameters that affect performance complete? 9. Have all parameters that affect performance been chosen as factors to be varied? 10. Is the experimental design efficient in terms of time and results? 11. Is the level of detail proper? 12. Is the measured data presented with analysis and interpretation? 13. Is the analysis statistically correct?
IN5060
# What to check 14. Has the sensitivity analysis been done? 15. Would errors in the input cause an insignificant change in the results? 16. Have the outliers in the input or output been treated properly? 17. Have the future changes in the system and workload been modeled? 18. Has the variance of input been taken into account? 19. Has the variance of the results been analyzed? 20. Is the analysis easy to explain? 21. Is the presentation style suitable for its audience? 22. Have the results been presented graphically as much as possible? 23. Are the assumptions and limitations of the analysis clearly documented?
IN5060
− What is the goals of the study? − What is the boundaries of the system you want to measure?
− Each system provides a set of services − When a user requests any of these services there are a number of possible outcomes − Some of the outcomes are desirable, some are not − This list will be useful when selecting the right metrics and workloads
IN5060
− Select the criteria used for comparing the performance
− Make a list of all the parameters that affect the performance − It might be useful to divide the list into system parameters and workload parameters − This list might grow as you learn from the first iterations of experiments and analysis.
IN5060
− The list of parameters can be divided into two parts: those that will be varied in the study and those that will not. − The parameters that are varied are called factors and their values are called levels − An important part of the work is to choose the factors so that the study will be possible to complete with the given resources
− Models, simulation or measurement
IN5060
7) Select workload
− The workload consists of a series of service requests to the system − You need to measure and understand the characteristics of a system in
− You can build on other people’s workload analysis, but beware the future==past trap.
8) Design experiments
− Once you have the list of factors and levels, you need to decide on a sequence of experiments that offer maximum information with minimal effort. − 2 phases can be useful: 1) Large number of factors, small number of levels to determine the relative effect of factors; 2) fewer factors / more levels for factors with significant impact
IN5060
9) Analyse and interpret data
− Choose appropriate statistical techniques − Try to make a fair evaluation between the systems
10) Present results
− Visualise the data in a way that fairly and clearly shows the differences in performance − A good metric for visualisation/presentation is how much effort it takes to read/understand the presentation. Easy = good
IN5060
Steps for a Performance Evaluation Study 1. State the goals of the study and define the system boundaries 2. List system services and possible outcomes 3. Select performance metrics 4. List system and workload parameters 5. Select factors and their values 6. Select evaluation techniques 7. Select the workload 8. Design the experiments 9. Analyse and interpret the data 10. Present the results. Start over if necessary.
IN5060
− introduce many of the main concepts of performance analysis − introduce the topics that form the basis of the graded assignments − provide example reports of good quality for you to study − be available on email for guidance and pointers
IN5060
− Go to the literature (and the web) for details and resources to help you on the way − Apply your own skills and judgement in the selection of metrics and methodology − Justify your choices and try to avoid making random or biased selections − You will face a lot of tradeoffs and difficult choices. Ask for
− This is what researchers and industry professionals are required to do in their practice