Systematic Evaluation of Complex Systems Acknowledgement: Parts of - - PDF document

1

Systematic Evaluation of Complex Systems

Acknowledgement: Parts of these slides are based on [Jai91]

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

2

Systematic Evaluation of Complex Systems

§

Motivation: Analysis of TCP Congestion Control

§

2k - Factorial Designs

§

2kr - Factorial Designs with Replications

§

2k-p – Fractional Factorial Designs

§

One Factor Experiments

§

Two Factor Experiments

§

Two Factor Experiments with Replications

§

General Full Factorial Designs

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

3

Quick Reminder: TCP Congestion Control

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

Slow start Additive increase, congestion avoidance New slow start to new threshold, then linear increase

20=40/2

Reset CW to 1, new threshold = CW/2

Initial congestion threshold Initial congestion window size

4

TCP Congestion Control Parameters

§

Performance affected by

• Initial Congestion Window
• Initial Congestion Window Threshold
• Timeout
• Enable Duplicate Acknowledgements
• Size of TCP-Buffers
• Maximum Segment Size (MSS)
• …
• MacOS 10.7: sysctl -a | grep tcp | wc –l => 79 Parameters
• MacOS 10.10: sysctl -a | grep tcp | wc –l => 116 Parameters
• MacOS 10.11: sysctl -a | grep tcp | wc –l => 120 Parameters
• Some Boolean other numeric

§

For different scenarios, e.g.

• Internet, LAN, DSL, Satellite, Congestion etc.

§

For different runs

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

5

Choosing Optimal Parameters

§

Choosing good parameters is a multi-criteria optimization problem

§

Finding optimal:

• 79 parameter (each assumed to have 2 values only)
• 5 scenarios
• 32 runs
• Requires 279 x 5 x 32 = 9.67 x 1025 simulations!

§

Combinatorial Explosion

§

Care must be taken in choice of examined settings!

→ Analysis of Variance (ANOVA)

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

6

Some Terminology

§

Response Variables:

• Outcome of experiment with regards to a performance index

§

Primary Factors:

• Freely chosen parameter that may influence the response variable

§

Secondary Factors:

• Parameters whose influence is not of interest (e.g. fixed ones)

§

Levels:

• Possible values of factors (e.g. true/false, 1Gb, 2Gb, 5s, …)

§

Replications:

• Number of runs performed

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

7

More Terminology

§

Interaction:

• Which factors influence each other?

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

A1 A2 B1 3 5 B2 6 8 A1 A2 B1 3 5 B2 6 9

8

Rules for the Planning of Experiments

§

Identify & Control all important parameters

§

Isolate effects of different parameters

• Do not changes too many parameters at once

§

Aggregate parameters changes reasonably

• Do not check each parameter combination

§

Treat interactions

• Check for parameters that influence each other (positive and negative!)

§

Regard variation of responses with fixed parameters

• Perform multiple runs
• Use confidence intervals

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

9

Systematic Evaluation of Complex Systems

§

Motivation: Analysis of TCP Congestion Control

§

2k - Factorial Designs

§

2kr - Factorial Designs with Replications

§

2k-p – Fractional Factorial Designs

§

One Factor Experiments

§

Two Factor Experiments

§

Two Factor Experiments with Replications

§

General Full Factorial Designs

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

10

2k - Factorial Designs

§

Examine effects of

• k factors F
• with two levels
• No replications
• Single scenario

§

Which of the factors has the largest impact?

• To get a feeling of the factors in the beginning of a study
• To find a point to start further optimization
• To reduce number of primary factors

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

11

2k - Factorial Designs – Approximation of Responses

§

Observation: Impact of factors often follows strictly monotone functions

§

Idea: Approximate effects with a linear function

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

Level 1 Level 2

12

2k - Factorial Designs – Example for a 22 design (I)

§

2 Factors at two levels

§

Response variable with

• Estimated average throughput (Buffer = 15Kb, MSS = 250 Bytes)
• Impact by changing MSS
• Impact of different buffer sizes
• Impact due to interaction of MSS and Buffers (0 iff no interaction)

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

TCP Throughput MSS = 100 Bytes MSS = 400 Bytes Buffer = 10Kb 15 45 Buffer = 20Kb 25 75

q0 y = q0 + qAxA + qBxB + qABxAxB qA qB qAB xA = ( −1 if MSS = 100 1 if MSS = 400 xB = ( −1 if Buffer = 10 1 if Buffer = 20

13

2k - Factorial Designs – Example for a 22 design (II)

§

Calculation of by linear equation system:

§

Influence of factor twice as high as

§

Positive correlation between both factors

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

q0, qA, qB, qAB y = 40 + 20xA + 10xB + 5xAxB 15 = y0 = q0 − qA − qB + qAB 45 = y1 = q0 + qA − qB − qAB 25 = y2 = q0 − qA + qB − qAB 75 = y3 = q0 + qA + qB + qAB xA xB

14

2k - Factorial Designs – Example for a 22 design (III)

§

Sign Table Method

§

Column AB is calculated by Column A * Column B

§

Total is calculated sum of yj values with sign of row

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

I A B AB yj +1

• 1
• 1

+1 15 +1 +1

• 1
• 1

45 +1

• 1

+1

• 1

25 +1 +1 +1 +1 75 160 80 40 20 Total 40 20 10 5 Total / 4

15

2k - Factorial Designs – Estimating the Variation

§

Variance may be a better indicator for variables with high impact

§

Simplification: compare changes of variation (without normalizing)

§

This is by definition ( , vectors x are orthogonal)

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

q0 = y SST = X

f∈P(F )

SSf = X

f∈P(F )

2kq2

f = 2k X f∈P(F )

q2

f

s2 = P2n

i=1(yi − y)2

2n − 1 SST =

2n

X

i=1

(yi − y)2

16

2k - Factorial Designs – Example for a 22 design (IV)

§

Total variation in the example:

§

Quadratic influence of the factors:

→ Conclusion for the study: evaluate A further!

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

SST = SSA + SSB + SSAB = 22q2

A + 22q2 B + 22q2 AB

= 22202 + 22102 + 2252 = 1600 + 400 + 100 = 2100 influence of A = SSA SST = 1600 2100 = 76.2% influence of B = SSB SST = 400 2100 = 19.0%

17

Exercise: Estimating the impact in a 23 design

§

Which of the factors in the following measurements require further analysis?

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

A1 A2 C1 C2 C1 C2 B1 100 15 120 10 B2 40 30 20 50

18

Exercise: Solution

A B C AB AC BC ABC y

!1 !1 !1 1 1 1 !1 100 !1 !1 1 1 !1 !1 1 15 !1 1 !1 !1 1 !1 1 40 !1 1 1 !1 !1 1 !1 30 1 !1 !1 !1 !1 1 1 120 1 !1 1 !1 1 !1 !1 10 1 1 !1 1 !1 !1 !1 20 1 1 1 1 1 1 1 50 15 '105 '175 '15 15 215 65 Total 1.875 '13.125 '21.875 '1.875 1.875 26.875 8.125 Total4/48 3.515625 172.265625 478.515625 3.515625 3.515625 722.265625 66.015625 28.125 1378.125 3828.125 28.125 28.125 5778.125 528.125 11596.875 0.2% 11.9% 33.0% 0.2% 0.2% 49.8% 4.6%

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

19

Systematic Evaluation of Complex Systems

§

Motivation: Analysis of TCP Congestion Control

§

2k - Factorial Designs

§

2kr - Factorial Designs with Replications

§

2k-p – Fractional Factorial Designs

§

One Factor Experiments

§

Two Factor Experiments

§

Two Factor Experiments with Replications

§

General Full Factorial Designs

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

20

2kr - Factorial Designs with Replications

§

Examine effects of

• k factors
• with two levels
• r replications
• Single scenario

§

How sure can we be that the factors are correctly estimated?

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

21

2kr - Factorial Designs with Replications

§

Response variables where i represents the factorial combination and j reflects the run

§

Measurements influenced by errors

§

Thus

§

The errors for each parameter set are expected to sum to zero, thus

§

Estimation of and the influence of the factors as in 2k design but by using

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

yi,j ei,j q0 qP(F ) ˆ yi yi,j = q0 + X

f∈P(F )

qfxf + ei,j ˆ yi = 1 r

r

X

j=1

yi,j

22

Variation in 2kr Designs

§

If we additionally assume that errors are statistically independent:

§

The quadratic impact is again

§

Replications also allow for estimation of errors done during estimation!

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

Impact of f = SSf SST SST = X

f∈P(F )

SSf + SSE = X

f∈P(F )

2krq2

f + 2k

X

i=1 r

X

j=1

e2

i,j

23

Confidence Intervals in 2kr Designs

§

Replications allow for assurance checks of effects

§

Variance for confidence intervals:

§

Q: Why not ?

§

As errors are assumed to be distributed independently:

§

Confidence intervals for :

§

Similar for responses

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

s2

e =

SSE 2k(r − 1) 2kr − 1 s2

q0 = s2 qA = s2 qf = s2 e

2kr qf qf ± t[1−α/2,2k(r−1)] · sf

ˆ yi

24

Assumptions on the errors

§

Assumptions about the errors

• Statistically independent
• Additive
• Normally distributed
• Constant standard deviation
• Effects of factors are additive

§

Must be verified for every response variable!

• Take system knowledge into account
• Or use statistical Tests: Chi-Squared-Test, F-Test
• Or use visual verification

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

25

Example for visual error verification

§

Consider the following measurements

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

A1 A2 B1 (15, 18, 12) (25, 28,19) B2 (45, 48, 51) (75, 81, 75)

26

Calculated Effects and Variations

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

I A B AB y yx1 yx2 yx3 ex1 ex2 ex3

1 "1 "1 1 15.00 &&&&&& 15 18 12 3 "3 1 "1 1 "1 48.00 &&&&&& 45 48 51 "3 3 1 1 "1 "1 24.00 &&&&&& 25 28 19 1 4 "5 1 1 1 1 77.00 &&&&&& 75 75 81 "2 "2 4 165.00 39.00 85.00 19.00 Total 41.25 9.75 21.25 4.75 Total7/74 SSE= 102 SE2= 12.75 Effect 380.25 1806.25 90.25 2378.75 SQ0= 3.092 Percent7of7 Variation 15.99% 75.93% 3.79% Conf= 7.131 Confidence 16.88 28.38 11.88 Intervals 2.62 14.12 "2.38

27

Plot Errors

§

Plot errors against projected y-value

§

Make sure: no direct relationship visible

→ Looks good!

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

• 6
• 5
• 4
• 3
• 2
• 1

1 2 3 4 5 50 100

28

Normal-Quantile-Quantile-Plot

§

Plot measured errors vs. where they are expected in a perfect normal distribution

§

Should follow the linear function y = x

§

Ensures errors follow a normal distribution

§

In Excel Speak: =NORMINV(RANG(P21,P$18: P$29,1)/(12+1),0,STABW(P$1 8:P$29))

→ Looks good!

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

• 6,00
• 4,00
• 2,00

0,00 2,00 4,00 6,00

• 5

5

29

Exercise: Discussion of a scenario

§

Take the following (hypothetical) performance measurements to determine if one of the operating systems is particularly well suited

• ne of the scenarios.

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

TCP Throughput Windows Linux Scenario 1 (9.55, 9.33, 11.22) (20.58, 19.51, 21.28) Scenario 2 (96.48, 97.99, 102.67) (395.39, 407.22, 366.43)

30

Exercise: Solution (1st Try)

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

I A B AB y yx1 yx2 yx3 ex1 ex2 ex3

1 "1 "1 1 10.00 9.55 9.33 11.13 "0.45 "0.67 1.13 1 "1 1 "1 99.05 96.48 97.99 102.67 "2.57 "1.06 3.63 1 1 "1 "1 20.46 20.58 19.51 21.28 0.12 "0.95 0.82 1 1 1 1 389.68 395.40 407.22 366.43 5.71 17.54 "23.25 519.31 301.21 458.15 280.06 Total 129.83 75.30 114.54 70.01 Total7/74 SSE= 905.348 SE2= 113.168 Effect 22681.87 52474.68 19607.94 95669.83 SQ0= 9.21284 Percent7of7 Variation 23.71% 54.85% 20.50% Conf= 21.2448 Confidence 96.55 135.78 91.26 Intervals 54.06 93.29 48.77

31

Exercise: Solution (1st Try)

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

• 30,00
• 25,00
• 20,00
• 15,00
• 10,00
• 5,00

0,00 5,00 10,00 15,00 20,00

• 100,00

200,00 300,00 400,00 500,00

Error vs. y

32

Exercise: Solution (1st Try)

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

• 30,00
• 25,00
• 20,00
• 15,00
• 10,00
• 5,00

0,00 5,00 10,00 15,00 20,00

• 15
• 10
• 5

5 10 15

Normal Quantile Quantile Plot

33

Exercise: Solution (2nd Try – Using logarithms)

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

I A B AB y yx1 yx2 yx3 ex1 ex2 ex3

1 "1 "1 1 1.00 0.98 0.97 1.05 "0.02 "0.03 0.05 1 "1 1 "1 2.00 1.98 1.99 2.01 "0.01 0.00 0.02 1 1 "1 "1 1.31 1.31 1.29 1.33 0.00 "0.02 0.02 1 1 1 1 2.59 2.6 2.61 2.56 0.01 0.02 "0.03 6.90 0.95 2.29 0.28 Total 1.72 0.24 0.57 0.07 Total7/74 SSE= 0.01 SE2= S0= 0.02 Effect 0.22 1.31 0.02 1.56 Conf= 0.05 Percent7of7 Variation 14.36% 84.02% 1.25% Confidence 0.29 0.63 0.12 Intervals 0.18 0.52 0.02

34

Exercise: Solution (2nd Try – Using logarithms)

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

• 0,04
• 0,03
• 0,02
• 0,01

0,01 0,02 0,03 0,04 0,05 0,06 0,5 1 1,5 2 2,5 3

Error vs. y

35

Exercise: Solution (2nd Try – Using logarithms)

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

• 0,04
• 0,03
• 0,02
• 0,01

0,01 0,02 0,03 0,04 0,05

• 0,04
• 0,02

0,00 0,02 0,04 0,06

36

2kr Designs – Lessons learned

§

False assumptions lead to false correlations

§

But replications allow for verification of experiments

§

Always perform F-Test or visual verification otherwise confidence intervals are useless!

§

Be sure to understand your system!

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

37

Systematic Evaluation of Complex Systems

§

Motivation: Analysis of TCP Congestion Control

§

2k - Factorial Designs

§

2kr - Factorial Designs with Replications

§

2k-p – Fractional Factorial Designs

§

One Factor Experiments

§

Two Factor Experiments

§

Two Factor Experiments with Replications

§

General Full Factorial Designs

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

38

2k-p – Fractional Factorial Designs

§

Examine effects of

• k factors
• with two levels
• without replications
• Single scenario
• But less experiments…

§

How can we save work?

§

Idea: Neglect relationship between factors (e.g. )

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

qABCD

39

Example for a 2k-p Design

§

Example for a 23-1 Design:

§

Factors AB, AC, BC, and ABC are neglected!

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

I A B C yj +1

• 1
• 1

+1 15 +1 +1

• 1
• 1

45 +1

• 1

+1

• 1

25 +1 +1 +1 +1 75 160 80 40 20 Total 40 20 10 5 Total / 4

40

Requirements on 2k-p Designs

§

Values of must form orthogonal vector space, s.t.

• Sum of any column j equals 0
• Sum of product of any two columns j and g equals 0
• Sum of the squares of any column j is 2k-p

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

xi,j

2k−p

X

i=1

xi,j = 0

2k−p

X

i=1

xi,jxi,g = 0

2k−p

X

i=1

x2

i,j = 2k−p

41

Example for a 2k-p Design

§

Effect calculated like before:

§

However, what about ?

§

Both effects are confounded!

§

Which effects confound depends on the choice of

§

Within the example:

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

qC = y1 − y2 − y3 + y4 4 qC qAB qAB = y1 − y2 − y3 + y4 4 xi,j I = ABC ⇒ A = A2BC = BC, B = AC, C = AB

42

Calculating the Variation in a 2k-p Design

§

Variation was defined as

§

Where SST is the variation of all factors and combinations, but not all might be calculated…

§

However due to all x adding to 0

§

Thus

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

Impact of f = SSf SST SSY = SS0 + SST SST = SSY − SS0 =

2k−p

X

i=1

y2

i − 2k−pq2

43

Exercise: A 24-1 Design

§

Calculate all effects and variations of effects. Which factors are negligible?

§

Which factors interact? How would you plan such an experiment?

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

A1 A2 C1 C2 C1 C2 B1 D1

• 1.5

10

• D2

4

• 3

B2 D1

• 2

12

• D2

1

• 5

44

Exercise: A 24-1 Design

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

I A B C D y

1 "1 "1 "1 1 4 1 "1 "1 1 "1 1.5 1 "1 1 "1 1 1 1 "1 1 1 "1 2 1 1 "1 "1 "1 10 1 1 "1 1 1 3 1 1 1 "1 "1 12 1 1 1 1 1 5 38.5 21.5 1.5

• 15.5
• 12.5 Total

4.8125 2.6875 0.1875

• 1.9375
• 1.5625 Total/8

57.78125 0.28125 30.03125 19.53125 115.96875 49.8% 0.2% 25.9% 16.8% SSY/= 301.25 SS0/= 185.28125 SST/=/ 115.96875

45

Exercise: Confoundings

§

I = ACD

§

A = CD

§

B = ABCD

§

C = AD

§

D = AC

§

BA = BCD

§

BC = BAD

§

BD = BAC

§

Different design would be better....

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

46

2k-p Designs – Lessons learned

§

Choose vector space wisely!

§

Do it only if interaction between factors is expected be negligible!

§

May safe quite a lot of runs….

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

47

Systematic Evaluation of Complex Systems

§

Motivation: Analysis of TCP Congestion Control

§

2k - Factorial Designs

§

2kr - Factorial Designs with Replications

§

2k-p – Fractional Factorial Designs

§

One Factor Experiments

§

Two Factor Experiments

§

Two Factor Experiments with Replications

§

General Full Factorial Designs

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

48

One Factor Experiments

§

Examine effects of

• ne factors F
• with a levels
• r replications
• Single scenario

§

Standard setup:

• We can measure each level separately
• Calculate a mean for each level
• Calculate confidence intervals for each level

§

But, we can do better! (if certain presumptions are fulfilled)

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

49

One Factor Experiments

§

Model:

§

Average response without levels

§

Influence of a level j:

§

Errors (independent of level)

§

Estimating

§

Estimating

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

r

X

i=1 a

X

j=1

yi,j = arµ + r

a

X

j=1

αj +

r

X

i=1 a

X

j=1

ei,j µ αj ei,j µ αj µ = Pr

i=1

Pa

j=1 yi,j

ar αj = 1 r

r

X

i=1

yi,j − µ

50

One Factor Experiments – Estimating Variations

§

Total variation explained by the factor

§

Helps to compare factors vs. errors

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

SSA SST = SSA SSY − SS0 = r Pa

j=1 α2 j

Pr

i=1

Pa

j=1 y2 i,j − arµ2

51

One Factor Experiments – Confidence Intervals

§

Error variance like before:

§

Variances of average and effects

§

Confidence intervals for response variables:

§

Again: Distribution & Independence must be verified for every response variable!

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

s2

e =

SSE a(r − 1) sµ = se √ar yj = µ + αj ± t[1−α/2,a(r−1)]sµ+α sα = r s2

e(a − 1)

ar = se r a − 1 ar

sµ+α = sµ + sα

52

Exercise: Comparing the approach to the naïve one

§

Compare the average throughput on the three machines

• by estimating a common µ
• by estimating the parameters separately

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

TCP Throughput Windows (206.9 144.7 95.3 198.1 103.9) Linux (172.4 122.2 157.7 163.5 149.8) MacOS (157.3 204.2 172.6 203.2 189)

53

Exercise: Solution

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

Y Mean Alpha

206.9 144.7 95.3 198.1 103.9 149.78 ,12.94 0.05 172.4 122.2 157.7 163.5 149.8 153.12 ,9.60 0.03 157.3 204.2 172.6 203.2 189 185.26 22.54 0.14 Total 162.72 Variation 22%

Errors

57.12 ,5.08 ,54.48 48.32 ,45.88 19.28 ,30.92 4.58 10.38 ,3.32 ,27.96 18.94 ,12.66 17.94 3.74 SSE 13800.47 SST= 17638.744 s t Var

VarNaive

e 33.91 1.78 76.32 mu 8.76 1.78 15.61 28.27 alpha 12.38 1.78 22.07 29.85 mu_alpha 15.17 1.78 27.03

54

Confidence Intervals: α-Values

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

• 40,00
• 30,00
• 20,00
• 10,00

0,00 10,00 20,00 30,00 40,00 50,00 0,5 1 1,5 2 2,5 3 3,5

55

Confidence Intervals: µ + α-Values

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

50 100 150 200 250 0,5 1 1,5 2 2,5 3 3,5

56

Confidence Intervals: Separate Estimation

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

50 100 150 200 250 0,5 1 1,5 2 2,5 3 3,5

57

One Factor Experiments - Lessons learned

§

Calculate effects based on common average estimation leads to cleaner results

§

Saves runs!

§

If and only if presumptions are met!

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

58

Systematic Evaluation of Complex Systems

§

Motivation: Analysis of TCP Congestion Control

§

2k - Factorial Designs

§

2kr - Factorial Designs with Replications

§

2k-p – Fractional Factorial Designs

§

One Factor Experiments

§

Two Factor Experiments

§

Two Factor Experiments with Replications

§

General Full Factorial Designs

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

59

Two Factor Experiments without replications

§

Examine effects of

• two factors F
• with a and b levels
• ne replication
• Single scenario

§

Full factorial design

§

Better estimation of parameter importance (no longer limited to two levels)

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

60

Two Factor Experiments - Model

§

Model: Additive effects & additive errors

§

Effects and errors sum up to 0, thus

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

yi,j = µ + αi + βj + ei,j µ = 1 ab

a

X

i=1 b

X

j=1

yi,j αi = 1 b

b

X

j=1

yi,j − µ βj = 1 a

a

X

i=1

yi,j − µ

61

Example: a 32 Design

§

Calculation of Effects in a 3x3 Design

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

A1 A2 A3 Mean Effect B1 15.9 33.9 45.4 31.73

• 7.8

B2 25.5 38.6 50.0 38.03

• 1.5

B3 33.5 47.2 65.8 48.83 9.3 Mean 24.97 39.90 53.73 39.53 Effect

• 14.57

0.37 14.20

62

Two Factor Experiments - Variations

§

By design (again):

§

Variation of factor A

§

Variation of factor B analogous

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

SSY = SS0 + SST SSA SST = SSA SSY − SS0 = b Pa

i=1 α2 i

Pa

i=1

Pb

j=1 y2 i,j − abµ2

63

Two Factor Experiments – Confidence Intervals

§

Error variance similar to designs before:

§

Variances of average and effects

§

Confidence intervals for response variables:

§

Yet again: Distribution & independence must be verified for every response variable!

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

s2

e =

SSE (a − 1)(b − 1) sµ = se √ ab sα = se r a − 1 ab yi,j = µ + αi + βj ± t[1−α/2,(a−1)(b−1)]sµ+α+β

sµ+α+β = sµ + sα + sβ

64

Exercise: A Two Factor Experiment

§

How large are the variations in the following experiment?

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

MSS 200 400 600 800 1000 Timeout 1s 153.5 139.8 184.7 231.1 266.5 2s 333.2 339.4 369.3 377.7 414.2 3s 194.7 239.3 221.1 281.9 306.5

65

Solution: A Two Factor Experiment

Values Mean Effect Variation 153.5 139.8 184.7 231.1 266.5 195.12 775.07 27.9% 333.2 339.4 369.3 377.7 414.2 366.76 96.57 46.2% 194.7 239.3 221.1 281.9 306.5 248.7 721.49 2.3% Mean 227.13 239.50 258.37 296.90 329.07 270.19 76.5% Effect 743.06 730.69 711.83 26.71 58.87 Variation 5.5% 2.8% 0.4% 2.1% 10.3% 21.2% SSY= 1195933.51 SS0= 1095066.56 SST= 100866.949 Errors 1.44 724.63 1.41 9.27 12.51 9.50 3.33 14.37 715.77 711.43 710.94 21.29 715.77 6.49 71.07 SSE= 2405.21067 Error5Variation 2.38%

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

66

Solution: A Two Factor Experiment

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

• 30,00
• 20,00
• 10,00

0,00 10,00 20,00 30,00 100 200 300 400 500

67

Solution: A Two Factor Experiment

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

• 25
• 20
• 15
• 10
• 5

5 10 15 20 25

• 30,00
• 20,00
• 10,00

0,00 10,00 20,00 30,00

68

Solution: A Two Factor Experiment

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

0,00 100,00 200,00 300,00 400,00 500,00 600,00 200 400 600 800 1000 1200

69

Two Factor Experiments – Lessons learned

§

Allows systematic evaluation of parameters

§

May calculate confidence intervals without replication!

§

Experiment must (of course) qualify

§

More complex evaluations possible

• Analysis of incomplete datasets (e.g. if your program crashes)
• Testing relevance of parameter variations
• Evaluating multiplicative effects, etc.
• See [Jai91] for details

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

70

Systematic Evaluation of Complex Systems

§

Motivation: Analysis of TCP Congestion Control

§

2k - Factorial Designs

§

2kr - Factorial Designs with Replications

§

2k-p – Fractional Factorial Designs

§

One Factor Experiments

§

Two Factor Experiments

§

Two Factor Experiments with Replications

§

General Full Factorial Designs

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

71

Two Factor Experiments with Replications

§

Examine effects of

• two factors F
• with a and b levels
• r replications
• Single scenario

§

Full factorial design

§

Determination of influences between two parameters

§

Even better confidence in correctness

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

72

Two Factor Experiments – Model (I)

§

Model: Additive effects & additive correlation between effects & additive errors

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

a

X

i=1 b

X

j=1 r

X

k=1

yi,j,k = abrµ + br

a

X

i=1

αi + ar

b

X

j=1

βj + r

a

X

i=1 b

X

j=1

γi,j +

a

X

i=1 b

X

j=1 r

X

k=1

ei,j,k

73

Two Factor Experiments – Model (II)

§

Effects and errors sum up to 0, thus

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

µ = 1 abr

a

X

i=1 b

X

j=1 r

X

k=1

yi,j,k βj = 1 ar

a

X

i=1 r

X

k=1

yi,j,k − µ γi,j = 1 r

r

X

k=1

yi,j,k − αi − βj − µ αi = 1 br

b

X

j=1 r

X

k=1

yi,j,k − µ

74

Two Factor Experiments – Variations

§

Variations are calculated similarly:

§

Degrees of freedom

§

Variances:

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

SSY = SS0 + SST = SS0 + SSA + SSB + SSAB + SSE abr = 1 + (a − 1) + (b − 1) + (a − 1)(b − 1) + ab(r − 1) se = s SSE ab(r − 1) sµ = se √ abr sα = se r a − 1 abr

sµ+α+β+γ = se 1 + √a − 1 + √ b − 1 + p (a − 1)(b − 1) √ abr

75

Exercises: Two-Factor Experiments

§

Which formulas can be used to calculate confidence intervals?

§

Which factor has more influence in the following design:

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

MSS 200 400 600 800 1000 Timeout 1s 320 315 325 470 474 466 320 322 318 510 520 500 680 676.5 683.5 2s 512 510 514 940 930 950 416 410 422 561 557.5 564.5 1224 1229 1219 3s 896 890 884 1974 1979 1969 736 730 742 2234 2244 2254 2856 2836 2876

76

Systematic Evaluation of Complex Systems

§

Motivation: Analysis of TCP Congestion Control

§

2k - Factorial Designs

§

2kr - Factorial Designs with Replications

§

2k-p – Fractional Factorial Designs

§

One Factor Experiments

§

Two Factor Experiments

§

Two Factor Experiments with Replications

§

General Full Factorial Designs

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

77

General Full Factorial Designs

§

Examine effects of

• k factors
• with different levels
• r replications

§

Full factorial design

§

Determination of influences between k factors

§

Even better confidence in correctness

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

78

Full Factorial Designs

§

Model:

• Sum of response values is sum of µ, factors, two-factor-interactions, three-

factor-interactions, …, and errors

• E.g. for three factors
• Calculations happen analogous to two factor design, but even more

complex!

• E.g.

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems

yijkl = µ + αi + βj + ξk + γABij + γACik + γBCjk + γABCijk + eijkl Pa

i=1

Pb

j=1

Pc

k=1

Pr

l=1 yijkl

abcr = µ

Pb

j=1

Pc

k=1

Pr

l=1 yijkl

bcr − µ = αi

79

Exercises: Full-Factor Experiments

§

Which formula can be used to calculate variations?

§

What is the variance of

• se
• sα

?

§

How can confidence intervals be calculated?

Telematics 2 / Performance Evaluation (WS 17/18): 11 – Systematic Evaluation of Complex Systems