Computer Science, Informatik 4 Communication and Distributed Systems Simulation “Discrete-Event System Simulation” Dr. Mesut Güneş
Computer Science, Informatik 4 Communication and Distributed Systems Chapter 11 Comparison and Evaluation of Alternative System Designs
Computer Science, Informatik 4 Communication and Distributed Systems Purpose Purpose: comparison of alternative system designs. � Approach: discuss a few of many statistical methods that can be � used to compare two or more system designs. Statistical analysis is needed to discover whether observed � differences are due to: • Differences in design or • The random fluctuation inherent in the models Dr. Mesut Güneş Chapter 11. Comparison and Evaluation of Alternative Designs 3
Computer Science, Informatik 4 Communication and Distributed Systems Outline For two-system comparisons: � • Independent sampling. • Correlated sampling (common random numbers). For multiple system comparisons: � • Bonferroni approach: confidence-interval estimation, screening, and selecting the best. Metamodels � Dr. Mesut Güneş Chapter 11. Comparison and Evaluation of Alternative Designs 4
Computer Science, Informatik 4 Communication and Distributed Systems Comparison of Two System Designs Goal: compare two possible configurations of a system � • Two possible ordering policies in a supply-chain system, two possible scheduling rules in a job shop. Approach: the method of replications is used to analyze the output � data. The mean performance measure for system i � • Denoted by θ i , i = 1,2. To obtain point and interval estimates for the difference in mean � performance, namely θ 1 – θ 2 . Dr. Mesut Güneş Chapter 11. Comparison and Evaluation of Alternative Designs 5
Computer Science, Informatik 4 Communication and Distributed Systems Comparison of Two System Designs Vehicle-safety inspection example: � • The station performs 3 jobs: (1) brake check, (2) headlight check, and (3) steering check. • Vehicles arrival: Possion with rate = 9.5/hour. • Present system: - Three stalls in parallel (one attendant makes all 3 inspections at each stall). - Service times for the 3 jobs: normally distributed with means 6.5, 6.0 and 5.5 minutes, respectively. • Alternative system: - Each attendant specializes in a single task, each vehicle will pass through three work stations in series - Mean service times for each job decreases by 10 % (5.85, 5.4, and 4.95 minutes). • Performance measure: mean response time per vehicle (total time from vehicle arrival to its departure). Dr. Mesut Güneş Chapter 11. Comparison and Evaluation of Alternative Designs 6
Computer Science, Informatik 4 Communication and Distributed Systems Comparison of Two System Designs • From replication r of system i , the simulation analyst obtains an estimate Y ir of the mean performance measure θ i . • Assuming that the estimators Y ir are (at least approximately) unbiased: θ 1 = E ( Y 1r ) , r = 1 , … , R 1 ; θ 2 = E ( Y 2 r ) , r = 1 , … , R 2 • Goal: compute a confidence interval for θ 1 – θ 2 to compare the two system designs • Confidence interval for θ 1 – θ 2 : - If CI is totally to the left of 0, strong evidence for the hypothesis that θ 1 – θ 2 < 0 ( θ 1 < θ 2 ). - If CI is totally to the right of 0, strong evidence for the hypothesis that θ 1 – θ 2 > 0 ( θ 1 > θ 2 ). - If CI contains 0, no strong statistical evidence that one system is better than the other – If enough additional data were collected (i.e., R i increased), the CI would most likely shift, and definitely shrink in length, until conclusion of θ 1 < θ 2 or θ 1 > θ 2 would be drawn. Dr. Mesut Güneş Chapter 11. Comparison and Evaluation of Alternative Designs 7
Computer Science, Informatik 4 Communication and Distributed Systems Comparison of Two System Designs In this chapter: � • A two-sided 100(1- α )% CI for θ 1 – θ 2 always takes the form of: − ± − Y Y t s e Y Y . .( ) α υ . 1 . 2 / 2 , . 1 . 2 Sample Degree Standard error mean for of of the estimator system i freedom • All three techniques assume that the basic data Y ir are approximately normally distributed. Dr. Mesut Güneş Chapter 11. Comparison and Evaluation of Alternative Designs 8
Computer Science, Informatik 4 Communication and Distributed Systems Comparison of Two System Designs Statistically significant versus practically significant � Y − • Statistical significance: is the observed difference Y larger than . 1 . 2 the variability in ? Y − Y . 1 . 2 • Practical significance: is the true difference θ 1 – θ 2 large enough to matter for the decision we need to make? • Confidence intervals do not answer the question of practical significance directly, instead, they bound the true difference within the range: − − − ≤ θ − θ ≤ − + − Y Y t s e Y Y Y Y t s e Y Y . .( ) . .( ) ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ α α υ υ 1 2 1 2 1 2 1 2 1 2 , , 2 2 • Whether a difference within these bounds is practically significant depends on the particular problem. Dr. Mesut Güneş Chapter 11. Comparison and Evaluation of Alternative Designs 9
Computer Science, Informatik 4 Communication and Distributed Systems Independent Sampling with Equal Variances � Different and independent random number streams are used to simulate the two systems • All observations of simulated system 1 are statistically independent of all the observations of simulated system 2. � The variance of the sample mean, , is: Y i . ( ) ( ) σ 2 V Y i i = = = V Y . i , , 1 2 i . R R i i � For independent samples: ( ) ( ) ( ) σ σ 2 2 − = + = + V Y Y V Y V Y 1 2 . 1 . 2 . 1 . 2 R R 1 2 Dr. Mesut Güneş Chapter 11. Comparison and Evaluation of Alternative Designs 10
Computer Science, Informatik 4 Communication and Distributed Systems Independent Sampling with Equal Variances If it is reasonable to assume that σ 2 2 (approximately) or if R 1 = 1 = σ 2 � R 2 , a two-sample- t confidence-interval approach can be used: Y − • The point estimate of the mean performance difference is: Y . 1 . 2 • The sample variance for system i is: R R ( ) 1 1 ∑ i ∑ i 2 = − = − 2 2 S Y Y Y R Y 2 i ri i ri i i − − . . R R 1 1 = = r r i i 1 1 The pooled estimate of σ 2 is: • − + − 2 2 R S R S ( 1 ) ( 1 ) = υ = + 2 S p 1 1 2 2 R R - , whe re 2 degrees of freedom + − 1 2 R R 2 1 2 − ± − • CI is given by: Y Y t s e Y Y . .( ) α υ . 1 . 2 / 2 , . 1 . 2 ( ) 1 1 • Standard error: − = + s e Y Y S . . p . 1 . 2 R R 1 2 Dr. Mesut Güneş Chapter 11. Comparison and Evaluation of Alternative Designs 11
Computer Science, Informatik 4 Communication and Distributed Systems Independent Sampling with Unequal Variances � If the assumption of equal variances cannot safely be made, an approximate 100(1- α ) % CI can be computed as: ( ) 2 2 S S − = + s e Y Y 1 2 . . . 1 . 2 R R 1 2 • With degrees of freedom: ( ) 2 + 2 2 S R S R / / υ = 1 1 2 2 , round to an interger ( ) ( ( ) ( ⎡ ⎤ ⎡ ⎤ ) ) 2 2 − + − 2 2 S R R S R R / / 1 / / 1 ⎢ ⎥ ⎢ ⎥ ⎣ 1 1 1 ⎦ ⎣ 2 2 2 ⎦ • In this case, the minimum number of replications R 1 > 7 and R 2 > 7 is recommended. Dr. Mesut Güneş Chapter 11. Comparison and Evaluation of Alternative Designs 12
Computer Science, Informatik 4 Communication and Distributed Systems Common Random Numbers (CRN) For each replication, the same random numbers are used to simulate � both systems � R 1 = R 2 = R . • For each replication r , the two estimates, Y r1 and Y r2 , are correlated. • However, independent streams of random numbers are used on different replications, so the pairs ( Y r1 , Y s2 ) are mutually independent for r ≠ s . Purpose: induce positive correlation between (for each r ) to � Y . , Y 1 . 2 reduce variance in the point estimator of . Y − Y . 1 . 2 Correlation: ( ) ( ) ( ) ( ) ρ 12 is positive − = + − V Y Y V Y V Y Y Y 2 cov , . 1 . 2 . 1 . 2 . 1 . 2 σ σ ρ σ σ 2 2 2 = + − 1 2 12 1 2 R R R Dr. Mesut Güneş Chapter 11. Comparison and Evaluation of Alternative Designs 13
Computer Science, Informatik 4 Communication and Distributed Systems Common Random Numbers (CRN) • Compare variance from independent sampling with variance from CRN: ρ σ σ 2 = − V V 12 1 2 CRN IND R • Variance of arising from CRN is less than that of Y − Y . 1 . 2 independent sampling (with R 1 =R 2 ). Dr. Mesut Güneş Chapter 11. Comparison and Evaluation of Alternative Designs 14
Recommend
More recommend
