Motivation Motivation 2 Typical e-commerce sites are complex, - - PDF document

SLIDE 1

A METHOD FOR A METHOD FOR EVALUATING THE IMPACT OF SOFTWARE CONFIGURATION PARAMETERS ON E PARAMETERS ON E- COMMERCE SITES

Monchai Sopitkamol Daniel A. Menascé

Kasetsart University

• Dept. of Computer Engineering

B k k Th il d George Mason University

• Dept. of Computer Science, MS 4A5

4400 University Drive Bangkok, Thailand fengmcs@ku.ac.th 4400 University Drive Fairfax, Virginia menasce@cs.gmu.edu

WOSP 2005, Spain

Motivation Motivation

2

Typical e-commerce sites are complex, consisting of

hundreds of machines with a large number of configurable parameters that may take many different values

The number of possible combinations of the values of

these parameters can be extremely large

An e-commerce site’s performance can be

significantly impacted by the settings of these significantly impacted by the settings of these parameters

SLIDE 2

The Problem The Problem

3

There is a need to study the performance impact of

the myriad of configurable parameters and their values for multi-tiered e-commerce sites.

The knowledge of which factors and their levels are

g relevant is critical and can be used to guide performance tuning efforts as well as the design of performance tuning efforts as well as the design of self-optimizing e-commerce sites.

Contributions of the Paper Contributions of the Paper

4

D i f ti l d h i t l th d l

Design of a practical, ad-hoc experimental methodology For evaluating the performance impact of parameters, their interactions,

and for ranking the parameters

Design of Experiments Conducted experiments on a site compliant with the TPC-W benchmark

for e-commerce

Experimental Data Analysis Use of ANOVA models to test various single-parameter and parameter

interaction hypotheses interaction hypotheses

Factor Rankings Designed a ranking methodology Ranked factors for individual e-commerce functions Ranked factors for all e-commerce functions combined

SLIDE 3

Outline Outline

5

Design of the Experiments Workload and Parameters Research Methodology E

i t l R lt

Experimental Results Performance Engineering Guidelines

Design of Experiments Design of Experiments

6

R V i bl

Response Variables Web Interaction Response Time (WIRT) Web Interaction per Second (WIPS) Web Interaction per Second (WIPS) Probability of Rejection (Prej) Factors Factors Factor A (Web Interaction Types): consists of 14 levels and is

prescribed by TPC-W specification f f f f

Factor B (Configurable factors): consists of 28 factors and

stems from the three-tiered architecture

Levels Levels Finite: discrete factors Infinite: continuous and range factors

g

SLIDE 4

Workload Characterization… Factor A Workload Characterization… Factor A

7

Web Interaction Type Frequency Search Request 20.00% Search Results 17.00% Product Detail 17 00% Product Detail 17.00% Home 16.00% Shopping Cart 11.60% New Products 5.00% Best Sellers 5.00% Customer Registration 3.00% Cus o e eg s a o 3 00% Buy Request * 2.60% Buy Confirm * 1.20% O

*

% Order Inquiry * 0.75% Order Display * 0.66% Admin Request 0.10% q Admin Confirm 0.09%

* = An SSL-connection Web interaction

Configurable Factors F

t B

Configurable Factors… Factor B

Application Server Database Server Web Server (IIS 5.0) Application Server

(Tomcat 4.1)

Database Server

(SQL Server 7.0) HTTP KeepAlive acceptCount Cursor Threshold A li i P i L l i P Fill F Application Protection Level minProcessors Fill Factor Connection Timeout maxProcessors Locks Number of Connections Max Worker Threads Logging Location Min Memory Per Query Resource Indexing Network Packet Size Performance Tuning Level Priority Boost Application Optimization Recovery Interval MemCacheSize Set Working Set Size MaxCachedFileSize Max Server Memory ListenBacklog Min Server Memory MaxPoolThreads User Connections worker ajp13 cachesize worker.ajp13.cachesize 8

SLIDE 5

Experimental Methodology Experimental Methodology

9

Experimental Methodology Experimental Methodology

10

SLIDE 6

Experimental Methodology I i i li

i

Experimental Methodology… Initialization

11

Goal: To find a “good” level of each factor so that the Search Request

interaction exhibits the smallest possible response times

Factors A B C D Levels (a1, a2) (b1, b2) (c1, c2) (d1, d2) Levels (a1, a2) (b1, b2) (c1, c2) (d1, d2) Pass One 1) (A= B=? C= D=?)

*

a

*

c

Pass Two 1) (A= B= C= D= )

*

a

*

c b d

1) (A= , B=?, C= , D=?)

1

a

2

c

1) (A= , B= , C= , D= )

1

a

2

c

2

b

2

d

2) ( , B=?, , D= )

* 1

a

* 2

c

1

d

2) ( , B=?, , D= )

* 1

a

* 2

c

2

d

3) ( B D ?)

*

a

*

c b

3) ( B D ?)

*

a

*

c b

3) ( , B= , , D=?)

1

a

2

c

1

b

4) ( , B= , , D= )

* 1

a

* 2

c

1

b

2

d

3) ( , B= , , D=?)

1

a

2

c

2

b

4) ( , B= , , D= )

* 1

a

* 2

c

2

b

2

d

Initialization Example

Experimental Methodology I i i li

i

Experimental Methodology… Initialization

12

Initialization Algorithm for SUT

SLIDE 7

Experimental Methodology Experimental Methodology

13

Experimental Methodology…

Data Collection

14

Goal: To generate experimental data for all factors for subsequent

statistical analyses (hypothesis tests and factor rankings) Factors A B C D Levels (a1, a2) (b1, b2) (c1, c2) (d1, d2) Levels (a1, a2) (b1, b2) (c1, c2) (d1, d2) (A= , B= , C= , D= ) Initialization Results

* 1

a

* 2

c

1

b

2

d

Factor A ⇒ (*.*, b1, c2*, d2) ( ,

1, 2 , 2)

Factor B ⇒ (a1*, *.*, c2*, d2) Factor C ⇒ (a1*, b1, *.*, d2) Factor D ⇒ (a1*, b1, c2, *.*)

Data Collection Example

SLIDE 8

Experimental Methodology… Data Collection Experimental Methodology… Data Collection

15

Algorithm for the Data Collection Process

Experimental Methodology Experimental Methodology

16

SLIDE 9

Experimental Methodology…

Hypothesis 1

17

Hypothesis 1: “No difference due to each configurable factor B”

factor a

• f

levels

• f

number

, :

. . . 2 . . 1 .

c =

= = =

c

H μ μ μ

• Example: Factor B = HTTP Keep-Alive

H0 : μ.1. = μ.2. F-Test statistic for Factor B: F = MSB/MSE Reject H0 if F > Fu (the upper-tail critical value from an F distribution with (c-1 = 1) degrees of freedom in numerator and (n-cr = n-28) degrees of freedom in denominator) That is different levels (enabled and disabled) of HTTP Keep Alive options yield back That is, different levels (enabled and disabled) of HTTP Keep-Alive options yield different system performance

Experimental Methodology…

Hypothesis 2

18

Hypothesis 2: “No interaction effects between each configurable factor (B) and fourteen Web interaction types (A)” (B) and fourteen Web interaction types (A)

H0 : The interaction of A and B = 0

Example: Factor B = HTTP Keep-Alive H0 : The interaction of A and B = 0 F-Test statistic for the interaction: F = MSAB/MSE Reject H0 if F > Fu (the upper-tail critical value from an F distribution with (r-1)(c-1) = (13)(1) = 13 degrees of freedom in numerator and (n-cr = n-28) degrees of freedom in denominator) That is there is interaction between HTTP Keep Alive factor and the fourteen Web back That is, there is interaction between HTTP Keep-Alive factor and the fourteen Web Interaction types.

SLIDE 10

Experimental Methodology…

Hypothesis 3

19

Hypothesis 3: “No difference due to each configurable factor B for each

• f the five most important Web interaction types”
• f the five most important Web interaction types

c

H μ μ μ = = =

• 2

1 0 :

Example: Factor B = HTTP Keep-Alive H0 : μ.1. = μ.2. F-Test statistic for Factor B: F = MSB/MSE Reject H0 if F > Fu (the upper-tail critical value from an F distribution with (c-1 = 1) degrees of freedom in numerator and (n-cr = n-28) degrees of freedom in denominator) That is different levels (enabled and disabled) of HTTP Keep Alive options yield back That is, different levels (enabled and disabled) of HTTP Keep-Alive options yield different system performance for certain Web interaction type

Experimental Methodology… p gy

Applying the ANOVA models

Three null hypothesis tests Hypothesis 1: “No difference due to

each configurable factor B” each configurable factor B

Hypothesis 2: “No interaction effects

b t h fi bl f t (B) between each configurable factor (B) and 14 Web interaction types (A)”

Hypothesis 3: “No difference due to

each factor for each of the five most important Web interaction types” p yp

20

SLIDE 11

Experimental Methodology… p gy

Two-way ANOVA with replications

28 factors under

–

r : # of levels of factor A (Web interaction

) 14

Notation

28 factors under study

type) = 14

–

c : # of levels of factor B (e.g., HTTP

KeepAlive) # f l f h l l f ∑ =

′ =

r i i

n c n

1

′

–

: # of replications for each level of factor A (assume that #

∑

′

r

i

n′ , j n n

i ij

∀ ′ = ′

) 1 c j ≤ ≤

–

n : total # of observations ( )

–

Rijk : kth observation for level i of factor A

and level j of factor B

∑ =

′ =

r i i

n c n

1

14 Web interaction types

21

Experimental Methodology Factor rankings

22

• Goal: Rank factors so that the highest ranking factor
• Goal: Rank factors so that the highest ranking factor

provides the most system performance improvement, or the most performance improvement for the five most important Web interaction types.

1.

Divide factors into three groups (1-3) where factors in group 1 k d hi h th th i 2 d R k i are ranked higher than those in group 2 and so on. Ranks are in reverse orders for throughput metric (i.e., factors in group 3 are ranked higher than those in group 2 and so on). See grouping criterion based on the these thresholds.

2.

Within each group, sort factors according to this decision tree.

SLIDE 12

Experimental Methodology… Factor rankings Experimental Methodology… Factor rankings

23

Factor grouping criteria back

Experimental Methodology… Factor rankings Experimental Methodology… Factor rankings

24

Thresholds for grouping factors For each Web interaction type For the whole system

Web Interaction 90% WIRT Constraint [TPC-W] 10% WIPS Constraint

• Prob. of

Rejection C i WIRT Constraint (secs) WIPS Constraint

• Prob. Of

Rejection Constraint 20EBs 100EBs

For each Web interaction type For the whole system

Interaction [TPC W] (secs) Constraint 20EBs 100EBs Buy Request 3 2.00 10.00 0.01% (secs) Constraint 8 1.33 6.67 0.01% Home 3 2.00 10.00 0.01% Product Detail 3 2.00 10.00 0.01% Search Request 3 2.00 10.00 0.01% Search Res lts 10 1.18 5.88 0.01%

Z WIPS EBs WIRT − = #

Results

back

SLIDE 13

Experimental Methodology… Factor rankings Experimental Methodology… Factor rankings

25

Factor ranking decision tree

back

Testbed System Configuration Testbed System Configuration

26

SLIDE 14

TPC-W Dynamic Page Generation TPC W Dynamic Page Generation

27

Experimental Setup Experimental Setup

28

N f E l t d

Workload model Workload configurations

Category Distribution

User Think Time Pareto: f(x) = ax-(a+1), a = 7/6, truncated at 70 secs

• No. of Emulated

Browsers

• No. of Books

20 10,000 truncated at 70 secs User Session Minimum Duration Negative exponential: f(x) = μe-μx, 1/μ = 15, truncated at 60 mins 100 100,000 Objects per Page Varied (under TPC-W specification) Varied HTML Objects Size Varied (under TPC-W specification) In-Line Object Size 45% = 5KB, 35% = 10KB, 15% = 50KB, 4% = 100KB, 1% = 200KB

SLIDE 15

Initial Database Sizes Initial Database Sizes

29

Number of Rows Table Name Number of Rows 20EBs/10K items 100EBs/100K items Customer 57,600 288,000 Country 92 92 Address 115,200 576,000 Orders 51 840 259 200 Orders 51,840 259,200 Item 10,000 100,000 Order_Line 155,520 777,600 Author 2,500 25,000 CC_XACTS 51,840 259,200

Experimental Results

two way ANOVA results

Experimental Results… two-way ANOVA results

20 EB /10K it 100 EB /100K it

30

WIRT WIPS Prej WIRT WIPS Prej Web Server HTTP KeepAlive Reject Reject Reject Reject Reject Reject Application Protection Level Accept Reject Reject Reject Reject Reject Connection Timeout Reject Reject Reject Reject Reject Reject 20 EBs/10K items Factor Name Accept Hypothesis 1? Accept Hypothesis 2? WIRT WIPS Prej WIRT WIPS Prej Web Server HTTP KeepAlive Reject Reject Reject Reject Reject Reject Application Protection Level Reject Reject Reject Reject Reject Reject Connection Timeout Reject Reject Reject Reject Reject Reject 100 EBs/100K items Factor Name Accept Hypothesis 1? Accept Hypothesis 2? Connection Timeout Reject Reject Reject Reject Reject Reject Number of Connections Reject Reject Reject Reject Reject Reject Logging Location Reject Reject Reject Reject Reject Reject Resource Indexing Reject Accept Reject Reject Reject Reject Performance Tuning Level Reject Reject Reject Reject Reject Reject Application Optimization Reject Reject Reject Reject Reject Reject MemCacheSize Reject Reject Reject Reject Reject Reject Connection Timeout Reject Reject Reject Reject Reject Reject Number of Connections Reject Reject Reject Accept Reject Reject Logging Location Accept Reject Reject Reject Reject Reject Resource Indexing Reject Reject Reject Reject Reject Reject Performance Tuning Level Reject Reject Reject Reject Reject Reject Application Optimization Reject Reject Reject Reject Reject Reject MemCacheSize Reject Reject Reject Reject Reject Reject MaxCachedFileSize Reject Reject Reject Reject Reject Reject ListenBacklog Reject Reject Reject Reject Reject Reject MaxPoolThreads Reject Reject Reject Reject Reject Reject worker.ajp13.cachesize Reject Reject Reject Reject Reject Reject Application Server acceptCount Reject Reject Reject Reject Reject Reject maxProcessors Reject Reject Reject Reject Reject Reject MaxCachedFileSize Reject Reject Reject Reject Reject Reject ListenBacklog Reject Reject Reject Reject Reject Reject MaxPoolThreads Accept Reject Reject Reject Reject Reject worker.ajp13.cachesize Reject Reject Reject Reject Reject Reject Application Server acceptCount Reject Reject Reject Reject Reject Reject maxProcessors Reject Reject Reject Reject Reject Reject maxProcessors Reject Reject Reject Reject Reject Reject minProcessors Reject Reject Reject Reject Reject Reject Database Server Cursor Threshold Reject Reject Reject Reject Reject Reject Fill Factor Reject Reject Reject Reject Reject Reject Locks Reject Reject Reject Reject Reject Reject Max Worker Threads Reject Reject Reject Reject Reject Reject maxProcessors Reject Reject Reject Reject Reject Reject minProcessors Reject Reject Reject Reject Reject Reject Database Server Cursor Threshold Reject Reject Reject Reject Reject Reject Fill Factor Reject Reject Reject Reject Reject Reject Locks Reject Reject Reject Reject Reject Reject Max Worker Threads Accept Reject Reject Reject Reject Reject a

• e

eads eject eject eject eject eject eject Min Memory Per Query Reject Reject Reject Reject Reject Reject Network Packet Size Reject Reject Reject Reject Reject Reject Priority Boost Reject Reject Reject Reject Reject Reject Recovery Interval Reject Reject Reject Reject Reject Reject Set Working Set Size Max Server Memory Reject Reject Reject Reject Reject Reject p j j j j j Min Memory Per Query Reject Reject Reject Reject Reject Reject Network Packet Size Reject Reject Reject Reject Reject Reject Priority Boost Reject Reject Reject Reject Reject Reject Recovery Interval Reject Reject Reject Reject Reject Reject Set Working Set Size Max Server Memory Reject Reject Reject Reject Reject Reject Min Server Memory User Connections Accept Reject Reject Accept Reject Reject Min Server Memory User Connections Reject Reject Reject Reject Reject Reject

SLIDE 16

Experimental Results

• ne way ANOVA results

Experimental Results… one-way ANOVA results

31

20 EBs/10K items Search Results Buy Request Home Product Detail Search Request 20 EBs/10K items WIRT WIPS Prej WIRT WIPS Prej WIRT WIPS Prej WIRT WIPS Prej WIRT WIPS Prej Web Server HTTP KeepAlive Accept Reject Accept Accept Reject Accept Accept Reject Accept Reject Reject Accept Reject Reject Reject Application Protection Level Accept Reject Accept Accept Reject Accept Accept Reject Reject Accept Reject Reject Accept Reject Reject Connection Timeout Accept Reject Accept Reject Reject Reject Accept Reject Reject Reject Reject Reject Accept Reject Reject Search Results Accept Hypothesis 3? Accept Hypothesis 3? Buy Request Home Product Detail Search Request Factor Name Accept Hypothesis 3? Accept Hypothesis 3? Accept Hypothesis 3? Connection Timeout Accept Reject Accept Reject Reject Reject Accept Reject Reject Reject Reject Reject Accept Reject Reject Number of Connections Accept Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Logging Location Reject Accept Accept Accept Reject Reject Accept Reject Reject Accept Reject Accept Accept Reject Reject Resource Indexing Accept Reject Accept Reject Reject Accept Reject Reject Accept Reject Reject Accept Reject Reject Reject Performance Tuning Level Accept Reject Accept Accept Reject Accept Accept Reject Reject Accept Reject Reject Accept Reject Reject Application Optimization Accept Reject Accept Reject Reject Accept Reject Reject Reject Reject Reject Accept Reject Reject Reject MemCacheSize Reject Reject Accept Reject Reject Reject Accept Reject Reject Accept Reject Reject Accept Accept Reject MemCacheSize Reject Reject Accept Reject Reject Reject Accept Reject Reject Accept Reject Reject Accept Accept Reject MaxCachedFileSize Accept Reject Accept Accept Reject Reject Accept Reject Reject Accept Accept Reject Accept Reject Reject ListenBacklog Accept Reject Accept Accept Reject Reject Accept Reject Reject Accept Reject Reject Accept Reject Reject MaxPoolThreads Reject Reject Accept Reject Reject Reject Accept Reject Reject Reject Reject Reject Accept Reject Reject worker.ajp13.cachesize Accept Reject Accept Reject Reject Reject Accept Reject Accept Accept Reject Accept Accept Reject Reject Application Server C acceptCount Accept Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Accept Reject Reject Reject maxProcessors Accept Reject Accept Reject Reject Accept Accept Reject Reject Accept Reject Reject Accept Reject Reject minProcessors Accept Reject Accept Accept Reject Reject Accept Reject Reject Accept Reject Accept Accept Reject Reject Database Server Cursor Threshold Accept Reject Accept Reject Reject Reject Accept Accept Reject Accept Reject Reject Accept Reject Reject Fill Factor Accept Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Locks Accept Reject Reject Accept Reject Accept Accept Reject Reject Reject Accept Reject Reject Accept Reject Max Worker Threads Accept Reject Reject Accept Reject Accept Accept Reject Reject Accept Reject Reject Accept Reject Reject Min Memory Per Query Accept Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Network Packet Size Reject Reject Accept Reject Reject Reject Reject Reject Accept Accept Reject Reject Reject Reject Reject Priority Boost Reject Reject Accept Accept Reject Reject Accept Reject Accept Reject Reject Reject Accept Reject Reject Recovery Interval Accept Reject Reject Accept Reject Reject Reject Reject Accept Reject Reject Reject Accept Reject Reject y p j j p j j j j p j j j p j j Set Working Set Size Max Server Memory Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Min Server Memory User Connections Dropped Reject Reject Dropped Reject Reject Dropped Reject Accept Dropped Reject Accept Dropped Reject Reject

Experimental Results… one-way ANOVA results p

y

32

100 EBs/100K items Search Results Buy Request Home Product Detail Search Request WIRT WIPS Prej WIRT WIPS Prej WIRT WIPS Prej WIRT WIPS Prej WIRT WIPS Prej Web Server HTTP KeepAlive Reject Reject Accept Accept Accept Reject Accept Reject Reject Accept Reject Reject Accept Reject Accept Application Protection Level Accept Reject Accept Reject Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Connection Timeout Reject Reject Accept Accept Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Factor Name Accept Hypothesis 3? Accept Hypothesis 3? Accept Hypothesis 3? Accept Hypothesis 3? Accept Hypothesis 3? y Connection Timeout Reject Reject Accept Accept Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Number of Connections Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Logging Location Accept Reject Accept Accept Reject Reject Accept Reject Reject Accept Reject Reject Accept Accept Reject Resource Indexing Accept Reject Accept Reject Accept Reject Reject Reject Reject Reject Reject Reject Accept Reject Accept Performance Tuning Level Accept Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Accept Reject Accept Application Optimization Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject MemCacheSize Reject Reject Accept Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject MaxCachedFileSize Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject ListenBacklog Accept Reject Accept Reject Reject Reject Reject Reject Reject Accept Reject Reject Reject Reject Reject MaxPoolThreads Reject Reject Accept Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject worker.ajp13.cachesize Reject Reject Accept Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Application Server Application Server acceptCount Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject maxProcessors Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject minProcessors Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Database Server Cursor Threshold Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Fill Factor Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Locks Accept Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Max Worker Threads Reject Reject Accept Accept Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Min Memory Per Query Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Network Packet Size Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Priority Boost Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Priority Boost Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Recovery Interval Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Set Working Set Size Max Server Memory Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Min Server Memory User Connections Reject Reject Accept Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject Reject

SLIDE 17

Factor Ranking Results … An example Factor Ranking Results … An example

33 Search R eques t Search R es ults R ank R ank

For 100EBs/100K

q Factor W IR T W IPS Prej Factor W IR T W IPS Prej C ursor T hreshold 1 11 10 Set W

• rking Set Size

1 1 1 M em C acheSize 2 14 11 Application Protection Lev el 2 21 2 Locks 3 7 7 M axPoolT hreads 3 11 3 Application Protection Lev el 4 15 12 M em C acheSize 4 14 4 N etw

• rk Packet Size

5 6 3 C ursor T hreshold 5 15 5 m inProcessors 6 22 13 Locks 6 20 6 M in M em

• ry Per Q

uery 7 13 14 N um ber of C

• nnections

7 13 7 ListenBacklog 8 20 15 M ax W

• rker T

hreads 8 9 8 w

• rker.ajp13.cachesize

9 18 16 M axC achedFileSize 9 6 9 Application O ptim ization 10 10 5 C

• nnection T

im eout 10 19 10 Priority Boost 10 10 4 m axProcessors 11 12 11 M axC achedFileSize 10 1 17 m inProcessors 12 16 12 H ttp KeepAliv e 10 10 17 R ecov ery Interv al 13 7 13 Logging Location 10 9 9 N etw

• rk Packet Size

14 3 14 Perform ance T uning Lev el 10 10 17 w

• rker.ajp13.cachesize

15 17 15 Set W

• rking Set Size

10 5 17 acceptC

• unt

16 4 16 M axPoolT hreads 10 16 1 ListenBacklog 17 10 17 N um ber of C

• nnections

10 17 8 M in M em

• ry Per Q

uery 18 2 18 m axProcessors 10 12 17 U ser C

• nnections

19 5 19 M ax W

• rker T

hreads 10 21 2 Application O ptim ization 20 9 20 R esource Indexing 10 10 6 H ttp KeepAliv e 20 9 20 R ecov ery Interv al 11 2 18 Logging Location 20 8 20 C

• nnection T

im eout 12 8 19 Perform ance T uning Lev el 20 9 20 U ser C

• nnections

13 4 20 Priority Boost 20 9 20 tC t 14 3 21 R I d i 20 9 20 acceptC

• unt

14 3 21 R esource Indexing 20 9 20 Fill Factor 15 19 22 Fill Factor 21 18 21

Factor Ranking Results … An example g

p

34 Factor WIRT LCI UCI CV Min Max Range Group Rank

Factor rankings of Search Request Web interaction for 20 EBs/10K configuration

Factor WIRT LCI UCI CV Min Max Range Group Rank Network Packet Size 0.130 0.112 0.149 2.656 0.025 5.063 5.038 2 1 Application Optimization 0.154 0.134 0.174 2.448 0.022 5.053 5.031 2 2 ListenBacklog 0.112 0.098 0.126 2.267 0.026 5.053 5.027 2 3 worker.ajp13.cachesize 0.115 0.101 0.129 2.262 0.026 5.277 5.251 2 4 Cursor Threshold 0.115 0.101 0.128 2.163 0.025 5.056 5.031 2 5 MemCacheSize 0.110 0.099 0.122 1.955 0.025 5.043 5.018 2 6 Priority Boost 0.116 0.105 0.128 1.912 0.025 5.044 5.019 2 7 MaxCachedFileSize 0.117 0.105 0.128 1.854 0.026 5.049 5.023 2 8 Application Protection Level 0 134 0 121 0 146 1 700 0 021 5 035 5 014 2 9 Application Protection Level 0.134 0.121 0.146 1.700 0.021 5.035 5.014 2 9 Http KeepAlive 0.134 0.121 0.146 1.700 0.021 5.035 5.014 2 9 Logging Location 0.134 0.121 0.146 1.700 0.021 5.035 5.014 2 9 Min Memory Per Query 0.134 0.121 0.146 1.700 0.021 5.035 5.014 2 9 Performance Tuning Level 0.134 0.121 0.146 1.700 0.021 5.035 5.014 2 9 Set Working Set Size 0.127 0.115 0.138 1.667 0.025 5.043 5.018 2 10 minProcessors 0.117 0.106 0.127 1.625 0.027 2.210 2.183 3 11 Recovery Interval 0.109 0.100 0.119 1.570 0.023 1.452 1.429 3 12 Locks 0.104 0.095 0.113 1.545 0.025 2.248 2.223 3 13 MaxPoolThreads 0 108 0 099 0 117 1 536 0 026 1 457 1 431 3 14 MaxPoolThreads 0.108 0.099 0.117 1.536 0.026 1.457 1.431 3 14 acceptCount 0.105 0.096 0.113 1.520 0.026 1.270 1.244 3 15 Number of Connections 0.103 0.094 0.111 1.510 0.025 1.528 1.503 3 16 maxProcessors 0.117 0.108 0.127 1.500 0.026 1.412 1.386 3 17 Max Worker Threads 0.114 0.105 0.123 1.473 0.027 1.352 1.325 3 18 Connection Timeout 0.120 0.111 0.129 1.441 0.024 1.489 1.465 3 19 Resource Indexing 0.198 0.184 0.212 1.315 0.022 2.046 2.024 3 20 Fill Factor 0.085 0.079 0.090 1.195 0.027 1.021 0.994 3 21 User Connections 0.058 0.048 0.067 0.674 0.023 0.292 0.269 3 22

SLIDE 18

Factor Ranking Results … Another example g

36

Factor WIPS LCI UCI CV MIN MAX Range Group Rank

WIPS Ranking Results for an SUT with the 20 EBs/10K configuration

g p Network Packet Size 0.535 0.528 0.541 0.501 0.005 12.739 12.734 2 1 Locks 0.533 0.527 0.539 0.460 0.005 9.091 9.086 2 2 Priority Boost 0.548 0.542 0.554 0.460 0.005 10.638 10.633 2 3 maxProcessors 0.532 0.527 0.538 0.444 0.005 8.000 7.995 2 4 C ti Ti t 0 537 0 531 0 542 0 422 0 004 6 410 6 407 2 5 Connection Timeout 0.537 0.531 0.542 0.422 0.004 6.410 6.407 2 5 MaxPoolThreads 0.550 0.544 0.555 0.422 0.003 5.650 5.647 2 6 Max Worker Threads 0.532 0.527 0.538 0.420 0.005 6.098 6.093 2 7 Min Memory Per Query 0.531 0.526 0.536 0.413 0.004 4.425 4.420 2 8 Resource Indexing 0.500 0.495 0.505 0.410 0.005 2.203 2.198 2 9 Resource Indexing 0.500 0.495 0.505 0.410 0.005 2.203 2.198 2 9 Application Optimization 0.548 0.543 0.554 0.407 0.004 4.926 4.922 2 10 Number of Connections 0.532 0.527 0.537 0.407 0.005 4.566 4.561 2 11 Cursor Threshold 0.528 0.523 0.533 0.402 0.004 4.237 4.233 2 12 Http KeepAlive 0.528 0.523 0.533 0.402 0.004 4.237 4.233 2 12 Recovery Interval 0.540 0.535 0.545 0.401 0.005 4.630 4.625 2 13 ListenBacklog 0.538 0.533 0.543 0.399 0.005 3.774 3.768 2 14 MaxCachedFileSize 0.547 0.542 0.552 0.397 0.004 3.198 3.194 2 15 Set Working Set Size 0.543 0.538 0.548 0.396 0.005 3.049 3.044 2 16 acceptCount 0 536 0 531 0 541 0 396 0 005 2 033 2 027 2 17 acceptCount 0.536 0.531 0.541 0.396 0.005 2.033 2.027 2 17 worker.ajp13.cachesize 0.554 0.549 0.559 0.395 0.005 1.563 1.557 2 18 Fill Factor 0.540 0.535 0.545 0.394 0.004 2.016 2.012 2 19 Application Protection Level 0.540 0.535 0.546 0.393 0.005 2.558 2.553 2 20 Logging Location 0.546 0.541 0.552 0.393 0.004 2.463 2.459 2 21 MemCacheSize 0.533 0.528 0.538 0.393 0.005 1.779 1.775 2 22 Performance Tuning Level 0.542 0.537 0.547 0.392 0.004 1.600 1.596 2 23 minProcessors 0.549 0.544 0.554 0.389 0.004 1.422 1.419 2 24 User Connections 0.570 0.565 0.575 0.358 0.025 5.814 5.789 2 25

Experimental Results

Factor rankings

Experimental Results… Factor rankings

38 WIRT 20 EBs/10K items 100 EBs/100K items Buy Request Fill Factor Cursor Threshold

Best ranking factors by Web interaction type for each SUT configuration

Buy Request Fill Factor Cursor Threshold Home Application Optimization Number of Connections Product Detail acceptCount Number of Connections Search Request Network Packet Size Cursor Threshold Database server Application server Search Results worker.ajp13.cachesize Set Working Set Size WIPS 20 EBs/10K items 100 EBs/100K items Buy Request Number of Connections Number of Connections Home Fill Factor HTTP KeepAlive Web server p Product Detail Application Optimization Set Working Set Size Search Request MemCacheSize MaxCachedFileSize Search Results Resource Indexing Set Working Set Size Prej 20 EBs/10K items 100 EBs/100K items Prej 20 EBs/10K items 100 EBs/100K items Buy Request Fill Factor Cursor Threshold Home Application Optimization Connection Timeout Product Detail acceptCount worker.ajp13.cachesize Search Request Network Packet Size MaxPoolThreads Search Results Set Working Set Size Set Working Set Size

SLIDE 19

Experimental Results… Factor rankings Experimental Results… Factor rankings

39

Overall top ranking factors by performance metric for each SUT configuration considering all 14 Web interactions combined

Performance Workload Levels Metric 20 EBs/10K 100 BEs/100K

WIRT Application Optimization Set Working Set Size WIPS Network Packet Size Set Working Set Size Prej Priority Boost Set Working Set Size

Engineering Guidelines for Performance Tuning of E- C Si Commerce Sites

I f R Ti M i

41 Important factors - Response Time Metric Light Workload Intensity >50% performance gain: Set Working Set Size (97%), Min Memory Per Query

(81%) U C (98%) d N b f C (54%) (81%), User Connections (98%), and Number of Connections (54%).

20-50% performance gain: Fill Factor (38%), acceptCount (25%), and Network Packet

Size (28%).

Heavy Workload Intensity Heavy Workload Intensity Number of Connections (33%), Min Memory Per Query (44%), and Set Working Set

Size (91%)

Important factors Throughput Metric Important factors - Throughput Metric Light Workload Intensity Fill Factor (11%), Min Memory Per Query (22%), Set Working Set Size (140%), and

User Connections (2192%) User Connections (2192%)

Heavy Workload Intensity Number of Connections (678%), MemCacheSize (12%), Locks (13%), Min Memory Per

Query (66%) Network Packet Size (24%) and Set Working Set Size (2125%) Query (66%), Network Packet Size (24%), and Set Working Set Size (2125%),

SLIDE 20

Engineering Guidelines for Performance Tuning of E- C Si Commerce Sites

42

Unimportant factors (<10% WIRT gain) Light Workload Intensity:

HTTP KeepAlive, Application Protection Level, Connection Timeout,

Logging Location, Performance Tuning Level, MaxCachedFileSize, ListenBacklog, worker.ajp13.cachesize, minProcessors, Max Worker ListenBacklog, worker.ajp13.cachesize, minProcessors, Max Worker Threads, Priority Boost, and Recovery Interval.

Heavy Workload Intensity:

HTTP KeepAlive, Application Protection Level, Connection Timeout,

Logging Location, Resource Indexing, Performance Tuning Level, Application Optimization, MemCacheSize, MaxCachedFileSize, Application Optimization, MemCacheSize, MaxCachedFileSize, ListenBacklog, MaxPoolThreads, worker.ajp13.cachesize, acceptCount, maxProcessors, minProcessors, CursorThreshold, FillFactor, Max Worker Threads Priority Boost Recovery Interval and User Worker Threads, Priority Boost, Recovery Interval, and User Connections.

Questions? Questions?

43

SLIDE 21

Call for Papers

Special Issue on

Application-Level QoS for Distributed Applications Application-Level QoS for Distributed Applications May/June 2006

Guest editors: Daniel A. Menascé (George Mason University) Daniel A. Menascé (George Mason University) and

• C. Murray Woodside (Carleton University)

Submissions due 1 October 2005.

T b it i t l t M i t C t l t

44

To submit a manuscript, log on to Manuscript Central at http://cs-ieee.manuscriptcentral.com