by : Raoufeh Hashemian R. Hashemian 1 , N. Carlsson 2 , D. - - PowerPoint PPT Presentation

▶
by raoufeh hashemian
SMART_READER_LITE
LIVE PREVIEW

by : Raoufeh Hashemian R. Hashemian 1 , N. Carlsson 2 , D. - - PowerPoint PPT Presentation

Jun 30, 2023 236 likes •472 views

by : Raoufeh Hashemian R. Hashemian 1 , N. Carlsson 2 , D. Krishnamurthy 1 , M. Arlitt 1 1. University of Calgary 2. Linkping University The 8th ACM/SPEC International Conference on Performance Engineering ICPE 2017 OUTLINE Motivation

slide-1
SLIDE 1
  • R. Hashemian1, N. Carlsson2, D. Krishnamurthy1, M. Arlitt1
  • 1. University of Calgary
  • 2. Linköping University

by: Raoufeh Hashemian

The 8th ACM/SPEC International Conference on

Performance Engineering

ICPE 2017

slide-2
SLIDE 2
  • Motivation
  • Related work
  • IRIS method
  • Evaluation
  • Tuning guideline
  • Conclusions

OUTLINE

IRIS: IteRative and Intelligent experiment Selection

ICPE17 2

slide-3
SLIDE 3
  • Benchmarking is not always cheap: time, resource limits

MOTIVATION

x

Measurements Piecewise fit Actual profile

1 x x x x x x x1 x2 x3 x4 x5 x6 d

Not all measurement points have the same value The position of points affect the accuracy of the fit Selecting points closer to step more accurate fit with less budget

IRIS: IteRative and Intelligent experiment Selection

ICPE17 3

Simple Scenario: Step function

Response variables Independent variable

slide-4
SLIDE 4

a more realistic case

MOTIVATION

Experiment results from a real server Removing points X2 to X5 has little effect on prediction accuracy

Independent variable

x x x x x x x x1 x2 x3 x4 x5 x6 x7

Response variable

IRIS: IteRative and Intelligent experiment Selection

ICPE17 4

slide-5
SLIDE 5
  • Response Surface Methodology

Select most effective parameters Find optimum point of the system function e.g. Box–Behnken, fractional factorial

  • Regression based, iterative function prediction

techniques

Build model in each iteration

  • 1. More costly due to model validation techniques
  • 2. Model error can propagate into future iterations

RELATED WORK

IRIS: IteRative and Intelligent experiment Selection

ICPE17 5

slide-6
SLIDE 6
  • The problem scope

Given the previously identified independent variables of interest, how to select the placement of experiment points?

  • Criteria

Should consider both independent and response variables when deciding about the next experiment point Scalability for scenarios with many independent variables

RELATED WORK

IRIS: IteRative and Intelligent experiment Selection

ICPE17 6

slide-7
SLIDE 7

IRIS

OVERVIEW

Two steps algorithm:

1) Initial Point Selection

Select a set of initial points to run the experiment based on:

  • An educated guess (e.g. a queueing model, …)
  • Or a linear assumption

2) Iterative Point Selection

Assumption: The experiment budget is limited

  • IRIS iteratively selects the next point to run the experiment, until

it runs out of budget

  • Each point is selected based on the results of all previous

experiments

IRIS: IteRative and Intelligent experiment Selection

ICPE17 7

slide-8
SLIDE 8

IRIS

INITIAL POINT SELECTION

A multi-core web server (load vs. response time)

IRIS: IteRative and Intelligent experiment Selection

ICPE17 9

An educated guess: a layered queueing model (LQM) for the system with estimated resource demands

slide-9
SLIDE 9

IRIS

ITERATIVE POINT SELECTION a list of already measured (xi ; yi) points where 1 ≤ i ≤ Ni Nt : total experiment budget α : gain trade-off factor

IRIS: IteRative and Intelligent experiment Selection

ICPE17 10

Inputs

List of all experimented points ( xj ; yj ) where 1 ≤ j ≤ Nt

  • utput
slide-10
SLIDE 10

IRIS

GAIN FORMULA

  • Gain for each interval

𝐻

𝑘 = 𝐵𝑘 𝛽 ∗ 𝑆𝑘 1−∝

  • 𝐵𝑘 = 𝑇𝑗𝑨𝑓 𝑝𝑔 𝑗𝑜𝑢𝑓𝑠𝑤𝑏𝑚
  • 𝑆𝑘 = |𝑆 𝑦𝑘+1 − 𝑆(𝑦𝑘)|
  • Trade-off factor: α

IRIS: IteRative and Intelligent experiment Selection

ICPE17 8

Independent variable

x x x x1 x2 x3

Response variable

Aj Rj

R X

slide-11
SLIDE 11

IRIS – ITERATIVE PHASE

  • 1. n=Ni , P = {pi |1<i<Ni }
  • 2. For each of thr n-1 intervals [xj : xj+1] where 1 ≤ j < n,

calculate Gj

  • 3. Find the interval [xk:xk+1] , where Gk= max{Gj}
  • 4. pn =

(xk+ xk+1) 2

, P = P ∩ { Pn }, n=n+1

  • 5. If (n ≤ Nt) then goto 2, else END

algorithm

IRIS: IteRative and Intelligent experiment Selection

ICPE17 11

slide-12
SLIDE 12

Delaunay triangulation to calculate Aj

  • A unique planar triangulation of the independent variable space
  • The resulting triangles consist of points with high proximity
  • Easy to calculate
  • Generalizes to multiple dimensions

IRIS

MULTI-DIMENSIONAL SCENARIO

Aj = Area of the triangles

IRIS: IteRative and Intelligent experiment Selection

ICPE17 12 Rj = Maximum difference in response variables of the 3 nodes in each triangle

slide-13
SLIDE 13

Equal Distance Point Selection (EQD)

  • Possible range of each independent variable is divided into N -1

equally sized intervals.

EVALUATION

BASE-LINE: EQUAL DISTANCE POINT SELECTION

Multi-stage EQD: available point budget is spent in multiple stages of EQD Single-stage EQD: all the budget is spent in a single round (penalty free) N= 9 N= 23 N= 25 N= 16 N= 9

IRIS: IteRative and Intelligent experiment Selection

ICPE17 13

slide-14
SLIDE 14

Average Absolute Error 𝐵𝐵𝐹 = | 𝑆𝑄𝑆𝐸 𝑌

𝑘 − 𝑆 𝑌 𝑘 | 𝑜 𝑘=1

𝑆(𝑌

𝑘) 𝑜 𝑘=1

Error Reduction Ratio 𝐹𝑆 = (𝐵𝐵𝐹𝑐𝑏𝑡𝑓𝑚𝑗𝑜𝑓 − 𝐵𝐵𝐹𝐽𝑆𝐽𝑇) 𝐵𝐵𝐹𝐽𝑆𝐽𝑇

EVALUATION

COMPARISON METRICS

IRIS: IteRative and Intelligent experiment Selection

ICPE17 14

slide-15
SLIDE 15

EVALUATION

SINGLE INDEPENDENT VARIABLE

System functions

  • An experimental system with web workload on a multi-core server

Error Reduction Ratio

  • Result: Higher ER ratio in the graph with larger flat region

IRIS: IteRative and Intelligent experiment Selection

ICPE17 15

Normalized x Normalized y Curves ER ratio

slide-16
SLIDE 16

EVALUATION

SINGLE INDEPENDENT VARIABLE

System functions Error Reduction Ratio

  • A group of bell-shaped synthetic functions representing normal distributions
  • Result: IRIS more effective for non-symmetric curves

IRIS: IteRative and Intelligent experiment Selection

ICPE17 16

Normalized x Normalized y Curves ER ratio

slide-17
SLIDE 17

EVALUATION

MULTIPLE INDEPENDENT VARIABLES

System functions

  • Load-response time dataset with two load parameters as independent

variables Error Reduction Ratio

  • Result: Lower ER due to large flat surface

IRIS: IteRative and Intelligent experiment Selection

ICPE17 17

Surfaces ER ratio

slide-18
SLIDE 18

EVALUATION

MULTIPLE INDEPENDENT VARIABLES

System functions Error Reduction Ratio

  • A group of three synthetic Gaussian surfaces with different means and

standard deviations

  • Result: higher ER in surfaces with larger slope

IRIS: IteRative and Intelligent experiment Selection

ICPE17 18

Surfaces ER ratio

slide-19
SLIDE 19

α α

Load- response time

AAE (%) AAE (%)

  • A convex sharp knee in the system function  Smaller α values
  • A concave and symmetric maximum point  Larger α values

Bell-Shaped

TUNING GUIDELINE

GAIN TRADE-OFF FACTOR

IRIS: IteRative and Intelligent experiment Selection

ICPE17 19

slide-20
SLIDE 20
  • IRIS can improve prediction in the

Region of Interest in the parameter space

  • Trade-off:

Slightly lower prediction accuracy for the rest of the parameter space

IRIS: IteRative and Intelligent experiment Selection

ICPE17 20

IRIS Multi-stage EQD

TUNING GUIDELINE

ERROR DISTRIBUTION

slide-21
SLIDE 21

IRIS outperforms equal distance for the majority of the evaluated systems Trade-off factor is tuned through initial system knowledge More reduction in Region of Interest In future, we are going to examine systems with higher dimensionality

CONCLUSIONS

IRIS: IteRative and Intelligent experiment Selection

ICPE17 21

slide-22
SLIDE 22

Thank you! Questions?

Raoufeh Hashemian r.hashemian@ucalgary.ca