Outline DM811 HEURISTICS AND LOCAL SEARCH ALGORITHMS 1. Experimental Algorithmics FOR COMBINATORIAL OPTIMZATION Definitions Performance Measures 2. Exploratory Data Analysis Lecture 13 Sample Statistics Scenarios of Analysis Experimental Analysis Guidelines for Presenting Data 3. Examples Marco Chiarandini Results Task 1 Results Task 2 4. Organizational Issues 2 Outline Contents and Goals Goals of this part of the course (to be continued in DM812): 1. Experimental Algorithmics Provide a view of issues in Experimental Algorithmics Definitions Performance Measures ◮ Exploratory data analysis ◮ Presenting results in a concise way with graphs and tables 2. Exploratory Data Analysis ◮ Organizational issues and Experimental Design Sample Statistics ◮ Basics of inferential statistics Scenarios of Analysis Guidelines for Presenting Data ◮ Sequential statistical testing: a methodology for tuning 3. Examples The goal of Experimental Algorithmics is not only producing a sound analysis Results Task 1 but also adding an important tool to the development of a good solver for a Results Task 2 given problem. Experimental Algorithmics is an important part in the algorithm production 4. Organizational Issues cycle, which is referred to as Algorithm Engineering 3 4
Experimental Algorithmics Experimental Algorithmics Mathematical Model Simulation Program (Algorithm) Goals ◮ Defining standard methodologies Experiment ◮ Comparing relative performance of algorithms so as to identify the best ones for a given application In empirical studies we consider simulation programs which are the ◮ Characterizing the behavior of algorithms implementation of a mathematical model (the algorithm) ◮ Identifying algorithm separators, i.e. , families of problem instances for [McGeoch, 1996] which the performance differ Algorithmic models of programs can vary according to their level of ◮ Providing new insights in algorithm design instantiation: ◮ minimally instantiated (algorithmic framework), e.g., simulated annealing ◮ mildly instantiated: includes implementation strategies (data structures) ◮ highly instantiated: includes details specific to a particular programming language or computer architecture 5 6 Definitions Fairness principle: being completely fair is perhaps impossible but try to For each general problem Π (e.g., TSP, GCP) we denote by C Π a set (or remove any possible bias class) of instances and by π ∈ C Π a single instance. ◮ possibly all algorithms must be implemented with the same style, with the same language and sharing common subprocedures and data The object of analysis are SLS algorithms, i.e. , randomized search heuristics structures (with no guarantee of optimality). ◮ the code must be optimized, e.g., using the best possible data structures ◮ single-pass heuristics (denoted A ⊣ ): have an embedded termination, for ◮ running times must be comparable, e.g., by running experiments on the example, upon reaching a certain state same computational environment (or redistributing them randomly) Eg, Construction heuristics, iterative improvement ◮ asymptotic heuristics (denoted A ∞ ): do not have an embedded termination and they might improve their solution asymptotically 7 9
Definitions Random Variables and Probability Statistics deals with random (or stochastic) variables. A variable is called random if, prior to observation, its outcome cannot be The most typical scenario considered predicted with certainty. The uncertainty is described by a probability distribution. Asymptotic heuristics with time (or iteration) limit decided a priori The algorithm A ∞ is halted when time expires. Discrete variables Continuous variables Probability distribution: Probability density function (pdf): Deterministic case: A ∞ on π Randomized case: A ∞ on π returns f ( v ) = dF ( v ) p i = P [ x = v i ] returns a solution of cost x . a solution of cost X , where X is a dv random variable. The performance of A ∞ on π is a Cumulative Distribution Function (CDF) Cumulative Distribution Function (CDF): The performance of A ∞ on π is the scalar y = x . � v � F ( v ) = P [ x ≤ v ] = p i F ( v ) = f ( v ) dv univariate Y = X . i − ∞ Mean Mean � � [This is not the only relevant scenario: to be refined later] µ = E [ X ] = x i p i µ = E [ X ] = xf ( x ) dx Variance Variance � σ 2 = E [( X − µ ) 2 ] = � ( x i − µ ) 2 p i σ 2 = E [( X − µ ) 2 ] = ( x − µ ) 2 f ( x ) dx 10 11 Generalization Sampling In experiments, 1. we sample the population of instances and On a specific instance, the random variable Y that defines the performance measure of an algorithm is described by its probability distribution/density 2. we sample the performance of the algorithm on each sampled instance function If on an instance π we run the algorithm r times then we have r replicates of Pr ( Y = y | π ) the performance measure Y , denoted Y 1 , . . . , Y r , which are independent and identically distributed (i.i.d.), i.e. It is often more interesting to generalize the performance r � on a class of instances C Π , that is, Pr ( y 1 , . . . , y r | π ) = Pr ( y j | π ) � j = 1 Pr ( Y = y, C Π ) = Pr ( Y = y | π ) Pr ( π ) π ∈ Π � Pr ( y 1 , . . . , y r ) = Pr ( y 1 , . . . , y r | π ) Pr ( π ) . π ∈ C Π 12 13
Instance Selection Statistical Methods In real-life applications a simulation of p ( π ) can be obtained by The analysis of performance is based on finite-sized sampled data. historical data. Statistics provides the methods and the mathematical basis to In simulation studies instances may be: ◮ describe, summarizing, the data (descriptive statistics) ◮ real world instances ◮ make inference on those data (inferential statistics) ◮ random variants of real world-instances ◮ online libraries Statistics helps to ◮ randomly generated instances ◮ guarantee reproducibility ◮ make results reliable They may be grouped in classes according to some features whose impact (are the observed results enough to justify the claims?) may be worth studying: ◮ extract relevant results from large amount of data ◮ type (for features that might impact performance) ◮ size (for scaling studies) In the practical context of heuristic design and implementation (i.e., ◮ hardness (focus on hard instances) engineering), statistics helps to take correct design decisions decisions with ◮ application (e.g., CSP encodings of scheduling problems), ... the least amount of experimentation Within the class, instances are drawn with uniform probability p ( π ) = c 14 15 Objectives of the Experiments Measures and Transformations On a single instance Alg. 1 Alg. 2 Alg. 3 Alg. 4 Alg. 5 0.4 0.3 Computational effort indicators 0.2 0.1 ◮ number of elementary operations/algorithmic iterations 0.0 ◮ Comparison : (e.g., search steps, objective function evaluations, number of visited −2 0 2 bigger/smaller, same/different, Response nodes in the search tree, consistency checks, etc.) Algorithm Configuration, ◮ total CPU time consumed by the process Component-Based Analysis Alg. 5 (sum of user and system times returned by getrusage ) ◮ Standard statistical methods: Alg. 4 experimental designs, test Alg. 3 hypothesis and estimation Alg. 2 Solution quality indicators Alg. 1 ◮ value returned by the cost function −2 0 2 ◮ Characterization : Response Interpolation: fitting models to data ◮ error from optimum/reference value Extrapolation: building models of Uniform random graphs ◮ gap | UB − LB | or | UB − LB | 3600 data, explaining phenomena p=0 p=0.1 p=0.2 p=0.5 UB UB 1000 + + + + + ◮ ranks ◮ Standard statistical methods: linear + + + + + + 100 + p=0.9 and non linear regression + + + + + Seconds + + 10 + + + + + + + + + 16 18 model fitting + + + + + + + + + + + +
Measures and Transformations Measures and Transformations On a class of instances On a class of instances Solution quality indicators ◮ Distance or error from a reference value Computational effort indicators (assume minimization case): ◮ no transformation if the interest is in studying scaling ◮ standardization if a fixed time limit is used e 1 ( x, π ) = x ( π ) − ¯ x ( π ) standard score ◮ geometric mean (used for a set of numbers whose values are meant to � ^ σ ( π ) be multiplied together or are exponential in nature), ◮ otherwise, better to group homogeneously the instances e 2 ( x, π ) = x ( π ) − x opt ( π ) relative error x opt ( π ) Solution quality indicators x ( π ) − x opt ( π ) e 3 ( x, π ) = invariant [Zemel, 1981] Different instances implies different scales ⇒ need for an invariant measure x worst ( π ) − x opt ( π ) (However, many other measures can be taken both on the algorithms and on ◮ optimal value computed exactly or known by instance construction the instances [McGeoch, 1996]) ◮ surrogate value such bounds or best known values ◮ Rank (no need for standardization but loss of information) 19 20 Outline Summary Measures for Sampled Data 1. Experimental Algorithmics Definitions Performance Measures Measures to describe or characterize a population ◮ Measure of central tendency, location 2. Exploratory Data Analysis ◮ Measure of dispersion Sample Statistics Scenarios of Analysis One such a quantity is Guidelines for Presenting Data ◮ a parameter if it refers to the population (Greek letters) ◮ a statistics if it is an estimation of a population parameter from the 3. Examples sample (Latin letters) Results Task 1 Results Task 2 4. Organizational Issues 21 23
Recommend
More recommend
