SLIDE 1
Performance Assessment in Optimization Anne Auger, CMAP & Inria - - PowerPoint PPT Presentation
Performance Assessment in Optimization Anne Auger, CMAP & Inria - - PowerPoint PPT Presentation
Performance Assessment in Optimization Anne Auger, CMAP & Inria Visualization and presentation of single runs Displaying 3 runs (three trials) Displaying 3 runs (three trials) Displaying 3 runs (three trials) Displaying 51 runs Which
SLIDE 2
SLIDE 3
Displaying 3 runs (three trials)
SLIDE 4
Displaying 3 runs (three trials)
SLIDE 5
Displaying 3 runs (three trials)
SLIDE 6
Displaying 51 runs
SLIDE 7
Which Statistics?
SLIDE 8
More problems with average / expectations
from Hansen GECCO 2019 Experimentation tutorial
SLIDE 9
Which Statistics?
SLIDE 10
Implications
from Hansen GECCO 2019 Experimentation tutorial
SLIDE 11
Benchmarking Black-Box Optimizers
Benchmarking: running an algorithm on several test functions in order to evaluate the performance of the algorithm
SLIDE 12
Why Numerical Benchmarking?
Evaluate the performance of optimization algorithms Compare the performance of different algorithms
understand strength and weaknesses of algorithms help in design of new algorithms
SLIDE 13
On performance measures …
SLIDE 14
Performance measure - What to measure?
CPU time (to reach a given target) drawbacks: depend on the implementation, on the language, on the machine time is spent on code optimization instead of science
Testing heuristics, we have it all wrong, J.N. Hooker, 1995 Journal of Heuristics
Prefer “absolute” value: # of function evaluations to reach a given target assumptions: internal cost of the algorithm negligible
- r measured independently
SLIDE 15
On performance measures - Requirements “Algorithm A is 10/100 times faster than Algorithm B to solve this type of problems”
SLIDE 16
“Algorithm A is 10/100 times faster than Algorithm B to solve this type of problems” quantitative measures On performance measures - Requirements As opposed to
displayed: mean f-value after 3.10^5 f-evals (51 runs) bold: statistically significant concluded: “EFWA significantly better than EFWA-NG”
Source: Dynamic search in fireworks algorithm, Shaoqiu Zheng, Andreas Janecek, Junzhi Li and Ying Tan CEC 2014
SLIDE 17
a performance measure should be quantitative, with a ratio scale well-interpretable with a meaning relevant in the “real world” simple
On performance measures - Requirements
SLIDE 18
Fixed Cost versus Fixed Budget - Collecting Data
SLIDE 19
Fixed Cost versus Fixed Budget - Collecting Data
Collect for a given target (several target), the number of function evaluations needed to reach a target Repeat several times: if algorithms are stochastic, never draw a conclusion from a single run if deterministic algorithm, repeat by changing (randomly) the initial conditions
SLIDE 20
ECDF: Empirical Cumulative Distribution Function of the Runtime
SLIDE 21
Definition of an ECDF
Let be real random variables. Then the empirical cumulative distribution function (ECDF) is defined as
(X1, …, Xn) ̂ F(t) = 1 n
n
∑
i=1
1Xi≤t
SLIDE 22
We display the ECDF of the runtime to reach target function values (see next slides for illustrations)
SLIDE 23
A Convergence Graph
A Convergence Graph
SLIDE 24
First Hitting Time is Monotonous
SLIDE 25
15 Runs
SLIDE 26
target
15 Runs ≤ 15 Runtime Data Points
SLIDE 27
Empirical CDF
1 0.8 0.6 0.4 0.2
the ECDF of run lengths to reach the target
- has for each
data point a vertical step of constant size
- displays for each
x-value (budget) the count of
- bservations to
the left (first hitting times)
Empirical Cumulative Distribution
SLIDE 28
Empirical CDF
1 0.8 0.6 0.4 0.2
interpretations possible:
- 80% of the runs
reached the target
- e.g. 60% of the
runs need between 2000 and 4000 evaluations
Empirical Cumulative Distribution
SLIDE 29
Aggregation
15 runs
SLIDE 30
Aggregation
15 runs 50 targets
SLIDE 31
Aggregation
15 runs 50 targets
SLIDE 32
15 runs 50 targets ECDF with 750 steps
Aggregation
SLIDE 33
We can aggregate over:
- different targets
- different functions and targets
We should not aggregate over dimension as functions of different dimensions have typically very different runtimes
SLIDE 34
ECDF aggregated over targets - single functions
ECDF for 3 different algorithms
SLIDE 35
ECDF aggregated over targets - single function
ECDF for a single algorithm different dimensions
SLIDE 36
ERT/ART: Average Runtime
SLIDE 37
Which performance measure ?
SLIDE 38
Which performance measure ?
SLIDE 39
Expected Running Time (restart algo)
ERT = E[RT r] = 1−ps
ps E[RTunsuccessful + E[RTsuccessful]
Estimator for ERT
b ps =
#succ #Runs
\ RTunsucc = Average Evals of unsuccessful runs \ RTsucc = Average Evals of successful runs ART =
#Evals #success
SLIDE 40
Example: scaling behavior A R T
A
SLIDE 41
On Test functions
SLIDE 42
Test functions
function testbed (set of test functions) should “reflect reality”: should model typical difficulties one is willing to solve Example: BBOB testbed (implemented in the COCO platform) the test functions are mainly non-convex and non-separable scalable with the search space dimension not too easy to solve, but yet comprehensible
SLIDE 43
Test functions (cont.)
If aggregation of results over all functions from a testbed (through ECDF):
- ne needs to be careful that some difficulties are not over-
represented or that not too many easy functions are present
SLIDE 44
- 24 functions in 5 groups:
- 6 dimensions: 2, 3, 5, 10, 20, (40 optional)
The bbob Testbed
SLIDE 45
BBOB testbed
Black-Box Optimization Benchmarking test suite noiseless / noisy testbed
http://coco.gforge.inria.fr/doku.php?id=start
noiseless testbed noisy testbed
SLIDE 46
COCO platform: automatizing the benchmarking process
SLIDE 47
https://github.com/numbbo/coco Step 1: download COCO
SLIDE 48
https://github.com/numbbo/coco Step 2: installation of post-processing
SLIDE 49
http://coco.gforge.inria.fr/doku.php?id=algorithms Step 3: downloading data for the moment: IPOP-CMA-ES
SLIDE 50