Bayesian Analysis for Algorithm Performance Comparison Is it - - PowerPoint PPT Presentation

Josu Ceberio

Bayesian Analysis for Algorithm Performance Comparison

Is it possible to compare optimization algorithms without hypothesis testing?

Is there a reproducibility crisis?

Fuente: Monya Baker (2016) Is there a reproducibility crisis? Nature, 533, 452-454

Hypothesis

Idea for solving a set

• f problems more

efficiently.

Questions

Is my algorithm better than the state-

• f-the-art?

On which problems is my algorithm better? Why is my algorithm better (or worse)?

Experimentation

Compare the performance

• f my algorithm with the-

state-of-the-art on some benchmark of problems. The analysis of the results should take into account the associated uncertainty.

Conclusions

What conclusions do we draw from the experimentation? How do we answer to the formulated questions?

Is there a reproducibility crisis?

The Questions

How l likely i is m my p proposal t to be be the b best a algorithm t to s solve a p problem? How l likely i is m my p proposal t to be be the b best a algorithm f from the c compared o

• nes?

The Point

STATISTICAL A ANALYSIS O OF EXPERIMENTAL R RESULTS NULL HYPOTHESIS STATISTICAL TESTING

WHAT N NHST C COMPUTES

p(t(x) > τ|H0)

Unknown Behaviour Observed Sample

The controversy with NHST

The controversy with NHST

We assume the null hypothesis, the average performance of the compared methods is the same. Then, the observed difference is computed from data and the probability of observing such a difference (or bigger) is estimated: the p-value. The p-value refers to the probability of erroneously assuming that there are differences when actually there are not. It is used to measure the magnitude of difference, as it decreases when the difference increases. WHAT N NHST C COMPUTES

p(t(x) > τ|H0)

1 − p(t(x) > τ|H0) = p(t(x) < τ|H0)

WHAT W WE W WOULD L LIKE T TO K KNOW

1 − p(H0|x) = p(H1|x)

p(H0|x)

The Point

Unknown Behaviour Observed Sample

Many alternatives to handle uncertainty associated with empirical results:

6WDWLVWLFDOQDO\VLV +DQGERRN

$&RPSUHKHQVLH+DQGERRNRI6DLVLFDO &RQFHSV7HFKQLTHVDQG6RIDUH7RROV (GLLRQ 'U0LFKDHO-GH6PLK

WHAT N NHST C COMPUTES

p(t(x) > τ|H0)

BAYESIAN STATISTICAL ANALYSIS

The Point

STATISTICAL A ANALYSIS O OF EXPERIMENTAL R RESULTS NULL HYPOTHESIS STATISTICAL TESTING Unknown Behaviour Observed Sample

The Bayesian Approach

The method focuses on estimating relevant information about the underlying performance parametric distribution represented by a set of parameters θ. This method asses the distribution of θ conditioned on a sample s drawn from the performance distribution. Instead of having a single probability distribution to model the underlying performance, Bayesian statistics considers all possible distributions and assigns a probability to each.

P(θ|s) ∝ P(s|θ)P(θ)

Posterior distribution

• f the parameters

Likelihood function Prior distribution

• f the parameters

HOW D DO W WE C COMPARE M MULTIPLE AL ALGORITHMS?

Minimizing some instances of a problem Minimizing a given instance of a problem

Algorithm f1

GA 100 PSO 90 ILP 135 SA 105 GP 95 . . . . . .

From Results to Rankings

Observed Sample

σ1

3 1 5 4 2 . . .

Algorithm f2

GA 130 PSO 80 ILP 135 SA 30 GP 300 . . . . . .

σ2

3 2 4 1 5 . . .

σ3

3 5 2 4 1 . . .

σ4

4 5 3 1 2 . . .

σ5

4 3 2 5 1 . . .

Algorithm f3

GA 37 PSO 352 ILP 19 SA 100 GP 10 . . . . . .

Algorithm f4

GA 566 PSO 756 ILP 101 SA 56 GP 57 . . . . . .

Algorithm f5

GA 256 PSO 125 ILP 89 SA 369 GP 36 . . . . . .

rankings, permutations

• Each algorithm in the comparison has a weight associated.
• The weights sum up 1.
• The weight associated to an algorithm represents its probability to appear at first rank.

Plackett-luce Model

P(σ) =

n

Y

i=1

wσi Pn

j=i wσj

!

#1 #2 #3 #4 #4

Plackett-luce Model

w1 = 0.3

w4 = 0.6

w3 = 0.17

w2 = 0.03

#1 #2 #3 #4

Plackett-luce Model

w1 = 0.3

w3 = 0.17

w2 = 0.03

#1

#2 #3 #4

Plackett-luce Model

w3 = 0.17

w2 = 0.03

#1 #2 #3

P(σ) = w4 w1 + w2 + w3 + w4 · w1 w1 + w2 + w3 · w3 w2 + w3 · w2 w2

The bayesian model

Posterior distribution of the weights Likelihood of the sample Prior distribution of the weights N

Y

k=1 n

Y

i=1

@ wσ(k)

i

Pn

j=i wσ(k)

j

1 A

R = {σ(1), . . . , σ(N)}

P(w|R) ∝

1 B

n

Y

i=1

wαi−1

i

B = Qn

i=1 Γ(αi)

Γ(Pn

i=1 αi)

No way to sample posterior distribution exactly à MCMC

Bayesian inference for algorithm ranking analysis

Instance #1 Instance #m Instance #2

• Inst. #1
• Inst. #m
• Inst. #2

Alg1 w1 w2 wn Alg2 Algn

Performance Matrix Weight Vector Sample Run the Algorithms Rank the Algorithms

• Inst. #1
• Inst. #m
• Inst. #2

Alg1 Alg2 Algn

Ranking Matrix MCMC Sampling Query Posterior

0.0 0.2 0.4 0.6

The Case of Study

23 F FUNCTIONS T TO O OPTIMIZE:

• OneMax (F1) and W-model extensions (F4-F10)
• LeadingOnes (F2) and W-model extensions (F11-

F17)

• Harmonic (F3)
• LABS: Low Autocorrelation Binary Sequences (F18)
• Ising-Ring (F19)
• Ising-Torus (F20)
• Ising-Triangular (F21)
• MIVS: Maximum Independent Vertex Set (F22)
• NQP: N-Queens problem (F23)

n ∈ {16, 64, 100, 625}

Pr Problem Si Size: 11 11 Me Metah aheuri ristic al algori rithms:

• greedy Hill Climber (gHC)
• Randomlized Local Search (RLS)
• (1+1) EA
• fast Genetic Algorithm (fGA)
• (1+10) EA
• (1+10) EAr/2,2r
• (1+10) EAnorm
• (1+10) EAvar
• (1+10) EAlog-n
• (1+(λ+λ)) GA
• “vanilla” GA (vGA)

Re Results of 1 11.132 ru runs ns are are co collect cted (23 x x 4 4 x x 1 11 x x 1 11)

• Aggregation of performances across 11 instances.
• Median performance across 11 repetitions.

Es Estim imate th the pr probability of

• f ea

each al algori rithm be being top top-rank ranked

• as its expected weight in the posterior distribution of weights

An Analyze th the un uncertainty ab about th the pr probabilities

• By estimating the 90% credible intervals of the posterior distribution of weights (5% and 95%)

Inference analyses & results

QUALITATIVE S SUMMARY Similar perf. (1+(λ+λ)) GA, (1+1)-EA, (1+10)-EAvar, (1+10)-Ealog-n, (1+10)-Eanorm,(1+10)-EAr/2,2r and fGA. Extreme perf. vGA and gHC. Easily treated instances are F1-F6, F8, F11-F13 and F15-16. Best solutions found for n=625

Inference analyses & results

Fix Fixed-ta target t pe perspe pect ctive – Record Running-time

(1+(,)) GA (1+1) EA gHC (1+10) EA_r/2,2r (1+10) EA (1+10) EA_log-n. (1+10) EA_norm. (1+1) EA_var. fGA vGA RLS 0.0 0.2 0.4 0.6

Probability of winning Algorithm

F17, n=625, φ=625 F19, n=100, φ=100

(1+(,)) GA (1+1) EA gHC (1+10) EA_r/2,2r (1+10) EA (1+10) EA_log-n. (1+10) EA_norm. (1+1) EA_var. fGA vGA RLS 0.0 0.1 0.2 0.3 0.4 0.5

Probability of winning Algorithm

Credible Intervals Only 11 samples to do inference à High uncertainty is expected! The more samples, the lower the uncertainty à Credibility intervals are more tight!

Expected probability High uncertainty

IN INTERP RPRE RETABIL ILIT ITY

Inference analyses & results

Fix Fixed-ta target t pe perspe pect ctive – Record Running-time – Set of easy functions

(1+(,)) GA (1+1) EA gHC (1+10) EA_r/2,2r (1+10) EA (1+10) EA_log-n. (1+10) EA_norm. (1+1) EA_var. fGA vGA RLS 0.00 0.25 0.50 0.75 1.00

Probability of winning Algorithm

n=625, all runs

(1+(,)) GA (1+1) EA gHC (1+10) EA_r/2,2r (1+10) EA (1+10) EA_log-n. (1+10) EA_norm. (1+1) EA_var. fGA vGA RLS 0.0 0.2 0.4 0.6

Probability of winning Algorithm

n=625, median

Credible Intervals Set of functions, two paths à (1) take all the runs, (2) take the median of the runs on each instance. gHC is the best in both cases à with more samples the uncertainty is lower

Inference analyses & results

Fix Fixed-ta target t pe perspe pect ctive – Record Running-time – Set of non-easy functions Credible Intervals Good estimations à credible intervals smaller than 0.05 Probabilities are similar à due to overlapping Uncertainty about which is the best à but not due to limitation of data, but due to equivalence in the algorithms

(1+(,)) GA (1+1) EA gHC (1+10) EA_r/2,2r (1+10) EA (1+10) EA_log-n. (1+10) EA_norm. (1+1) EA_var. fGA vGA RLS 0.050 0.075 0.100 0.125 0.150

Probability of winning Algorithm

n=625, all runs

Inference analyses & results

Fix Fixed-bu budget pe perspe pect ctive – Evolution winning probability - %90 credibility intervals

0.0 0.2 0.4 0.6 300 600 900

Budget Winning probability

(1+(,)) GA (1+1) EA gHC (1+10) EA_r/2,2r (1+10) EA (1+10) EA_log-n. (1+10) EA_norm. (1+1) EA_var. fGA vGA RLS

F21, n=100

gHC is the best, but probability decreases while the rest improve. gHC becomes better, as the budget increases.

3 4 5 6 7 8 9 10 11

Algorithms ranked with average data Wilcoxon test for pairwise comparisons, and shaffer’s method for p-value correction.

BAYESIAN A ANALYSIS ESTIMATED P PROBABILITY A AND NOTION O OF U UNCERTAINTY I IN T THE FORM O OF C CREDIBLE I INTERVAL

Inference analyses & results

Im Impact of

• f th

the pr prior di distribution – Comparison of three different priors

0.0 0.2 0.4 0.6 ( 1 + (

• ,
• )

) G A ( 1 + 1 ) E A g H C ( 1 + 1 ) E A _ r / 2 , 2 r ( 1 + 1 ) E A ( 1 + 1 ) E A _ l

• g
• n

. ( 1 + 1 ) E A _ n

• r

m . ( 1 + 1 ) E A _ v a r . f G A v G A R L S

Algorithm Winning probability Prior

Unifor Empirical Deceptive

F9, n=100, φ=100

Empirical data favours the best performing algorithms Neligible effect (even when median values are considered)

Discussion

Bayesian inference using Plackett-Luce for analysis of algorithms’ performance ranking Include it in the practical EC performance comparison’ tool set à IOHProfiler Strong points Ability to handle multiple algorithms Interpretability Exact description of the uncertainty WEAKNESSES Aggregating performances into rankings we loose information about the magnitude of differences Limitations of the Plackett-Luce model à From n! to n parameters. How do we deal with ties?

scmamp: Statistical Comparison of Multiple Algorithms in Multiple Problems

Josu Ceberio

Bayesian Analysis for Algorithm Performance Comparison

Thank you very much for your attention!