On the Sustainability of the Extreme Value Theory for WCET - PowerPoint PPT Presentation

On the Sustainability of the Extreme Value Theory for WCET Estimation Luca Santinelli , J´ erˆ ome Morio, Guillaume Dufour, Damien Jacquemart

Plan 1 Motivations & Problem Statement 2 EVT Applicability: what are the hypotheses to apply EVT? 3 EVT complexity: what is complex while applying EVT? 4 EVT robustness 2/19

Plan 1 Motivations & Problem Statement 2 EVT Applicability: what are the hypotheses to apply EVT? 3 EVT complexity: what is complex while applying EVT? 4 EVT robustness 3/19

Motivations & Problem Statement Extreme Value Theory (EVT) in combination with measurements: Measurement-Based Probabilistic Timing Analysis (MBPTA). 1 Measurements of system execution times: reproducible & reliable. 2 Rare Events (where the worst-case should be): Extreme Value Theory. Statistical analysis in between: stationarity, time series (new - necessary to make MBPTA more formal). 4/19

Motivations & Problem Statement 1e+00 1e−03 Probability 1e−06 1e−09 1e−12 EVT 1−CDF Obs 1−CDFF 13600 13650 13700 13750 Execution Time 4/19

Motivations & Problem Statement Extreme Value Theory in combination with measurements: { X 1 , X 2 , X 3 , . . . , X N } → probabilistic estimation of Worst-Case Execution Time (pWCET). The need for it - with complex (modeling) system: it reduces cost and pessimism. Pros: easy to apply , near zero modeling (near 0 info required). Cons: reliability - worst-case : where is the worst-case? Safety? strict hypotheses (not realistic hypotheses). 4/19

Motivations & Problem Statement Measurements: � � C k C = Measurements p k = P {C = C k } C k ∈{ 1 , ··· , K } probabilistic Worst-Case HP Verification Measurements Execution Time (pWCET): C EVT � � EVT C C j C = p j = P {C = C j } j ∈{ 1 , ··· , J } 4/19

Extreme Value Theory Block Maxima (BM) EVT approach Peak over Thresholds (PoT) EVT approach 5/19

Extreme Value Theory EVT BM The maxima of an independent and identical distributed sequence converges to a Generalized Extreme Value (GEV) distribution G ξ under some general conditions:  exp( − exp( − x )) , if ξ = 0   G ξ ( x ) = � � − (1 + ξ x ) − 1 exp , if ξ � = 0 . ξ   GEV distributions characterized by ξ = 0, ξ > 0 and ξ < 0 corresponding to Gumbel, Fr´ echet and Weibull. Suppose there exist a N and b N , with a N > 0 such that, for all � X ( N ) − b N � = G N ( a N y + b N ) N →∞ y ∈ R P ≤ y − → G ( y ), where G is a non degenerate a N CDF, then G is a GEV distribution G ξ . In this case, one denotes G ∈ MDA ( ξ ) (MDA=Maximum Domain of Attraction). Block size b selected arbitrarily: it affects accuracy and safety of the EVT BM estimation 5/19

Extreme Value Theory EVT PoT PoT considers the largest samples X i to estimate the probability P {X > S } . Let us assume that the distribution function G of independent and identical distributed samples X 1 , ..., X N is continuous. Set y ∗ = sup { y , G ( y ) < 1 } = inf { y , G ( y ) = 1 } . Then, the next two assertions are equivalent: a) G ∈ MDA ( ξ ), and b) there exists a positive and measurable function u �→ β ( u ) such that |G u ( y ) − H ξ,β ( u ) ( y ) | = 0. G u ( y ) = P {X − u ≤ y |X > u } , and H ξ,β ( u ) lim sup u �→ y ∗ 0 < y < y ∗ − u is the CDF of a generalized Pareto distribution (GPD) with shape parameter ξ and scale parameter β ( u ). Threshold u selected arbitrarily: affects accuracy and safety of the EVT PoT estimation. 5/19

Extreme Value Theory BM and PoT comparison: 1e+00 Obs BM b=10 BM b=100 PoT u=0.98 PoT u=0.95 1e−03 Probability 1e−06 1e−09 1e−12 13600 13700 13800 13900 14000 Execution Time 5/19

Plan 1 Motivations & Problem Statement 2 EVT Applicability: what are the hypotheses to apply EVT? 3 EVT complexity: what is complex while applying EVT? 4 EVT robustness 6/19

EVT applicability Classically: independence and identical distribution (i.i.d.) 7/19

EVT applicability Long range independence EVT Let { X n } be a stationary sequence such that M n = max { X 1 , . . . X n } has a non-degenerate limiting distribution G as in P { a n ( M n − b n ) ≤ x } d → G ( x ) , for some constants a n > 0, b n . Suppose that D ( u n ) : | F i 1 ,..., i p , j 1 ,..., j q ( u n ) − F i 1 ,..., i p ( u n ) · F j 1 ,..., j q ( u n ) | ≤ α n , l , where lim l →∞ lim n →∞ α n , l = 0, holds for all sequences u n given by u n = x / a n + b n , −∞ < x < ∞ . Then G is one of the three classical types: Weibull, Frechet, Gumbel. 7/19

EVT applicability Extremal independence EVT Let { X n } be a stationary sequence with marginal distribution function F such that M n = max { X 1 , . . . X n } , and { u n } a sequence of constants such that D ( u n ), D ′ ( u n ) hold. Let 0 ≤ τ < ∞ , then P { M n ≤ u n } d → exp ( − τ ) iff n · [1 − F ( u n )] → τ. D ′ ( u n ) : lim n →∞ sup n · � n / k j =2 P { X 1 > u n , X j > u n } → 0, assure independence between close-in-time observations. 7/19

EVT applicability Independence → extremal independence → stationarity C the pWCET EVT estimation in case of stationarity, � C the pWCET EVT estimation in case of independence. Supposing execution time measurements follow the same marginal distribution, then C = � C θ , C � � C . 7/19

EVT hypotheses: identical distribution A collection of random variables is identically distributed (i.d.) if each random variable has the same probability distribution. Kolmogorov-Smirnov test for comparing two sets of observations. The trace of observations is divided into two sets which are compared, to verify whether they represent the same distribution. 8/19

EVT hypotheses: independence A collection of random variables is independent (i.) if all the random variables are mutually independent. Independence proven or tested for time series based on autoregression and autocorrelation . Verifying stationarity (Statistic analysis): more info about observation traces; characterize system execution behavior looking for the worst-case execution conditions. 9/19

EVT hypotheses: extremal independence Extremogram ρ : dependence at the extremes is estimated. Analogue of the autocorrelation function, depending only on the extreme values in the sequence of observations; based on estimators. Extremal index θ : another tool for measuring the dependency of extreme values (to compare with the independence case). Make use of the blocks test to compute; based on estimators. 10/19

EVT hypotheses: stationarity A process X is stationary if its mean variance and autocovariance structure do not change over time. Weak form of stationarity: flat-looking observations, no trend, constant variance over time, and no periodic fluctuations or autocorrelation. Autocorrelation : similarity between observations as a function of the time lag between them. Sample Auto Correlation Function (ACF) : important assessment tools for detecting data dependence and fitting models to data. The model is not faced at first, the observed data { X 1 , . . . , X N } are known . 11/19

Plan 1 Motivations & Problem Statement 2 EVT Applicability: what are the hypotheses to apply EVT? 3 EVT complexity: what is complex while applying EVT? 4 EVT robustness 12/19

EVT complexity Parameter effect on WCET estimation: single path BM 1.0 Obs b=5 b=10 0.8 b=20 b=50 b=100 0.6 Probability b=200 0.4 0.2 0.0 13600 13650 13700 13750 13800 Execution time b selection! 13/19

EVT complexity Parameter effect on WCET estimation: multi-path BM 1.0 0.8 0.6 Probability 0.4 Obs b=5 b=10 0.2 b=20 b=50 b=100 0.0 b=200 100 200 500 1000 2000 5000 Execution time b selection! 13/19

EVT complexity Parameter effect on WCET estimation: single path PoT 1.0 Obs u=.7 u=.8 0.8 u=.9 u=.95 u=.98 0.6 Probability u=.9999 0.4 0.2 0.0 13600 13650 13700 13750 13800 Execution time u selection! 13/19

EVT complexity Parameter effect on WCET estimation: multi-path PoT 1.0 0.8 0.6 Probability 0.4 Obs u=.7 u=.8 0.2 u=.9 u=.95 u=.98 0.0 u=.9999 100 200 500 1000 2000 5000 Execution time u selection! 13/19

Plan 1 Motivations & Problem Statement 2 EVT Applicability: what are the hypotheses to apply EVT? 3 EVT complexity: what is complex while applying EVT? 4 EVT robustness 14/19

EVT robustness How can we define robustness? Still a fuzzy word/definition for EVT, but something to play with and extend. error/confidence: bootstrap & relative error. Multi-path: all the paths have to be included. 15/19

Confidence - relative error Bootstrapping and looking for independence: relative error with ”artificial independence”. 1 0.8 PWCET 0.6 0.4 0.2 0 1.36 1.365 1.37 1.375 1.38 Execution Time 4 x 10 16/19

Confidence - relative error Bootstrapping and looking for independence: relative error with ”artificial independence”. Relative error in % 1000 750 500 250 0 1.36 1.365 1.37 1.375 1.38 �Exectution TIme 4 x 10 16/19

Confidence - relative error Bootstrapping and looking for independence: relative error with ”artificial independence”. 1 0.8 pWCET 0.6 0.4 0.2 0 0 2000 4000 6000 8000 10000 Execution Time 16/19