Compositional Solution Space Quantification for Probabilistic - - PowerPoint PPT Presentation

▶
compositional solution space quantification for
SMART_READER_LITE
LIVE PREVIEW

Compositional Solution Space Quantification for Probabilistic - - PowerPoint PPT Presentation

Jul 08, 2023 152 likes •757 views

Compositional Solution Space Quantification for Probabilistic Software Analysis Mateus Borges, Marcelo dAmorim (UFPE) Antonio Filieri (Stuttgart) Corina Pasareanu (CMU SV and NASA Ames) Willem Visser (Stellenbosch) Uncertain Environments

slide-1
SLIDE 1

Compositional Solution Space Quantification for Probabilistic Software Analysis

Mateus Borges, Marcelo d’Amorim (UFPE) Antonio Filieri (Stuttgart) Corina Pasareanu (CMU SV and NASA Ames) Willem Visser (Stellenbosch)

slide-2
SLIDE 2

Uncertain Environments

slide-3
SLIDE 3

Uncertain Environments

slide-4
SLIDE 4

Quantitative Properties

Not restricted to boolean values Establish non-functional requirements ➔ Reliability, performance...

slide-5
SLIDE 5

Probabilistic Model Checking

PMC Engine

(e.g. PRISM...)

Probabilistic Model + Property Specification Violation Probability

slide-6
SLIDE 6

Probabilistic Model Checking

Problem: can be expensive!

➔ You need to learn a new modelling language ➔ You need to model the system

We would like to analyze code

slide-7
SLIDE 7

Probabilistic Software Analysis

Analysis Engine

Target Event + Program + Input Profile Event Probability + Accuracy

slide-8
SLIDE 8

Probabilistic Software Analysis

PCs S

  • u

r c e C

  • d

e Input Profile Target Event Symbolic Execution Probabilistic Analysis Probability + Accuracy

slide-9
SLIDE 9

Probabilistic Software Analysis

PCs S

  • u

r c e C

  • d

e Input Profile Target Event Symbolic Execution Probabilistic Analysis Probability + Accuracy

Collect path conditions leading to target event

slide-10
SLIDE 10

Obstacle: Quantification

slide-11
SLIDE 11

Integration Methods

Symbolic

➔ very expensive, restricted

Numerical

➔ expensive with multi-dimensional domains

Statistical

➔ approximate results

slide-12
SLIDE 12

Challenge

Quantifying the solution space of complex mathematical functions

Example constraint from TSAFE module (Tactical Separation Assisted Flight Environment)

sqrt(pow(((x1 + (e1 * (cos(x4) – cos((x4 + (((1.0 * (((c1 * x5) * (e2/c2)) / x6)) * x2) / e1)))))) – (((e2/c2)) * (1.0 – cos((c1 * x5))))), 2.0)) > 999.0 & (c1 * x5) > 0.0 & x3 > 0.0 & x6 > 0.0 & c1 = 0.017… & c2 = 68443.0 & e1 = ((pow(x2,2.0) / tan((c1*x3)))/c2) & e2 = pow(x6,2.0) / tan (c1*x3)

slide-13
SLIDE 13

Contribution

qCORAL Supports arbitrarily complex constraints Computes accurate estimates efficiently Path Conditions + Input Profile Event Probability + Accuracy

slide-14
SLIDE 14

High Level View: Divide

Path Conditions PC1 PC2 PC3 Input Profile

slide-15
SLIDE 15

High Level View: Divide

Path Conditions PC1 PC2 PC3

pt1 pt2 pt3 pt7 pt4 pt5 pt6

Input Profile

slide-16
SLIDE 16

High Level View: Divide

Path Conditions PC1 PC2 PC3

pt1 pt2 pt3 pt7 pt4 pt5 pt6 e_pt1 e_pt2 e_pt3 e_pt7 e_pt4 e_pt5 e_pt6 Monte Carlo

Input Profile

slide-17
SLIDE 17

High Level View: Conquer

e_pt1 e_pt2 e_pt3 e_pt7 e_pt4 e_pt5 e_pt6

slide-18
SLIDE 18

High Level View: Conquer

e_pt1 e_pt2 e_pt3 e_pt7 e_pt4 e_pt5 e_pt6

e_PC1 e_PC2 e_PC3

slide-19
SLIDE 19

High Level View: Conquer

e_pt1 e_pt2 e_pt3 e_pt7 e_pt4 e_pt5 e_pt6

e_PC1 e_PC2 e_PC3 Probability Estimate

slide-20
SLIDE 20

PCT

Working With Disjunctions

All elements in PCT are disjoint Estimates can be computed individually PC1 ... PC3 PC2

slide-21
SLIDE 21

Working With Conjunctions

... c1 c2 c2 c3 c4 c3

PC1 PC2 PC3

slide-22
SLIDE 22

Working With Conjunctions

... c1 c2 c2 c3 c4 c3

PC1 PC2 PC3

slide-23
SLIDE 23

Working With Conjunctions

... c1 c2 c2 c3 c4 c3

PC1 PC2 PC3

pt1: c1 pt2: c2 pt3: c3 pt4: c3 && c4

Contains dependent variables

slide-24
SLIDE 24

Working With Conjunctions

... c1 c2 c2 c3 c4 c3 Partitions can be analyzed faster Estimates can be efficiently re-used

PC1 PC2 PC3 Contains dependent variables

pt1: c1 pt2: c2 pt3: c3 pt4: c3 && c4

slide-25
SLIDE 25

Quantifying Constraints

c2

slide-26
SLIDE 26

Quantifying Constraints

c2 Domain Solution Space

slide-27
SLIDE 27

Hit-or-Miss Monte Carlo

c2 E[X] = #hits / #samples

slide-28
SLIDE 28

Stratified Sampling

Remove infeasible areas with RealPaver c2 Domain c2 Solution Space Boxes returned by RealPaver

slide-29
SLIDE 29

Stratified Sampling

c2 Increase precision with Stratified Sampling Remove infeasible areas with RealPaver

slide-30
SLIDE 30

SPF Toolchain (with qCORAL)

qCORAL

slide-31
SLIDE 31

Illustrative Example

// 0 <= x,y,z <= 9 f(x,y,z): if x < 5: if y < 3: abort() elif z + y > 10: abort()

Probability that f(x,y,z) calls abort()?

slide-32
SLIDE 32

Illustrative Example

// 0 <= x,y,z <= 9 f(x,y,z): if x < 5: if y < 3: abort() elif z + y > 10: abort()

Probability that f(x,y,z) calls abort()?

pc1: x < 5 && y < 3 pc2: x < 5 && y >= 3 && z + y > 10

slide-33
SLIDE 33

Illustrative Example

qCORAL

//0 <= x,y,z <= 9 pc1: x < 5 && y < 3 pc2: x < 5 && y >= 3 && z + y > 10

slide-34
SLIDE 34

Illustrative Example

x < 5 y < 3 x < 5 y >= 3 z + y > 10 pc1: x < 5 && y < 3 pc2: x < 5 && y >= 3 && z + y > 10

slide-35
SLIDE 35

Illustrative Example

x < 5 y < 3 x < 5 y >= 3 z + y > 10 pc1: x < 5 && y < 3 pc2: x < 5 && y >= 3 && z + y > 10

slide-36
SLIDE 36

Illustrative Example

x < 5 y < 3 x < 5 y >= 3 z + y > 10 pc1: x < 5 && y < 3 pc2: x < 5 && y >= 3 && z + y > 10 x < 5 y < 3 y >= 3 z + y > 10 pt1 pt2 pt3

slide-37
SLIDE 37

Illustrative Example

x < 5 y < 3 x < 5 y >= 3 z + y > 10 pc1: x < 5 && y < 3 pc2: x < 5 && y >= 3 && z + y > 10 x < 5 y < 3 y >= 3 z + y > 10 pt1 pt2 pt3 pt1: E = 0.5001 Var = 0.00008 pt2: E = 0.3000 Var = 0.00003 pt3: E = 0.3806 Var = 0.00009

slide-38
SLIDE 38

Illustrative Example

pc1: x < 5 && y < 3 pc2: x < 5 && y >= 3 && z + y > 10 pt1: E = 0.5001 Var = 0.00008 pt2: E = 0.3000 Var = 0.00003 pt3: E = 0.3806 Var = 0.00009

slide-39
SLIDE 39

Illustrative Example

pc1: x < 5 && y < 3 pc2: x < 5 && y >= 3 && z + y > 10 pt1: E = 0.5001 Var = 0.00008 pt2: E = 0.3000 Var = 0.00003 pt3: E = 0.3806 Var = 0.00009 pc1: E = 0.1501 Var = 0.00013 pc2: E = 0.1927 Var = 0.00022

slide-40
SLIDE 40

Illustrative Example

pc1: x < 5 && y < 3 pc2: x < 5 && y >= 3 && z + y > 10 pt1: E = 0.5001 Var = 0.00008 pt2: E = 0.3000 Var = 0.00003 pt3: E = 0.3806 Var = 0.00009 pc1: E = 0.1501 Var = 0.00013 pc2: E = 0.1927 Var = 0.00022

Estimate: 0.3403 Variance: <= 0.0005

slide-41
SLIDE 41

Estimate: 0.3403 Variance: <= 0.0005

Illustrative Example

qCORAL

//0 <= x,y,z <= 9 pc1: x < 5 && y < 3 pc2: x < 5 && y >= 3 && z + y > 10

slide-42
SLIDE 42

RQ1: qCORAL is competitive with other tools? RQ2: qCORAL features help with complex constraints?

Evaluation

slide-43
SLIDE 43

RQ1: qCORAL is competitive?

VolComp Benchmark (PLDI’13) Techniques/Tools:

➔ Mathematica (NIntegrate) ➔ VolComp ➔ qCORAL

slide-44
SLIDE 44

RQ1: qCORAL is competitive?

VolComp Benchmark (PLDI’13) Techniques/Tools:

➔ Mathematica (NIntegrate) ➔ VolComp ➔ qCORAL Baseline

slide-45
SLIDE 45

RQ1: qCORAL is competitive?

NIntegrate VolComp qCORAL solution bounds

  • avg. est.
  • avg. σ

ARTRIAL 0.9350 [0.9340, 0.9364] 0.9352 1.63e-04 CART 0.9826 [0.9470, 1.0000] 0.9818 1.11e-02 CORONARY 0.0001 [0.0001, 0.0001] 0.0001 4.29e-07 EGFR-EPI 0.1264 [0.1264, 0.1264] 0.1262 3.29e-04 PACK 0.2462 [0.2522, 0.2800] 0.2663 2.72e-05 VOL 1.0005 [0.0000, 1.0000] 1.0001 5.18e-03

slide-46
SLIDE 46

RQ1: qCORAL is competitive?

NIntegrate VolComp qCORAL solution bounds

  • avg. est.
  • avg. σ

ARTRIAL 0.9350 [0.9340, 0.9364] 0.9352 1.63e-04 CART 0.9826 [0.9470, 1.0000] 0.9818 1.11e-02 CORONARY 0.0001 [0.0001, 0.0001] 0.0001 4.29e-07 EGFR-EPI 0.1264 [0.1264, 0.1264] 0.1262 3.29e-04 PACK 0.2462 [0.2522, 0.2800] 0.2663 2.72e-05 VOL 1.0005 [0.0000, 1.0000] 1.0001 5.18e-03

slide-47
SLIDE 47

RQ1: qCORAL is competitive?

NIntegrate VolComp qCORAL solution bounds

  • avg. est.
  • avg. σ

ARTRIAL 0.9350 [0.9340, 0.9364] 0.9352 1.63e-04 CART 0.9826 [0.9470, 1.0000] 0.9818 1.11e-02 CORONARY 0.0001 [0.0001, 0.0001] 0.0001 4.29e-07 EGFR-EPI 0.1264 [0.1264, 0.1264] 0.1262 3.29e-04 PACK 0.2462 [0.2522, 0.2800] 0.2663 2.72e-05 VOL 1.0005 [0.0000, 1.0000] 1.0001 5.18e-03

slide-48
SLIDE 48

RQ1: qCORAL is competitive?

NIntegrate VolComp qCORAL time time

  • avg. time

ARTRIAL 4,179.36 771.10 4.14 CART 7.66 33.74 4.39 CORONARY 0.86 1.99 0.57 EGFR EPI 1.98 0.60 1.61 PACK 5,066.20 104.80 68.79 VOL 1,245.30 3.76 821.11

slide-49
SLIDE 49

RQ1: Observations

qCORAL estimates: ➔ are very close to the results reported by NIntegrate ➔ almost always fall within the VolComp interval

slide-50
SLIDE 50

RQ2: Evaluation

➔ Subjects from the aerospace domain ➔ Picked 70% of the paths to avoid bias ➔ Reported results for 30 executions (avg. estimate and standard error)

slide-51
SLIDE 51

RQ2: Evaluated configurations

qCORAL {} qCORAL {STRAT} qCORAL {STRAT, PCACHE}

+ ICP Stratified Sampling + Partitioning Caching

slide-52
SLIDE 52

RQ2: Subjects Considered

Subject LOC #pcs analyzed (70%) complex functions Apollo ~2,600 5,779 sqrt TSAFE - Conflict ~50 23

cos,pow, sin, sqrt,tan

TSAFE - Turnlogic ~50 225 atan2

slide-53
SLIDE 53

RQ2: Conclusions

Impact of features depends on the subject {STRAT} can reduce variance (x50 in Conflict) ➔ There is a time overhead, however {PCACHE} can reduce time (x2 in Apollo)

➔ Savings increase with number of samples

slide-54
SLIDE 54

(Most Recent) Related Work

Sankaranarayanan et al. (PLDI’13) ➔ Supports only linear constraints Adje et al. (VSTTE’13) ➔ Supports only the four basic arithmetic

  • perations
slide-55
SLIDE 55

Conclusions

qCORAL

New approach to solution space quantification Acceleration procedure improves accuracy More details at pan.cin.ufpe.br/qcoral

slide-56
SLIDE 56

Extra Slides

slide-57
SLIDE 57

Probability of a Target Event P(event) = Sum of the probabilities of the

paths that lead to the event Ratio of the size of solution space by the size of domain

P(path) =

And if the number

  • f paths is infinite?

Bound the symbolic execution and measure the confidence!

(see Filieri et al, ICSE 2013)

slide-58
SLIDE 58

And the Variance?

Use Chebyshev’s inequality: “...at least 1 - 1/k2 of the distribution's values are within k standard deviations of the mean”

slide-59
SLIDE 59

Target application

Sometimes knowing only if an event happens is not very useful!

➔ randomized behavior ➔ probabilistic profile of the environment