A Storm is Coming! A New Probabilistic Model Checker Joost-Pieter - - PowerPoint PPT Presentation

A Storm is Coming!

A New Probabilistic Model Checker

Joost-Pieter Katoen

IFIP WG 1.8 Open Problems in Concurrency Theory, June 26, 2017

joint with: Christian Dehnert, Sebastian Junges and Matthias Volk

Probabilistic Model Checking

2

“A promising new direction in formal methods research these days is the development of probabilistic models, with associated tools for quantitative evaluation of system performance along with correctness”

Theory in practice for system design and verification. ACM SIGLOG News, 3, 2015 The birth of model checking. 2008

“probabilistic model checking is

• ne of the main challenges for the future”

Probabilistic Model Checking

3

First approaches soon after the birth of model checking

mostly focused on almost-sure events

Second generation focused on quantitative properties

• mega-regular events, probabilistic CTL

Since the early 2000s, powerful model checkers exist Bright future: many application areas, e.g. robotics

Probabilistic Model Checkers

4

Model Checkers: PRISM MRMC LiQuor iscasMC MoDeST MARCIE GreatSPN IMCA PASS PARAM ……… Applications:

Reliability Engineering Performance Evaluation Dependability Analysis Systems Biology Robotics Software Engineering Model Repair ……… plus all statistical “model checkers”

HVC Award 2016

• M. Kwiatkowska

D.Parker

• G. Norman

> 10,000 downloads New: the STORM model checker

This Talk

• B. STORM’s performance compared to PRISM
• A. The ins and outs of the STORM model checker
• C. STORM’s performance compared to other competitors

5

• D. STORM’s support for Markov automata with multiple objectives

STORM’s Characteristics

6

It supports several native input languages Models: Markov chains and MDPs, and Markov automata Supports explicit state, fully symbolic and hybrid engines It has a modular set-up: easy exchange of solvers

currently: 15 solvers, CUDD and Sylvan

STORM’s Characteristics

7

Supports a Python interface for rapid prototyping Hosts many functionalities under a single roof:

(high-level) counterexample synthesis permissive scheduler synthesis conditional probabilities and rewards game-based abstraction of infinite-state MDPs long-run averages on MDPs

Mostly faster than all competitors

STORM’s Architecture

8

STORM comprises about 100,000 C++ code lines

STORM’s Solvers

9

STORM’s Input Languages — Probabilistic Programs

10

Cccp : cp := [0, . . . , 0]; i := 1; x := N; while (x > 0) { while (cp[i] 6= 0) {i :⇡ Unif[1 . . . N]}; cp[i] := 1; x := x1 }

Programs in probabilistic GCL + observe-statements
 Automated abstraction techniques for unbounded variables
 Used in security, machine learning, AI, etc.

STORM’s Input Languages — Dynamic Fault Trees

11

Dugan’s DFTs with, p-FDEPs, nested SPAREs
 Tailored state-space generation and reduction techniques
 One of the—if not the — most prominent models in reliability engineering

STORM’s Input Languages — Generalised Stochastic Petri Nets

12

Petri Nets with “exponential” 
 and “immediate” transitions Supports confused GSPNs 
 One of the—if not the — most 
 prominent models in performance 
 and dependability analysis

n

• t

i n M A R C I E , G r e a t S P N , S m a r t e t c .

This Talk

• B. STORM’s performance compared to PRISM
• A. The ins and outs of the STORM model checker
• C. STORM’s performance compared to other competitors

13

• D. STORM’s support for Markov automata with multiple objectives

Performance Comparison with PRISM

14

Compare best engines (left) and exact arithmetic engines (right)

Performance Comparison with PRISM

15

All PRISM benchmark models with all 84 properties Compare engines that are conceptually similar
 8-core proc (2.0 GHz) with 8GB RAM; timeout = 1800 s

This Talk

• B. STORM’s performance compared to PRISM
• A. The ins and outs of the STORM model checker
• C. STORM’s performance compared to other competitors

16

• D. STORM’s support for Markov automata with multiple objectives

How Many Problems Can be Solved in Time?

17

Compare best engines of EPMC, PRISM and STORM

Markov Automata

18

STORM’s Performance on Markov Automata

19

Compare IMCA against STORM (sparse) on all IMCA models Reachability, expected rewards, and long-run rewards Time-bounded and reward-bounded reachability

No STORM Engine Prevails

20

STORM’s DFT State Space Generation

21

Monolithic state-space generation Don’t care propagation, symmetry and partial-order reduction Modularisation: analyse independent sub-DFTs separately

Performance of DFT Analysis

22

Performance of Parameter Synthesis

23

Comparison to PARAM and PRISM on parametric Markov chains

This Talk

• B. STORM’s performance compared to PRISM
• A. The ins and outs of the STORM model checker
• C. STORM’s performance compared to other competitors

24

• D. STORM’s support for Markov automata with multiple objectives

Outlook: Markov Automata with Multiple Objectives

25

Stochastic job-shop scheduling schedule n jobs on k machines under pre-emptive scheduling each job has an exponential duration LEPT scheduling optimal to minimise expected completion time How to schedule if multiple constraints are imposed? the expected completion time of all jobs below a threshold and finish 50% of all jobs quickly too. Trade-off! Pareto

STORM’s Performance

26

STORM’s Performance Comparison

27

Take-Home Message

28

STORM is modular. STORM is fast(er). https://stormchecker.org/ STORM is extendible.