Random Testing and Model Checking: Building a Common Framework for - PowerPoint PPT Presentation

Random Testing and Model Checking: Building a Common Framework for Nondeterministic Exploration Jet Propulsion Laboratory, California Institute of Technology Alex Groce and Rajeev Joshi WODA 2008 July 21, 2008 1

Background & Motivation � LaRS (Laboratory for Reliable Software) at JPL has been building, verifying, and testing flash file systems for space mission use � This work grows out of that experience WODA 2008 July 21, 2008 2

Background & Motivation � MSAP • Two flash file systems, one RAM file system, one critical parameter storage module • Approach: random testing [ICSE’07,ASE’08] � MSL (Mars Science Laboratory) • One flash file system, one RAM file system, one low-level flash interface (critical parameter storage) • Approach: model checking/random testing WODA 2008 July 21, 2008 3

Random Testing � I think we all know what random testing is: • Operations and parameters generated at random to test a program • Possibly with some bias or feedback to help with the problem of irrelevant/redundant operations WODA 2008 July 21, 2008 4

Model Checking and Dynamic Analysis � (Software) model checking • (In principle exhaustive) exploration of a program’s state space � Dynamic analysis (what we’re here for today) • Analysis of a running program • Usually instrumentation or execution in virtual environment – e.g. Valgrind, Daikon • Testing is a dynamic analysis : program is executed in order to learn about its behaviors • We’re looking at the kind of model checking that is essentially a dynamic analysis WODA 2008 July 21, 2008 5

Many Software Model Checkers BLAST SPIN CRunner CMC CBMC JPF2 SLAM MAGIC Bogor VeriSoft WODA 2008 July 21, 2008 6

Two Approaches Execution of actual code (dynamic: like testing) BLAST SPIN CRunner CRunner CMC CBMC JPF2 Our focus in this talk SLAM MAGIC Bogor VeriSoft Analysis of derived transition system (“static”) WODA 2008 July 21, 2008 7

Model Checking as State-Based Testing � Model-checking by executing the program • Backtracking search for all states m m kdi r / a kdi r / a Will explore, as a side-effect, many executions (like random testing) m m kdi r / b kdi r / b but the goal is to explore states m m kdi r / c kdi r / c r m r m di r / a di r / a m m kdi r / a kdi r / a CFG State already visited! Backtrack and try a Done with test! different operation State already visited! Backtrack and try a Backtrack and try a different operation different operation WODA 2008 July 21, 2008 8

SPIN and Model-Driven Verification � SPIN compiles a PROMELA model into a C program: it’s a model checker generator • Embed C code in transitions by executing the compiled C code • Take advantage of all SPIN features – hashing, multicore exploration, etc. � Requires the ability to restore a running program to an earlier execution state • Difficult engineering problem, handled by CIL- based automatic code instrumentation [VMCAI’08] WODA 2008 July 21, 2008 9

SPIN and Model-Driven Verification Execute C code � When SPIN backtracks, until control returns to SPIN it uses information on how to restore the state Push tracked & matched of the C program: state on stack • Tracked memory is restored on backtrack Has state been • Matched memory is also visited before? used to determine if a N Y state has been visited Backtrack: before Store matched pop stack & state in state restore tracked & table matched state WODA 2008 July 21, 2008 10

SPIN and Model-Driven Verification Execute C code � (Unsound) abstraction until control returns to SPIN by matching on an abstraction of the Push tracked & matched tracked concrete state state on stack • E.g. track the pointers/contents of a Has state been linked list visited before? • Match on a sorted array N Y copy only (if order Backtrack: doesn’t matter for Store matched pop stack & state in state property in question) restore tracked & table matched state WODA 2008 July 21, 2008 11

A Common Goal � Program state spaces are typically too large to explore fully even after (unsound) abstraction � Random testing and model checking are both methods for nondeterministically exploring a program’s state space • A series of random walks • vs. systematic exploration with backtracking WODA 2008 July 21, 2008 12

Which is Better? � Conventional wisdom (exaggerated): • Random testing is probably less effective than model checking • BUT model checking is much more difficult to apply than random testing, scales poorly, crashes a lot, makes your ears bleed, and may cause temporary paralysis Test engineer using a model checker on a C program? WODA 2008 July 21, 2008 13

How True is the Conventional Wisdom? � Realistically, the state spaces for real programs are huge • Model checking will almost certainly use unsound abstractions, and still be only partial exploration • Systematically missing some states that could expose errors • Are we sure this is better than smart random testing for fault detection / coverage? WODA 2008 July 21, 2008 14

How True is the Conventional Wisdom? � On the other hand, explicit-state model checking is not that difficult to apply • PROMELA is a nice language for expressing nondeterministic choice & test structure • Provides test-case playback, minimization, and other things often build by hand for testing • Scales quite well if memory usage is (a) limited (no 5GB memory footprint) and (b) well-defined • Often true for embedded systems WODA 2008 July 21, 2008 15

Using SPIN for True Random Testing � Want to apply both methods • For research purposes (comparison) • Due diligence in testing! This stuff is going to Mars… � But why write two testers? – one for random testing, one for model checking • Basic harness looks the same, property checks look the same, etc. • Annoying redundant work , better to spend time improving the harness or running more tests WODA 2008 July 21, 2008 16

A Quick Primer: Using SPIN for Random Testing, in Five Slides OR Almost All the PROMELA You Ever Need to Know WODA 2008 July 21, 2008 17

Simple PROMELA Code i nt i nt x; x; i nt i nt y; y; act i ve p t i ve pr oct ype m r oct ype m ai n ( ai n ( ) { { Start simple 1 This model has 7 states i f i f 2 : : : : x = 1 x = 1 What are they? : : : : x = 2 x = 2 3 f i ; f i ; State = (PC, x, y) 5 asser t ( ser t ( x = == y y) ; 7 } SPIN’s nondeterministic choice construct Picks any one of the choices that is enabled Not mutually exclusive! i f i f How do we guard a choice? : : : : ( x < 10) ( x < 10) - > y = 1 - > y = 1 : : : : ( x < 5) ( x < 5) - > - > y = 3 y = 3 : : : : ( x > 1) ( x > 1) - > - > y = 4 y = 4 f i ; f i ; WODA 2008 July 21, 2008 18

Simple PROMELA Code i nt x; i nt x; i nt y; i nt y; act i ve pr oct ype m act i ve pr oct ype m ai n ( ) { ai n ( ) i f i f 1 : : x = : : x = 1 This model has 17 states 2 : : x = : : x = 2 3 f i ; f i ; 5 i f i f What are they? : : y = : : y = 1 7 : : y = 2 : : y = State = (PC, x, y) 9 f i ; f i ; 13 i f i f Er… : : x > y - > x = y : : x > y - > x = y 14 : : y > x - > y = x : : y > x - > y = x 15 : : el se - > ski p : : el se - > ski p 17 f i ; f i ; Don’t worry about state-counting too asser t asser t ( x == y) ; ( x == y) ; much – SPIN has various automatic reductions and atomicity choices that } can make that difficult WODA 2008 July 21, 2008 19

Simple PROMELA Code i nt i nt x; x; act i ve p t i ve pr oct ype m r oct ype m ai n ( ai n ( ) { { Only a couple more PROMELA x = 0; x = 0; constructs to learn for building test harnesses: the do do loop do do : : ( x < : : ( x < 10) - > x++ 10) - > x++ : : br eak : : br eak Like i f i f , except it introduces a loop to the top – br eak br eak choice od od can exit the loop / * H / * Her e, x er e, x i s a anyt hi ng b nyt hi ng bet ween et ween This nondeterministically assigns x 0 a and nd 9 i i ncl usi ve ncl usi ve * / a value in the range 0…9 WODA 2008 July 21, 2008 20

Simple PROMELA Code i nl i ne p l i ne pi ck ( i ck ( var , M var , M AX) AX) i nl i ne gives us a macro facility i nl i ne var var = = 0; 0; do do As you can imagine, this is a : : ( var < : : ( var < M M AX) - > var ++ AX) - > var ++ useful macro for building a test harness! : : br eak : : br eak od od WODA 2008 July 21, 2008 21