Finding Minimal Unsatisfiable Cores of Declarative Specifications - PowerPoint PPT Presentation

Finding Minimal Unsatisfiable Cores of Declarative Specifications Emina Torlak, Felix Chang and Daniel Jackson Formal Methods ’08 · Turku, Finland · May 30, 2008

Testing alone is not enough to establish correctness … testing: a few cases of arbitrary size Testing can be used to show the presence of bugs, but never to show their absence. (EWD249) 2

What about more systematic analyses? (s 0 ∧ s 1 ∧ … ∧ s n ) ∧ ¬p system property theorem proving and model model finding and bmc: all cases checking: all cases within a small scope 3

What about more systematic analyses? (s 0 ∧ s 1 ∧ … ∧ s n ) ∧ ¬p if ∧ ¬ system property is unsatisfiable, is the system correct? theorem proving and model model finding and bmc: all cases checking: all cases within a small scope 3

What about more systematic analyses? (s 0 ∧ s 1 ∧ … ∧ s n ) ∧ ¬p the scope may be too small theorem proving and model model finding and bmc: all cases checking: all cases within a small scope 3

What about more systematic analyses? (s 0 ∧ s 1 ∧ … ∧ s n ) ∧ ¬p the scope may be too small the system may be too constrained theorem proving and model model finding and bmc: all cases checking: all cases within a small scope 3

What about more systematic analyses? (s 0 ∧ s 1 ∧ … ∧ s n ) ∧ ¬p the scope may be too small the system may be too constrained the property may be too weak theorem proving and model model finding and bmc: all cases checking: all cases within a small scope 3

What about more systematic analyses? (s 0 ∧ s 1 ∧ … ∧ s n ) ∧ ¬p the scope may be may succeed for the wrong reasons too small the system may be too constrained the property may be too weak theorem proving and model model finding and bmc: all cases checking: all cases within a small scope 3

Expected coverage vs. proof coverage expected coverage F 0 F 1 user’s expectation of coverage: F 2 “most of” F 0 … F 3 needed to establish F 5 . F 3 F 4 how to check if the analysis worked as F 5 expected? 4

Expected coverage vs. proof coverage expected coverage proof coverage F 0 F 0 F 1 F 1 minimal unsatisfiable core: an user’s expectation of coverage: unsatisfiable subset of the F 2 F 2 “most of” F 0 … F 3 needed to original constraints that establish F 5 . becomes satisfiable if any of its F 3 F 3 members are removed. F 4 F 4 how to check if the analysis worked as show proof coverage! F 5 F 5 expected? 4

Exposing coverage bugs with minimal unsatisfiable cores expected coverage success! axioms too strong property too weak F 0 F 0 F 0 F 0 F 1 F 1 F 1 F 1 F 2 F 2 F 2 F 2 F 3 F 3 F 3 F 3 F 4 F 4 F 4 F 4 F 5 F 5 F 5 F 5 property minimal cores 5

Example: minimal cores in Alloy sig File { } sig Dir { contents: set File + Dir } one sig Root extends Dir {} fact hierarchy { no contents.Root File + Dir in Root.*contents all obj: File + Dir | one contents.obj } check { all d: Dir | not d in d.^contents } 6

Example: minimal cores in Alloy Dir and File are sets. contents sig File { } Dir File contents sig Dir { contents: set File + Dir } one sig Root extends Dir {} contents maps directories to files or directories. Root fact hierarchy { no contents.Root Root is a singleton subset of Dir. File + Dir in Root.*contents all obj: File + Dir | one contents.obj } check { all d: Dir | not d in d.^contents } 6

Example: minimal cores in Alloy sig File { } sig Dir { contents: set File + Dir } ✘ one sig Root extends Dir {} fact hierarchy { no contents.Root Root has no parent. File + Dir in Root.*contents all obj: File + Dir | one contents.obj } check { all d: Dir | not d in d.^contents } 7

Example: minimal cores in Alloy sig File { } sig Dir { contents: set File + Dir } one sig Root extends Dir {} fact hierarchy { ✘ Everything is reachable no contents.Root from the Root. File + Dir in Root.*contents all obj: File + Dir | one contents.obj } check { all d: Dir | not d in d.^contents } 8

Example: minimal cores in Alloy sig File { } sig Dir { contents: set File + Dir } one sig Root extends Dir {} fact hierarchy { ✘ no contents.Root File + Dir in Root.*contents all obj: File + Dir | one contents.obj Every file and directory } has one parent. check { all d: Dir | not d in d.^contents } 9

Example: minimal cores in Alloy sig File { } sig Dir { contents: set File + Dir } one sig Root extends Dir {} fact hierarchy { ? no contents.Root File + Dir in Root.*contents all obj: File + Dir | one contents.obj } check { all d: Dir | not d in d.^contents } Check that the containment hierarchy is acyclic. 10

Example: minimal cores in Alloy sig File { } sig Dir { contents: set File + Dir } one sig Root extends Dir {} fact hierarchy { no contents.Root Root has no parent. File + Dir in Root.*contents all obj: File + Dir | one contents.obj Every file and directory has one parent. } check { all d: Dir | not d in d.^contents } 11

Example: minimal cores in Alloy sig File { } sig Dir { contents: set File + Dir } axioms too strong one sig Root extends Dir {} fact hierarchy { no contents.Root Root has no parent. File + Dir in Root.*contents all obj: File + Dir | one contents.obj Every file and directory has one parent. } check { all d: Dir | not d in d.^contents } 11

Example: minimal cores in Alloy sig File { } sig Dir { contents: set File + Dir } axioms too strong one sig Root extends Dir {} fact hierarchy { no contents.Root Root has no parent. File + Dir in Root.*contents at most all obj: File + Dir | one contents.obj Every file and directory has one parent. } lone check { all d: Dir | not d in d.^contents } 11

Example: minimal cores in Alloy sig File { } sig Dir { contents: set File + Dir } one sig Root extends Dir {} success! fact hierarchy { no contents.Root File + Dir in Root.*contents all obj: File + Dir | lone contents.obj } check { all d: Dir | not d in d.^contents } 12

A resolution-based analysis framework minimal core constraint F 0 F 0 irrelevant constraint F 1 F 1 translation relation translation clause F 2 F 2 resolvent (learned) clause conflict F 3 F 3 resolution relation F 4 F 4 unsatisfiable core F 5 F 5 constraints cnf resolution refutation translate constraints to a clausal logic and use a resolution engine to determine (un)satisfiability 13

Minimal cores in a resolution-based analysis framework challenge F 0 F 0 how to use the proof at the clause level to find F 1 F 1 a minimal core at the specification level when ‣ clause proof is not minimal F 2 F 2 ‣ minimal clause core may map to a large specification core F 3 F 3 F 4 F 4 F 5 F 5 constraints cnf resolution refutation 14

Minimal cores in a resolution-based analysis framework challenge F 0 F 0 how to use the proof at the clause level to find F 1 F 1 a minimal core at the specification level when ‣ clause proof is not minimal F 2 F 2 ‣ minimal clause core may map to a large specification core F 3 F 3 naive core extraction F 4 F 4 F 5 F 5 easy problems constraints cnf resolution refutation 14

Minimal cores in a resolution-based analysis framework challenge F 0 F 0 how to use the proof at the clause level to find F 1 F 1 a minimal core at the specification level when ‣ clause proof is not minimal F 2 F 2 ‣ minimal clause core may map to a large specification core F 3 F 3 naive core extraction simple core extraction F 4 F 4 F 5 F 5 easy problems medium problems constraints cnf resolution refutation 14

Minimal cores in a resolution-based analysis framework challenge F 0 F 0 how to use the proof at the clause level to find F 1 F 1 a minimal core at the specification level when ‣ clause proof is not minimal F 2 F 2 ‣ minimal clause core may map to a large specification core F 3 F 3 naive core extraction simple core extraction recycling core extraction F 4 F 4 F 5 F 5 easy problems medium problems hard problems constraints cnf resolution refutation 14

Naive core extraction (NCE): prune constraints one at a time K � { F 0 … F n } M � {} yes F 0 K ⊆ M K no F 1 c � choose(K \ M) M � M ∪ {c} F 2 K' � K \ {c} F 3 G � solve(K') F 4 no ! ∈ G F 5 yes K � K' 15

Naive core extraction (NCE): prune constraints one at a time K � { F 0 … F n } M � {} K yes F 0 K ⊆ M K no F 1 c � choose(K \ M) M � M ∪ {c} F 2 K' � K \ {c} F 3 G � solve(K') F 4 no ! ∈ G F 5 yes K � K' 15

Naive core extraction (NCE): prune constraints one at a time K � { F 0 … F n } M � {} K yes F 0 K ⊆ M K no F 1 c � choose(K \ M) M � M ∪ {c} F 2 K' � K \ {c} F 3 G � solve(K') F 4 no ! ∈ G F 5 yes K � K' 15

Naive core extraction (NCE): prune constraints one at a time K � { F 0 … F n } M � {} K yes F 0 F 0 c K ⊆ M K no F 1 c � choose(K \ M) M � M ∪ {c} F 2 K' � K \ {c} F 3 G � solve(K') F 4 no ! ∈ G F 5 K´ yes K � K' 15

Naive core extraction (NCE): prune constraints one at a time K � { F 0 … F n } M � {} K yes F 0 F 0 c K ⊆ M K no G F 1 c � choose(K \ M) M � M ∪ {c} F 2 K' � K \ {c} F 3 G � solve(K') F 4 no ! ∈ G F 5 K´ yes K � K' 15

Naive core extraction (NCE): prune constraints one at a time K � { F 0 … F n } M � {} K yes F 0 F 0 c K ⊆ M K no G F 1 c � choose(K \ M) M � M ∪ {c} F 2 K' � K \ {c} F 3 G � solve(K') F 4 no ! ∈ G F 5 K´ yes K � K' 15