Finding Minimal Unsatisfiable Cores of Declarative Specifications - - PowerPoint PPT Presentation

▶
finding minimal unsatisfiable cores of declarative
SMART_READER_LITE
LIVE PREVIEW

Finding Minimal Unsatisfiable Cores of Declarative Specifications - - PowerPoint PPT Presentation

Sep 05, 2023 261 likes •934 views

Finding Minimal Unsatisfiable Cores of Declarative Specifications Emina Torlak, Felix Chang and Daniel Jackson Formal Methods 08 Turku, Finland May 30, 2008 Testing alone is not enough to establish correctness testing: a few cases

slide-1
SLIDE 1

Finding Minimal Unsatisfiable Cores of Declarative Specifications

Emina Torlak, Felix Chang and Daniel Jackson

Formal Methods ’08 · Turku, Finland · May 30, 2008
slide-2
SLIDE 2

Testing alone is not enough to establish correctness …

2 testing: a few cases of arbitrary size Testing can be used to show the presence of bugs, but never to show their absence. (EWD249)

slide-3
SLIDE 3

(s0 ∧ s1 ∧ … ∧ sn) ∧ ¬p

What about more systematic analyses?

3 model finding and bmc: all cases within a small scope theorem proving and model checking: all cases

system property
slide-4
SLIDE 4

(s0 ∧ s1 ∧ … ∧ sn) ∧ ¬p

What about more systematic analyses?

3 model finding and bmc: all cases within a small scope theorem proving and model checking: all cases

system property

if ∧ ¬ is unsatisfiable, is the system correct?

slide-5
SLIDE 5

(s0 ∧ s1 ∧ … ∧ sn) ∧ ¬p

What about more systematic analyses?

3 model finding and bmc: all cases within a small scope theorem proving and model checking: all cases the scope may be too small

slide-6
SLIDE 6

(s0 ∧ s1 ∧ … ∧ sn) ∧ ¬p

What about more systematic analyses?

3 model finding and bmc: all cases within a small scope theorem proving and model checking: all cases the scope may be too small the system may be too constrained

slide-7
SLIDE 7

(s0 ∧ s1 ∧ … ∧ sn) ∧ ¬p

What about more systematic analyses?

3 model finding and bmc: all cases within a small scope theorem proving and model checking: all cases the scope may be too small the system may be too constrained the property may be too weak

slide-8
SLIDE 8

(s0 ∧ s1 ∧ … ∧ sn) ∧ ¬p

What about more systematic analyses?

3 model finding and bmc: all cases within a small scope theorem proving and model checking: all cases the scope may be too small the system may be too constrained the property may be too weak

may succeed for the wrong reasons

slide-9
SLIDE 9

Expected coverage vs. proof coverage

4 user’s expectation of coverage: “most of” F0 … F3 needed to establish F5. expected coverage how to check if the analysis worked as expected?

F0 F1 F2 F3 F4 F5
slide-10
SLIDE 10

Expected coverage vs. proof coverage

4 user’s expectation of coverage: “most of” F0 … F3 needed to establish F5. minimal unsatisfiable core: an unsatisfiable subset of the

  • riginal constraints that

becomes satisfiable if any of its members are removed. expected coverage proof coverage how to check if the analysis worked as expected? show proof coverage!

F0 F1 F2 F3 F4 F5 F0 F1 F2 F3 F4 F5
slide-11
SLIDE 11

Exposing coverage bugs with minimal unsatisfiable cores

5 expected coverage axioms too strong property too weak success! minimal cores

property F0 F1 F2 F3 F4 F5 F0 F1 F2 F3 F4 F5 F0 F1 F2 F3 F4 F5 F0 F1 F2 F3 F4 F5
slide-12
SLIDE 12

Example: minimal cores in Alloy

sig File { } sig Dir { contents: set File + Dir }

  • ne sig Root extends Dir {}

fact hierarchy { no contents.Root File + Dir in Root.*contents all obj: File + Dir | one contents.obj } check { all d: Dir | not d in d.^contents } 6

slide-13
SLIDE 13

Dir Root File

contents contents

Root is a singleton subset of Dir. Dir and File are sets. contents maps directories to files or directories.

Example: minimal cores in Alloy

sig File { } sig Dir { contents: set File + Dir }

  • ne sig Root extends Dir {}

fact hierarchy { no contents.Root File + Dir in Root.*contents all obj: File + Dir | one contents.obj } check { all d: Dir | not d in d.^contents } 6

slide-14
SLIDE 14

✘

Root has no parent.

Example: minimal cores in Alloy

7 sig File { } sig Dir { contents: set File + Dir }

  • ne sig Root extends Dir {}

fact hierarchy { no contents.Root File + Dir in Root.*contents all obj: File + Dir | one contents.obj } check { all d: Dir | not d in d.^contents }

slide-15
SLIDE 15

✘

Everything is reachable from the Root. sig File { } sig Dir { contents: set File + Dir }

  • ne sig Root extends Dir {}

fact hierarchy { no contents.Root File + Dir in Root.*contents all obj: File + Dir | one contents.obj } check { all d: Dir | not d in d.^contents }

Example: minimal cores in Alloy

8

slide-16
SLIDE 16

✘

Every file and directory has one parent.

Example: minimal cores in Alloy

9 sig File { } sig Dir { contents: set File + Dir }

  • ne sig Root extends Dir {}

fact hierarchy { no contents.Root File + Dir in Root.*contents all obj: File + Dir | one contents.obj } check { all d: Dir | not d in d.^contents }

slide-17
SLIDE 17

Check that the containment hierarchy is acyclic.

?

sig File { } sig Dir { contents: set File + Dir }

  • ne sig Root extends Dir {}

fact hierarchy { no contents.Root File + Dir in Root.*contents all obj: File + Dir | one contents.obj } check { all d: Dir | not d in d.^contents }

Example: minimal cores in Alloy

10

slide-18
SLIDE 18

Root has no parent. Every file and directory has one parent. sig File { } sig Dir { contents: set File + Dir }

  • ne sig Root extends Dir {}

fact hierarchy { no contents.Root File + Dir in Root.*contents all obj: File + Dir | one contents.obj } check { all d: Dir | not d in d.^contents }

Example: minimal cores in Alloy

11

slide-19
SLIDE 19

Root has no parent. Every file and directory has one parent. sig File { } sig Dir { contents: set File + Dir }

  • ne sig Root extends Dir {}

fact hierarchy { no contents.Root File + Dir in Root.*contents all obj: File + Dir | one contents.obj } check { all d: Dir | not d in d.^contents }

Example: minimal cores in Alloy

11

axioms too strong

slide-20
SLIDE 20

Root has no parent. Every file and directory has one parent. sig File { } sig Dir { contents: set File + Dir }

  • ne sig Root extends Dir {}

fact hierarchy { no contents.Root File + Dir in Root.*contents all obj: File + Dir | one contents.obj } check { all d: Dir | not d in d.^contents }

Example: minimal cores in Alloy

11

axioms too strong

at most lone

slide-21
SLIDE 21

sig File { } sig Dir { contents: set File + Dir }

  • ne sig Root extends Dir {}

fact hierarchy { no contents.Root File + Dir in Root.*contents all obj: File + Dir | lone contents.obj } check { all d: Dir | not d in d.^contents }

Example: minimal cores in Alloy

12

success!

slide-22
SLIDE 22

A resolution-based analysis framework

13 constraints cnf resolution refutation

F0 F1 F2 F3 F4 F5

translate constraints to a clausal logic and use a resolution engine to determine (un)satisfiability

F0 F1 F2 F3 F4 F5 minimal core constraint irrelevant constraint translation clause resolvent (learned) clause conflict resolution relation translation relation unsatisfiable core
slide-23
SLIDE 23

Minimal cores in a resolution-based analysis framework

14

F0 F1 F2 F3 F4 F5 F0 F1 F2 F3 F4 F5

challenge

how to use the proof at the clause level to find a minimal core at the specification level when

  • clause proof is not minimal
  • minimal clause core may map to a large

specification core constraints cnf resolution refutation

slide-24
SLIDE 24

Minimal cores in a resolution-based analysis framework

14

F0 F1 F2 F3 F4 F5 F0 F1 F2 F3 F4 F5

challenge

how to use the proof at the clause level to find a minimal core at the specification level when

  • clause proof is not minimal
  • minimal clause core may map to a large

specification core constraints cnf resolution refutation

naive core extraction

easy problems
slide-25
SLIDE 25

Minimal cores in a resolution-based analysis framework

14

F0 F1 F2 F3 F4 F5 F0 F1 F2 F3 F4 F5

challenge

how to use the proof at the clause level to find a minimal core at the specification level when

  • clause proof is not minimal
  • minimal clause core may map to a large

specification core constraints cnf resolution refutation

naive core extraction simple core extraction

easy problems medium problems
slide-26
SLIDE 26

Minimal cores in a resolution-based analysis framework

14

F0 F1 F2 F3 F4 F5 F0 F1 F2 F3 F4 F5

challenge

how to use the proof at the clause level to find a minimal core at the specification level when

  • clause proof is not minimal
  • minimal clause core may map to a large

specification core constraints cnf resolution refutation

naive core extraction simple core extraction recycling core extraction

easy problems medium problems hard problems
slide-27
SLIDE 27 K { F0 … Fn } M {} K yes no G solve(K') ! ∈ G no K K' yes K ⊆ M M M ∪ {c} c choose(K \ M) K' K \ {c} F0 F1 F3 F5 F2 F4

Naive core extraction (NCE): prune constraints one at a time

15

slide-28
SLIDE 28 K { F0 … Fn } M {} K yes no G solve(K') ! ∈ G no K K' yes K ⊆ M M M ∪ {c} c choose(K \ M) K' K \ {c} F0 F1 F3 F5 F2 F4

Naive core extraction (NCE): prune constraints one at a time

15 K

slide-29
SLIDE 29 K { F0 … Fn } M {} K yes no G solve(K') ! ∈ G no K K' yes K ⊆ M M M ∪ {c} c choose(K \ M) K' K \ {c} F0 F1 F3 F5 F2 F4

Naive core extraction (NCE): prune constraints one at a time

15 K

slide-30
SLIDE 30 K { F0 … Fn } M {} K yes no G solve(K') ! ∈ G no K K' yes K ⊆ M M M ∪ {c} c choose(K \ M) K' K \ {c} F0 F1 F3 F5 F2 F4

Naive core extraction (NCE): prune constraints one at a time

15 K c

F0

K´

slide-31
SLIDE 31 K { F0 … Fn } M {} K yes no G solve(K') ! ∈ G no K K' yes K ⊆ M M M ∪ {c} c choose(K \ M) K' K \ {c} F0 F1 F3 F5 F2 F4

Naive core extraction (NCE): prune constraints one at a time

15 K c G

F0

K´

slide-32
SLIDE 32 K { F0 … Fn } M {} K yes no G solve(K') ! ∈ G no K K' yes K ⊆ M M M ∪ {c} c choose(K \ M) K' K \ {c} F0 F1 F3 F5 F2 F4

Naive core extraction (NCE): prune constraints one at a time

15 K c G

F0

K´

slide-33
SLIDE 33 K { F0 … Fn } M {} K yes no G solve(K') ! ∈ G no K K' yes K ⊆ M M M ∪ {c} c choose(K \ M) K' K \ {c} F2

Naive core extraction (NCE): prune constraints one at a time

16 K

F0 F1 F3 F5 F4 F0
slide-34
SLIDE 34 K { F0 … Fn } M {} K yes no G solve(K') ! ∈ G no K K' yes K ⊆ M M M ∪ {c} c choose(K \ M) K' K \ {c} F2

Naive core extraction (NCE): prune constraints one at a time

16 K c

F0 F1 F3 F5 F4 F0 F2

K´

slide-35
SLIDE 35 K { F0 … Fn } M {} K yes no G solve(K') ! ∈ G no K K' yes K ⊆ M M M ∪ {c} c choose(K \ M) K' K \ {c} F2

Naive core extraction (NCE): prune constraints one at a time

16 K G c

F0 F1 F3 F5 F4 F0 F2

K´

slide-36
SLIDE 36 K { F0 … Fn } M {} K yes no G solve(K') ! ∈ G no K K' yes K ⊆ M M M ∪ {c} c choose(K \ M) K' K \ {c} F2

Naive core extraction (NCE): prune constraints one at a time

16 K G c

F0 F1 F3 F5 F4 F0 F2

K´

slide-37
SLIDE 37 K { F0 … Fn } M {} K yes no G solve(K') ! ∈ G no K K' yes K ⊆ M M M ∪ {c} c choose(K \ M) K' K \ {c} F2

Naive core extraction (NCE): prune constraints one at a time

16 K G c

F0 F1 F3 F5 F4 F0 F2

K´

slide-38
SLIDE 38 K { F0 … Fn } M {} K yes no G solve(K') ! ∈ G no K K' yes K ⊆ M M M ∪ {c} c choose(K \ M) K' K \ {c} F2

Naive core extraction (NCE): prune constraints one at a time

16 K G

F0 F1 F3 F5 F4 F0 F2
slide-39
SLIDE 39

K´

K { F0 … Fn } M {} K yes no G solve(K') ! ∈ G no K K' yes K ⊆ M M M ∪ {c} c choose(K \ M) K' K \ {c}

From naive to simple core extraction: using the core clauses

17 K G

F0 F1 F3 F5 F4 F0 F2

c

slide-40
SLIDE 40

K´

K { F0 … Fn } M {} K yes no G solve(K') ! ∈ G no K K' yes K ⊆ M M M ∪ {c} c choose(K \ M) K' K \ {c}

From naive to simple core extraction: using the core clauses

17 K G

F0 F1 F3 F5 F4 F0

F4 not needed to prove the unsatisfiability

  • f K´ = F0 ∧ F1 ∧ F3 ∧ F4 ∧ F5
F2

c

slide-41
SLIDE 41 G solve(F0 … Fn) M {} K yes no G' solve(K') ! ∈ G' no G G' yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

Simple core extraction (SCE)

18

F0 F1 F3 F5 F2 F4
slide-42
SLIDE 42 G solve(F0 … Fn) M {} K yes no G' solve(K') ! ∈ G' no G G' yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

Simple core extraction (SCE)

18

F0 F1 F3 F5 F2 F4

G

slide-43
SLIDE 43 G solve(F0 … Fn) M {} K yes no G' solve(K') ! ∈ G' no G G' yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

Simple core extraction (SCE)

18

F0 F1 F3 F5 F2 F4

G K

slide-44
SLIDE 44 G solve(F0 … Fn) M {} K yes no G' solve(K') ! ∈ G' no G G' yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

Simple core extraction (SCE)

18

F0 F1 F3 F5 F2 F4

K G

slide-45
SLIDE 45 G solve(F0 … Fn) M {} K yes no G' solve(K') ! ∈ G' no G G' yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

Simple core extraction (SCE)

18

F0 F1 F3 F5 F2 F4

K G c

F2

K´

slide-46
SLIDE 46 G solve(F0 … Fn) M {} K yes no G' solve(K') ! ∈ G' no G G' yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

Simple core extraction (SCE)

18

F0 F1 F3 F5 F2 F4

K G c

F2

G´ K´

slide-47
SLIDE 47 G solve(F0 … Fn) M {} K yes no G' solve(K') ! ∈ G' no G G' yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

Simple core extraction (SCE)

18

F0 F1 F3 F5 F2 F4

K G c

F2

G´ K´

slide-48
SLIDE 48 G solve(F0 … Fn) M {} K yes no G' solve(K') ! ∈ G' no G G' yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

Simple core extraction (SCE)

18

F0 F1 F3 F5 F2 F4

K c

F2

G K´

slide-49
SLIDE 49 G solve(F0 … Fn) M {} K yes no G' solve(K') ! ∈ G' no G G' yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

Simple core extraction (SCE)

18

F0 F1 F3 F5 F2 F4

K

F2

G

slide-50
SLIDE 50 G solve(F0 … Fn) M {} K yes no G' solve(K') ! ∈ G' no G G' yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

From simple to recycling core extraction: using the resolvents

19 K c

F2

G´ G

F0 F1 F3 F5 F4

K´

slide-51
SLIDE 51 G solve(F0 … Fn) M {} K yes no G' solve(K') ! ∈ G' no G G' yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

From simple to recycling core extraction: using the resolvents

19 K c

F2

G´ G Re-deriving clauses already learned for G.

F0 F1 F3 F5 F4

K´

slide-52
SLIDE 52 G solve(F0 … Fn) M {} K yes no G' solve(learnable(K', G)) ! ∈ G' no G fix(G', G) yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

Recycling core extraction (RCE)

20

F0 F1 F3 F5 F2 F4
slide-53
SLIDE 53 G solve(F0 … Fn) M {} K yes no G' solve(learnable(K', G)) ! ∈ G' no G fix(G', G) yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

Recycling core extraction (RCE)

20

F0 F1 F3 F5 F2 F4

G

slide-54
SLIDE 54 G solve(F0 … Fn) M {} K yes no G' solve(learnable(K', G)) ! ∈ G' no G fix(G', G) yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

Recycling core extraction (RCE)

20

F0 F1 F3 F5 F2 F4

G K

slide-55
SLIDE 55 G solve(F0 … Fn) M {} K yes no G' solve(learnable(K', G)) ! ∈ G' no G fix(G', G) yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

Recycling core extraction (RCE)

20

F0 F1 F3 F5 F2 F4

G K

slide-56
SLIDE 56 G solve(F0 … Fn) M {} K yes no G' solve(learnable(K', G)) ! ∈ G' no G fix(G', G) yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

K´

Recycling core extraction (RCE)

20

F0 F1 F3 F5 F2 F4

G K c

F2
slide-57
SLIDE 57 G solve(F0 … Fn) M {} K yes no G' solve(learnable(K', G)) ! ∈ G' no G fix(G', G) yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

K´

Recycling core extraction (RCE)

20

F0 F1 F3 F5 F2 F4

G K c

F2

learnable(K´, G)

slide-58
SLIDE 58 G solve(F0 … Fn) M {} K yes no G' solve(learnable(K', G)) ! ∈ G' no G fix(G', G) yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

K´

Recycling core extraction (RCE)

20

F0 F1 F3 F5 F2 F4

G K G c

F2

´ Resolvents of G are sources of G´.

slide-59
SLIDE 59 G solve(F0 … Fn) M {} K yes no G' solve(learnable(K', G)) ! ∈ G' no G fix(G', G) yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

K´

Recycling core extraction (RCE)

20

F0 F1 F3 F5 F2 F4

G K G c

F2

´ Resolvents of G are sources of G´.

slide-60
SLIDE 60 G solve(F0 … Fn) M {} K yes no G' solve(learnable(K', G)) ! ∈ G' no G fix(G', G) yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

K´

Recycling core extraction (RCE)

20

F0 F1 F3 F5 F2 F4

G K G c

F2

´ Resolvents of G are sources of G´.

slide-61
SLIDE 61 G solve(F0 … Fn) M {} K yes no G' solve(learnable(K', G)) ! ∈ G' no G fix(G', G) yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

K´

Recycling core extraction (RCE)

20

F0 F1 F3 F5 F2 F4

G K G c

F2

´ fix(G´, G)

slide-62
SLIDE 62 G solve(F0 … Fn) M {} K yes no G' solve(learnable(K', G)) ! ∈ G' no G fix(G', G) yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

K´

Recycling core extraction (RCE)

20

F0 F1 F3 F5 F2 F4

G K c

F2
slide-63
SLIDE 63 G solve(F0 … Fn) M {} K yes no G' solve(learnable(K', G)) ! ∈ G' no G fix(G', G) yes K ⊆ M K sourcesOf(core(G)) M M ∪ {c} c choose(K \ M) K' K \ {c}

Recycling core extraction (RCE)

20

F0 F1 F3 F5 F2 F4

G K

F2
slide-64
SLIDE 64 NUM374 SET943 SET967 TOP020 Trees COM008 RingElection Hotel Lists.empties ALG212 LAT258 Lists.reflexive GEO092 Lists.symmetric SET948 GEO158 GEO159 MED007 MED009 GEO115 GEO091

Simple / Recycling Log Plot

NUM374 SET943 SET967 Trees COM008 Hotel RingElection ALG212 Lists.empties LAT258 Lists.symmetric GEO092 Lists.reflexive SET948 GEO158 GEO159 TOP020 GEO115 GEO091 MED007 MED009

Naive / Recycling Log Plot

Experimental results

21

90.00 29.22 9.49 3.08 0.00 1.48x 2.45x 29.04x 50.00 18.80 7.07 2.66 0.00 1.08x 1.27x 18.41x easy problems medium problems hard problems
slide-65
SLIDE 65

Conclusions and future work

22

M a y 3 , 2 8

kodkod with RCE Alloy4 ConfigAssure

  • faster minimal cores with recycling

core extraction

  • currently used with Alloy4 for

coverage analysis of hand-crafted formal models

  • more speed with new recycling

heuristics and some imprecision

  • use with other applications of

Kodkod such as code checking (Forge) and declarative configuration (configAssure)

slide-66
SLIDE 66

http://web.mit.edu/emina/www/kodkod.html http://alloy.mit.edu/

kodkod.jar SAT binaries javadocs papers examples alloy4.jar tutorial papers courses case studies