GuidedSampler: Coverage-guided Sampling of SMT Solutions Rafael - - PowerPoint PPT Presentation

▶
guidedsampler coverage guided sampling of smt solutions
SMART_READER_LITE
LIVE PREVIEW

GuidedSampler: Coverage-guided Sampling of SMT Solutions Rafael - - PowerPoint PPT Presentation

Jan 05, 2023 326 likes •1.02k views

GuidedSampler: Coverage-guided Sampling of SMT Solutions Rafael Dutra, Jonathan Bachrach, Koushik Sen EECS Department UC Berkeley Formal Methods in Computer-Aided Design October 25, 2019 Constraint Sampling Input: SMT formula A SMT

slide-1
SLIDE 1

GuidedSampler: Coverage-guided Sampling

  • f SMT Solutions

Rafael Dutra, Jonathan Bachrach, Koushik Sen EECS Department UC Berkeley

Formal Methods in Computer-Aided Design October 25, 2019

slide-2
SLIDE 2

A SMT solver can generate one solution:

Constraint Sampling

2

mem[0] mem[1] σ0 1 1 ∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where x = mem[0], y = mem[1], mem’ = store(mem, mem[0], -1 * mem[mem[0]])

Input: SMT formula φ

slide-3
SLIDE 3

Goal: Generate many solutions to φ

Constraint Sampling

3

mem[0] mem[1] σ0 σ1 σ2 σ3 σ4 σ5 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where x = mem[0], y = mem[1], mem’ = store(mem, mem[0], -1 * mem[mem[0]])

Input: SMT formula φ

slide-4
SLIDE 4
  • Synthesis
  • Symbolic execution

Motivation: Sampling Solutions

  • Thoroughly exercising some target functionality
  • Constrained-Random Verification

4

int4 x, y, z, w; int4 mem[4] = {x, y, z, w}; for (int4 i = 0; i < 4; ++i) { mem[mem[i]] *= -1; }

i < 4 mem[0] < 0 ∨ mem[0] ≥ 4

slide-5
SLIDE 5

SMT: Satisfiability Modulo Theories

5

SMT formula φ

∧(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1 * mem[mem[0]]) mem ∈ Array(BV[4], BV[4])

slide-6
SLIDE 6

mem ∈ Array(BV[4], BV[4])

SMT: Satisfiability Modulo Theories

6

SMT formula φ Bit-vector

∧(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1 * mem[mem[0]])

slide-7
SLIDE 7

SMT: Satisfiability Modulo Theories

7

SMT formula φ Bit-vector Array

mem ∈ Array(BV[4], BV[4]) ∧(mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1 * mem[mem[0]])

slide-8
SLIDE 8

State of the art

  • SMTSampler (our prior work)

○ Efficient generation of solutions for SMT formulas

  • Markov Chain Monte Carlo (MCMC)

○ Works for linear constraints and can generate biased solutions

  • Constraint solver heuristics

○ Can be expensive, requiring one solver call per solution

  • Universal hashing

○ Expensive, but can guarantee exact distribution of solution

  • Weighted Sampling

○ Literal-weighted distributions: WAPS

8

slide-9
SLIDE 9

Goal: Generate solutions to φ

SMTSampler

9

mem[0] mem[1] σ0 σ1 σ2 σ3 σ4 σ5 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where x = mem[0], y = mem[1], mem’ = store(mem, mem[0], -1 * mem[mem[0]])

Input: SMT formula φ

slide-10
SLIDE 10

Goal: Generate solutions to φ

Coverage-guided Sampling

10

mem[0] mem[1] σ0 σ1 σ2 σ3 σ4 σ5 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where x = mem[0], y = mem[1], mem’ = store(mem, mem[0], -1 * mem[mem[0]]) mem’[1] < 0

Input: SMT formula φ Input: Coverage predicates

mem’[1] ≥ 4 mem’[0] < 0 ψ1 ψ2 ψ3

slide-11
SLIDE 11

Goal: Generate solutions to φ such that the predicates ψ1, ψ2, …, ψn are covered uniformly

Coverage-guided Sampling

11

mem[0] mem[1] ψ1 ψ2 ψ3 σ0 σ1 σ2 σ3 σ4 σ5 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where x = mem[0], y = mem[1], mem’ = store(mem, mem[0], -1 * mem[mem[0]]) mem’[1] < 0

Input: SMT formula φ

1 1 1 1 1 1 1 1 mem’[1] ≥ 4 mem’[0] < 0

Input: Coverage predicates

ψ1 ψ2 ψ3

slide-12
SLIDE 12

Goal: Generate solutions to φ such that the predicates ψ1, ψ2, …, ψn are covered uniformly

Coverage-guided Sampling

12

mem[0] mem[1] ψ1 ψ2 ψ3 σ0 σ1 σ2 σ3 σ4 σ5 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ∧(x + y = 4 ∧ x ≥ 0 ∧ x < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where x = mem[0], y = mem[1], mem’ = store(mem, mem[0], -1 * mem[mem[0]]) mem’[1] < 0

Input: SMT formula φ

1 1 1 1 1 1 1 1 mem’[1] ≥ 4 mem’[0] < 0

Input: Coverage predicates

ψ1 ψ2 ψ3

slide-13
SLIDE 13

Uniformity over Coverage Classes

13

slide-14
SLIDE 14

Uniformity over Coverage Classes

14

slide-15
SLIDE 15

Uniformity over Coverage Classes

15

slide-16
SLIDE 16

Challenges

  • Coverage of the formula might still not be ideal even using state-of-the-art

approaches, such as SMTSampler

  • User might be interested in a specific notion of coverage for the produced

solutions

16

slide-17
SLIDE 17

GuidedSampler

17

slide-18
SLIDE 18

GuidedSampler

Our goals:

  • Sample solutions from a formula φ, but

have the distribution determined by the coverage predicates ψ1, ψ2, …, ψn

  • Uniformly sample solutions from the

different coverage classes

  • Uniformly sample within each

coverage class Our approach:

  • Compute simple mutations that can be

applied to one solution to generate another solution from a different class

  • Combine those mutations together to

generate a large number of new solutions

18

slide-19
SLIDE 19

19

Formula φ (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) Coverage Predicates ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ψ1 ψ2 ψ3

slide-20
SLIDE 20

20

Formula φ x = mem[0] y = mem[1] 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 Coverage Predicates ψ1 ψ2 ψ3

slide-21
SLIDE 21

21

Formula φ x = mem[0] y = mem[1] 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) ψ1 ψ2 ψ3 1 1 Random Class ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 Coverage Predicates ψ1 ψ2 ψ3

slide-22
SLIDE 22

22

Solution σ 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 MAX-SMT Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) ψ1 ψ2 ψ3 1 1 ψ1 ψ2 ψ3 1 ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 Coverage Predicates ψ1 ψ2 ψ3 Random Class

slide-23
SLIDE 23

23

Solution σ 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 MAX-SMT Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) ψ1 ψ2 ψ3 1 MAX-SMT ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 Coverage Predicates ψ1 ψ2 ψ3

slide-24
SLIDE 24

24

Solution σ 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 MAX-SMT Hard constraints

  • φ
  • ψ1 ≠ 0

Soft constraints

  • ψ2 = 1
  • ψ3 = 0
  • x1 = 0
  • x2 = 0
  • x3 = 0
  • y0 = 1
  • y1 = 0
  • y2 = 0
  • y3 = 0

Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) ψ1 ψ2 ψ3 1 MAX-SMT ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 Coverage Predicates ψ1 ψ2 ψ3

slide-25
SLIDE 25

σ1

25

Solution σ 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 MAX-SMT MAX-SMT Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) ψ1 ψ2 ψ3 1 1 1 ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 Coverage Predicates ψ1 ψ2 ψ3

slide-26
SLIDE 26

σ1

26

Solution σ 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 MAX-SMT Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) ψ1 ψ2 ψ3 1 MAX-SMT ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 Coverage Predicates ψ1 ψ2 ψ3

slide-27
SLIDE 27

σ1

27

Solution σ 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 MAX-SMT Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 Coverage Predicates ψ1 ψ2 ψ3

slide-28
SLIDE 28

σ1

28

Solution σ 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 σ2 MAX-SMT Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 Coverage Predicates ψ1 ψ2 ψ3

slide-29
SLIDE 29

σ1

29

Solution σ 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 σ2 MAX-SMT Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 Coverage Predicates ψ1 ψ2 ψ3

slide-30
SLIDE 30

σ1

30

Solution σ 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 σ2 MAX-SMT Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 Coverage Predicates ψ1 ψ2 ψ3

slide-31
SLIDE 31

σ1

31

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 σ2 σ3 MAX-SMT Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 1 Coverage Predicates ψ1 ψ2 ψ3

slide-32
SLIDE 32

σ1

32

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 σ2 σ3 MAX-SMT Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ... Coverage Predicates ψ1 ψ2 ψ3

slide-33
SLIDE 33

σ1

33

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 1 1 σ2 σ3 δ3 1 1 δ2 δ1 MAX-SMT = σ ⊕ σ1 = σ ⊕ σ2 = σ ⊕ σ3 Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ... Coverage Predicates ψ1 ψ2 ψ3

slide-34
SLIDE 34

δ12 σ1

34

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 1 1 σ2 σ3 δ3 1 1 δ2 δ1 1 1 1 1 MAX-SMT = δ1 ∨ δ2 Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ... Coverage Predicates ψ1 ψ2 ψ3

slide-35
SLIDE 35

σ12 δ12 σ1

35

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 1 1 σ2 σ3 δ3 1 1 δ2 δ1 1 1 1 1 1 1 1 1 MAX-SMT = σ ⊕ δ12 Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ψ1 ψ2 ψ3 1 ... Coverage Predicates ψ1 ψ2 ψ3

slide-36
SLIDE 36

σ12 δ12 σ1

36

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 1 1 σ2 σ3 δ3 1 1 δ2 δ1 1 1 1 1 1 1 1 1 MAX-SMT = σ ⊕ δ12 Why does it work?

  • δ1 and δ2 are a minimal set of bits that can be

flipped and preserve the satisfiability of the formula

  • It’s likely that the formula has some clauses

establishing a relation between those bits

  • Those clauses will likely still be satisfied when

flipping both the bits in δ1 and δ2 Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ψ1 ψ2 ψ3 1 ... Coverage Predicates ψ1 ψ2 ψ3

slide-37
SLIDE 37

σ12 δ12 σ1

37

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 1 1 σ2 σ3 δ3 1 1 δ2 δ1 1 1 1 1 1 1 1 1 MAX-SMT = σ ⊕ δ12 ⇐ And new sample σ12 is likely from a new coverage class Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ψ1 ψ2 ψ3 1 ... Coverage Predicates ψ1 ψ2 ψ3

slide-38
SLIDE 38

σ12 δ12 σ1

38

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 1 1 σ2 σ3 δ3 1 1 δ2 δ1 1 1 1 1 1 1 1 1 MAX-SMT = σ ⊕ δ12 Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ψ1 ψ2 ψ3 1 ... Coverage Predicates ψ1 ψ2 ψ3

slide-39
SLIDE 39

σ12 δ12 σ1

39

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 1 1 σ2 σ3 δ3 1 1 δ2 δ1 1 1 1 1 1 1 1 1 MAX-SMT Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) δ13 1 1 1 1 = δ1 ∨ δ3 MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ... Coverage Predicates ψ1 ψ2 ψ3

slide-40
SLIDE 40

σ12 δ12 δ13 σ13 σ1

40

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 1 1 1 1 1 1 1 1 σ2 σ3 δ3 1 1 δ2 δ1 1 1 1 1 1 1 1 1 MAX-SMT = σ ⊕ δ13 Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ψ1 ψ2 ψ3 1 1 ... Coverage Predicates ψ1 ψ2 ψ3

slide-41
SLIDE 41

σ12 δ12 δ13 δ23 σ13 σ1

41

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 1 1 1 1 1 1 1 1 σ2 σ3 δ3 1 1 δ2 δ1 1 1 1 1 1 1 1 1 1 1 1 1 MAX-SMT = δ2 ∨ δ3 Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ... Coverage Predicates ψ1 ψ2 ψ3

slide-42
SLIDE 42

σ12 δ12 δ13 δ23 σ13 σ1

42

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 1 1 1 1 1 1 1 1 σ2 σ3 δ3 1 1 δ2 δ1 1 1 1 1 1 1 1 1 1 1 1 1 MAX-SMT σ23 1 1 1 1 = σ ⊕ δ23 Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ψ1 ψ2 ψ3 1 ... Coverage Predicates ψ1 ψ2 ψ3

slide-43
SLIDE 43

σ12 σ13 σ1

43

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 σ2 σ3 1 1 1 1 MAX-SMT σ23 1 1 1 1 Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ... Coverage Predicates ψ1 ψ2 ψ3

slide-44
SLIDE 44

σ12 σ13 σ1

44

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 σ2 σ3 1 1 1 1 MAX-SMT σ23 1 1 1 1 17 / 18 valid solutions Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ... Coverage Predicates ψ1 ψ2 ψ3

slide-45
SLIDE 45

σ12 σ13 σ1

45

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 σ2 σ3 1 1 1 1 MAX-SMT σ23 1 1 1 1 σ123 1 1 1 1 Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ψ1 ψ2 ψ3 1 1 ... Coverage Predicates ψ1 ψ2 ψ3

slide-46
SLIDE 46

σ12 σ13 σ1

46

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 σ2 σ3 1 1 1 1 MAX-SMT σ23 1 1 1 1 σ123 1 1 1 1 Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ψ1 ψ2 ψ3 1 1 Repeated class ... Coverage Predicates ψ1 ψ2 ψ3

slide-47
SLIDE 47

σ12 σ13 σ1

47

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 σ2 σ3 1 1 1 1 MAX-SMT σ23 1 1 1 1 Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ... Coverage Predicates ψ1 ψ2 ψ3

slide-48
SLIDE 48

σ12 σ13 σ1

48

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 σ2 σ3 1 1 1 1 MAX-SMT σ23 1 1 1 1 Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) At most n generate atomic mutations =O(n6) mutations: NO MAX-SMT

( )

n 6

samples by combining MAX-SMT calls to MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ... Coverage Predicates ψ1 ψ2 ψ3

slide-49
SLIDE 49

σ12 σ13 σ1

49

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 σ2 σ3 1 1 1 1 MAX-SMT σ23 1 1 1 1 Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) At most 50 generate atomic mutations 15 890 700 mutations: NO MAX-SMT samples by combining MAX-SMT calls to MAX-SMT ψ1 ψ2 ψ3 1 ... ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ... Coverage Predicates ψ1 ψ2 ψ3

slide-50
SLIDE 50

σ12 σ13 σ1

50

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 σ2 σ3 1 1 1 1 MAX-SMT σ23 1 1 1 1 Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 ... Coverage Predicates ψ1 ψ2 ψ3

slide-51
SLIDE 51

... mem’[1] < 0 mem’[1] ≥ 4 mem’[0] < 0 σ12 σ13 σ1

51

Solution σ 1 1 1 1 1 1 1 1 1 1 1 1 1 Random assignment σ’ x0 x1 x2 x3 y0 y1 y2 y3 1 1 σ2 σ3 1 1 1 1 MAX-SMT σ23 1 1 1 1 Formula φ x = mem[0] y = mem[1] (mem[0] ≥ 0 ∧ mem[0] < 4) ∧ (mem’[1] < 0 ∨ mem’[1] ≥ 4), where mem’ = store(mem, mem[0], -1* mem[mem[0]]) MAX-SMT ψ1 ψ2 ψ3 1 ... Coverage Predicates ψ1 ψ2 ψ3

slide-52
SLIDE 52

Random assignment σ′

52

Key Ideas

slide-53
SLIDE 53

σ Random assignment Base solution σ′

53

Key Ideas

  • M3: In the MAX-SMT query

to generate σ, set coverage predicates to random values

slide-54
SLIDE 54

σ2 σ1 σ Random assignment Base solution Closest solutions σ′

54

σ3 σ4

Key Ideas

  • M3: In the MAX-SMT query

to generate σ, set coverage predicates to random values

  • M1: Find neighboring

solutions that flip coverage predicates

slide-55
SLIDE 55

Random assignment Base solution Closest solutions Generated samples σ2 σ12 σ1 σ σ′

55

σ3 σ4

Key Ideas

  • M3: In the MAX-SMT query

to generate σ, set coverage predicates to random values

  • M1: Find neighboring

solutions that flip coverage predicates

  • M2: Whenever generating

a new sample, discard it if it’s from a repeated coverage class

slide-56
SLIDE 56

σ2 σ12 σ1 σ Random assignment Base solution Closest solutions Generated samples σ′

56

σ3 σ4

Key Ideas

  • M3: In the MAX-SMT query

to generate σ, set coverage predicates to random values

  • M1: Find neighboring

solutions that flip coverage predicates

  • M2: Whenever generating

a new sample, discard it if it’s from a repeated coverage class

slide-57
SLIDE 57

σ2 σ12 σ1 σ Random assignment Base solution Closest solutions Generated samples σ′

57

σ3 σ4

Key Ideas

  • M3: In the MAX-SMT query

to generate σ, set coverage predicates to random values

  • M1: Find neighboring

solutions that flip coverage predicates

  • M2: Whenever generating

a new sample, discard it if it’s from a repeated coverage class

slide-58
SLIDE 58

Implementation

  • Implemented in C++ using Z3 as the constraint solver
  • https://github.com/RafaelTupynamba/GuidedSampler

58

slide-59
SLIDE 59

Experiments on SMT-LIB

We evaluated GuidedSampler on 213 industrial benchmarks from 22 classes.

59

Benchmark Class Average # Nodes Average # Bits QF_AUFBV/ecc 179 1931 QF_ABV/bmc-arrays 855 53 QF_ABV/stp_samples 1139 192 QF_BV/bmc-bv-svcomp14 7518 7607 QF_BV/tacas07 8812 16620 QF_BV/sage/app8 978 1047

slide-60
SLIDE 60

Experiments

We compared 6 approaches for SMT sampling:

  • BH: Baseline with hard constraints
  • BS: Baseline with soft constraints
  • S0: SMTSampler [1]
  • S1 = S0 + M1 (flipping coverage predicates to generate neighboring solutions)
  • S2 = S0 + M1 + M2 (discarding solutions from repeated classes)
  • S3 = S0 + M1 + M2 + M3: GuidedSampler (randomize class of base solution)

[1] Rafael Dutra, Jonathan Bachrach and Koushik Sen. 2018. SMTSampler: Efficient Stimulus Generation from Complex SMT Constraints. In ICCAD 2018.

60

slide-61
SLIDE 61

Coverage Predicates

  • Internal Predicates

○ Look at values of internal nodes ○ Analogous to internal wires in a circuit ○ General notion of coverage from the formula itself

  • Random Predicates

○ Random formulas generated from a grammar including variables of φ ○ Problem-specific notion of coverage

61

slide-62
SLIDE 62

Experiments: Unique Coverage Classes

62

Higher is better GuidedSampler vs. BH Baseline Number of unique coverage classes per time

slide-63
SLIDE 63

Experiments: Unique Coverage Classes

63

Higher is better GuidedSampler vs. BS Baseline Number of unique coverage classes per time

slide-64
SLIDE 64

Experiments: Unique Coverage Classes

64

Higher is better GuidedSampler vs. SMTSampler Number of unique coverage classes per time

slide-65
SLIDE 65

Experiments: Unique Coverage Classes

65

Higher is better S3 = GuidedSampler S0 = SMTSampler BS, BH: baselines Number of unique coverage classes per time

slide-66
SLIDE 66

Experiments: Uniformity over Coverage Classes

66

S3 = GuidedSampler S0 = SMTSampler BS, BH: baselines

→ GuidedSampler generated > 100 000 classes

slide-67
SLIDE 67

Discussion

  • The most important modification is

M1, which allows covering 3.1 times more classes in average

  • M1 and M2 are also essential for

producing a more uniform distribution over coverage classes.

  • Similar results for internal predicates

and random predicates

67

Modifications:

  • M1: Flipping coverage predicates to

compute neighboring solutions

  • M2: Discarding new solutions that

repeat a previously seen coverage class

  • M3: Randomizing coverage class of

initial base solution

slide-68
SLIDE 68

Conclusion

  • Generating lots of solutions

efficiently given an SMT formula

  • Generate millions of

solutions with tens of solver calls

  • Achieve better coverage of

the constraint space, even for user-defined coverage classes

68

σ12 = σ ⊕ δ12 δ12 = δ1 ∨ δ2 σ1 Solution σ 1 1 1 1 1 1 1 1 1 1 σ2 1 1 δ2 δ1 1 1 1 1 1 1 1 1 ψ1 ψ2 ψ3 1 ψ1 ψ2 ψ3 1 MAX-SMT x0 x1 x2 x3 y0 y1 y2 y3