Generating Good Generators for Inductive Relations POPL 2018 - - PowerPoint PPT Presentation

▶

Mar 06, 2023 156 likes •506 views

Generating Good Generators for Inductive Relations POPL 2018 Leonidas Lampropoulos 1 Zoe Paraskevopoulou 2 Benjamin Pierce 1 1 University of Pennsylvania 2 Princeton University Generating Good Generators for Inductive Relations in a

SLIDE 1

Generating Good Generators for Inductive Relations

POPL 2018 Leonidas Lampropoulos1 Zoe Paraskevopoulou2 Benjamin Pierce1

1University of Pennsylvania 2Princeton University

SLIDE 2

Generating Good Generators for Inductive Relations in a property-based random testing tool for Coq

1 / 21

SLIDE 3

Testing with QuickChick

∀(x : A). P(x)

P(a)? More confidence! Counterexample!

✗

2 / 21

SLIDE 4

Testing with QuickChick

∀(x : A).Q(x) ⇒ P(x)

Q(a)? P(a)? More confidence! Counterexample!

✗ ✓ ✗

3 / 21

SLIDE 5

Sparse Preconditions Q

4 / 21

SLIDE 6

∀ → →

Random data generator for
Type inference function
as a function
Decidability for

5 / 21

SLIDE 7

∀ → →

More confidence?

6 / 21

SLIDE 8

λ N

λ (N→N)→N.x6 λ N λ N

λ N

λ (N→N)→N λ N

λ N

λ N λ N λ (N→N)→(N→N)λ N λ N→N λ N λ N→N

λ N
λ Nλ Nλ N→N

7 / 21

SLIDE 9

λ N

λ (N→N)→N.x6 λ N λ N

λ N

λ (N→N)→N λ N

λ N

λ N λ N λ (N→N)→(N→N)λ N λ N→N λ N λ N→N

λ N
λ Nλ Nλ N→N

7 / 21

SLIDE 10

Testing with Good Generators

∀(x : A).Q(x) ⇒ P(x)

P(a)? More confidence! Counterexample!

such that Q() ✓ ✗

8 / 21

SLIDE 11

Q

Counterexample

9 / 21

SLIDE 12

∀(x : A).Q(x) ⇒ P(x)

is good if

Soundness x ∈ range(gen) ⇒ Q(x)

and

Completeness x ∈ Q(x) ⇒ range(gen)

10 / 21

SLIDE 13

Generate only well-typed terms! → → such that e ∈ range(gen_term Γ t) ⇒ Γ ⊢ e : t Make sure that all of them can be generated! Γ ⊢ e : t ⇒ e ∈ range(gen_term Γ t)

11 / 21

SLIDE 14

←

← ←

Testing an Optimising Compiler by Generating Random Lambda Terms.

Michal H. Palka, Koen Claessen, Alejandro Russo, and John Hughes. AST ’11

12 / 21

SLIDE 15

←

← ←

Testing an Optimising Compiler by Generating Random Lambda Terms.

Michal H. Palka, Koen Claessen, Alejandro Russo, and John Hughes. AST ’11

12 / 21

SLIDE 16

Generating Good Generators

1 →2 → 2 1 → and ∀ a1, range(gen_A2 a1) ≡ {a2 | Q a1 a2}

13 / 21

SLIDE 17

Generating Good Generators

→→→ →→

14 / 21

SLIDE 18

Γ :=

15 / 21

SLIDE 19

Γ :=

∀ Γ Γ →

Γ

15 / 21

SLIDE 20

Γ :=

∀ Γ Γ →

Γ

15 / 21

SLIDE 21

Γ :=

∀ Γ Γ →

Γ

15 / 21

SLIDE 22

Γ := ∀ Γ Γ → Γ

15 / 21

SLIDE 23

Γ := ∀ Γ Γ → Γ

15 / 21

SLIDE 24

Γ := ← Γ ∀ Γ Γ → Γ

15 / 21

SLIDE 25

∀ → →

16 / 21

SLIDE 26

Generating Provably Good Generators

∀Γt, range(genterm Γ t) ≡ {e | Γ ⊢ e : t}

17 / 21

SLIDE 27

Generating Provably Good Generators

∀Γt, range(genterm Γ t) ≡ {e | Γ ⊢ e : t}

17 / 21

SLIDE 28

Generating Provably Good Generators

∀Γt, range(genterm Γ t) ≡ {e | Γ ⊢ e : t}

17 / 21

SLIDE 29

Evaluation

Is the class of inductive definitions large/general/useful? Are the generators efficient? Do they achieve good coverage and distribution of test cases?

18 / 21

SLIDE 30

Evaluation

Applicability

Tested specifications from textbook
83% of suitable-for-testing theorems could be tested with our approach

19 / 21

SLIDE 31

Evaluation

Applicability

Tested specifications from textbook
83% of suitable-for-testing theorems could be tested with our approach

19 / 21

SLIDE 32

Evaluation

Applicability

Tested specifications from textbook
83% of suitable-for-testing theorems could be tested with our approach

∀ → →

19 / 21

SLIDE 33

Evaluation

Performance

Compared to handwritten generators used in IFC case study by Hritcu et al.

(2013, 2016)

1.75× slower that handwritten generators
Same bug-finding performance (counterexamples/sec)

20 / 21

SLIDE 34

Conclusion

Sound and complete generators for inductive relations for free! What’s next?

larger class of inductive definitions
derive decidability instances
derive shrinkers

Find us on GitHub!

21 / 21