Sled gehammer Hell The Day after Jud gment Jasmin C. Blanchette TU - PowerPoint PPT Presentation

Sled gehammer Hell The Day after Jud gment Jasmin C. Blanchette TU München

Larry Paulson Jia Meng Kong Susanto Claire Quigley Markus Wenzel Fabian Immler Philipp Meyer Sascha Böhme

Sledgehammer

Sledgehammer Relevance filter

Sledgehammer Relevance filter ATP translation

Sledgehammer Relevance filter ATP translation E SPASS Vampire SInE Metis Metis Metis Metis proof proof proof proof

Sledgehammer Relevance filter SMT tr. SMT translation ATP translation Z3 CVC3 Yices E SPASS Vampire SInE Metis Metis Metis Metis Metis Metis Metis or SMT or SMT or SMT proof proof proof proof proof proof proof

Sledgehammer Sledgehammer Relevance filter Relevance filter SMT tr. SMT translation ATP translation Z3 CVC3 Yices E SPASS Vampire SInE Metis Metis Metis Metis Metis Metis Metis or SMT or SMT or SMT proof proof proof proof proof proof proof

rev [a, b] = [b, a]

rev [a, b] = [b, a] by (metis Cons_eq_appendI eq_Nil_appendI rev.simps(2) rev_singleton_conv)

2010

2010 3 ATPs x 30s 46% 54%

2010 3 ATPs x 30s 3 ATPs x 30 s nontrivial goals 46% 66% 34% 54%

2010 3 ATPs x 30s 3 ATPs x 30 s nontrivial goals 46% 66% 34% 54% 2011 (4 ATPs + 3 SMTs) x 30s (4 ATPs + 3 SMTs) x 30s nontrivial goals 39% 54% 46% 61%

Issue #1: Too Many Facts Issue #2: Encoding Overhead Issue #3: Large Terms

Issue #1: Too Many Facts

Conjecture: … c … d … e ...

Conjecture: … c … d … e ... Lemma 1: … c … f ... Lemma 2: … f … g ... Lemma 3: … g … h ...

Conjecture: … c … d … e ... ✓ Lemma 1: … c … f ... Lemma 2: … f … g ... Lemma 3: … g … h ...

Conjecture: … c … d … e ... ✓ Lemma 1: … c … f ... ✓ Lemma 2: … f … g ... Lemma 3: … g … h ...

Conjecture: … c … d … e ... ✓ Lemma 1: … c … f ... ✓ Lemma 2: … f … g ... ✓ Lemma 3: … g … h ...

The More Facts...

E 1.2 Success Rate 0 100 200 300 400 500 600 700 800 900 1000 The More Facts...

E 1.2 Success Rate 0 100 200 300 400 500 600 700 800 900 1000 The More Facts...

E 1.2 Success Rate 0 100 200 300 400 500 600 700 800 900 1000 The More Facts...

SPASS 3.7 Success Rate 0 100 200 300 400 500 600 700 800 900 1000 The More Facts...

SPASS 3.7 Success Rate 0 100 200 300 400 500 600 700 800 900 1000 The More Facts...

Vampire 1.0 Success Rate 0 100 200 300 400 500 600 700 800 900 1000 The More Facts...

Vampire 1.0 Success Rate 0 100 200 300 400 500 600 700 800 900 1000 The More Facts...

Z3 2.15 Success Rate 0 100 200 300 400 500 600 700 800 900 1000 The More Facts...

Z3 2.15 Success Rate 0 100 200 300 400 500 600 700 800 900 1000 The More Facts...

E SPASS Vampire Z3

E SPASS Vampire Z3

How Effective is the Relevance Filter?

Uses 0-9 100-109 200-209 300-309 400-409 490-499 How Effective is the Relevance Filter?

Experiments

Z3 Weights Experiments

Z3 Weights ……..……… +0.4 pp Experiments

Z3 Weights ……..……… +0.4 pp Z3 Triggers Experiments

Z3 Weights ……..……… +0.4 pp Z3 Triggers ……..……… +0.1 pp Experiments

Z3 Weights ……..……… +0.4 pp Z3 Triggers ……..……… +0.1 pp Z3 “Slicing” Experiments

Z3 Weights ……..……… +0.4 pp Z3 Triggers ……..……… +0.1 pp Z3 “Slicing” ........….…… +2.1 pp Experiments

Z3 Weights ……..……… +0.4 pp Z3 Triggers ……..……… +0.1 pp Z3 “Slicing” ........….…… +2.1 pp E Weights Experiments

Z3 Weights ……..……… +0.4 pp Z3 Triggers ……..……… +0.1 pp Z3 “Slicing” ........….…… +2.1 pp E Weights ..……………. +1.4 pp Experiments

Issue #2: Encoding Overhead

HOL to FOL

Application Operatorxxxxx suc(N) app(suc, N) HOL to FOL

Application Operatorxxxxx suc(N) app(suc, N) Type Information suc(N) ti(suc(ti(N, nat)), nat) HOL to FOL

Application Operatorxxxxx suc(N) app(suc, N) Type Information suc(N) ti(suc(ti(N, nat)), nat) ti(app(ti(suc, fun(nat, nat)), ti(N, nat)), nat) HOL to FOL

Problem Size No App. App. No Types ... 32K 41K Types ... 60K 107K

Problem Size No App. App. No Types ... 32K 41K Types ... 60K 107K Solving Time (E 1.2 ) No App. App. No Types ... 1 s 15 s Types ... 172 s ???? s

E SPASS Vampire Z3

E SPASS Vampire Z3 70% 60% 50% 40% 30% 20% 10% 0% Arrow FFT FTA Hoare Jinja NS QE S2S SN All

E SPASS Vampire Z3 70% 60% 50% 40% 30% 20% 10% 0% Arrow FFT FTA Hoare Jinja NS QE S2S SN All Claim: Sorts Rock x (Types)

E SPASS Vampire Z3 70% 60% 50% 40% 30% 20% 10% 0% Arrow Jinja NS SN Claim: Sorts Rock x (Types)

Issue #3: Large Terms

1 + … + 1 = ( n ::nat) rev [x 1 , …, x n ] = [x n , …, x 1 ] map f [x 1 , …, x n ] = [f x 1 , …, f x n ]

0,8 s Vampire SPASS 0,6 s E 0,4 s 0,2 s Z3 0 s 1 2 3 4 5 6 7 8 9 10 1 + … + 1 = ( n ::nat) rev [x 1 , …, x n ] = [x n , …, x 1 ] map f [x 1 , …, x n ] = [f x 1 , …, f x n ]

0,8 s 30,0 s Vampire SPASS Vampire 0,6 s 22,5 s E 0,4 s 15,0 s E 0,2 s 7,5 s Z3 SPASS Z3 0 s 0 s 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 + … + 1 = ( n ::nat) rev [x 1 , …, x n ] = [x n , …, x 1 ] map f [x 1 , …, x n ] = [f x 1 , …, f x n ]

0,8 s 30,0 s Vampire SPASS Vampire 0,6 s 22,5 s E 0,4 s 15,0 s E 0,2 s 7,5 s Z3 SPASS Z3 0 s 0 s 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 + … + 1 = ( n ::nat) rev [x 1 , …, x n ] = [x n , …, x 1 ] 10,0 s 7,5 s SPASS 5,0 s 2,5 s E Vampire Z3 0 s 1 2 3 4 5 6 7 8 9 10 map f [x 1 , …, x n ] = [f x 1 , …, f x n ]

0,8 s 30,0 s Vampire SPASS Vampire 0,6 s 22,5 s E 0,4 s 15,0 s E 0,2 s 7,5 s Z3 SPASS Z3 0 s 0 s 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 + … + 1 = ( n ::nat) rev [x 1 , …, x n ] = [x n , …, x 1 ] 10,0 s 7,5 s (but simp can solve SPASS 5,0 s all of these 2,5 s E within 10 ms…) Vampire Z3 0 s 1 2 3 4 5 6 7 8 9 10 map f [x 1 , …, x n ] = [f x 1 , …, f x n ]

Future Work

Isabelle ATP / SMT Future Work

Isabelle ATP / SMT Improve xx Relevance Filter xx Lighten xx Translation xx Provide xx Extralogical Info. xx Future Work

Isabelle ATP / SMT Improve xx Handle Large Relevance Filter xx Axiom Bases Lighten xx Support Translation xx Types Provide xx Exploit Extralogical Info. xx Extralogical Info. Future Work

Sled gehammer Hell The Day after Jud gment Jasmin C. Blanchette blanchette@in.tum.de