Learning-Assisted Reasoning within Interactive Theorem Provers - - PowerPoint PPT Presentation

Learning-Assisted Reasoning within Interactive Theorem Provers

Thibault Gauthier May 17, 2019

1 / 44

black box proof conjecture

2 / 44

black box proof conjecture library

2 / 44

Object white, spherical, many petals red, star-shaped, five petals Properties

3 / 44

Proof Assistant Theorems Constants Mizar 51086 9172 Coq 23320 4841 HOL4 16476 2247 HOL Light 16191 820 Isabelle/HOL 14814 1076 Matita 1712 629

4 / 44

∀ x . x + = x logical operators variables constants

5 / 44

proof conjecture library

6 / 44

Demo

7 / 44

provers proof conjecture library

8 / 44

translation provers proof conjecture library

8 / 44

translation theorem prediction provers proof conjecture library

8 / 44

Formula Syntactic features Conjecture ∀x,y. (x +y)×(x −y) = x2 −y2 ∀x,y,z. x ×(y +z) = x ×y +x ×z ∀x,y. x +y = y +x Library ∀x,y. x ×y = y ×x eiπ +1 = 0 (x2)′ = 2×x

9 / 44

Formula Syntactic features Conjecture ∀x,y. (x +y)×(x −y) = x2 −y2 +,×,

2

∀x,y,z. x ×(y +z) = x ×y +x ×z ×,+ ∀x,y. x +y = y +x + Library ∀x,y. x ×y = y ×x × eiπ +1 = 0 e,i,×,π,+,1,0 (x2)′ = 2×x

′,2,×, 2

9 / 44

49 12 71 85 conjecture theorem → rule lemma

10 / 44

49 12 71 72 12 71 85 conjecture theorem → rule lemma

10 / 44

49 12 71 72 12 71 85 49 12 71 conjecture theorem → rule lemma

10 / 44

49 12 71 72 12 71 85 49 12 71 conjecture theorem → rule lemma Theorem Dependencies 85 49 12 71 102 51 45 86 12 . . . . . .

10 / 44

Re-proving

Tested library Benchmark Success standard library 40% judgement day 77% flyspeck 39% standard library 50% standard library 41%

11 / 44

Demo

12 / 44

proof conjecture library

13 / 44

tactic prediction proof search proof conjecture library

14 / 44

Tactics Useful for Solvers linear system, differential equations Simplifiers irreducible fraction, differentiation Induction natural numbers, lists, trees

15 / 44

Tactic Prediction Formal library Proof recording Knowledge base Training Predictors Proof search Conjecture Search tree Tactic policy Tactic value Proof minimization Proof

16 / 44

Tactic Prediction Formal library Proof recording Knowledge base Training Predictors Proof search Conjecture Search tree Tactic policy Tactic value Proof minimization Proof

16 / 44

axiom conjecture → rule lemma

17 / 44

axiom conjecture → rule lemma

17 / 44

axiom conjecture → rule lemma

17 / 44

axiom conjecture → tactic goal

18 / 44

axiom conjecture → tactic goal

18 / 44

axiom conjecture → tactic goal

18 / 44

REWRITE TAC INDUCT TAC METIS TAC

19 / 44

THENL tactical composes the effect of tactics.

20 / 44

THENL tactical composes the effect of tactics.

INDUCT TAC

20 / 44

THENL tactical composes the effect of tactics.

REWRITE TAC METIS TAC INDUCT TAC

20 / 44

THENL tactical composes the effect of tactics.

REWRITE TAC METIS TAC INDUCT TAC

20 / 44

THENL tactical composes the effect of tactics.

REWRITE TAC METIS TAC INDUCT TAC THENL [REWRITE TAC,METIS TAC] INDUCT TAC

20 / 44

Demo

21 / 44

Tactic Prediction Formal library Proof recording Knowledge base Training Predictors Proof search Conjecture Search tree Tactic policy Tactic value Proof minimization Proof

22 / 44

Proof recording

Original proof:

INDUCT TAC

THENL [REWRITE TAC, METIS TAC]

Modified proof:

(R numLib.INDUCT TAC)

THENL

[R boolLib.REWRITE TAC, R metisLib.METIS TAC]

Database of tactics:

R (f n) (f (SUC n)) ⇒ transitive R: INDUCT TAC n ∗ m ≤ n ∗ p ⇒ (n = 0) ∨ m ≤ p : REWRITE TAC INJ f U(:num) s ⇒ INFINITE s : METIS TAC ...

23 / 44

Tactic Prediction Formal library Proof recording Knowledge base Training Predictors Proof search Conjecture Search tree Tactic policy Tactic evaluation Proof minimization Proof

24 / 44

Prediction algorithm

Algorithm: Nearest neighbor weighted by TF-IDF heuristics Effect: Order goals from the database according to their distance to a target goal. Remark: This is algorithm performs premise selection. How do we adapt it to predict tactics?

25 / 44

Policy

Database of tactics is a map from goals to tactics.

R (f n) (f (SUC n)) ⇒ transitive R: INDUCT TAC n ∗ m ≤ n ∗ p ⇒ (n = 0) ∨ m ≤ p : REWRITE TAC INJ f U(:num) s ⇒ INFINITE s : METIS TAC ...

An order on goals induces an order on tactics. New goal appearing during proof search:

LENGTH (MAP f l) = LENGTH l

Policy for the new goal:

Rank Tactic Policy 1 REWRITE TAC 0.5 2 METIS TAC 0.25 ... 4 INDUCT TAC 0.0625 ...

26 / 44

Value

Database of lists of goals: ◮ Positive examples: appears in human proofs. ◮ Negative examples: produced during TacticToe search but do not appear in the final proof.

27 / 44

Plan

Tactic Prediction Formal library Proof recording Knowledge base Training Predictors Proof search Conjecture Search tree Tactic policy Tactic value Proof minimization Proof

28 / 44

Training

Improve recorded data to create better predictions during search.

29 / 44

Training: orthogonalization

Issue: Many tactics are doing the same job on a goal g. Solution: Competition for g where the most popular tactic wins.

30 / 44

Training: orthogonalization

Recorded goal-tactic pair:

LENGTH (MAP f l) = LENGTH l: INDUCT TAC

Competition:

Progress Coverage INDUCT TAC Yes 136 REWRITE TAC No 2567 METIS TAC Yes 694

Added to the database:

LENGTH (MAP f l) = LENGTH l: METIS TAC

Result: 6 % improvement.

31 / 44

Training: abstraction

Issue: Some theorems are never used inside tactics. Solution: Abstract all lists of theorems in a tactic and instantiate them depending on the target goal.

32 / 44

Training: abstraction

Abstraction algorithm:

Original : REWRITE TAC [T1,T2] Abstraction : REWRITE TAC X Instantiation: REWRITE TAC [T67, T1, T43, ... ]

Question: Dow we keep the original or the abstraction ? Answer: Let them compete during orthogonalization. Result: 15% improvement

33 / 44

Training: preselection

Issue: Predictions are too slow during proof search. Solution: Preselect 1000 suitable tactics by importing proofs (many tactics) from related goals.

34 / 44

Tactic Prediction Formal library Proof recording Knowledge base Training Predictors Proof search Conjecture Search tree Tactic policy Tactic evaluation Proof minimization Proof

35 / 44

Proof search: search tree

gi ... gn ... g0 a0 ... aj ... a1 t1 tj tm

36 / 44

Proof search: advanced tree search

0.25 0.5 0.125

37 / 44

Proof search: advanced tree search

0.7 0.7 0.25 0.5 0.125 0.25 0.5 0.125

37 / 44

Proof search: advanced tree search

0.7,0.3 0.7,0.3 0.25 0.5 0.125 0.3 0.25 0.5 0.125

37 / 44

Proof search: advanced tree search

0.7,0.3,0.8 0.8 0.7,0.3 0.25 0.5 0.125 0.3 0.25 0.5 0.125

37 / 44

Re-proving

Tested library Proof automation Success 50% 66%

38 / 44

Re-proving: HOL4 proofs found in less than x seconds

10 20 30 40 50 60 1,000 2,000 3,000 4,000 TacticToe E prover

39 / 44

Re-proving: percentage of solved HOL4 proof of size x

2 4 6 8 10 12 14 16 18 20 20 40 60 80 100 TacticToe E prover

40 / 44

Tactic Prediction Formal library Proof recording Knowledge base Training Predictors Proof search Conjecture Search tree Tactic policy Tactic value Proof minimization Proof

41 / 44

Before:

boolLib.REWRITE TAC [DB.fetch "list" "EVERY_CONJ",... ]

THEN

BasicProvers.Induct on [HolKernel.QUOTE "l"]

THENL

[BasicProvers.SRW TAC [] [], simpLib.ASM SIMP TAC (BasicProvers.srw ss ()) [boolLib.DISJ IMP THM, DB.fetch "list" "MAP", DB.fetch "list" "CONS_11", boolLib.FORALL AND THM]]

After:

Induct on ‘l‘

THENL

[SRW TAC [] [], ASM SIMP TAC (srw ss ()) [DISJ IMP THM, FORALL AND THM]]

42 / 44

Summary: TacticToe learns from human proofs to solve new goals. Advantages over ATPs (E prover) for ITP (HOL4) users: ◮ Includes domain specific automation found in the ITP. ◮ Generated proofs are human-level proofs. ◮ No translation or reconstruction needed.

43 / 44

Demo

44 / 44