SMAC and XGBoost your Theorem Prover Edvard K. Holden Konstantin - - PowerPoint PPT Presentation

▶
smac and xgboost your theorem prover
SMART_READER_LITE
LIVE PREVIEW

SMAC and XGBoost your Theorem Prover Edvard K. Holden Konstantin - - PowerPoint PPT Presentation

Sep 12, 2023 766 likes •1.04k views

SMAC and XGBoost your Theorem Prover Edvard K. Holden Konstantin Korovin The University of Manchester 1 Theorem Proving in First-Order Logic Proof Axioms + Conjecture iProver | Counter model | Timeout 2 Heuristics - The Key to Success

slide-1
SLIDE 1

SMAC and XGBoost your Theorem Prover

Edvard K. Holden Konstantin Korovin The University of Manchester

1

slide-2
SLIDE 2

Theorem Proving in First-Order Logic

iProver Axioms + Conjecture

2

Proof | Counter model | Timeout

slide-3
SLIDE 3

Heuristics - The Key to Success

  • Controls the proving process
  • Crucial for performance
  • No single optimal heuristic
  • Manual exploration is infeasible

3

slide-4
SLIDE 4

Heuristics - iProver ~100 Options

...

  • -instantiation_flag true
  • -inst_lit_sel [+prop;+sign;+ground;-num_var;-num_symb]
  • -inst_lit_sel_side num_symb
  • -inst_solver_per_active 1400
  • -inst_passive_queues [[-conj_dist;+conj_symb;-num_var];[+age;-num_symb]]
  • -inst_passive_queues_freq [25;2]

…

  • -res_passive_queues [[+conj_symb;-num_symb];[+age;-num_symb]]
  • -res_passive_queues_freq [15;5]
  • -res_forward_subs full

…

4

slide-5
SLIDE 5

Proving Problems

iProver

5

slide-6
SLIDE 6

Proving Problems

iProver

6

slide-7
SLIDE 7

Proving Problems

iProver

Solved Black 3 / 3 Blue 1 / 2 Red 0 / 3

7

slide-8
SLIDE 8

Proving Problems

iProver

Heuristic 1 Heuristic 2 Heuristic 3

8

slide-9
SLIDE 9

Proving Problems

iProver

Heuristic 1 Heuristic 2 Heuristic 3

9

slide-10
SLIDE 10

Proving Problems

iProver

Heuristic 1 Heuristic 2 Heuristic 3

∴ All Problems Solved

10

slide-11
SLIDE 11

Proving Problems

iProver

Heuristic 1 Heuristic 2 Heuristic 3

How to group? What are the heuristics? How to map?

11

slide-12
SLIDE 12

Heuristic Challenges

Phase 1

  • Discover good heuristics

Phase 2

  • Select the right heuristic

12

slide-13
SLIDE 13

Phase 1

Learning and discovering efficient heuristics

13

slide-14
SLIDE 14

Heuristic Learning - Optimisation

14

iProver Optimiser [Heuristic] Feedback:= #Problems Solved

slide-15
SLIDE 15

Heuristic Learning - SMAC

Sequential Model-Based Algorithm Configuration

  • Construct the heuristics
  • Optimisation Parameters: ordinal, categorical, real
  • Optimise with Random Forest
  • Maximise number of solved problems

15

slide-16
SLIDE 16

Heuristic Learning - Optimisation & Clustering

iProver Optimiser [Heuristic] [Feedback] iProver Optimiser [Heuristic] [Feedback] iProver Optimiser [Heuristic] [Feedback]

slide-17
SLIDE 17

Heuristic Learning - Clustering Problems

17

Clustering

(Problem Properties) (Heuristic Evaluation)

slide-18
SLIDE 18

Heuristic Learning - Overview

18

Features (Re) Cluster

Opt Loop Opt Loop Opt Loop

Heuristic 1 . . . Heuristic n [Evaluation Features]

Heuristic Evaluation

slide-19
SLIDE 19

Heuristic Learning - Results

19

  • 500 CASC FOF Problems
  • Default solves 207
  • Optimise ~2 days
  • Optimise instantiation options
slide-20
SLIDE 20

Phase 2

Selecting the best heuristic

20

slide-21
SLIDE 21

Heuristic Mapping - Supervised Learning

21

Features Model Label

slide-22
SLIDE 22

Heuristic Mapping - Overview

22 22

Problem XGBoost Heuristic iProver Features ML Model Label

slide-23
SLIDE 23

Heuristic Mapping: Labelling

23

AVG Label Time: 27 s AVG Label Time: 42 s

Optimal Time Mapping Temporal Property Mapping

slide-24
SLIDE 24

Heuristic Mapping - Model Results

24

10-Fold-Cross-Validation Test Accuracy 86% ± 2% Ratio of solved problems 88% ± 2%

slide-25
SLIDE 25

Heuristic Mapping - Prover Results

25

Default Heuristic Best Optimised Heuristic Heuristic Mapping* Solved: 207 217 248 AVG Time in intersection: 27.9 28.7 26.0 *Trained with 30-70 split

slide-26
SLIDE 26

Conclusion

Heuristic evaluation to learn heuristics

  • Solves 24% more problems
  • Reduces solving times by 60%

Multi-class heuristic selection

  • Specialised and diverse heuristics
  • Solves nearly all solvable problems
  • 16.3% speed improvement over

default heuristic

26