Portfolio-based Algorithm Selection for SAT Holger H. Hoos BETA Lab - - PowerPoint PPT Presentation

▶
portfolio based algorithm selection for sat
SMART_READER_LITE
LIVE PREVIEW

Portfolio-based Algorithm Selection for SAT Holger H. Hoos BETA Lab - - PowerPoint PPT Presentation

Nov 10, 2023 217 likes •715 views

Portfolio-based Algorithm Selection for SAT Holger H. Hoos BETA Lab Department of Computer Science University of British Columbia Canada based on joint work with Lin Xu, Frank Hutter and Kevin Leyton-Brown ECAI 2012 CoCoMile Workshop

slide-1
SLIDE 1

Portfolio-based Algorithm Selection for SAT

Holger H. Hoos

BETA Lab Department of Computer Science University of British Columbia Canada based on joint work with Lin Xu, Frank Hutter and Kevin Leyton-Brown

ECAI 2012 CoCoMile Workshop Montpellier, France, 2012/08/27

slide-2
SLIDE 2

What is SAT?

Given: A formula F in propositional logic Objective: Determine whether F is satisfiable (i.e., can be made true by assigning truth values to the variables in F)

Why SAT?

I Drosophila problem for computing science (and beyond)

I prototypical NP-hard problem I prominent in various areas of CS and beyond I important applications I conceptual simplicity aids solver design / development

I hotbed for the kind of research discussed here

Holger Hoos: Portfolio-based Algorithm Selection for SAT 2

slide-3
SLIDE 3

What fuels progress in SAT solving?

I Insights into SAT (theory) I Creativity of algorithm designers

heuristics

I Advanced debugging techniques

fuzz testing, delta-debugging

I Principled experimentation

statistical techniques, experimentation platforms

I SAT competitions

Holger Hoos: Portfolio-based Algorithm Selection for SAT 3

slide-4
SLIDE 4

2009: 5 of 27 medals 2011: 28 of 54 medals

Holger Hoos: Portfolio-based Algorithm Selection for SAT 4

slide-5
SLIDE 5

2012 SAT Challenge

Rank Application Hard Combinatorial Random (SAT only) 1 SATzilla2012 App SATzilla2012 COMB CCASat 2 SATzilla2012 ALL SATzilla2012 ALL SATzilla2012 RAND 3 Industrial SAT Solver ppfolio2012 SATzilla2012 ALL 4 lingeling (2011) interactSAT_c Sparrow (2011) 5 interactSAT pfolioUZK EagleUP (2011) 6 glucose aspeed-crafted sattime2012 7 SINN Clasp-crafted ppfolio2012

2/3 first, 3/3 second, 3/3 third places

Holger Hoos: Portfolio-based Algorithm Selection for SAT 5

slide-6
SLIDE 6

Meta-algorithmic techniques

I algorithms that operate upon other algorithms (SAT solvers)

6= meta-heuristics

I here: algorithms whose inputs include one or more SAT solvers

I configurators (e.g., ParamILS, GGA, SMAC) I selectors (e.g., SATzilla, 3S) I schedulers (e.g., aspeed; also: 3S, SATzilla) I (parallel) portfolios (e.g., ManySAT, ppfolio) I run-time predictors I experimentation platforms (e.g., EDACC, HAL) Holger Hoos: Portfolio-based Algorithm Selection for SAT 6

slide-7
SLIDE 7

Why are meta-algorithmic techniques important?

I no one knows how to best solve SAT (or any other

NP-hard problem) no single dominant solver

I state-of-the-art performance often achieved by combinations

  • f various heuristic choices (e.g., pre-processing;

variable/value selection heuristic; restart rules; data structures; . . . ) complex interactions, unexpected behaviour

I performance can be tricky to assess due to

I differences in behaviour across problem instances I stochasticity

potential for suboptimal choices in solver development, applications

Holger Hoos: Portfolio-based Algorithm Selection for SAT 7

slide-8
SLIDE 8

Why are meta-algorithmic techniques important?

I human intuitions can be misleading, abilities are limited

substantial benefit from augmentation with computational techniques

I use of fully specified procedures (rather than intuition /

ad hoc choices) can improve reproducibility, facilitate scientific analysis / understanding

Holger Hoos: Portfolio-based Algorithm Selection for SAT 8

slide-9
SLIDE 9

Outline

  • 1. Introduction
  • 2. Portfolio-based Algorithm Selection
  • 3. SATzilla
  • 4. Beyond Algorithm Selection for SAT
  • 5. Conclusions & Outlook

Holger Hoos: Portfolio-based Algorithm Selection for SAT 9

slide-10
SLIDE 10

Instance-based algorithm selection (Rice 1976):

I Given: set S of algorithms for a problem, problem instance π I Objective: select from S the algorithm expected to solve π

most efficiently, based on (cheaply computable) features of π

Note:

Best case performance bounded by oracle, which selects the best s 2 S for each π = virtual best solver (VBS)

Holger Hoos: Portfolio-based Algorithm Selection for SAT 10

slide-11
SLIDE 11

Instance-based algorithm selection

algorithms

Holger Hoos: Portfolio-based Algorithm Selection for SAT 11

slide-12
SLIDE 12

Instance-based algorithm selection

selector component algorithms

Holger Hoos: Portfolio-based Algorithm Selection for SAT 11

slide-13
SLIDE 13

Instance-based algorithm selection

selector component algorithms

Holger Hoos: Portfolio-based Algorithm Selection for SAT 11

slide-14
SLIDE 14

Instance-based algorithm selection

selector component algorithms feature extractor

Holger Hoos: Portfolio-based Algorithm Selection for SAT 11

slide-15
SLIDE 15

Instance-based algorithm selection

selector component algorithms feature extractor

Holger Hoos: Portfolio-based Algorithm Selection for SAT 11

slide-16
SLIDE 16

Key components:

I set of (state-of-the-art) solvers I set of cheaply computable, informative features I efficient procedure for mapping features to solvers (selector) I training data I procedure for building good selector based on training data

(selector builder)

Holger Hoos: Portfolio-based Algorithm Selection for SAT 12

slide-17
SLIDE 17

Methods for instance-based selection:

I classification-based: predict the best solver, using:

I decision trees

(Guerri & Milano 2004)

I case-based reasoning

(Gebruers, Guerri, Hnich, Milano 2004; OMahony, Hebrard, Holland, Nugent, OSullivan 2008)

I (weighted) k-nearest neighbours

(Malitsky, Sabharwal, Samulowitz, Sellmann 2011; Kadioglu, Malitsky, Sabharwal, Samulowitz, Sellmann 2011)

I pairwise cost-sensitive decision forests + voting

(Xu, Hutter, HH, Leyton-Brown 2012) I regression-based: predict running time for each solver,

select the one predicted to be fastest

(Leyton-Brown, Nudelman, Shoham 2003; Xu, Hutter, HH, Leyton-Brown 2007–9)

Holger Hoos: Portfolio-based Algorithm Selection for SAT 13

slide-18
SLIDE 18

Instance features:

I Use generic and problem-specific features that correlate with

performance and can be computed (relatively) cheaply:

I number of clauses, variables, . . . I constraint graph features I local & complete search probes

I Use as features statistics of distributions,

e.g., variation coefficient of node degree in constraint graph

I Consider combinations of features (e.g., pairwise products

quadratic basis function expansion).

Holger Hoos: Portfolio-based Algorithm Selection for SAT 14

slide-19
SLIDE 19

SATzilla 2007–9 (Xu, Hutter, HH, Leyton-Brown):

I use state-of-the-art complete (DPLL/CDCL) and incomplete

(local search) SAT solvers

I extract (up to) 84 polytime-computable instance features I use ridge regression on selected features to predict solver

run-times from instance features (one model per solver)

I run solver with best predicted performance

Holger Hoos: Portfolio-based Algorithm Selection for SAT 15

slide-20
SLIDE 20

Some bells and whistles:

I use pre-solvers to solve ‘easy’ instances quickly I build run-time predictors for various types of instances,

use classifier to select best predictor based on instance features.

I predict time required for feature computation; if that time

is too long (or error occurs), use back-up solver

I use method by Schmee & Hahn (1979) to deal with

censored run-time data prizes in 5 of the 9 main categories of the 2009 SAT Solver Competition (3 gold, 2 silver medals)

Holger Hoos: Portfolio-based Algorithm Selection for SAT 16

slide-21
SLIDE 21

The problem with standard classification approaches

Crucial assumption: solvers behave similarly on instances with similar features But do they really?

I uninformative features I correlated features I feature normalisation (tricky!) I cost of misclassification

Holger Hoos: Portfolio-based Algorithm Selection for SAT 17

slide-22
SLIDE 22

SATzilla 2011–12 (Xu, Hutter, HH, Leyton-Brown 2012):

I uses cost-based decision forests to directly select solver

based on features

I one predictive model for each pair of solvers (which is better?) I majority voting (over pairwise predictions) to select

solver to be run

Holger Hoos: Portfolio-based Algorithm Selection for SAT 18

slide-23
SLIDE 23

2011 SAT Competition Data (Inst. Solved) Solver Application Crafted Random Oracle (VBS) 84.7% 76.3% 82.2% SATzilla 2011 75.3% 66.0% 80.8% SATzilla 2009 70.3% 63.0% 80.3% Gold medalist (SBS) 71.7% 54.3% 68.0% (SATzilla assessed by 10-fold cross-validation)

Holger Hoos: Portfolio-based Algorithm Selection for SAT 19

slide-24
SLIDE 24

2012 SAT Challenge Results

Rank Application Hard Combinatorial Random (SAT only) 1 SATzilla2012 App SATzilla2012 COMB CCASat 2 SATzilla2012 ALL SATzilla2012 ALL SATzilla2012 RAND 3 Industrial SAT Solver ppfolio2012 SATzilla2012 ALL 4 lingeling (2011) interactSAT_c Sparrow (2011) 5 interactSAT pfolioUZK EagleUP (2011) 6 glucose aspeed-crafted sattime2012 7 SINN Clasp-crafted ppfolio2012

SATzilla: 2/3 first, 3/3 second, 2/3 third places

Holger Hoos: Portfolio-based Algorithm Selection for SAT 20

slide-25
SLIDE 25

2012 SAT Challenge: Single best solver vs SATzilla

10 10

1

10

2

10 10

1

10

2

lingeling2011 runtime [CPU sec] SATzilla2012 APP runtime [CPU sec]

Holger Hoos: Portfolio-based Algorithm Selection for SAT 21

slide-26
SLIDE 26

Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection

Xu, HH, Leyton-Brown (2010)

Note:

I SATzilla builds algorithm selector based on given set

  • f SAT solvers

but: success entirely depends on quality of given solvers

I Automated algorithm configuration produces solvers that work

well on average on a given set of SAT instances

(e.g., SATenstein – KhudaBukhsh, Xu, HH, Leyton-Brown 2009)

but: may have to settle for compromises for broad, heterogenous sets Idea: Combine the two approaches portfolio-based selection from set of automatically constructed solvers

Holger Hoos: Portfolio-based Algorithm Selection for SAT 22

slide-27
SLIDE 27

Configuration + Selection = Hydra

parametric algorithm Holger Hoos: Portfolio-based Algorithm Selection for SAT 23

slide-28
SLIDE 28

Configuration + Selection = Hydra

parametric algorithm (multiple configurations) Holger Hoos: Portfolio-based Algorithm Selection for SAT 23

slide-29
SLIDE 29

Configuration + Selection = Hydra

parametric algorithm (multiple configurations) selector feature extractor Holger Hoos: Portfolio-based Algorithm Selection for SAT 23

slide-30
SLIDE 30

Configuration + Selection = Hydra

parametric algorithm (multiple configurations) selector feature extractor Holger Hoos: Portfolio-based Algorithm Selection for SAT 23

slide-31
SLIDE 31

Configuration + Selection = Hydra

parametric algorithm (multiple configurations) selector feature extractor Holger Hoos: Portfolio-based Algorithm Selection for SAT 23

slide-32
SLIDE 32

Simple combination:

  • 1. build solvers for various types of instances using automated

algorithm configuration

  • 2. construct portfolio-based selector from these

Drawback: Requires suitably defined sets of instances

Better solution:

iteratively build & add solvers that improve performance

  • f given portfolio

Hydra Note: Builds portfolios solely using

I generic, highly configurable solver (e.g., SATenstein) I instance features (as used in SATzilla)

Holger Hoos: Portfolio-based Algorithm Selection for SAT 24

slide-33
SLIDE 33

Results on mixture of 6 well-known benchmark sets

10

  • 2

10 10

2

10

  • 2

10

  • 1

10 10

1

10

2

10

3

Hydra[BM, 1] PAR Score Hydra[BM, 7] PAR Score

Holger Hoos: Portfolio-based Algorithm Selection for SAT 25

slide-34
SLIDE 34

Results on mixture of 6 well-known benchmark sets

1 2 3 4 5 6 7 8 50 100 150 200 250 300 Number of Hydra Steps PAR Score Hydra[BM] training Hydra[BM] test SATenFACT on training SATenFACT on test

Holger Hoos: Portfolio-based Algorithm Selection for SAT 26

slide-35
SLIDE 35

Note:

I Hydra can use arbitrary algorithm configurators,

selector builders

I different approaches are possible: e.g., ISAC, based on

feature-based instance clustering, distance-based selection

(Kadioglu, Malitsky, Sellmann, Tierney 2010)

Holger Hoos: Portfolio-based Algorithm Selection for SAT 27

slide-36
SLIDE 36

The next step: Programming by Optimisation (PbO)

HH (2010–12)

Key idea:

I specify large, rich design spaces of solver for given problem

I avoid premature, uninformed, possibly detrimental

design choices

I active development of promising alternatives for

design components

I automatically make choices to obtain solver optimised

for given use context

Holger Hoos: Portfolio-based Algorithm Selection for SAT 28

slide-37
SLIDE 37

solver

Holger Hoos: Portfolio-based Algorithm Selection for SAT 29

slide-38
SLIDE 38

solver design space

  • f solvers

Holger Hoos: Portfolio-based Algorithm Selection for SAT 29

slide-39
SLIDE 39

application context solver design space

  • f solvers

Holger Hoos: Portfolio-based Algorithm Selection for SAT 29

slide-40
SLIDE 40

application context solver

  • ptimised

solver design space

  • f solvers

Holger Hoos: Portfolio-based Algorithm Selection for SAT 29

slide-41
SLIDE 41

application context solver

  • ptimised

solver instance- based selector design space

  • f solvers

Holger Hoos: Portfolio-based Algorithm Selection for SAT 29

slide-42
SLIDE 42

application context planner planner solver

  • ptimised

solver parallel portfolio instance- based selector design space

  • f solvers

Holger Hoos: Portfolio-based Algorithm Selection for SAT 29

slide-43
SLIDE 43

application context planner planner

  • ptimised

solver parallel portfolio instance- based selector design space

  • f solvers

Holger Hoos: Portfolio-based Algorithm Selection for SAT 29

slide-44
SLIDE 44

Levels of PbO:

Level 4: Make no design choice prematurely that cannot be justified compellingly. Level 3: Strive to provide design choices and alternatives. Level 2: Keep and expose design choices considered during software development. Level 1: Expose design choices hardwired into existing code (magic constants, hidden parameters, abandoned design alternatives). Level 0: Optimise settings of parameters exposed by existing software.

Holger Hoos: Portfolio-based Algorithm Selection for SAT 30

slide-45
SLIDE 45

Success in optimising speed:

Application, Design choices Speedup PbO level SAT-based software verification (Spear), 41

Hutter, Babi´ c, HH, Hu (2007)

4.5–500 × 2–3 AI Planning (LPG), 62

Vallati, Fawcett, Gerevini, HH, Saetti (2011)

3–118 × 1 Mixed integer programming (CPLEX), 76

Hutter, HH, Leyton-Brown (2010)

2–52 ×

... and solution quality:

University timetabling, 18 design choices, PbO level 2–3 new state of the art; UBC exam scheduling

Fawcett, Chiarandini, HH (2009) Holger Hoos: Portfolio-based Algorithm Selection for SAT 31

slide-46
SLIDE 46

Communications of the ACM, 55(2), pp. 70–80, February 2012

www.prog-by-opt.net

slide-47
SLIDE 47

Conclusions & Outlook

I State of the art in SAT solving:

portfolio-based algorithm selectors; powered by machine learning, component solvers, features

I Further progress likely:

component solvers, features, insights

Holger Hoos: Portfolio-based Algorithm Selection for SAT 33

slide-48
SLIDE 48

I Portfolio-based algorithm selectors and other meta-algorithmic

techniques will become dominant in most (all?) areas of AI.

I Programming by Optimisation (or a closely related approach)

will greatly facilitate this development.

Of interest to CoCoMile / ML: Auto-WEKA: Automated Selection and Hyper-Parameter Optimization

  • f Classification Algorithms (Thornton, Hutter, HH, Leyton-Brown 2012)

http://arxiv.org/abs/1208.3719 I Driven by SAT, SMT, PbO and advances in automated

testing / debugging, program synthesis from higher-level designs will become practical and widely used.

Holger Hoos: Portfolio-based Algorithm Selection for SAT 34