Understanding Random SAT Understanding Random SAT Beyond the - - PowerPoint PPT Presentation

Understanding Random SAT Understanding Random SAT

Beyond the Clauses-to-Variables Ratio

Eugene Nudelman Eugene Nudelman Stanford University

joint work with…

Kevin Leyton-Brown Kevin Leyton-Brown Holger Hoos Holger Hoos University of British Columbia Alex Devkar Alex Devkar Yoav Shoham Yoav Shoham Stanford University

Introduction Introduction

• SAT is one of the most studied

most studied problems in CS

• Lots known about its worst-case

worst-case complexity

– But often, particular instances of NP-hard problems like SAT are easy in practice easy in practice

• “Drosophila” for average-case

average-case and empirical empirical (typical-case) complexity studies

• (Uniformly) random SAT provides a way to bridge

analytical and empirical work

CP 2004

Previously Previously…

• Easy-hard-less

Easy-hard-less hard hard transitions discovered in the behaviour

• f DPLL-type solvers [Selman, Mitchell, Levesque]

– Strongly correlated with phase transition in solvability – Spawned a new enthusiasm for using empirical methods to study algorithm performance

• Follow up included study of:

– Islands of tractability [Kolaitis et. al.] – SLS search space topologies [Frank et.al., Hoos] – Backbones [Monasson et.al., Walsh and Slaney] – Backdoors [Williams et. al.] – Random restarts [Gomes et. al.] – Restart policies [Horvitz et.al, Ruan et.al.]

– …

• 2
• 1.5
• 1
• 0.5

CP 2004

Empirical Hardness Models Empirical Hardness Models

• We proposed building regression models

regression models as a disciplined way of predicting and studying algorithms’ behaviour

[Leyton-Brown, Nudelman, Shoham, CP-02]

• Applications

Applications of this machine learning approach:

1) Predict running time

• Useful to know how long

how long an algorithm will run

2) Gain theoretical understanding

• Which variables are important

important to the hardness model?

3) Build algorithm portfolios

• Can select the right algorithm on a per-instance

per-instance basis

4) Tune distributions for hardness

• Can generate harder

harder benchmarks by rejecting easy instances

CP 2004

Outline Outline

• Features

Features

• Experimental Results

–Variable Size Data –Fixed Size Data

CP 2004

Features: Local Search Probing Features: Local Search Probing

200 400 600 800 1000 1200 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Step N Step Number mber BEST # Unsat BEST # Unsat Clauses lauses

Long Plateau Short Plateau

CP 2004

Features: Local Search Probing Features: Local Search Probing

200 400 600 800 1000 1200 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Step N Step Number mber BEST # Unsat BEST # Unsat Clauses lauses

Best Solution (mean, CV)

CP 2004

Features: Local Search Probing Features: Local Search Probing

200 400 600 800 1000 1200 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Step N Step Number mber BEST # Unsat BEST # Unsat Clauses lauses

Number of Steps to Optimal (mean, median, CV, 10%.90%)

CP 2004

Features: Local Search Probing Features: Local Search Probing

200 400 600 800 1000 1200 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Step N Step Number mber BEST # Unsat BEST # Unsat Clauses lauses

• Ave. Improvement To

Best Per Step (mean, CV)

CP 2004

Features: Local Search Probing Features: Local Search Probing

200 400 600 800 1000 1200 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Step N Step Number mber BEST # Unsat BEST # Unsat Clauses lauses

First LM Ratio (mean, CV)

CP 2004

Features: Local Search Probing Features: Local Search Probing

200 400 600 800 1000 1200 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Step N Step Number mber BEST # Unsat BEST # Unsat Clauses lauses

BestCV (CV of Local Minima) (mean, CV)

CP 2004

Features: DPLL, LP Features: DPLL, LP

• DPLL

DPLL search space size estimate

– Random probing Random probing with unit propagation – Compute mean depth till contradiction – Estimate log(#nodes)

• Cumulative number of unit propagations

unit propagations at different depths (DPLL with Satz heuristic)

• LP relaxation

LP relaxation

– Objective value – stats of integer slacks – #vars set to an integer

CP 2004

Other Features Other Features

Var Var

Var

Clause Clause

• Problem Size

Problem Size:

– v (#vars) – c (#clauses) – Powers of c/v, v/c, |c/v — 4.26|

• Graphs

Graphs:

– Va Variable-Clause riable-Clause (VCG, bipartite) – Variable Variable (VG, edge whenever two variables occur in the same clause) – Clause Clause (CG, edge iff two clauses share a variable with opposite sign)

• Balance

Balance

– #pos vs. #neg literals – unary, binary, ternary clauses

• Proximity to Horn formula

Horn formula

}

used for normalizing many other features

Var Var

Var Var Var

Clause Clause Clause Clause

CP 2004

Outline Outline

• Features

Features

• Experimental Results

Experimental Results

–Variable Size Data –Fixed Size Data

CP 2004

Experimental Setup Experimental Setup

• Uniform random 3-SAT, 400 vars
• Datasets

Datasets (20000 instances each)

– Variable-ratio Variable-ratio dataset (1 CPU-month)

• c/v uniform in [3.26, 5.26] (∴ c ∈[1304,2104])

– Fixed-ratio Fixed-ratio dataset (4 CPU-months)

• c/v=4.26 (∴ v=400, c=1704)
• Solvers

Solvers

– Kcnfs [Dubois and Dequen] – OKsolver [Kullmann] – Satz [Chu Min Li]

• Quadratic

Quadratic regression egression with logistic response function

• Training : test : validation split – 70 : 15 : 15

CP 2004

Kcnfs Data

• 2
• 1.5
• 1
• 0.5

0.5 1 1.5 2 3.3 3.5 3.7 3.9 4.1 4.3 4.5 4.7 4.9 5.1 5.3

c / / v

4 * Pr(SAT) - 2 log(Kcnfs runtime) CP 2004

Kcnfs Kcnfs Data Data

0.01 0.1 1 10 100 1000 3.26 3.76 4.26 4.76 5.26 Clauses-to-Variables Ratio Runtime(s)

CP 2004

Kcnfs Kcnfs Data Data

0.01 0.1 1 10 100 1000 3.26 3.76 4.26 4.76 5.26 Clauses-to-Variables Ratio Runtime(s)

CP 2004

Kcnfs Kcnfs Data Data

0.01 0.1 1 10 100 1000 3.26 3.76 4.26 4.76 5.26 Clauses-to-Variables Ratio Runtime(s)

CP 2004

Kcnfs Kcnfs Data Data

0.01 0.1 1 10 100 1000 3.26 3.76 4.26 4.76 5.26 Clauses-to-Variables Ratio Runtime(s)

CP 2004

Variable Ratio Prediction (Kcnfs) Variable Ratio Prediction (Kcnfs)

0.01 0.1 1 10 100 1000 0.01 0.1 1 10 100 1000 Actual Runtime [CPU sec] Predicted Runtime [CPU sec]

CP 2004

Variable Ratio - Variable Ratio - UNSAT UNSAT

0.01 0.1 1 10 100 1000 0.01 0.1 1 10 100 1000 Actual Runtime [CPU sec] Predicted Runtime [CPU sec]

CP 2004

Variable Ratio - Variable Ratio - SAT AT

0.01 0.1 1 10 100 1000 0.01 0.1 1 10 100 1000 Actual Runtime [CPU sec] Predicted Runtime [CPU sec]

CP 2004

Kcnfs Kcnfs vs. Satz

• vs. Satz (UNSAT)

(UNSAT)

0.01 0.1 1 10 100 1000 0.01 0.1 1 10 100 1000 Kcnfs time [CPU sec] Satz time [CPU sec]

CP 2004

Kcnfs Kcnfs vs. Satz

• vs. Satz (SAT)

(SAT)

0.01 0.1 1 10 100 1000 0.01 0.1 1 10 100 1000 Kcnfs time [CPU sec] Satz time [CPU sec]

CP 2004

Feature Importance Feature Importance – Variable Ratio Variable Ratio

• Subset selection

Subset selection can be used to identify features sufficient sufficient for approximating full model performance

• Other (correlated) sets could potentially achieve similar

performance

Variable Variable Cost of Cost of Omission Omission |c/v-4.26| 100 |c/v-4.26|2 69

(v/c)2 × SapsBestCVMean

53 |c/v-4.26| × SapsBestCVMean 33

CP 2004

Feature Importance Feature Importance – Variable Ratio Variable Ratio

• Subset selection

Subset selection can be used to identify features sufficient sufficient for approximating full model performance

• Other (correlated) sets could potentially achieve similar

performance

Variable Variable Cost of Cost of Omission Omission |c/v-4.26

• 4.26|

100 |c/v-4.26

• 4.26|2

69

(v/c)2 × SapsBestCVMean

53 |c/v-4.26

• 4.26| × SapsBestCVMean

33

CP 2004

Feature Importance Feature Importance – Variable Ratio Variable Ratio

• Subset selection

Subset selection can be used to identify features sufficient sufficient for approximating full model performance

• Other (correlated) sets could potentially achieve similar

performance

Variable Variable Cost of Cost of Omission Omission |c/v-4.26| 100 |c/v-4.26|2 69

(v/c)2 × SapsBestCVMean SapsBestCVMean

53 |c/v-4.26| × SapsBestCVMean

SapsBestCVMean

33

CP 2004

Fixed Ratio Data Fixed Ratio Data

0.01 0.1 1 10 100 1000 3.26 3.76 4.26 4.76 5.26 Clauses-to-Variables Ratio Runtime(s)

CP 2004

Fixed Ratio Prediction (Kcnfs) Fixed Ratio Prediction (Kcnfs)

0.01 0.1 1 10 100 1000 0.01 0.1 1 10 100 1000 Actual Runtime [CPU sec] Predicted Runtime [CPU sec]

CP 2004

Feature Importance Feature Importance – Fixed Ratio Fixed Ratio

Variable Variable Cost of Cost of Omission Omission

SapsBestSolMean2

100

SapsBestSolMean × MeanDPLLDepth

74

GsatBestSolCV × MeanDPLLDepth

21

VCGClauseMean × GsatFirstLMRatioMean

9

CP 2004

Feature Importance Feature Importance – Fixed Ratio Fixed Ratio

CP 2004

Variable Variable Cost of Cost of Omission Omission

SapsBestSolMean SapsBestSolMean2

100

SapsBestSolMean SapsBestSolMean × MeanDPLLDepth

74

GsatBestSolCV GsatBestSolCV × MeanDPLLDepth

21

VCGClauseMean × GsatFirstLMRatioMean GsatFirstLMRatioMean

9

Feature Importance Feature Importance – Fixed Ratio Fixed Ratio

Variable Variable Cost of Cost of Omission Omission

SapsBestSolMean2

100

SapsBestSolMean × MeanDPLLDepth MeanDPLLDepth

74

GsatBestSolCV × MeanDPLLDepth MeanDPLLDepth

21

VCGClauseMean VCGClauseMean × GsatFirstLMRatioMean

9

CP 2004

SAT vs. UNSAT SAT vs. UNSAT

• Training models separately

Training models separately for SAT and UNSAT instances:

– good models require fewer features fewer features – model accuracy improves accuracy improves – c/v no longer an important feature for VR data – Completely different features different features are useful for SAT than for UNSAT

• Feature importance on SAT

SAT instances:

– Local Search Local Search features sufficient

• 7 features for good VR model
• 1 feature for good FR model (SAPSBestSolCV x SAPSAveImpMean)

– If LS features omitted, LP + DPLL LP + DPLL search space search space probing

• Feature importance on UNSAT

UNSAT instances:

– DPLL search space DPLL search space probing – Clause graph Clause graph features

CP 2004

Beyond Ratio: Weighted CG C Beyond Ratio: Weighted CG Clustering Coefficient lustering Coefficient

• Byproduct of our analysis: a very strong correlation

strong correlation between weighted CG clustering coefficient and v/c

• Clustering coefficient is a more fundamental concept than

v/c, since it describes the structure of the constraints structure of the constraints explicitly, not implicitly.

– correlation between (unweighted) CC and hardness has been shown for other constraint problems

• ther constraint problems

(e.g., graph coloring, combinatorial auctions)

• We have a proof sketch

proof sketch of this correlation

CP 2004

Conclusions Conclusions

• Can construct good models

good models for DPLL solvers

• These models can be analyzed

analyzed to gain understanding about what makes instances hard or easy for solvers

• Algorithm portfolios

Algorithm portfolios can be constructed (Satzilla) (Satzilla)

• More specifically:

– Strong relationship between LS and DPLL LS and DPLL search spaces – Our approach automatically identified automatically identified importance of c/v – SAT/UNSAT instances SAT/UNSAT instances have very different performance characteristics; it helps to model them separately – Clustering Coefficient Clustering Coefficient explains why c/v is important in terms of local properties of constraint graph

CP 2004