The SAT 2005 Competition Industrial category Certified UNSAT - PowerPoint PPT Presentation

The SAT 2005 Competition What’s new this year The benchmarks First stage results All categories Random category Crafted category Industrial category Second stage results Random category Crafted category The SAT 2005 Competition Industrial category Certified UNSAT Special track Fourth Edition Non clausal special track Next contest? Daniel Le Berre and Laurent Simon Pseudo Boolean evaluation Eighth International Conference on Theory and Applications of Satisfiability Testing, SAT’05 1 / 55

Agenda The SAT 2005 Competition What’s new this year What’s new this The benchmarks year The benchmarks First stage results First stage results All categories All categories Random category Random category Crafted category Industrial category Crafted category Second stage Industrial category results Random category Crafted category Second stage results Industrial category Random category Certified UNSAT Special track Crafted category Non clausal special Industrial category track Next contest? Certified UNSAT Special track Pseudo Boolean Non clausal special track evaluation Next contest? Pseudo Boolean evaluation 2 / 55

They support us The SAT 2005 Competition Thank you! What’s new this year The benchmarks First stage results All categories Random category Crafted category Industrial category Second stage results Random category Crafted category Industrial category Certified UNSAT Special track Non clausal special track Next contest? Pseudo Boolean evaluation 3 / 55

The new judges The SAT 2005 Competition What’s new this year The benchmarks First stage results Armin Biere Specialist about industrial benchmarks and All categories Random category solvers. Crafted category Industrial category Olivier Kullmann Specialist about k-SAT. Generated all the Second stage results benchmarks for the random category. Random category Crafted category Allen van Gelder Well aware of the CASC competition. Industrial category Proposed the new scoring scheme. Managed Certified UNSAT Special track the certified unsat special track. Non clausal special track All the decisions were taken in agreement with the judges Next contest? Pseudo Boolean evaluation 4 / 55

The special tracks The SAT 2005 Competition Certified UNSAT a specific category in which the solvers What’s new this year must output a certificate of unsatisfiability. The benchmarks The proof format and a proof checker were First stage results provided by Allen van Gelder. All categories Random category Only two participants: zchaff and ttsp-3.0 Crafted category Industrial category Pseudo Boolean evaluation dedicated to solvers managing Second stage pseudo-boolean constraints and optimization results Random category functions. Crafted category Industrial category Managed by Vasco Manquinho and Olivier Certified UNSAT Special track Roussel Non clausal special http://www.cril.univ-artois.fr/PB05/ track 8 solvers (17 variants) from 8 submitters. Next contest? Pseudo Boolean Non clausal evaluation dedicated to solvers able to take evaluation gates as input. The input format was provided by Fahiem Bacchus and Toby Walsh. No solver submission. One benchmark submission. 5 / 55

What’s new in the rules The SAT 2005 Competition What’s new this year The benchmarks ◮ Competition and Demonstration divisions. First stage results All categories Competition the source code of the solver must be Random category Crafted category Industrial category available after the competition. Second stage Demonstration a binary version of the solver must be results Random category available for research purpose. Crafted category Industrial category ◮ Participation to the competition must benefit to the Certified UNSAT community Special track ◮ By providing source code, binary or benchmarks Non clausal special track ◮ By supporting the conference and the competition Next contest? Pseudo Boolean evaluation 6 / 55

The new scoring scheme The SAT 2005 Competition What’s new this year The benchmarks First stage results All categories Benchmark purse to be divided equally among the solvers Random category Crafted category able to solve it. Industrial category Second stage Speed purse to be divided unequally among the solvers able results to solve a given benchmark. Random category Crafted category Industrial category Series an extra credit is given for each series solved. Certified UNSAT Special track Solver his score is the sum of the credits obtained per Non clausal special benchmarks solved. track Next contest? Pseudo Boolean evaluation 7 / 55

The new award scheme The SAT 2005 Competition What’s new this year The benchmarks First stage results All categories Random category Crafted category ◮ Three categories: industrial, crafted and random Industrial category Second stage ◮ Three specialties: SAT, UNSAT and SAT+UNSAT results Random category ◮ Three medals: gold, silver and bronze Crafted category Industrial category So we have a total of 27 awards this year! Certified UNSAT Special track Non clausal special track Next contest? Pseudo Boolean evaluation 8 / 55

Invariants The SAT 2005 Competition What’s new this year The benchmarks First stage results All categories Random category Crafted category ◮ Only 3 solvers per submitter can enter the first stage, Industrial category competition division. Second stage results Random category ◮ Only 1 solver per submitter can enter the second stage, Crafted category Industrial category competition division. Certified UNSAT Special track Non clausal special track Next contest? Pseudo Boolean evaluation 9 / 55

Random category The SAT 2005 Competition What’s new this year The benchmarks First stage results All categories Random category Crafted category ◮ 3-SAT, 5-SAT, 7-SAT Industrial category Second stage ◮ From 400 to 10000 variables. results Random category ◮ 285 SAT and 105 UNSAT benchmarks Crafted category Industrial category ◮ Answers known in advance Certified UNSAT Special track Non clausal special track Next contest? Pseudo Boolean evaluation 10 / 55

Industrial category The SAT 2005 Competition What’s new this year The benchmarks First stage results Zarpas New formal verification benchmarks from IBM All categories Random category (FV 2004) Crafted category Industrial category Velev Known VLIW-SAT (2.0 and 4.0), Second stage VLIW-UNSAT 2.0 and Liveness UNSAT 2.0 results Random category Crafted category Grieu VMPC invertion, open cryptographic problem Industrial category Certified UNSAT Narain VPN models generated from Alloy Special track Maris Planning benchmarks Non clausal special track Wider range of problems than in previous edition. Next contest? Pseudo Boolean evaluation 11 / 55

Crafted category The SAT 2005 Competition What’s new this year Sat’04 Previous year hard, unsolved benchmarks The benchmarks Biere LinvRinv benchmarks (proposed by Cook last First stage results year) All categories Random category Crafted category Sabharwal Counting/Ordering/Pebbling problems Industrial category Second stage Jarvisalo Based on 3-Regular graphs results Random category Lynce Social Golfer problem (A golf problem in St Crafted category Industrial category Andrews?) Certified UNSAT Special track Sorge Algebraic benchmarks Non clausal special Markstrom Problems generating long learned clauses. track Next contest? Roussel PHNF form of previous year medium Pseudo Boolean benchmarks evaluation Wider range of problems than in previous edition. 12 / 55

Environment The SAT 2005 Competition What’s new this The hardware: year The benchmarks LRI 16 Athlon 1800+ with 1GB RAM First stage results UC 8 Athlon 1800+ with 2 GB RAM All categories Random category 32 Pentium III 450 with 1GB of RAM Crafted category Industrial category Second stage ◮ Running GNU Linux (RH flavor). results Random category ◮ Solvers compiled with GCC 3.3.5. Crafted category Industrial category ◮ Java solver using Java 1.5.0 02 JVM. Certified UNSAT Special track Provided by: Non clausal special track ◮ LINC Lab, Department of ECECS, University of Next contest? Cincinnati Pseudo Boolean evaluation ◮ LRI, Universit´ e de Paris-Sud 13 / 55

The first stage The SAT 2005 Competition What’s new this year The benchmarks First stage results All categories Random category ◮ Aim: to detect the most promising solvers for a given Crafted category Industrial category (category,specialty) Second stage results ◮ 20 minutes timeout (Greater than in previous years) Random category Crafted category ◮ Solvers answering incorrectly move to demonstration Industrial category Certified UNSAT division Special track Non clausal special track Next contest? Pseudo Boolean evaluation 14 / 55