Anatomy of a Scheduling Competition Marco Benedetti, Federico Pecora - - PowerPoint PPT Presentation
Anatomy of a Scheduling Competition Marco Benedetti, Federico Pecora and Nicola Policella University of Orl eans, marco.benedetti@univ-orleans.fr Inst. for Cognitive Science and Technology, federico.pecora@istc.cnr.it European Space Agency,
Marco Benedetti, Federico Pecora and Nicola Policella
University of Orl´ eans, marco.benedetti@univ-orleans.fr
European Space Agency, nicola.policella@esa.int
ICAPS Workshop on “Scheduling a Scheduling Compoetition” Providence (RI), September 22nd, 2007
1
– temporal constraints (e.g., generalized precedence constraints, . . . ) – non-temporal constraints (e.g., limited capacity resources, load, . . . )
Anatomy of a Scheduling Competition
2
theoretical and applications appeal
– drives research in Artificial Intelligence, Operations Research, Constraint Programming, Management Science – fosters development of decision support systems (e.g., production planning, supply chain management)
evaluations
scheduling systems/algorithms regularly appear in the literature (e.g., [Beck and Fox, 2000], [Godard et al., 2005], [Barbulescu et al., 2006], . . . )
– paper-specific evaluations usually ad-hoc and limited to the scope of the paper – difficult to see the big picture, easy to duplicate results
Anatomy of a Scheduling Competition
3
– to collectively discuss the issue of creating a common forum for comparatively evaluating different approaches to scheduling – in other words: why so we need a scheduling competition? what is it? how to organize it?
– puts forth a series of specific questions regarding the scheduling competition – outline some tentative answers over the backdrop of current competitions in Computer Science (CS)
– a blueprint for a scheduling competition which is well-formed and operative, and which is inclusive with respect to the broad scope of the scheduling community
Anatomy of a Scheduling Competition
4
advantages, pitfalls, and trends
underpinning the establishment of a competition
– A. Motivation: motivation underlying the organization of the competition. May be purely academic (promoting the comparison of specific algorithms, methods or approaches to better understand the theoretical aspects of the computational problem); may include industry-oriented aspects (e.g., fitness for real-world problems, usability, impact on existing processes, etc.) – B. Knowledge Representation: the formal representation (or lack thereof) in which competition benchmarks (in) and competitor results (out) are expressed
Anatomy of a Scheduling Competition
5
– C. Tracks: the organization of the competition into tracks. Tracks defined as competition sub-divisions which are determined by differences in how the problem is approached and/or how the problem is defined. – D. Benchmarks: the nature of the benchmark source. Indicates whether problems are contributed by the participants (contrib), taken from a community-maintained repository (library), or disclosed “on-line” during the competition through a competition server. – E. Measure: the evaluation criteria employed to determine system ranking. f(σ, τ, φ, ω), where:
∗ σ : the degree to which a system solves the given benchmark(s) → number of solved problems in SAT competition ∗ τ : the amount of time taken by the system to complete the benchmark(s) → CPU time to find a plan in IPC ∗ φ : a measure related to the quality of solutions found → number of satisfied soft constraints in CSP competition ∗ ω : other measures related to the use of the participating system → system’s ease of use, portability, etc. in ICKEPS
– F. Disclosure: whether or not systems need to be completely disclosed in order to
Anatomy of a Scheduling Competition
6
submission, and therefore made public), binary (source code submission not required, binaries made public), remote (systems are run on participants’ computational resources and/or accessed remotely during the competition). – G. Participation: the type and number of participants. Indicates whether the current state-of-the-art is conceived purely for academic evaluation (AC) and/or if the technology has industrial potential (IND).
Anatomy of a Scheduling Competition
7
Competition KR Bench. Tracks Measure Part. Disc. Description/Motivation CASC 1996 (11) in, out library 5 divisions, 13 categories f(σ, τ, φ) ac (20) source To stimulate Automated Theorem Proving (ATP) research and system development, and to expose ATP systems within and beyond the ATP community (held in conjunction with CADE). CLIMA 2005 (3) N/A N/A N/A f(σ, ω) ac (6) remote To stimulate research in the area
multi-agent systems by identifying key problems and collecting suitable benchmarks that can serve as milestones for testing new approaches and techniques from computational logics. CSP 2005 (2) in, out library, contrib 5 categories f(σ, τ, φ) ac (21) binary To improve understanding of the sources of Constraint Satisfaction Problem (CSP) solver efficiency, and the options that should be considered in crafting solvers.
Anatomy of a Scheduling Competition
8
Competition KR Bench. Tracks Measure Part. Disc. Description/Motivation GGP 2005 (3) in,
{WIN, LOOSE} server N/A f(σ) ac (12) remote To assess state-of-the-art in General Game Playing (GGP) systems, i.e., automated systems which can accept a formal description of an arbitrary game and, without further human interaction, can play the game effectively. A $10,000 prize is awarded to the winning team. ICGA 1977 (30) N/A N/A 32 games f(σ) ac/ind (60) remote The International Computer Games Association (ICGA) was founded by computer chess programmers in 1977 to organise championship events for computer programs. The ICGA Tournament aims to facilitate contacts between Computer Science and Commercial Organisations, as well as the International Chess Federation.
Anatomy of a Scheduling Competition
9
Competition KR Bench. Tracks Measure Part. Disc. Description/Motivation ICKEPS 2005 (2) N/A server N/A f(σ, φ, ω) ac (7) remote To promote the knowledge-based and domain modeling aspects of Planning and Scheduling (P&S), to accelerate knowledge engineering research in AI P&S, to encourage the development and sharing
prototype tools
software platforms that promise more rapid, accessible, and effective ways to construct reliable and efficient P&S systems. IPC 1998 (5) in, out library 2 parts, 3 tracks f(σ, τ, φ) ac (12) binary To provide a forum for empirical comparison
planning systems, to highlight challenges to the community in the form of problems at the edge of current capabilities, to propose new directions for research and to provide a core of common benchmark problems and a representation formalism to aid the comparison and evaluation of planning systems.
Anatomy of a Scheduling Competition
10
Competition KR Bench. Tracks Measure Part. Disc. Description/Motivation ITC 2003 (1) in, out library N/A f(σ, φ) ac (11) binary The International Timetabling Competition was designed in order to promote research into automated methods for timetabling. It was not designed as a comparison of methods, and discourages drawing strict scientific conclusions from the results. A prize of ¤300 + free registration to PATAT 2004 was awarded to the winner. PB- Eval 2005 (3) in,
{YES, NO, ?} library N/A f(σ, τ, φ) ac (10) binary The goal of the Pseudo-Boolean (PB) Evaluation is to assess the state of the art in the field of PB solvers. QBF 2007 (5) in,
{YES, NO, ?} library, contrib 3 tracks f(σ, τ) ac (12) binary Assessing the state of the art in the field of QBF solvers and QBF-based applications.
Anatomy of a Scheduling Competition
11
Competition KR Bench. Tracks Measure Part. Disc. Description/Motivation RoboCup 1997 (10) N/A N/A 4 leagues, 7 divisions f(σ, ω) ac/ind (440) binary To foster AI and intelligent robotics research by providing a standard problem where wide range
technologies can be integrated and examined, as well as being used for intergrated project-oriented education. SAT 2002 (5) in,
{YES, NO, ?} library, contrib 3 tracks f(σ, τ) ac (47) source To identify new challenging benchmarks and to promote new solvers for the propositional SATisfiability problem (SAT) as well as to compare them with state-of- the-art solvers. SMT- COMP 2005 (3) in,
YES, NO, ?} library 11 divisions f(σ) ac/ind (12) binary Push state-of-the-art in Satisfiability Modulo Theories (SMT) for verification applications, in which background theories are used to express verification conditions (e.g., empty theory, real/integer arithmetic, theories of program or hardware structures).
Anatomy of a Scheduling Competition
12
Competition KR Bench. Tracks Measure Part. Disc. Description/Motivation TAC 2002 (6) in server 3 scenarios f(φ) ac/ind (23) remote An international forum designed to promote and encourage high quality research into the trading agent problem. TANCS 1999 (2) N/A library 4 divisions f(σ, τ, φ) ac (6) binary To compare the performance of fully automatic, non classical ATP systems (based on tableaux, resolution, rewriting, etc.) in an experimental setting and promote the experimental study on theorem proving and satisfiability testing in non classical logics. Temination 2004 (4) in,
{YES, NO, ?} library, contrib 3 categories f(σ) ac (6) binary A competition for termination- proving systems.
Anatomy of a Scheduling Competition
13
competition
1The answers provided are intended to be questionable and provocative!
Anatomy of a Scheduling Competition
14
– ICGA also accepts participants whose systems derive from commersial products – RoboCup oriented towards showcasing industrial products, and sposors integrated project-
– TAC draws motivation from e-trading and is overseen by an industrail advisory board
participation and/or industrial benchmarks
– competition stems from need to push state-of-the-art in algorithms and/or promote new challenging benchmarks – competitive context has fostered theoretical innovation as well as consistent development effort
Anatomy of a Scheduling Competition
15
Q1: Which are the academic and industrial motivations for a scheduling competition?
– The scheduling problem has been widely studied in AI, OR, CP and MS. A scheduling competition can foster cross-fertilization among these different areas, so we can achieve a deeper understanding of the scheduling techniques across a wide set of problems
– Development of commercial solvers is backed by software engineering solutions which facilitate development of market-grade products; research prototypes are seldom developed beyond the “demonstrator” level : a competition can bring high-quality implementation to research and high-quality research to industry.
Anatomy of a Scheduling Competition
16
Q1: Which are the academic and industrial motivations for a scheduling competition? (cont.)
– A regular comparative evaluation can accelerate the process of finding new challenges, producing the shared goal of finding solutions to these challenges.
– A competition can contribute to rationalizing and avoiding duplication of research results; it can lead to a more comprehensive global picture of the state-of-the-art in scheduling.
Anatomy of a Scheduling Competition
17
CASC, CSP, IPC, QBF, SAT):
– reference problem is described by precise formal attributes, e.g., variables and constraints in CSP, CNF formulae in SAT
(e.g., TANCS, RoboCup, ICKEPS, ICGA):
– no need for or impossible to define formal benchmark representation
Anatomy of a Scheduling Competition
18
– the competition was established through the collaborative deliberations which brought to PDDL [Ghallab et al., 1998] – narrow initial focus has facilitated competition establishment – there remains no general agreement that the planning problem should be expressed in this formalism – PDDL has introduced further bias in planning research
Anatomy of a Scheduling Competition
19
Q2: Should the scheduling competition provide one or more formal languages to represent benchmark instances?
– different forms of scheduling vary widely – a number
formalisms exists: project scheduling [Schwindt, 1995], job-shop with due dates and dynamic job arrivals [Demirkol et al., 1998], permutation flow- shop [Watson et al., 2002] – many are sufficiently expressive to represent a number of variants of the scheduling problem
to use a representation which is well understood and supported
Anatomy of a Scheduling Competition
20
Q2: Should the scheduling competition provide one or more formal languages to represent benchmark instances? (cont.)
systems into clearly defined categories, and the definition of tracks would not be biased by existing sub-communities
language; scheduling already has a wealth of accepted formalisms
– avoiding the pitfalls of the PDDL approach: (a) providing more than one formalism and/or a super-set; (b) establishing a formal syntax and semantics; (c) providing open source parsers and/or XML-based format; (d) ensuring a continuous, open revision process to keep up with new challenges
Anatomy of a Scheduling Competition
21
Q3: What rationale should be followed to define tracks?
TANCS), in solving strategies (e.g., CASC), or both (e.g., IPC)
– in planning, systems compete in tracks in which distinctions are made wrt problem, solution and solver characteristics
as well as different problem formulations
– E.g., Precedence Constraint Posting and Genetic Algorithm based approaches [Cesta et al., 2002, Mendes et al., 2005] for resource constrained project scheduling: PCP slower but provides information-rich solution; GA faster, but returns “flat” solution
Anatomy of a Scheduling Competition
22
Q4: Which tracks should be provided in a scheduling competition?
– problems described by a network of activities, which establishes the temporal relations among the different activities of the problem, and a set of limited capacity resources, which are required in order to execute the different activities.
– problems consisting of a set of activities which compete for the same set of resources; typically the available resources are not sufficient to satisfy all the activity requests, thus a solution has to sacrifice some activities while preserving schedule quality.
Anatomy of a Scheduling Competition
23
Q4: Which tracks should be provided in a scheduling competition? (cont.)
– problems such as the above where the input instance becomes available over time during solving; solvers thus have to react to new requests (e.g., allocating jobs to machines) with
– the problem of maintaining the consistency of a pre-defined schedule during its execution in a real or simulated environment; schedule revision must be quick, and sometimes solution quality must be given secondary priority as the execution of the schedule does not allow for time-intensive computation and/or solution continuity is preferred over drastic on-line changes of the schedule.
Anatomy of a Scheduling Competition
24
problem instances for solver comparison. The competition changed the way systems were evaluated:
– random formulae – structured problems – industrial benchmarks
understanding of solving approaches by exposing new problem characteristics which are not present in random problems Q5: Should we follow the SAT competition approach?
Anatomy of a Scheduling Competition
25
Q6: Should the competition include benchmarks for “dynamic” scheduling problems, such as on-line scheduling and schedule execution monitoring?
be published on-line — see server mechanisms in TAC, GGP, ICKEPS Q7: Are currently available benchmark libraries sufficient to capture the features of scheduling aplications?
dynamic job-shop [Uzsoy, 1998], permutation flow shop [Watson, 2002], RCPSP [Schwindt, 1995], . . .
Anatomy of a Scheduling Competition
26
Q8: Should the scheduling competition evaluate algorithms, systems, or both?
implementation is difficult . . . ask John Hooker later!
– Evaluation in CASC, CSP, IPC, PB-Eval, SAT, QBF, TANCS depends on number of solved instances (σ) and time (τ) – Indeed, implementation has become a determining factor in recent editions of the SAT competition – Other competitions include metrics related to solution quality (φ), e.g., CASC, CSP, IPC, ITC (see Timetabling presentation); few competitions measure features of the system, e.g., ICKEPS
Anatomy of a Scheduling Competition
27
Q9: Which measure should be adopted in the scheduling competition?
straightforward measures such as makespan, lateness, (weighted) tardiness, . . .
structure, such as
– order strength: quantifies restrictiveness [Mastor, 1970] – resource strength: quantifies relationship between resource availability and demand [Schwindt, 1998] – schedule fluidity: average slack between activities in the schedule [Cesta et al., 1998] – flow time metrics, maximum stretch, disruptibility [Policella et al., 2004]
Anatomy of a Scheduling Competition
28
Q10: What level of disclosure should the competition require?
fosters cross fertilization
Anatomy of a Scheduling Competition
29
Q10: What level of disclosure should the competition require? (cont.)
– guarantees that the technology is sufficiently mature to be used by third parties; – it enables others to autonomously replicate results and test the scope of applicability of the technology in other domains; – it can safeguard against distorted claims, as it implies that anyone contributing a new algorithm must provide a reproducible comparison with relevant solvers; – commercially interested parties can evaluate binary prototypes in view of potential further investment in the technology.
Anatomy of a Scheduling Competition
30
answer the previous questions
– it should not be assertive in “proclaiming winners” because the outcome should showcase the most successful and novel approaches; – it should be assertive in “proclaiming winners” because researchers are human beings, and humans thrive under competition
– a competition which is not sufficiently inclusive can bias research (e.g., the effect of a restricted benchmark specification language) – an excessive focus on incremental details can hinder real progress in the field
Anatomy of a Scheduling Competition
31
Anatomy of a Scheduling Competition
32
References
[Barbulescu et al., 2006] Barbulescu, L., Howe, A. E., Whitley, L. D., and Roberts, M. (2006). Understanding Algorithm Performance on an Oversubscribed Scheduling Application. Journal of Artificial Intelligence Research, 27:577–615. [Beck and Fox, 2000] Beck, J. and Fox, M. (2000). Constraint-directed techniques for scheduling alternative
[Cesta et al., 2002] Cesta, A., Oddi, A., and Smith, S. (2002). A Constrained-Based Method for Project Scheduling with Time Windows. Journal of Heuristics, 8(1):109–135. [Cesta et al., 1998] Cesta, A., Oddi, A., and Smith, S. (June, 1998). Profile-Based Algorithms to Solve Multi-Capacitated Metric Scheduling Problems. In Proceedings of the 5th International Conference on Artificial Intelligence Planning Systems. [Demirkol et al., 1998] Demirkol, E., Mehta, S., and Uzsoy, R. (1998). Benchmarks for Shop Scheduling Problems. European Journal of Operational Research, 109(1):137–141. [Ghallab et al., 1998] Ghallab, M., Howe, A., Knoblock, C., McDermott, D., Ram, A., Veloso, M., Weld, D., and Wilkins, D. (1998). PDDL — The Planning Domain Definition Language, AIPS 98 Planning Competition Committee. [Godard et al., 2005] Godard, D., Laborie, P., and Nuitjen, W. (2005). Randomized Large Neighborhood Search for Cumulative Scheduling. In Proceedings of the International Conference on Automated Planning & Scheduling (ICAPS 2005), pages 81–89.
Anatomy of a Scheduling Competition
33
[Hooker, 1994] Hooker, J. (1994). Needed: An empirical science of algorithms. Operations Research, 42(2):201–212. [Mastor, 1970] Mastor, A. (1970). An Experimental and Comparative Evaluation of Production Line Balancing
[Mendes et al., 2005] Mendes, J., Gon¸ calves, J., and Resende, M. (2005). A Random Key Based Genetic Algorithm for the Resource Constrained Project Scheduling Problem. Technical report, AT&T Labs. AT&T Labs Research Technical Report TD-6DUK2C. [Policella et al., 2004] Policella, N., Smith, S. F., Cesta, A., and Oddi, A. (2004). Generating Robust Schedules through Temporal Flexibility. In Proceedings of the 14th International Conference on Automated Planning & Scheduling, ICAPS’04, pages 209–218. AAAI. [Schwindt, 1995] Schwindt, C. (1995). Project Generator ProGen/max and PSP/max-library, University Karlsruhe, Institute for Economic Theory and Operations Research. [Schwindt, 1998] Schwindt, C. (1998). Generation of Resource-Constrained Project Scheduling Problems Subject to Temporal Constraints. Report WIOR-543, Universit¨ at Karlsruhe. [Uzsoy, 1998] Uzsoy, R. (1998). Purdue Electronics Manufacturing Research Group repository: http://cobweb. ecn.purdue.edu/~uzsoy2/Problems/main.html. Accessed April, 2007. [Watson et al., 2002] Watson, J., Barbulescu, L., Whitley, L., and Howe, A. (2002). Contrasting Structured and Random Permutation Flow-Shop Scheduling Problems: Search-Space Topology and Algorithm Performance. Informs Journal on Computing, 14(2):98–123. [Watson, 2002] Watson, J.-P. (2002). Structured versus Random Benchmarks for the Permutation Flow-Shop
Anatomy of a Scheduling Competition
34
Scheduling Problem: http://www.cs.colostate.edu/sched/generatorNew/index.html. Accessed April, 2007.
Anatomy of a Scheduling Competition