Motivation Phase transition Phase Transitions in Classical Planning: Formalization An Experimental Study Experiments Approaches 1st test series 2nd test series Jussi Rintanen Discussion Conclusions Albert-Ludwigs-Universität Freiburg, Germany June 7, ICAPS’04
Motivation Almost all of the standard benchmarks are solvable Motivation by simple polynomial-time problem-specific Phase transition algorithms. Formalization Experiments ◦ Narrow class, not representative (in general; Approaches applications)! 1st test series ◦ Say little about performance of planners in general! 2nd test series How were difficult instances obtained: increase the Discussion number of packages, airplanes, ... ( ≥ 2000 state Conclusions variables, ≥ 40000 operators, ) Actually, 20 state variables and 40 operators is a challenge to many planners!!!
How to get challenging benchmarks? Analogy: SAT benchmarks Motivation Phase transition Notoriously difficult to come by just by inventing 1 Formalization some. Experiments Prove that for any algorithm the problem is difficult 2 Approaches (pigeon-hole formulas for DPLL/resolution!): not very 1st test series interesting... 2nd test series Discussion Go to Intel and ask for problems that resist solution. 3 Conclusions (Which company is the Intel of planning?) Experiment with the set of all instances, identifying 4 problem parameters that make planning difficult.
Planning phase transition 1111 Motivation Phase transition 0111 1011 1101 1110 Formalization Experiments Approaches 1st test series 0011 0101 0110 1010 1001 1100 2nd test series Discussion Conclusions 0001 0010 0100 1000 0000
How to solve the easiest problems Bylander 1996: Bylander 1996: Motivation insolubility by solvable by a Phase transition a simple syntactic simple hill−climbing Formalization test algorithm Experiments Approaches 1st test series 2nd test series Discussion Conclusions
Problem instances Motivation Characterized by the following parameters. Phase transition number n of state variables (size of state space) 1 Formalization number of operators 2 Experiments Approaches number of effect literals in operators ( our 3 1st test series experiments: 2 ) 2nd test series number of precondition literals ( our experiments: 3 ) 4 Discussion number of goal literals ( our experiments: n ) 5 Conclusions number of goal literals with value differing from the 6 initial value ( our experiments: n ).
Further restrictions Motivation Phase transition Model B (Bylander 1996): no restrictions. Formalization Model C: each literal occurs as effect at least once. Experiments Otherwise very likely some goal literals cannot be Approaches made true: many trivially insoluble instances. 1st test series Model A: each literal occurs as effect about the same 2nd test series Discussion number of times. Conclusions Model C does not fully fix the problem in Model B, so we go a bit further in Model A.
Experimental set-up Motivation Fix other parameters, and vary the number of Phase transition operators. Formalization ⇒ What happens to difficulty when the number of = Experiments Approaches arcs ( ∼ operators) in the transition graph is varied? 1st test series Number of instances for given parameter values is 2nd test series astronomic, so we sample the space of all problem Discussion instances. Conclusions Evaluate runtimes and plan lengths of different planners.
Approach: satisfiability planning Motivation First developed by Kautz and Selman (1992, 1996) Phase transition Translate planning into formulae, find plans with a Formalization SAT solver. Experiments The commercially most successful planning Approaches SAT Planning technology ( outside planning !!!): bounded State-space search LPG model-checking since 1999 a leading technology for 1st test series model-checking, mega-USD business 2nd test series Has not been considered competitive on current Discussion Conclusions benchmarks. Main reason: “faster” planners give no quality guarantees.
Planner: SP Motivation Our own (here: SP , for Satisfiability Planning) Phase transition Improved problem encodings: formula size often ≤ 1 Formalization 5 of BLACKBOX and runtimes 1 1 1 Experiments 10 , 100 , 1000 on big Approaches problems. SAT Planning State-space search With novel evaluation strategies very good on LPG standard benchmarks without any 1st test series benchmark-specific tricks!! See ECAI’04 paper. 2nd test series Discussion BLACKBOX about as good as SP on the small Conclusions problem instances we discuss in this talk.
Approach: heuristic state-space search Motivation Phase transition Formalization Heuristic search in the state space + distance Experiments heuristics Approaches SAT Planning State-space search Reference: Bonet and Geffner (2001) LPG 1st test series Favored by the planning competition community. 2nd test series Discussion Conclusions
Planners: HSP an FF Motivation Phase transition Formalization HSP (Bonet and Geffner, 2001) 1 Experiments FF (Hoffmann and Nebel, 2001) Approaches 2 SAT Planning additional techniques inspired by the standard State-space search LPG benchmarks 1st test series very good on standard benchmarks 2nd test series Discussion Conclusions
LPG: planning graphs + heuristic search Motivation Phase transition Developed by Gerevini and Serina (1999-) Formalization Basic data structure: planning graph from Graphplan Experiments (Blum & Furst, 1995) Approaches SAT Planning Local search with incomplete plans ( ∼ planning State-space search LPG graphs) 1st test series Advantage over earlier planning graph approaches: 2nd test series length increased dynamically during search Discussion (optimality given up!) Conclusions
First test series Motivation Model A (Results on Model C are similar.) Phase transition 20 state variables, from 36 to 120 operators at Formalization Experiments interval ∼ 6 Approaches About 500 soluble instance for each operators / 1st test series variable ratio (about 8000 soluble instances out of Runtimes Plan lengths 100000, identified by a BDD-based breadth-first 2nd test series search planner) Discussion Measure runtimes and plan lengths (timeout 10 Conclusions minutes)
Runtimes: SP Model A: Distribution of runtimes on SP Motivation Phase transition 100 Formalization runtime in seconds Experiments Approaches 10 1st test series Runtimes Plan lengths 2nd test series 1 Discussion Conclusions 0.1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 ratio # operators / # state variables
Runtimes: LPG Model A: Distribution of runtimes on LPG Motivation Phase transition 100 Formalization runtime in seconds Experiments Approaches 10 1st test series Runtimes Plan lengths 1 2nd test series Discussion Conclusions 0.1 0.01 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 ratio # operators / # state variables
Runtimes: FF Model A: Distribution of runtimes on FF Motivation Phase transition 100 Formalization runtime in seconds Experiments Approaches 10 1st test series Runtimes Plan lengths 1 2nd test series Discussion Conclusions 0.1 0.01 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 ratio # operators / # state variables
Runtimes: HSP Model A: Distribution of runtimes on HSP Motivation Phase transition 100 Formalization runtime in seconds Experiments Approaches 10 1st test series Runtimes Plan lengths 1 2nd test series Discussion Conclusions 0.1 0.01 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 ratio # operators / # state variables
Plan lengths: SP Model A: Distribution of plan lengths on SP 250 Motivation Phase transition Formalization 200 number of operators Experiments Approaches 150 1st test series Runtimes Plan lengths 100 2nd test series Discussion Conclusions 50 0 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 ratio # operators / # state variables
Plan lengths: LPG Model A: Distribution of plan lengths on LPG 250 Motivation Phase transition Formalization 200 number of operators Experiments Approaches 150 1st test series Runtimes Plan lengths 100 2nd test series Discussion Conclusions 50 0 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 ratio # operators / # state variables
Plan lengths: FF Model A: Distribution of plan lengths on FF 250 Motivation Phase transition Formalization 200 number of operators Experiments Approaches 150 1st test series Runtimes Plan lengths 100 2nd test series Discussion Conclusions 50 0 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 ratio # operators / # state variables
Further tests: scalability Motivation 20, 40 and 60 state variables ( ∼ 10 6 , 10 12 , 10 18 Phase transition Formalization states) Experiments No efficient insolubility test: could not distinguish Approaches between insoluble and very difficult instances. 1st test series Main results for SP only (SP scales up by far the 2nd test series Phase transition best.) Runtimes Plan lengths LPG, HSP and FF: proportion of solved instances LPG, HSP , FF Discussion wrt SP (timeout 10 minutes) Conclusions
Recommend
More recommend
