Anytime Best First search: Empirical evaluation Natalia Flerova - - PowerPoint PPT Presentation

Anytime Best First search: Empirical evaluation

Natalia Flerova Radu Marinescu Rina Dechter

University of California Irvine IBM Research Ireland

Anytime Repairing AOBF (wR-AOBF)

(based on ARA* [Likhachev et al. 2003] ) Main idea:

 Run search iteratively, decreasing w (like wAOBF)  BUT re-use the results of previous iterations!

 Consider some starting weight w, put start node in OPEN  until w=1 or out of time

 Search for solution in AOBF manner, expanding nodes on OPEN with best

f(n), but only if f(n) is better than the current best cost

 keep track of nodes that are already on CLOSED, but whose g(n) has

changed (INCONS list)

 output the solution found by Weighted A*  Decrease w by fixed positive value δ  Move all nodes from INCONS to OPEN  re-compute f(n) for all nodes in OPEN with new w

Experiments

• 3 algorithms:

– wAOBF – wR-AOBF – BRAOBB [Otten, Dechter'11]

• Experimental settings:

– I-bounds attempted{2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22} – Time limit:

• Main dataset: 1 hour
• Pascal2 dataset: 10 hours
• We also consider short term results: 20 sec, 1 min, 10 min

– Memory limit:

• Small memory: 4 Gb
• Large memory: 80 Gb

– –

Weight schedule:

Experiments

Benchmarks:

– Main set (17 hard problems):

• Pedigrees
• Binary grids
• WCSP

– Pascal2 set:

• Alchemy (1 instance)
• CSP (61 instances)
• DBN (116 instances)
• Grids (21 instances)
• Imagealignment (10 instances)
• Objectdetection (37 instances)
• Promedas (86 instances)
• Proteinfolding (10 instances)
• Proteinprotein (11 instances)
• Segmentation (100 instances)

The main conclusions

Impossible to claim absolute dominance of any

• f 3 algorithms.
• The performance greatly depends on:

– benchmark – heuristic strength – memory limit

The main conclusions

Memory limit:

– Best First schemes (wAOBF and wR-AOBF)

greatly benefit from additional memory.

– With 80 Gb memory limit they:

• can find solutions to many instances, for which no

solutions were found for 4 Gb

• Can find optimal solutions for problems, for which only

loose approximations are found for 4 Gb – BRAOBB is less hurt by lack of memory, having

similar performance for both 4 Gb and 80 Gb

Benchmarks:

• Main data set:

– wAOBF and wR-AOBF perform better

compared to BRAOBB on instances with a lot

• f determinism (pedigrees and grids)

– BRAOBB performs better on instances with

little determinism (WSCP)

The main conclusions

Benchmarks:

• Pascal2 data set: (only results for 4 Gb)

–

CSP:

• BF schemes manage to find solutions on considerably fewer instances

than BRAOBB (e.g. i=2: wAOBF 13 vs BRAOBB 61 instances)

–

DBN:

• No solutions by wAOBF and wR-AOBF

. BRAOBB finds solutions for 108 (i=2) to 60 instances (i=22)

–

Grids:

• No solutions by wAOBF and wR-AOBF

. BRAOBB finds solutions for 21 (i=2) to 13 instances (i=22)

–

Image alignment:

• BRAOBB finds solutions for more instances than wAOBF and wR-AOBF

.

–

The main conclusions

Given a particular instance: (e.g.pedigree31, C*=-130.461)

Impact of heuristics

I=8: MBE-ROOT = -123.324 BRAOBB: 1st sol: [0] -164.472 Last sol: [1075] -140.293 wAOBF-sqrt: 1st sol: [421] -176.6877 Last sol: [5400] -143.4375 I=10: MBE-ROOT = -124.32 BRAOBB: 1st sol: [0] -151.657 Last sol: [655] -138.952 wAOBF-sqrt: 1st sol: [0] -172.7116 Last sol: [6035] -136.6440

'Our' grids vs pascal2 grids

75-20-5: MBE-ROOT = -8.24529 C*=-12.7195 BAYES, k=2, ar=3, n=400, f=400, e=1120 w*=27, h=99 A lot of determinism wAOBF-sqrt: 1st sol: [0] -23.2776 Last sol: [6479] -12.7195 Grid20x20.f10: MBE-ROOT = -1506.39 C*=-1309.72 MARKOV, k=2, ar=2, n=400, f=1200, e=800 w*=44, h=68 No determinism wAOBF-sqrt: OOM

Why it takes so much time to find the first solution?

wAOBF-sqrt: 1st sol: [88] -3669.6776 Nodes: 61248 Last sol: [1200] -2999.9340 Nodes: 377165 No C*, OOM wAOBF-sqrt: 1st sol: [226] -2006.2421 Nodes: 228571 Last sol: [661] -1547.1252 Nodes: 996572 C*=-1547.1252