Visualizations to Summarize Search Behavior Ian S. Howell !,# , - PowerPoint PPT Presentation

Visualizations to Summarize Search Behavior Ian S. Howell !,# , Berthe Y. Choueiry $ , and Hongfeng Yu % Constraint Systems Laboratory $ Visualizations Laboratory % Department of Computer Science & Engineering University of Nebraska-Lincoln Constraint Systems Laboratory 9/8/20 CP 2020 1

Introduction • Backtrack search when solving a constraint satisfaction problem (CSP) suffers from thrashing • Enforcing a consistency property reduces thrashing at the cost of further processing • The tradeoff of running a higher-level consistency (HLC) is poorly understood. • We propose to summarize search Constraint Systems Laboratory 9/8/20 CP 2020 2

Background • A constraint satisfaction problem (CSP) consists of a set of variables, the variables’ domains, and constraints. • A solution is a domain to variable assignment that satisfies all constraints. • Backtrack search is only sound and complete method of solving a CSP. Constraint Systems Laboratory 9/8/20 CP 2020 3

Background • Thrashing is the main malady of search – Repeatedly performing same logic • Applying a consistency algorithm after variable instantiation reduces thrashing • Past visualizations focus on debugging and inspecting search – Detect thrashing through isomorphic subtrees – Debug individual issues by investigating the state of variables, propagators, etc. Constraint Systems Laboratory 9/8/20 CP 2020 4

Background • Woodward et al. propose to measure: – thrashing with Backtracks per Depth (BpD) – HLC cost with consistency calls per depth (CpD) 35,000 35,000 60 GAC APOAC CPU Time: 140.81 sec. CPU Time: 23.77 sec. 30,000 30,000 #NV: 3,978,074 50 #NV: 59,181 # Backtracks #BT: 3,348,330 #BT: 53,212 25,000 25,000 # Calls to POAC #Calls POAC: 11,142 40 # Backtracks 20,000 20,000 #BTs 30 15,000 15,000 Filter #BTs No Filter 20 10,000 10,000 Wipeout 10 5,000 5,000 Depth Depth 0 0 0 1 26 51 76 101 126 151 176 201 226 251 276 301 326 351 376 1 26 51 76 101 126 151 176 201 226 251 276 301 326 351 376 Constraint Systems Laboratory 9/8/20 CP 2020 5

Contributions 1. Criteria for computing distance between two time samples based on the BpD 2. A clustering technique for summarizing search into a history of qualitatively distinct regimes 3. A new visualization that examines the behavior of variable ordering heuristics Constraint Systems Laboratory 9/8/20 CP 2020 6

Outline 1. Introduction 2. Background 3. Contributions 4. Outline 5. Distance between timestamps 6. Summarizing search history 7. Example: Analyzing structure 8. Visualizing variable ordering 9. Example: Variable ordering 10. Conclusions and discussion Constraint Systems Laboratory 9/8/20 CP 2020 7

Distance between timestamps • Design goal: capture the “shape” change #BT at time !′ #BT at time ! Depth Constraint Systems Laboratory 9/8/20 CP 2020 8

Distance between timestamps • Adapt Kulbak-Liebler Divergence • BpD distribution: – Use additive smoothing to account for zero- probability intervals Constraint Systems Laboratory 9/8/20 CP 2020 9

Distance between timestamps Constraint Systems Laboratory 9/8/20 CP 2020 10

Summarizing search history • Create a clustering tree – Use agglomerative, hierarchical clustering – Only merge temporally adjacent clusters – Each cluster’s representative is the middle of the interval of included timestamps Divergence time t 0 t 7 t 8 t 9 t 10 t 11 t 1 t 2 t 3 t 4 t 5 t 6 Constraint Systems Laboratory 9/8/20 CP 2020 11

Summarizing search history • Create a summarization tree – Cut the tree at a user-defined ! or – Cut the tree to include a " regimes Divergence δ r 1 r 2 r 3 r 4 r 5 time t 0 t 7 t 8 t 9 t 10 t 11 t 1 t 2 t 3 t 4 t 5 t 6 Constraint Systems Laboratory 9/8/20 CP 2020 12

Example: Analyzing structure Constraint Systems Laboratory 9/8/20 CP 2020 13

Example: Analyzing structure 0.1-summarization of GAC on MUG 100-25-3 Constraint Systems Laboratory 9/8/20 CP 2020 14

Example: Analyzing structure Primal graph of MUG 100-25-3 Constraint Systems Laboratory 9/8/20 CP 2020 15

Example: Analyzing structure Many large cycles Primal graph of MUG 100-25-3 Constraint Systems Laboratory 9/8/20 CP 2020 16

Example: Analyzing structure BpD of GAC and DPPC+ on MUG 100-25-3 Constraint Systems Laboratory 9/8/20 CP 2020 17

Visualizing variable ordering • Variable Instantiations per Depth (VIpD) – "($, &, ') is the number of instantiations of variable $ at depth & and time ' • Order the variables of the VIpD according to each variable’s weighted depth: Constraint Systems Laboratory 9/8/20 CP 2020 18

Example: Variable ordering VIpD of GAC and POAC with dom/wdeg on MUG 100-1-3 Constraint Systems Laboratory 9/8/20 CP 2020 19

Example: Variable ordering VIpD of POAC with dom/deg on MUG 100-1-3 Constraint Systems Laboratory 9/8/20 CP 2020 20

Example: Variable ordering 0.1-summarization of POAC with dom/wdeg on MUG 100-1-3 Constraint Systems Laboratory 9/8/20 CP 2020 21

Example: Variable ordering 0.1-summarization of POAC with dom/deg on MUG 100-1-3 Constraint Systems Laboratory 9/8/20 CP 2020 22

Conclusions and discussion • Summarizations can be used to help explain the (changing) behavior of search • Researchers and developers can use these tools iteratively to better study the impact of a strategy on a given problem • Summarizations can catch small and large behavior that a human could not Constraint Systems Laboratory 9/8/20 CP 2020 23

Conclusions and discussion • Initial results of BpD with binary branching give similar results • We currently provide a ‘post-mortem’ analysis. Future work includes an ‘in-vivo’ analysis that enables human experts Constraint Systems Laboratory 9/8/20 CP 2020 24

Acknowledgements • This research is supported by NSF Grant No. RI-1619344 and NSF CAREER Award No. III-1652846. • The experiments were completed utilizing the Holland Computing Center of the University of Nebraska, which receives support from the Nebraska Research Initiative. Constraint Systems Laboratory 9/8/20 CP 2020 25