The More the Merrier?! Evaluating the Effect of Landmark Extraction - PowerPoint PPT Presentation

The More the Merrier?! Evaluating the Effect of Landmark Extraction Algorithms on Landmark-Based Goal Recognition Kin Max Piamolini Gusm˜ ao Ramon Fraga Pereira Felipe Meneguzzi Pontifical Catholic University of Rio Grande do Sul - Brazil February 8, 2020 Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 1 / 21

Summary Motivation Goal Recognition and Landmark-Based Goal Recognition Experimentation Methodology Heuristics and Algorithms; Dataset; Evaluation Metrics Results Missing and Full Observations Missing, Noisy and Full Observations Conclusions Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 2 / 21

Motivation Explore different landmark extraction algorithms with landmark-based heuristics for goal recognition; Evaluate wether more landmarks lead to a more precise recognition; Inform future fine-tuning of algorithms to enhance recognition precision; Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 3 / 21

Goal Recognition and Landmark-Based Goal Recognition Goal Recognition : recognize agent’s goal based on it’s interactions with the environment; Landmarks : necessary facts or actions that must be present in any solution plan; Landmark-Based Goal Recognition : goal recognition techniques that leverage on landmarks. Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 4 / 21

Experimentation Methodology Heuristics and Algorithms Dataset Evaluation Metrics Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 5 / 21

Experimentation Methodology: Heuristics and Algorithms 2 landmark-based heuristics for goal recognition 1 : Goal Completion Heuristic ( h gc ); Landmark Uniqueness Heuristic ( h uniq ). 5 landmark extraction algorithms: Exhaust ; h m 2 ; RHW 3 ; Zhu & Givan 4 ; Hoffamnn et al. 5 2 threshold values: 0% and 10%. 1 Pereira et al., Landmark-Based Heurristics for Goal Recognition. AAAI, 2017. 2 Keyder et al., Sound and complete landmarks for and/or graphs. ECAI, 2010. 3 Richter et al., Landmarks revisited. AAAI 2008. 4 Zhu L. e Givan R., Landmark extraction via planning graph propagation, 2003. 5 Hoffmann et al., Ordered landmarks in planning. JAIR, 2004. Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 6 / 21

Experimentation Methodology: Dataset Dataset with goal and plan recognition problems 6 ; Problems from 15 classical planning domains; 6313 problems with missing and full observations with 5 observability levels (10%, 30%, 50%, 70% and 100%); 2850 problems with missing, noisy and full observations with 4 observability levels (25%, 50%, 75% and 100%). 6 Pereira F. R. e Meneguzzi F., Goal and plan recognition datasets using classical planning domains. 2017. Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 7 / 21

Experimentation Methodology: Evaluation Metrics Percentage of extracted landmarks; Accuracy (%); Spread in G ; Accuracy/Spread in G ratio; Recognition time (s). Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 8 / 21

Results Missing and Full Observations Missing, Noisy and Full Observations Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 9 / 21

Results: Missing and Full Observations 100 Exhaust h m RHW Zhu/Givan Hoffmann 80 Landmark Percentage 60 40 20 0 Algorithms Figure: Percentage of extracted landmarks. Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 10 / 21

Results: Missing and Full Observations 100 Exhaust h m RHW Zhu/Givan 80 Hoffmann Accuracy/Spread 60 40 20 0 10 30 50 70 100 Observability Figure: Accuracy/Spread in G ratio for h gc . Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 11 / 21

Results: Missing and Full Observations 100 Exhaust h m RHW Zhu/Givan 80 Hoffmann Accuracy/Spread 60 40 20 0 10 30 50 70 100 Observability Figure: Accuracy/Spread in G ratio for h uniq . Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 12 / 21

Results: Missing and Full Observations 45 Exhaust h m 40 RHW Zhu/Givan Hoffmann 35 30 Recognition Time (s) 25 20 15 10 5 0 0 10 20 30 40 50 60 Observation Length Figure: Recognition time for h gc . Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 13 / 21

Results: Missing and Full Observations 45 Exhaust h m 40 RHW Zhu/Givan Hoffmann 35 30 Recognition Time (s) 25 20 15 10 5 0 0 10 20 30 40 50 60 Observation Length Figure: Recognition time for h uniq . Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 14 / 21

Results: Missing, Noisy and Full Observations 100 Exhaust h m RHW Zhu/Givan Hoffmann 80 Landmark Percentage 60 40 20 0 Algorithms Figure: Percentage of extracted landmarks. Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 15 / 21

Results: Missing, Noisy and Full Observations 100 Exhaust h m RHW Zhu/Givan 80 Hoffmann Accuracy/Spread 60 40 20 0 25 50 75 100 Observability Figure: Accuracy/Spread in G ratio for h gc . Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 16 / 21

Results: Missing, Noisy and Full Observations 100 Exhaust h m RHW Zhu/Givan 80 Hoffmann Accuracy/Spread 60 40 20 0 25 50 75 100 Observability Figure: Accuracy/Spread in G ratio for h uniq . Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 17 / 21

Results: Missing, Noisy and Full Observations 35 Exhaust h m RHW 30 Zhu/Givan Hoffmann 25 Recognition Time (s) 20 15 10 5 0 0 5 10 15 20 25 30 35 Observation Length Figure: Recognition time for h gc . Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 18 / 21

Results: Missing, Noisy and Full Observations 25 Exhaust h m RHW Zhu/Givan 20 Hoffmann Recognition Time (s) 15 10 5 0 0 5 10 15 20 25 30 35 Observation Length Figure: Recognition time for h uniq . Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 19 / 21

Conclusions Quantity is not more important than quality; Algorithms with higher extraction capability obtained better performance with h uniq ; Quantity matters more when dealing with noisy observations. Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 20 / 21

Thank You! Questions? Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 21 / 21