The More the Merrier?! Evaluating the Effect of Landmark Extraction - - PowerPoint PPT Presentation

▶

Feb 02, 2023 224 likes •442 views

The More the Merrier?! Evaluating the Effect of Landmark Extraction Algorithms on Landmark-Based Goal Recognition Kin Max Piamolini Gusm ao Ramon Fraga Pereira Felipe Meneguzzi Pontifical Catholic University of Rio Grande do Sul - Brazil

SLIDE 1

The More the Merrier?! Evaluating the Effect of Landmark Extraction Algorithms on Landmark-Based Goal Recognition

Kin Max Piamolini Gusm˜ ao Ramon Fraga Pereira Felipe Meneguzzi

Pontifical Catholic University of Rio Grande do Sul - Brazil

February 8, 2020

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 1 / 21

SLIDE 2

Summary

Motivation Goal Recognition and Landmark-Based Goal Recognition Experimentation Methodology

Heuristics and Algorithms; Dataset; Evaluation Metrics

Results

Missing and Full Observations Missing, Noisy and Full Observations

Conclusions

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 2 / 21

SLIDE 3

Motivation

Explore different landmark extraction algorithms with landmark-based heuristics for goal recognition; Evaluate wether more landmarks lead to a more precise recognition; Inform future fine-tuning of algorithms to enhance recognition precision;

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 3 / 21

SLIDE 4

Goal Recognition and Landmark-Based Goal Recognition

Goal Recognition: recognize agent’s goal based on it’s interactions with the environment; Landmarks: necessary facts or actions that must be present in any solution plan; Landmark-Based Goal Recognition: goal recognition techniques that leverage on landmarks.

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 4 / 21

SLIDE 5

Experimentation Methodology

Heuristics and Algorithms Dataset Evaluation Metrics

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 5 / 21

SLIDE 6

Experimentation Methodology: Heuristics and Algorithms

2 landmark-based heuristics for goal recognition1:

Goal Completion Heuristic (hgc); Landmark Uniqueness Heuristic (huniq).

5 landmark extraction algorithms:

Exhaust; hm2; RHW 3; Zhu & Givan4; Hoffamnn et al.5

2 threshold values: 0% and 10%.

1Pereira et al., Landmark-Based Heurristics for Goal Recognition. AAAI, 2017. 2Keyder et al., Sound and complete landmarks for and/or graphs. ECAI, 2010. 3Richter et al., Landmarks revisited. AAAI 2008. 4Zhu L. e Givan R., Landmark extraction via planning graph propagation, 2003. 5Hoffmann et al., Ordered landmarks in planning. JAIR, 2004. Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 6 / 21

SLIDE 7

Experimentation Methodology: Dataset

Dataset with goal and plan recognition problems6; Problems from 15 classical planning domains; 6313 problems with missing and full observations with 5 observability levels (10%, 30%, 50%, 70% and 100%); 2850 problems with missing, noisy and full observations with 4

bservability levels (25%, 50%, 75% and 100%).

6Pereira F. R. e Meneguzzi F., Goal and plan recognition datasets using classical planning domains. 2017. Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 7 / 21

SLIDE 8

Experimentation Methodology: Evaluation Metrics

Percentage of extracted landmarks; Accuracy (%); Spread in G; Accuracy/Spread in G ratio; Recognition time (s).

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 8 / 21

SLIDE 9

Results

Missing and Full Observations Missing, Noisy and Full Observations

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 9 / 21

SLIDE 10

Results: Missing and Full Observations

20 40 60 80 100

Landmark Percentage Algorithms Exhaust hm RHW Zhu/Givan Hoffmann

Figure: Percentage of extracted landmarks.

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 10 / 21

SLIDE 11

Results: Missing and Full Observations

20 40 60 80 100 10 30 50 70 100

Accuracy/Spread Observability Exhaust hm RHW Zhu/Givan Hoffmann

Figure: Accuracy/Spread in G ratio for hgc.

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 11 / 21

SLIDE 12

Results: Missing and Full Observations

20 40 60 80 100 10 30 50 70 100

Accuracy/Spread Observability Exhaust hm RHW Zhu/Givan Hoffmann

Figure: Accuracy/Spread in G ratio for huniq.

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 12 / 21

SLIDE 13

Results: Missing and Full Observations

5 10 15 20 25 30 35 40 45 10 20 30 40 50 60

Recognition Time (s) Observation Length Exhaust hm RHW Zhu/Givan Hoffmann

Figure: Recognition time for hgc.

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 13 / 21

SLIDE 14

Results: Missing and Full Observations

5 10 15 20 25 30 35 40 45 10 20 30 40 50 60

Recognition Time (s) Observation Length Exhaust hm RHW Zhu/Givan Hoffmann

Figure: Recognition time for huniq.

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 14 / 21

SLIDE 15

Results: Missing, Noisy and Full Observations

20 40 60 80 100

Landmark Percentage Algorithms Exhaust hm RHW Zhu/Givan Hoffmann

Figure: Percentage of extracted landmarks.

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 15 / 21

SLIDE 16

Results: Missing, Noisy and Full Observations

20 40 60 80 100 25 50 75 100

Accuracy/Spread Observability Exhaust hm RHW Zhu/Givan Hoffmann

Figure: Accuracy/Spread in G ratio for hgc.

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 16 / 21

SLIDE 17

Results: Missing, Noisy and Full Observations

20 40 60 80 100 25 50 75 100

Accuracy/Spread Observability Exhaust hm RHW Zhu/Givan Hoffmann

Figure: Accuracy/Spread in G ratio for huniq.

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 17 / 21

SLIDE 18

Results: Missing, Noisy and Full Observations

5 10 15 20 25 30 35 5 10 15 20 25 30 35

Recognition Time (s) Observation Length Exhaust hm RHW Zhu/Givan Hoffmann

Figure: Recognition time for hgc.

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 18 / 21

SLIDE 19

Results: Missing, Noisy and Full Observations

5 10 15 20 25 5 10 15 20 25 30 35

Recognition Time (s) Observation Length Exhaust hm RHW Zhu/Givan Hoffmann

Figure: Recognition time for huniq.

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 19 / 21

SLIDE 20

Conclusions

Quantity is not more important than quality; Algorithms with higher extraction capability obtained better performance with huniq; Quantity matters more when dealing with noisy observations.

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 20 / 21

SLIDE 21

Thank You! Questions?

Kin Max Piamolini Gusm˜ ao The More the Merrier?! February 8, 2020 21 / 21