SLIDE 1
testbed in the Do we need a testbed in the Do we need a COIN - - PowerPoint PPT Presentation
testbed in the Do we need a testbed in the Do we need a COIN - - PowerPoint PPT Presentation
testbed in the Do we need a testbed in the Do we need a COIN community and for what ? COIN community and for what ? COIN @ AAMAS Held with AAMAS May 12, 2008, Estoril, Portugal Examples of Testbeds and Contest Examples of Testbeds and
SLIDE 2
SLIDE 3
Objectives Objectives
Motivation & Challenge
to find out solutions for a complex problem
Comparison & evaluation
qualitative, quantitative models, languages, platforms
Improvement of the quality of tools in the domain
SLIDE 4
What to compare? What to compare?
Languages and Models of
Coordination Organisations Institutions Norms
Platforms of COIN
results from platforms can be extended to the underlying model (?)
depends of how to compare
SLIDE 5
Which metrics? Which metrics?
expressiveness evolution & adaptation flexibility reactiveness requirements for the agent's cognitive capabilities constraints x autonomy scalability reliability maturity
#case studies
cost
#messages
...
SLIDE 6
How to compare? How to compare?
by developing Testbed
Dist. Vehicle Monitoring Testbed, ART Testbed
by organising Contest
Agent Contest, ART Contest
Based on the performance in a scenario Avoid the Turing Test problem (?)
Considering just the “result” (e.g. ART Contest) Considering both the “result” and the “method” (e.g. Agent Contest)
SLIDE 7
Scenario – assumptions Scenario – assumptions
Open
agents can enter and leave the systems
Any agent architecture
no access to the “internals” of the agents
SLIDE 8
Scenario – desired properties Scenario – desired properties
Cooperation
no single agent can have a good result
Explicit rules must be available
to allow agents to reason about them even violating them
Require different dimensions
structural, functional, dialogical, normative, ...
SLIDE 9
Task accomplishment scenario (TAS) Task accomplishment scenario (TAS)
The platform should help the collective accomplishment of some task Purposeful
describe goals, activities, objectives, etc
Measurable
objective ways to evaluate the performance of the platform
Feasible
a task that may be accomplished by software agents
SLIDE 10
Metrics for TAS Metrics for TAS
time to accomplish allocation of goals to capable agents cost of coordination constraints detection of violations sanctions e.g. remove bad agents from the system scalability e.g. size and number of tasks ...
SLIDE 11
agent
Method I Method I
given by the contest given by the participant
agent agent ....... agent
Platform
given by the contest
agent agent agent agent
Evaluator
SLIDE 12
Method II Method II
given by the participant given by the contest
agent agent agent
Platform
agent
Evaluator
SLIDE 13
agent
Method III Method III
given by the participant given by the contest
agent agent ....... agent
Platform
given by the participant
agent agent agent agent
Evaluator
SLIDE 14