Reasoning about Hypothetical Agent Behaviours and their Parameters - - PowerPoint PPT Presentation

▶
reasoning about hypothetical agent behaviours and their
SMART_READER_LITE
LIVE PREVIEW

Reasoning about Hypothetical Agent Behaviours and their Parameters - - PowerPoint PPT Presentation

Nov 10, 2023 104 likes •704 views

Introduction Approach Experiments Reasoning about Hypothetical Agent Behaviours and their Parameters Stefano Albrecht and Peter Stone Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 1

slide-1
SLIDE 1

Introduction Approach Experiments

Reasoning about Hypothetical Agent Behaviours and their Parameters

Stefano Albrecht and Peter Stone

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 1

slide-2
SLIDE 2

Introduction Approach Experiments

Introduction

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 2

slide-3
SLIDE 3

Introduction Approach Experiments

Motivation: Ad Hoc Teamwork

Design individual agent which can collaborate effectively with other agents, without pre-coordination Flexibility – ability to collaborate with different teammates Efficiency – find effective policy quickly AAAI 2010 Challenge Paper (Stone et al.)

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 3

slide-4
SLIDE 4

Introduction Approach Experiments

Motivation: Ad Hoc Teamwork

Design individual agent which can collaborate effectively with other agents, without pre-coordination Flexibility – ability to collaborate with different teammates Efficiency – find effective policy quickly AAAI 2010 Challenge Paper (Stone et al.)

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 4

Multiagent Interaction without Prior Coordination JAAMAS Special Issue on MIPC AAMAS’17 Workshop on MIPC

mipc.inf.ed.ac.uk

slide-5
SLIDE 5

Introduction Approach Experiments

Type-Based Method

Hypothesise possible types of other agents: Each type θj ∈ Θj is blackbox behaviour specification

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 5

P(aj|Ht, θj)

slide-6
SLIDE 6

Introduction Approach Experiments

Type-Based Method

Hypothesise possible types of other agents: Each type θj ∈ Θj is blackbox behaviour specification Compute belief over types based on interaction history Ht P(θj|Ht) ∝ P(Ht|θj)P(θj)

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 5

P(aj|Ht, θj)

slide-7
SLIDE 7

Introduction Approach Experiments

Type-Based Method

Hypothesise possible types of other agents: Each type θj ∈ Θj is blackbox behaviour specification Compute belief over types based on interaction history Ht P(θj|Ht) ∝ P(Ht|θj)P(θj) Plan own action with respect to belief over types

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 5

P(aj|Ht, θj)

slide-8
SLIDE 8

Introduction Approach Experiments

Type-Based Method

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 6

slide-9
SLIDE 9

Introduction Approach Experiments

Type-Based Method

HBA (Albrecht & Ramamoorthy, AIJ’16) PLASTIC (Barrett & Stone, AIJ’16)

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 6

slide-10
SLIDE 10

Introduction Approach Experiments

Type-Based Method and Parameters

Type-based method useful for ad hoc teamwork: Flexible – can hypothesise any types Efficient – can learn true type with few observations But...

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 7

slide-11
SLIDE 11

Introduction Approach Experiments

Type-Based Method and Parameters

Type-based method useful for ad hoc teamwork: Flexible – can hypothesise any types Efficient – can learn true type with few observations But...

Limitation: method does not recognise parameters in types!

Complex behaviours often have parameters If we want to reason about n parameter settings, have to store n copies of same type with different parameter settings ⇒ Inefficient, does not scale

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 7

slide-12
SLIDE 12

Introduction Approach Experiments

Type-Based Method and Parameters

Goal in this work

Devise method which allows agent to reason about both: Relative likelihood of types and Values of bounded continuous parameters in types

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 8

slide-13
SLIDE 13

Introduction Approach Experiments

Type-Based Method and Parameters

Goal in this work

Devise method which allows agent to reason about both: Relative likelihood of types and Values of bounded continuous parameters in types Keep blackbox nature of types (can be any model) Work with any continuous parameters in types

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 8

slide-14
SLIDE 14

Introduction Approach Experiments

Approach

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 9

slide-15
SLIDE 15

Introduction Approach Experiments

Approach

For each θj ∈ Θj, maintain parameter estimate p ∈ [pmin, pmax]n Update estimates after new observations

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 10

slide-16
SLIDE 16

Introduction Approach Experiments

Approach

For each θj ∈ Θj, maintain parameter estimate p ∈ [pmin, pmax]n Update estimates after new observations

Updating estimate incurs two computational costs:

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 10

slide-17
SLIDE 17

Introduction Approach Experiments

Approach

For each θj ∈ Θj, maintain parameter estimate p ∈ [pmin, pmax]n Update estimates after new observations

Updating estimate incurs two computational costs:

Computing new parameter estimate Types are blackboxes: must sample effects of parameters ⇒ Need general, efficient estimation methods

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 10

slide-18
SLIDE 18

Introduction Approach Experiments

Approach

For each θj ∈ Θj, maintain parameter estimate p ∈ [pmin, pmax]n Update estimates after new observations

Updating estimate incurs two computational costs:

Computing new parameter estimate Types are blackboxes: must sample effects of parameters ⇒ Need general, efficient estimation methods Adjusting internal state of type May depend on history of observations and parameter values ⇒ New estimate may introduce model inconsistency

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 10

slide-19
SLIDE 19

Introduction Approach Experiments

Approach: Selective Parameter Updating

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 11

Observe action at−1

j

  • f agent j
slide-20
SLIDE 20

Introduction Approach Experiments

Approach: Selective Parameter Updating

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 12

Observe action at−1

j

  • f agent j

Select types Φ ⊂ Θj for updating

slide-21
SLIDE 21

Introduction Approach Experiments

Approach: Selective Parameter Updating

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 13

Observe action at−1

j

  • f agent j

Select types Φ ⊂ Θj for updating For each θj ∈ Φ, update estimate pt−1 → pt

slide-22
SLIDE 22

Introduction Approach Experiments

Approach: Selective Parameter Updating

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 14

Observe action at−1

j

  • f agent j

Select types Φ ⊂ Θj for updating For each θj ∈ Φ, update estimate pt−1 → pt Update beliefs: P(θj|Ht) ∝ P(at−1

j

|Ht−1, θj, pt) P(θj|Ht−1)

slide-23
SLIDE 23

Introduction Approach Experiments

Approach: Selective Parameter Updating

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 15

Observe action at−1

j

  • f agent j

Select types Φ ⊂ Θj for updating For each θj ∈ Φ, update estimate pt−1 → pt Update beliefs: P(θj|Ht) ∝ P(at−1

j

|Ht−1, θj, pt) P(θj|Ht−1) Plan own action

slide-24
SLIDE 24

Introduction Approach Experiments

Approach: Selective Parameter Updating

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 16

Observe action at−1

j

  • f agent j

Select types Φ ⊂ Θj for updating For each θj ∈ Φ, update estimate pt−1 → pt Update beliefs: P(θj|Ht) ∝ P(at−1

j

|Ht−1, θj, pt) P(θj|Ht−1) Plan own action

slide-25
SLIDE 25

Introduction Approach Experiments

Updating Parameter Estimates

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 17

Given type θj, update parameter estimate pt−1 → pt Type defines action likelihoods

P(at−1

j

| Ht−1, θj, p)

P(a0

j|H0, θj, p1, p2)

P(a1

j|H1, θj, p1, p2)

  • 5

P(a2

j|H2, θj, p1, p2)

5

p1 p2

5

  • 5
slide-26
SLIDE 26

Introduction Approach Experiments

Approximate Bayesian Updating (ABU)

Idea: construct Bayesian update using polynomials

Maintain prior P(p|Ht−1, θj), represented as polynomial

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 18

slide-27
SLIDE 27

Introduction Approach Experiments

Approximate Bayesian Updating (ABU)

Idea: construct Bayesian update using polynomials

Maintain prior P(p|Ht−1, θj), represented as polynomial Approximate likelihood f (p) = P(at−1

j

|Ht−1, θj, p) as polynomial by sampling over p

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 18

slide-28
SLIDE 28

Introduction Approach Experiments

Approximate Bayesian Updating (ABU)

Idea: construct Bayesian update using polynomials

Maintain prior P(p|Ht−1, θj), represented as polynomial Approximate likelihood f (p) = P(at−1

j

|Ht−1, θj, p) as polynomial by sampling over p Take convolution of prior and likelihood, refit to original degree, normalise to get posterior P(p|Ht, θj)

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 18

slide-29
SLIDE 29

Introduction Approach Experiments

Approximate Bayesian Updating (ABU)

Idea: construct Bayesian update using polynomials

Maintain prior P(p|Ht−1, θj), represented as polynomial Approximate likelihood f (p) = P(at−1

j

|Ht−1, θj, p) as polynomial by sampling over p Take convolution of prior and likelihood, refit to original degree, normalise to get posterior P(p|Ht, θj) Get parameter estimate pt by taking maximum or sampling from posterior

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 18

slide-30
SLIDE 30

Introduction Approach Experiments

Approximate Bayesian Updating (ABU) – Example

0.2 0.4 0.6 0.8 1

p

1 2 3 4 5 6

Belief density

P(p|Ht−1

i

, θj) p∗

Prior

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 19

slide-31
SLIDE 31

Introduction Approach Experiments

Approximate Bayesian Updating (ABU) – Example

0.2 0.4 0.6 0.8 1

p

1 2 3 4 5 6

Belief density

P(p|Ht−1

i

, θj) p∗

Prior Past action at−1

j Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 19

slide-32
SLIDE 32

Introduction Approach Experiments

Approximate Bayesian Updating (ABU) – Example

0.2 0.4 0.6 0.8 1

p

1 2 3 4 5 6

Belief density

P(p|Ht−1

i

, θj) p∗

Prior Past action at−1

j

0.2 0.4 0.6 0.8 1

p

  • 0.5

0.5 1 1.5

ˆ f(p)

Samples from f Fitted ˆ f

Likelihood of at−1

j

given type θj f (p) = P(at−1

j

|Ht−1, θj, p)

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 19

slide-33
SLIDE 33

Introduction Approach Experiments

Approximate Bayesian Updating (ABU) – Example

0.2 0.4 0.6 0.8 1

p

1 2 3 4 5 6

Belief density

P(p|Ht−1

i

, θj) p∗

Prior Past action at−1

j

0.2 0.4 0.6 0.8 1

p

  • 0.5

0.5 1 1.5

ˆ f(p)

Samples from f Fitted ˆ f

Likelihood of at−1

j

given type θj f (p) = P(at−1

j

|Ht−1, θj, p)

0.2 0.4 0.6 0.8 1

p

  • 1

1 2 3 4

Belief density

Samples from ˆ g Fitted ˆ h P(p|Ht

i , θj)

p∗

Posterior (blue)

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 19

slide-34
SLIDE 34

Introduction Approach Experiments

Approximate Bayesian Updating (ABU) – Example

0.2 0.4 0.6 0.8 1

p

1 2 3 4 5 6

Belief density

P(p|Ht−1

i

, θj) p∗

Prior Past action at−1

j

0.2 0.4 0.6 0.8 1

p

  • 0.5

0.5 1 1.5

ˆ f(p)

Samples from f Fitted ˆ f

Likelihood of at−1

j

given type θj f (p) = P(at−1

j

|Ht−1, θj, p)

0.2 0.4 0.6 0.8 1

p

  • 1

1 2 3 4

Belief density

Samples from ˆ g Fitted ˆ h P(p|Ht

i , θj)

p∗

Posterior (blue) ← Generate estimate pt from posterior

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 19

slide-35
SLIDE 35

Introduction Approach Experiments

Exact Global Optimisation (EGO)

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 20

Estimation as Global Optimisation: arg max

p t

  • τ=1

P(aτ−1

j

|Hτ−1, θj, p) Solve with Bayesian Optimisation

P(a0

j|H0, θj, p1, p2)

P(a1

j|H1, θj, p1, p2)

  • 5

P(a2

j|H2, θj, p1, p2)

p1 p2

5

  • 5
slide-36
SLIDE 36

Introduction Approach Experiments

Approach: Selective Parameter Updating

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 21

Observe action at−1

j

  • f agent j

Select types Φ ⊂ Θj for updating For each θj ∈ Φ, update estimate pt−1 → pt Update beliefs: P(θj|Ht) ∝ P(at−1

j

|Ht−1, θj, pt) P(θj|Ht−1) Plan own action

slide-37
SLIDE 37

Introduction Approach Experiments

Selecting Types for Parameter Updates

Expensive to update all types after each observation...

Idea: let agent decide which types to update

Focus on types which are “most useful” to update Two selection methods: Posterior selection Bandit selection

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 22

slide-38
SLIDE 38

Introduction Approach Experiments

Posterior Selection

Focus on types which are believed to be most likely Don’t waste time on unlikely types But: can lead to premature convergence of belief to wrong type... Occasionally update types which are less likely

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 23

slide-39
SLIDE 39

Introduction Approach Experiments

Posterior Selection

Focus on types which are believed to be most likely Don’t waste time on unlikely types But: can lead to premature convergence of belief to wrong type... Occasionally update types which are less likely Tradeoff: sample Φ from belief P(θj|Ht−1)

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 23

slide-40
SLIDE 40

Introduction Approach Experiments

Bandit Selection

Assumption: parameter estimates converge Focus on types which are expected to make largest leap toward convergence Don’t waste time on estimates that wouldn’t change much

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 24

slide-41
SLIDE 41

Introduction Approach Experiments

Bandit Selection

Assumption: parameter estimates converge Focus on types which are expected to make largest leap toward convergence Don’t waste time on estimates that wouldn’t change much

Frame as multi-armed bandit problem:

Each type θj is an arm Pulling arm (= updating type) θj gives reward rt = η−1

n

  • k=1

|pt

k − pt−1 k

|, η =

n

  • k=1

pmax

k

− pmin

k

Can solve efficiently using bandit algorithm (e.g. UCB1)

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 24

slide-42
SLIDE 42

Introduction Approach Experiments

Experiments

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 25

slide-43
SLIDE 43

Introduction Approach Experiments

Level-Based Foraging

Blue = our agent, red = other agent Goal: collect all items in minimal time Agents and items have skill levels ∈ [0, 1] ⇒ Have to coordinate skills

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 26 0.15 0.58 0.83 0.53 0.23 0.48 0.55

slide-44
SLIDE 44

Introduction Approach Experiments

Level-Based Foraging

Red has one of 4 types: θL1

j : Search for item, try to load

θL2

j : Search for feasible item, try to load

θF1

j : Search for agent, load item closest to agent

θF2

j : Search for agent, load closest feasible item

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 27 0.15 0.58 0.83 0.53 0.23 0.48 0.55

p1 p2 p3

slide-45
SLIDE 45

Introduction Approach Experiments

Level-Based Foraging

Red has one of 4 types: θL1

j : Search for item, try to load

θL2

j : Search for feasible item, try to load

θF1

j : Search for agent, load item closest to agent

θF2

j : Search for agent, load closest feasible item

Each type has 3 parameters: level p1 view radius p2 view angle p3

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 27 0.15 0.58 0.83 0.53 0.23 0.48 0.55

p1 p2 p3

slide-46
SLIDE 46

Introduction Approach Experiments

Level-Based Foraging

Red has one of 4 types: θL1

j : Search for item, try to load

θL2

j : Search for feasible item, try to load

θF1

j : Search for agent, load item closest to agent

θF2

j : Search for agent, load closest feasible item

Each type has 3 parameters: level p1 view radius p2 view angle p3 Blue does not know true type, parameter values, or meaning of parameters Uses MCTS to plan own actions

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 27 0.15 0.58 0.83 0.53 0.23 0.48 0.55

p1 p2 p3

slide-47
SLIDE 47

Introduction Approach Experiments

Videos

2 agents, 5 items, 10x10 world Starting with random parameter estimates First video without updating Second video with updating, using bandit selection and EGO 3 agents, 10 items, 15x15 world Starting with random parameter estimates First video without updating Second video with updating, using bandit selection and EGO

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 28

slide-48
SLIDE 48

Introduction Approach Experiments

Results

15x15 world, 10 items, 3 agents Averaged over 500 random instances

AGA ABU EGO 30 40 50 60 70 80

% Completed

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 29

← knows parameter values for true type ← random estimates, no updating

slide-49
SLIDE 49

Introduction Approach Experiments

Results

15x15 world, 10 items, 3 agents Averaged over 500 random instances

AGA ABU EGO 30 40 50 60 70 80

% Completed

All Post Bandit

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 30

← knows parameter values for true type ← updating one type after each observation ← random estimates, no updating

slide-50
SLIDE 50

Introduction Approach Experiments

Results

AGA ABU EGO 0.01 0.1 1

Seconds (log)

Average seconds (log-scale) needed per parameter update for single type

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 31

slide-51
SLIDE 51

Introduction Approach Experiments

Results

1 2 3 4 5 6 7 8 9 10 11 12 13 14 end

Time step

0.20 0.25 0.30 0.35 0.40

Mean error

AGA ABU EGO

Mean error in estimates of view radius p2 for true type in 15x15 world (updating all types in each time step)

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 32

slide-52
SLIDE 52

Introduction Approach Experiments

Results

1 2 3 4 5 6 7 8 9 end

Time step

0.2 0.4 0.6 0.8 1

Probability

AGA ABU EGO Cor Rnd

Average belief P(θ∗

j |Ht) for true type θ∗ j in 10x10 world

(updating all types in each time step)

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 33

slide-53
SLIDE 53

Introduction Approach Experiments

Conclusion

Updating single type after each observation already achieves substantial improvements over random estimates

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 34

slide-54
SLIDE 54

Introduction Approach Experiments

Conclusion

Updating single type after each observation already achieves substantial improvements over random estimates Posterior selection tends to select more greedily than Bandit selection, premature convergence of beliefs

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 34

slide-55
SLIDE 55

Introduction Approach Experiments

Conclusion

Updating single type after each observation already achieves substantial improvements over random estimates Posterior selection tends to select more greedily than Bandit selection, premature convergence of beliefs EGO best estimation, can detect parameter correlation, but also most expensive

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 34

slide-56
SLIDE 56

Introduction Approach Experiments

Conclusion

Updating single type after each observation already achieves substantial improvements over random estimates Posterior selection tends to select more greedily than Bandit selection, premature convergence of beliefs EGO best estimation, can detect parameter correlation, but also most expensive Future work: improved methods for type selection; theoretical understanding of interaction between parameter estimates and belief evolution

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 34

slide-57
SLIDE 57

Introduction Approach Experiments Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 35

Thank you

slide-58
SLIDE 58

Introduction Approach Experiments

Algorithm: Selective Parameter Updating

Given: type space Θj, initial belief P(θj|H0) and parameter estimate p0 for each θj ∈ Θj Repeat for each t > 0:

1: Observe action at−1

j

  • f agent j

2: Select a subset Φ ⊂ Θj for parameter updates 3: For each θj ∈ Φ: 4:

Obtain new parameter estimate pt for θj

5:

Adjust internal state of θj wrt pt

6: Set pt = pt−1 for all θj ∈ Φ 7: For each θj ∈ Θj, update belief:

P(θj|Ht) ∝ P(at−1

j

|Ht−1, θj, pt) P(θj|Ht−1)

Stefano Albrecht, Peter Stone Reasoning about Hypothetical Agent Behaviours and their Parameters 36