Multi-agent learning Gerard Vreeswijk , Intelligent Systems Group, - - PowerPoint PPT Presentation
Multi-agent learning Multi-agent reinforcement learning Multi-agent reinfo rement lea rning Multi-agent learning Gerard Vreeswijk , Intelligent Systems Group, Computer Science Department, Faculty of Sciences, Utrecht University, The
Multi-agent learning Multi-agent reinforcement learning
Gerard Vreeswijk, Intelligent Systems Group, Computer Science Department, Faculty of Sciences, Utrecht University, The Netherlands. Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 1
Multi-agent learning Multi-agent reinforcement learning
(a)
Indep endent Lea rners (IL) Agents that attempt to learn(b)
Joint A tion Lea rners (JAL) Agents that attempt to learn bothdo they converge to equilibria? Are these equilibria optimal?
structure and action selection strategies? Claus et al. address some of these questions in a limited setting, namely, A repeated cooperative two-player multiple-action game in strategic form.
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 2
Multi-agent learning Multi-agent reinforcement learning
Claus and Boutilier (1998). “The Dynamics of Reinforcement Learning in Cooperative Multia- gent Systems” in: Proc. of the Fifteenth National Conf. on Artificial Intelligence, pp. 746-752.
The paper on which this presentation is mostly based on.
Watkins and Dayan (1992). “Q-learning”. Machine Learning, Vol. 8, pp. 279-292.
Mainly the result that Q-learning converges to the optimum action-values with probability one as long as all actions are repeatedly sampled in all states and the action-values are represented discretely.
Fudenberg, D. and D. Kreps (1993): “Learning Mixed Equilibria,” Games and Economic Behavior,
Mainly Proposition 6.1 and its proof pp. 342-344. Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 3
Multi-agent learning Multi-agent reinforcement learning
is multi-state and amounts to continuously updating the various Q(s, a) with r(s, a, s′) + γ· max
a
Q(s′, a) (1)
game G) so that (1) reduces to r(s, a, s) which may be abbreviated to r(a)
learning rule: Qnew(a) = (1 − λ)Qold(a) + λ· r
convergence in Q-learning (Watkins, Dayan, 1992):
through time such that ∑t λ is divergent and ∑t λ2 is convergent.
infinitely often.
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 4
Multi-agent learning Multi-agent reinforcement learning
Convergence on Q-learning does not depend on the exploration strategy
learning.
Exploitive explo ration Even during exploration, there is a probabilistic bias toexploring optimal actions.
response function): eQ(a)/T ∑a′ eQ(a′)/T with T > 0. Letting T → 0 establishes convergence conditions (1) and (2) as mentioned above (Watkins, Dayan, 1992).
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 5
Multi-agent learning Multi-agent reinforcement learning
if the agents learn Q-values for their individual actions.
form ai, r(ai) where ai is the action performed by i and r(ai) is a reward for action ai.
Qnew(a) = (1− λ)Qold(a) + λ· r(a) ILs perform their actions, obtain a reward and update their Q-values without regard to the actions performed by other agents.
Independent Learning: – An agent is unaware of the existence of other agents. – It cannot identify other agent’s actions, or has no reason to believe that other agents are acting strategically. Of course, even if an agent can learn through joint actions, it may still choose to ignore information about the other agents’ behaviour.
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 6
Multi-agent learning Multi-agent reinforcement learning
rewards for joint actions. For a 2 × 2 game an agent would have to maintain Q(T, L), Q(T, R), Q(B, L), and Q(B, R).
not opponent’s actions L, R. Let ai be an action of player i. A
is a set of joint actions a−i such that a = ai ∪ a−i is a complete joint action profile.
estimated through forecast by, e.g., fictitious play: fi(a−i) =Def Πj=iφj(a−i) where φj(a−i) is i’s empirical distribution of j’s actions on a−i.
Q-values, weighed by the estimated probability of the associated complementary joint action profiles: EV(ai) =
a−i∈A−i
Q(ai ∪ a−i) fi(a−i)
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 7
Multi-agent learning Multi-agent reinforcement learning
Case 1: the coordination game T B L R
10
Q-values of different joint actions a = ai ∪ a−i.
information is circumscribed by the limited freedom of its own actions ai ∈ Ai.
the strategy being played by other agents through fictitious play, and plays a softmax best response. A JAL computes singular Q-values by means of explicit belief distributions on joint Q-values. Thus, EV(ai) =
a−i∈A−i
Q(ai ∪ a−i) fi(a−i) is more or less the same as the Q-values learned by ILs.
fairly sure of the relative Q-values
cannot really benefit from this.
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 8
Multi-agent learning Multi-agent reinforcement learning
Figure 1:Convergence
coordi- nation for ILs and JALs (aver- aged
100 trials).
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 9
Multi-agent learning Multi-agent reinforcement learning
Case 1: the coordination game T B L R
10
Q-values of different joint actions a = ai ∪ a−i.
information is circumscribed by the limited freedom of its own actions ai ∈ Ai.
the strategy being played by other agents (fictitious play) and plays a softmax best response. A JAL computes single Q-values by means of explicit belief distributions on joint Q-values. Thus, EV(ai) =
a−i∈A−i
Q(ai ∪ a−i) fi(a−i) is more or less the same as the Q-values learned by ILs.
fairly sure of the relative Q-values
cannot really benefit from this.
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 10
Multi-agent learning Multi-agent reinforcement learning
T C B L M R 10 k 2 k 10 Suppose penalty k = −100. The following stories are entirely symmetrical for Row and Column. IL 1. Initially, Column explores.
B on average very unattractive, and will converge to C.
B slightly less attractive, and will converge to C as well. JAL 1. Initially, Column explores.
to T and B. Plays C the most.
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 11
Multi-agent learning Multi-agent reinforcement learning
Figure 2:Likelihood
conver- gence to opt. equilib- rium as a func- tion of penalty k (100 trials).
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 12
Multi-agent learning Multi-agent reinforcement learning
T C B L M R 11
7 6 5
[Show Netlogo assign- ments Inl. Adaptieve Systemen.]
almost always going to begin to play the non-equilibrium strategy profile (B, R).
long as exploration continues, Row will soon find C to be more attractive, so long as Col continues to primarily choose R.
Row’s move and begins to perform action M. Once this equilibrium is reached, the agents remain there.
general, allowing one to conclude that the multiagent Q-learning schemes we have proposed will converge to equilibria almost surely.
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 13
Multi-agent learning Multi-agent reinforcement learning
Figure 3:A’s strat- egy in climb- ing game.
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 14
Multi-agent learning Multi-agent reinforcement learning
Figure 4:B’s strat- egy in climb- ing game.
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 15
Multi-agent learning Multi-agent reinforcement learning
Figure 5:Joint actions in climb- ing game.
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 16
Multi-agent learning Multi-agent reinforcement learning
Recall from fictitious and Bayesian play the notion of a
p redi tive strategy withforecast function fi : H → ∆(A−i) and behaviour rule gi : H → ∆(Ai).
empirical frequencies of play with probability one.
player i’s choice of action at every history given gt
i goes to zero as t
proceeds: ui( f t
i , gt i) ր max{ui( f t i , ai) | ai ∈ Ai}
as t → ∞, where ui denotes expected payoff. – Being asymptotically myopic is less demanding than a behaviour rule that assigns positive probability only to pure strategies that eventually come close to maximising expected payoff.
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 17
Multi-agent learning Multi-agent reinforcement learning
Being asymptotically myopic includes:
with a small probability (regardless
with a large probability (regardless).
called
stable if joint behaviour is alimit point of a with probability one.
1993, p. 343): Let forecast fi be asymptotically empirical, and let behaviour rule gi be asymptotically
profile is a Nash equilibrium. Proof (Outline.) Suppose a is stable. Then empirical frequencies eventually converge (!) to a. Because fi is asymptotically empirical, so will the fi, for all i, with probability one. Convergence of fi together with asymptotic myopia of gi implies that the gi converge as well. This situation is in effect a Nash equilibrium.
Last modified on April 3rd, 2014 at 13:17 Slide 18
Multi-agent learning Multi-agent reinforcement learning
∑t λ2 is convergent.
t(a) of agent i choosing action a is nonzero.
at all times.
lim
t→∞ Pi t(Xt) = 0,
where Xt is a random variable denoting the event that ( fi, gi) prescribe a sub-optimal action.
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 19
Multi-agent learning Multi-agent reinforcement learning
Optimistic Boltzmann (OB): For agent i, action ai ∈ Ai, let MaxQ(ai) =Def max∏−iQ(Π−i, ai). Choose actions with Boltzmann exploration (another exploitive strategy would suffice) using MaxQ(ai) as the value of ai. Weighted OB (WOB): Explore using Boltzmann using factors MaxQ(ai)· Pr(optimal match ∏−i for ai). Combined: Let C(ai) = ρMaxQ(ai) + (1 − ρ)EV(ai), for some 0 ≤ ρ ≤ 1. Choose actions using Boltzmann exploration with C(ai) as value of ai.
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 20
Multi-agent learning Multi-agent reinforcement learning
Figure 6:Sliding average reward in the penalty game.
Gerard Vreeswijk. Last modified on April 3rd, 2014 at 13:17 Slide 21