Multi-agent learning Fictitious Play Gerard Vreeswijk , Intelligent - - PowerPoint PPT Presentation
Multi-agent learning Fictitious Play Multi-agent learning Fictitious Play Gerard Vreeswijk , Intelligent Systems Group, Computer Science Department, Faculty of Sciences, Utrecht University, The Netherlands. Gerard Vreeswijk. Slides last
Multi-agent learning Fictitious Play
Gerard Vreeswijk, Intelligent Systems Group, Computer Science Department, Faculty of Sciences, Utrecht University, The Netherlands. Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 1
Multi-agent learning Fictitious Play
behaviour of your opponent(s), and respond optimally.
projected on a single mixed strategy.
Nash equilibrium play. In terms of current use, the name is a bit of a misnomer, since play actually occurs (Berger, 2005).
the most important, representative of a follower strategy.
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 2
Multi-agent learning Fictitious Play
Part I. Best reply strategy 1. Pure fictitious play.
Part II. Extensions and approximations of fictitious play
memory and are inert [tend to stick to their current strategy] (Peyton Young, XXX).
Shoham et al. (2009): Multi-agent Systems. Ch. 7: “Learning and Teaching”.
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 3
Multi-agent learning Fictitious Play
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 4
Multi-agent learning Fictitious Play
Players receive payoff p > 0 iff they coordinate. This game possesses three Nash equilibria, viz. (0, 0), (0.5, 0.5), and (1, 1). Round A’s action B’s action A’s beliefs B’s beliefs 0.
* 1. A B
2. B A
* 3. A B
4. B A
* 5. B B
6. B B
7. B B
. . . . . . . . . . . . . . .
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 5
Multi-agent learning Fictitious Play
Definition (Steady state). An action profile a is a steady state (or absorbing state) of fictitious play if it is the case that whenever a is played at round t it is also played at round t + 1.
(possibly weak) Nash equilibrium in the stage game.
model converges to s−i, for all i. If s would not be Nash, one of the players would deviate from si, which would contradict our assumption that s is a Nash equiibrium.a
equilibrium is unlikely to maintain the process in a steady state.
aAd absurdum is not a preferred route. But sometimes it is more intuitive.
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 6
Multi-agent learning Fictitious Play
it is a steady state of fictitious play in the repeated game. Notice the use of terminology: “pure strategy profile” for Nash equilibria; “action profile” for steady states.
deterministic (not a mix). Suppose s is played at round t. Because s is Nash, a best response to s−i is action si. (There might be others!) Because s is a strict equilibrium, si is the unique best response to s−i. Because this argument holds for each i, action profile s will be played in round t + 1 again.
Pure strict Nash ⇒ Steady state ⇒ Pure Nash. But what if pure Nash equilibria do not exist?
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 7
Multi-agent learning Fictitious Play
Zero sum game. A’s goal is to have pennies matched. Round A’s action B’s action A’s beliefs B’s beliefs 0.
1. T T
2. T H
3. T H
4. H H
5. H H
6. H H
7. H T
8. H T
. . . . . . . . . . . . . . .
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 8
Multi-agent learning Fictitious Play
fictitious play, then it converges to a Nash equilibrium.
players would deviate from si, which would contradict the convergence of the empirical distribution. Remarks:
converge asymptotically to the actual distribution.
converge (hence, converge to a Nash equilibrium), the actually played responses per stage need not be Nash equilibria of the stage game.
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 9
Multi-agent learning Fictitious Play
Repeated Coordination Game. Players receive payoff p > 0 iff they coordinate. Round A’s action B’s action A’s beliefs B’s beliefs 0.
1. B A
2. A B
3. B A
4. A B
. . . . . . . . . . . . . . .
expected payoffs 1, 0.5, and 1, respectively.
rather than p/2.
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 10
Multi-agent learning Fictitious Play
Rock-paper-scissors. Winner receives payoff p > 0. Else, payoff zero.
expected payoff p/3. Round A’s action B’s action A’s beliefs B’s beliefs 0.
1. Rock Scissors
2. Rock Paper
3. Rock Paper
4. Scissors Paper
5. Scissors Paper
. . . . . . . . . . . . . . .
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 11
Multi-agent learning Fictitious Play
Slide 12
Multi-agent learning Fictitious Play
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 13
Multi-agent learning Fictitious Play
Build forecasts, not on complete history, but on
Give not necessarily best responses, but
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 14
Multi-agent learning Fictitious Play
A forecasting rule for player i is a function that maps a history to a probability distribution over the opponents’ actions in the next round: fi : H → ∆(X−i). A response rule for player i is a function that maps a history to a probability distribution over i’s own actions in the next round: gi : H → ∆(Xi). A predictive learning rule for player i is the combination of a forecasting rule and a response rule. This is typically written as ( fi, gi).
prediction.
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 15
Multi-agent learning Fictitious Play
Let ht ∈ Ht be a history of play up to and including round t and φjt =Def the empirical distribution of j’s actions up to and including round t. Then the fictitious play forecasting rule is given by fi(ht) =Def ∏
j=i
φjt. Let fi be a fictitious play forecasting rule. Then gi is said to be a fictitious play response rule if all values are best responses to values of fi. Remarks:
the number of times each action is played by j.
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 16
Multi-agent learning Fictitious Play
Notation: p−i : strategy profile of opponents as predicted by fi in round t. ui(xi, p−i) : expected utility of action xi, given p−i. qi : strategy profile of player i in round t + 1. I.e., gi. Task: define qi given p−i and ui(xi, p−i). Idea: Respond randomly, but (somehow) proportional to expected payoff. Elaborations of this idea: a) Strictly proportional: qi(xi | p−i) =Def ui(xi, p−i) ∑x′
i∈Xi ui(x′
i, p−i).
b) Through, what is called, mixed logit: qi(xi | p−i) =Def eui(xi,p−i)/γi ∑x′
i∈Xi eui(x′ i,p−i)/γi .
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 17
Multi-agent learning Fictitious Play
logit(di) =Def edi/γ ∑j edj/γ where γ > 0.
γ ↓ 0 : logit “shares” 1 among all maximal di γ = 1 : logit is strictly proportional γ → ∞ : logit “spreads” 1 among all di evenly Mixed logit can be justified in different ways. a) On the basis of information and entropy arguments. b) By assuming the dj are i.i.d. extreme value distributed. This distribution arises as the limit of the maximum of n independent random variables (each exponentially distributed).
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 18
Multi-agent learning Fictitious Play
In this digression we try to answer the following question: Why does play according to a diversified strategy yield more information than play according to a strategy where only a few options are played?
encoded messages would cost 3
001, . . . 111.
we would need log2 16 = 4 bits.
we would need ⌈log2 20⌉ =
If some messages are send more frequently than others, it pays off to search for a code such that messages that occur more frequently are represented by short code words (at the expense of messages that are send less frequently, that must then be represented by the remaining longer code words).
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 19
Multi-agent learning Fictitious Play
Example. Suppose persons A and B work on a dark terrain. They are separated, and can only communicate by morse through a flashlight. A and B have agreed to send only the following messages: m1 Yes m2 No m3 All well? m4 Shall I come to you? A possible encoding could be Code 1: m1
00 m2
01 m3
10 m4
11
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 20
Multi-agent learning Fictitious Play
Another encoding could be Code 2: m1
m2
10 m3
110 m4
111 To prevent ambiguity, no code word may be a prefix of some other code word. A useless coding would be Code 3: m1
m2
1 m3
00 m4
01 Under Code 3, the sequence 0101 may mean different things, such as m1, m2, m1, m2, or m1, m2, m4. (There are still other possibilities.)
efficient encoding, i.e., an encoding that minimises the number of bits per message.
messages is known, we can for every code compute the expected number of bits per message, and hence its efficiency.
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 21
Multi-agent learning Fictitious Play
The following would be a plausible probability distribution: m1 Yes 1/2 m2 No 1/4 m3 All well? 1/8 m4 Shall I come to you? 1/8 For Code 2, E[number of bits] = 1 2· 1 + 1 4· 2 + 1 8· 3 + 1 8· 3 = 1.75 For Code 1, the expected number of bits is 2.0. Therefore, Code 2 is more efficient than Code 1. Theorem (Noiseless Coding Theo- rem, Shannon) p1 log2(1/p1) + . . . + pn log2(1/pn) is a lower bound for the expected number of bits in an encoding of n messages with expected occur- rence (p1, . . . , pn). This number is called the entropy of
The entropy of Code 2 is equal to 1.75. Therefore, Code 2 is optimal.
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 22
Multi-agent learning Fictitious Play
Smoothed fictitious play is a generalisation of mixed logit. Let wi : ∆i → R be a function that “grades” i’s probability distributions (over actions) under the following conditions.
(steep) whenever grading approaches the boundary of ∆i (whenever distributions become extremely uneven). Let Ui(qi, p−i) =Def ui(qi, p−i) + γi· wi(qi) Let fi be fictitious forecasting and let gi correspond to a best response based
wi and smoothing parameter γi.
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 23
Multi-agent learning Fictitious Play
Theorem (Fudenberg & Levine, 1995). Let G be a finite game and let ǫ > 0. If a given player uses smoothed fictitious play with a sufficiently small smoothing parameter, then with probability one his regrets are bounded above by ǫ. – Peyton Young does not reproduce the proof of Fudenberg et al., but shows that in this case ǫ-regret can be derived from a later and more general result of Hart and Mas-Colell in 2001. – This later result identifies a large family of rules that eliminate regret, based on an extension of Blackwell’s approachability theorem. – Roughly, Blackwell’s approachability theorem generalises maxmin reasoning to vector-valued payoffs.
Fudenberg & Levine, 1995. “Consistency and cautious fictitious play,” Journal of Economic Dynamics and Control, Vol. 19 (5-7), pp. 1065-1089. Hart & Mas-Colell, 2001. “A General Class of Adaptive Strategies,” Journal of Economic Theory, Vol. 98(1),
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 24
Multi-agent learning Fictitious Play
in expected payoff. A coarse correlated ǫ-equilibrium (ǫ-CCE) is a probability distribution on strategy profiles, such that no player can opt out and gain more in expectation than ǫ. Theorem (Fudenberg & Levine, 1995). Let G be a finite game and let ǫ > 0. If all players use smoothed fictitious play with sufficiently small smoothing parameters, then with probability one empirical play will converge to the set of coarse correlated ǫ-equilibria. Summary of the two theorems: smoothed fictitious play limits regret and converges to ǫ-CCE. There is another learning method with no regret and convergence to zero-CCE . . .
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 25
Multi-agent learning Fictitious Play
Let j : action j, where 1 ≤ j ≤ k ¯ ut
i : i’s realised average payoff up to and including round t
φ−it : the realised joint empirical distribution of i’s opponents ¯ ui(j, φ−it) : i’s hypothetical average payoff for playing action j against φ−it ¯ rit : player i’s regret vector in round t, i.e., ¯ ui(j, φ−it) − ¯ ut
i
Exponentiated regret matching (PY, p. 59) is defined as qi(t+1)
j
∝ [¯ rit
j ]a
+
where a > 0. (For a = 1 we have ordinary regret matching.) An extended theorem on regret matching (Mas-Colell et al., 2001) ensures that individual players have no regret with probability one, and empirical distribution of play converges to the set of coarse correlated equilibria (PY,
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 26
Multi-agent learning Fictitious Play
FP Smoothed FP Exponentiated RM Depends on past play of
Depends on own past payoffs
Puts zero probabilities on sub-optimal responses
Best response
when smoothing parameter γi ↓ 0 when exponent a → ∞ Individual no regret
Within ǫ > 0, almost always (PY, p. 82) Exact, almost always (PY, p. 60) Collective convergence to coarse correlated equilibria
Within ǫ > 0, almost always (PY, p. 83) Exact, almost always (PY, p. 60)
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 27
Multi-agent learning Fictitious Play
Fictitious play Plays best responses.
Smoothed fictitious play Plays sub-optimal responses, e.g., softmax-proportional to their estimated payoffs.
Exponentiated regret matching Plays regret suboptimally, i.e., proportional to a power of positive regret.
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 28
Multi-agent learning Fictitious Play
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 29
Multi-agent learning Fictitious Play
learning rules rely on the entire history of play.
have a finite memory. (On the
discounted payoffs this is no real problem.)
distant past are apt to be less relevant than more recent ones.
memory of m rounds.
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 30
Multi-agent learning Fictitious Play
constantly re-evaluated, discontinuities in behaviour are likely to occur. Example: the asymmetric coordination game.
less likely to lead to equilibria of any sort.
action as in the previous round with probability λ.
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 31
Multi-agent learning Fictitious Play
E = { (x, y) | for some i : y−i = x−i and ui(yi, y−i) > ui(xi, y−i) }
Nash equilibrium.
better replies if every node is connected to a sink.
(1, 1) (2, 4) (4, 2) (1, 1) (4, 2) (2, 4) (3, 3) (1, 1) (1, 1)
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 32
Multi-agent learning Fictitious Play
Coordination games Two-person games with identical actions for all players, where best responses are formed by the diagonal of the joint action space. Potential games (Monderer and Shapely, 1996). There is a function ρ : X → R, called the potential, such that for every player i and every action profile x, y ∈ X: y−i = x−i ⇒ ui(yi, y−i) − ui(xi, y−i) = ρ(y) − ρ(x) true : The potential function increases along every path.
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 33
Multi-agent learning Fictitious Play
with finite memory and inertia converges to a pure Nash equilibrium of G.
consists of identical action profiles x. Such a state is denoted by x. Z∗ =Def { homogeneous states }.
infinitely many times .
that the overall probability to play any action is bounded away from zero.
that the set of absorbing states is identical to the set of homogeneous states that consist
and (4), the process eventually lands in an absorbing state which, due to (5), is a repeated pure Nash equilibrium.
Slide 34
Multi-agent learning Fictitious Play
Let inertia be determined by λ > 0. Pr(all players play their previous action) = λn. Hence, Pr(all players play their previous action during m subsequent rounds) = λnm. If all players play their previous action during m subsequent rounds, then the process arrives at a homogeneous state. But also conversely. Hence, for all t, Pr(process arrives at a homogeneous state in round t + m) = λnm. Recall the converse Borel-Cantelli lemma: if {En}n are independent events, and ∑∞
n=1 Pr(En) is unbounded, then Pr( an infinite number of En’s occur ) = 1.
This lemma applies if an infinite selection of disjoint histories is considered. Since history length ≤ m, this is always possible.
Slide 35
Multi-agent learning Fictitious Play
A better-reply learning method from states (finite histories) to strategies (probability distributions on actions) γi : Z → ∆(Xi) possesses the following important properties: i) It is deterministic. ii) Every action is played with positive probability.
γi = inf{γi(z)(xi) | z ∈ Z, xi ∈ Xi} Since Z and Xi are finite, the “inf” is a “min,” and γi > 0.
γ = inf{γi | 1 ≤ i ≤ n}. Since there are finitely many players, the “inf” is a “min,” and γ > 0.
Slide 36
Multi-agent learning Fictitious Play
Suppose the process is in x.
because response functions are deterministic better replies.
be an edge x → y in the better reply graph. Suppose this edge concerns action xi of player i. We now know that xi is played with probability at least γ, irrespective
Further probabilities:
playing the same action : λn−1.
γλn−1.
another m − 1 rounds, so as to arrive at state y : λn(m−1).
γλn−1· λn(m−1) = γλnm−1.
better reply-path x(1), . . . , x(l) is followed to a sink : ≥ (γλnm−1)L, where L is the length of a longest better-reply path. Since Z∗ is encountered infinitely
Slide 37
Multi-agent learning Fictitious Play
represented by, or projected on) a mixed strategy.
sub-optimal actions.
better-reply learning rules, that i) play sub-optimal actions, and ii) can be brought arbitrary close to fictitious play.
games, every better-reply process with finite memory and inertia converges to a pure Nash equilibrium.
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 38
Multi-agent learning Fictitious Play
Bayesian play :
mixed strategy.
Gradient dynamics :
strategies.
Varoufakis, 2004), all models of mixed strategies are correct. (I.e., q−i = s−i, for all i.)
the payoff space.
Gerard Vreeswijk. Slides last processed on Tuesday 2nd March, 2010 at 13:53h. Slide 39