L ECTURE 32-34: G AME T HEORY 7-9 / E VOLUTIONARY G AME T HEORY 4-6 I - - PowerPoint PPT Presentation
15-382 C OLLECTIVE I NTELLIGENCE S18 L ECTURE 32-34: G AME T HEORY 7-9 / E VOLUTIONARY G AME T HEORY 4-6 I NSTRUCTOR : G IANNI A. D I C ARO G AMES AGAINST THE FIELD In this population game model, each individual plays against an
15781 Fall 2016: Lecture 22
2
§ In this population game model, each individual plays against an ‘environment’ and not against other individuals. The environment represents an average (isotropic) property that determines the payoffs of the individuals based on the strategy they play (i.e., that they are programmed for)
15781 Fall 2016: Lecture 22
3
§ Payoffs are not linear functions of the population profile! § In pairwise contest games they always are linear functions: 𝜌 𝑡$,𝑦 = 𝑦𝜌 𝑡$,𝑡( + (1 − 𝑦) 𝜌 𝑡$,𝑡.
15781 Fall 2016: Lecture 22
4
15781 Fall 2016: Lecture 22
5
15781 Fall 2016: Lecture 22
6
15781 Fall 2016: Lecture 22
7
§ In monomorphic populations:
8
15781 Fall 2016: Lecture 22
9
§ 𝑂 individuals (phenotypes) programmed to use only pure strategies from a finite set 𝑇 = {𝑡(,𝑡.,…,𝑡3} § 𝑜$ = proportion of individuals using strategy 𝑡$, 𝑂 = ∑ 𝑜$
3 $7(
§ Proportion of individuals in the population using 𝑡$ : 𝑦$ =
89 :
§ Population state: 𝒚 = (𝑦(, 𝑦.,… ,𝑦3) § 𝛾 = per capita birth rate of the population (independent from the game) § 𝜀 = per capita death rate of the population (independent from the game) § Rate of change of the # of individuals using pure strategy 𝑡$: 𝑜̇ $ = (𝛾 − 𝜀 + 𝜌 𝑡$,𝒚 )𝑜$ § à Rate of change of the # of individuals in population: § 𝑂̇ = ∑ 𝑜̇ $
3 $7(
à 𝑂̇ = 𝛾 − 𝜀 + 𝜌 ? 𝒚 𝑂, 𝜌 ? 𝒚 = ∑ 𝑦$
3 $7(
𝜌 𝑡$,𝒚
15781 Fall 2016: Lecture 22
10
§ 𝑂̇ = 𝐷𝑂 à 𝑂 can grow or decline exponentially § The behavior can be improved by letting 𝛾 and 𝜀 depending on 𝑂 … § Game-theoretic interesting question: how proportions 𝑜$ change over time? § From 𝑦$ =
89 : , by differentiating: 𝑜̇ $ = 𝑂𝑦̇$ + 𝑦$𝑂̇ à …
§ Replicator dynamics: 𝑦̇$ = 𝜌 𝑡$,𝒚 − 𝜌 ? 𝒚 𝑦$ The proportion of individuals with phenotype / strategy 𝑡$ increases (decreases) if its payoff is bigger (smaller) than the average payoff in the population § We said that an ESS is an end-point in evolution § A fixed point of the replicator dynamics is a population that satisfies 𝑦̇$ = 0, ∀𝑗 At a fixed point the population is not evolving anymore
15781 Fall 2016: Lecture 22
11
§ Pairwise contest population with action set 𝐵 = {𝐹, 𝐺} and payoffs: 𝜌 𝐹,𝐹 = 1, 𝜌 𝐹, 𝐺 = 1, 𝜌 𝐺, 𝐹 = 2, 𝜌 𝐺, 𝐺 = 0 § à 𝜌 𝐹,𝒚 = 𝑦( + 𝑦. 𝜌 𝐺,𝒚 = 2𝑦( § 𝜌 ? 𝒚 = 𝑦( 𝑦( + 𝑦. + 𝑦. 2𝑦( = 𝑦(. + 3𝑦(𝑦. § Replicator dynamics: § 𝑦̇( = 𝑦( 𝑦( + 𝑦. − 𝑦(. − 3𝑦(𝑦. § 𝑦̇. = 𝑦. 2𝑦( − 𝑦(. − 3𝑦(𝑦. § Fixed points § 𝑦( = 0,𝑦. = 1 , 𝑦( = 1,𝑦. = 1 , 𝑦( = 1/2,𝑦. = 1/2 § (𝑦( = 0, 𝑦. = 0) is also a fixed point but not an interesting one since the population disappears
15781 Fall 2016: Lecture 22
12
§ Let’s consider the case of only two pure strategies 𝑇 = 𝑡(,𝑡. , population state 𝒚 = 𝑦(,𝑦. , representing the fraction of individuals using strategies 𝑡( and 𝑡., and let’s use pairwise contest games for evolving population § The general payoff matrix for each pairwise game: Π = 𝜌 𝑡(,𝑡( ,𝜌 𝑡(,𝑡( 𝜌 𝑡(,𝑡. ,𝜌 𝑡.,𝑡( 𝜌 𝑡.,𝑡( ,𝜌 𝑡(,𝑡. 𝜌 𝑡.,𝑡. ,𝜌(𝑡.,𝑡.) because of symmetry it can be simplified considering only row player: Π = 𝜌 𝑡(,𝑡( 𝜌 𝑡(,𝑡. 𝜌 𝑡.,𝑡( 𝜌 𝑡.,𝑡. § Expected payoff for playing pure strategies 𝑡( and 𝑡. in population 𝒚 = (𝑦(,𝑦.) 𝜌 𝑡(,𝒚 = 𝑦(𝜌 𝑡(,𝑡( + 𝑦.𝜌 𝑡(,𝑡. , 𝜌 𝑡.,𝒚 = 𝑦(𝜌 𝑡.,𝑡( + 𝑦.𝜌 𝑡.,𝑡. § Average payoff in the population: 𝜌 ? 𝒚 = 𝑦(𝜌 𝑡(,𝒚 + 𝑦.𝜌 𝑡.,𝒚
§ Replicator dynamics: 𝑦̇$ = 𝜌 𝑡$,𝒚 − 𝜌 ? 𝒚 𝑦$
§ We can simplify the notation and use only one variable: 𝑦 = 𝑦( 𝑦. = 1 − 𝑦 à 𝑦̇. = −𝑦̇( 𝑦̇ = 𝑦(1 − 𝑦)(𝜌 𝑡(,𝒚 − 𝜌 𝑡.,𝒚 )
15781 Fall 2016: Lecture 22
13
§ Let’s consider the case of only two pure strategies pairwise contest population with action set 𝑇 = 𝐵,𝐶 § Let 𝑦 = 𝑦L 𝑦M = 1 − 𝑦 à 𝑦̇M = −𝑦̇L § Dynamics of population state: 𝑦̇ = 𝑦(1 − 𝑦)(𝜌 𝐵, 𝒚 − 𝜌 𝐶, 𝒚 )
Fitness of strategy 𝐵 Fitness of strategy 𝐶
§ If 𝑔
L 𝑦 > 𝑔 M 𝑦 → 𝑦̇ > 0
à 𝐵 increases § If 𝑔
L 𝑦 < 𝑔 M 𝑦 → 𝑦̇ < 0
à 𝐵 decreases § If 𝑔
L 𝑦 = 𝑔 M 𝑦 → 𝑦̇ = 0
à Slope of 𝑔
L 𝑦 − 𝑔 M 𝑦
defines the stability § Fixed points: (𝑦 = 0, 𝑦 = 1, 𝜌 𝐵,𝒚 = 𝜌 𝐶, 𝒚
15781 Fall 2016: Lecture 22
14
§ Let’s consider the pairwise contest Prisoner’s dilemma: 𝑇 = 𝐷,𝐸 , 𝜌 𝐷, 𝐷 = 3 𝜌 𝐷,𝐸 = 0 𝜌 𝐸,𝐷 = 5 𝜌 𝐸,𝐸 = 1 § Let 𝑦 be the proportion of individuals using 𝐷: à 𝜌 𝐷, 𝒚 = 3𝑦 + 0 1 − 𝑦 = 3𝑦 𝜌 𝐸, 𝒚 = 5𝑦 + 1 1 − 𝑦 = 1 + 4𝑦 § 𝑦̇ = 𝑦 1 − 𝑦 𝜌 𝐷,𝒚 − 𝜌 𝐸, 𝒚 = 𝑦 1 − 𝑦 3𝑦 − (1 + 4𝑦) § = 𝑦 1 − 𝑦 (1 + 𝑦) à Fixed points: 𝑦∗ = 0 𝐸,𝐸 , 𝑦∗ = 1 (𝐷, 𝐷) § Nash equilibrium: (𝐸, 𝐸)
§ Since 𝑦̇ < 0 for all 𝑦 except the fixed points, any population that is not at a fixed point evolves toward the Nash equilibrium, and away from the other equilibrium § Phase portrait with 𝑦( (playing C), and 𝑦( (playing D) . The grayed area doesn’t correspond to any feasible strategy. Indeed only the points on the diagonal between (1,0) and (0,1) represent feasible population profiles 𝑦( 𝑦.
15781 Fall 2016: Lecture 22
15
§ Theorem: In a two-player games, every symmetric NE corresponds to a fixed point in the replicator dynamics § The reverse is not necessarily true: a fixed point of the dynamics is not necessarily a NE § Can we identify a consistent relation between NE and fixed points? § 𝑇 = 𝐵,𝐶 , 𝜌 𝐵, 𝐵 = 3 𝜌 𝐶,𝐶 = 1 𝜌 𝐵,𝐶 = 𝜌 𝐶, 𝐵 = 0 § ESS are 𝜏 = 1,0 , or = 0,1 everyone either plays 𝐵 or 𝐶 § The mixed strategy 𝜏 = (
( W , X W) that is a fixed point is not an ESS
§ If 𝑦 corresponds to the portion of the population playing 𝐵: § 𝑦̇ = 𝑦 1 − 𝑦 𝜌 𝐵,𝒚 − 𝜌 𝐶,𝒚 = 𝑦 1 − 𝑦 3𝑦 − 1 − 𝑦)) = 𝑦(1 − 𝑦)(4𝑦 − 1) § Fixed points are 𝑦∗ = 0, 𝑦∗ = 1, 𝑦∗ = 1/4
15781 Fall 2016: Lecture 22
16
§ Fixed points for the prisoners’ dilemma: 𝑦∗ = 0, 𝑦∗ = 1, 𝑦∗ = 1/4 § 𝑦̇ > 0 if 𝑦 > 1/4, and 𝑦̇ < 0 if 𝑦 < 1/4, that is the slope changes in 𝑦∗ = 1/4, such that only the pure strategies population behaviors are stable equilibria à Evolutionary end points § If population starts in a state with more than 25% individuals playing 𝐵, it evolves until everyone uses 𝐵, less than 25% à everyone will use 𝐶 § In the phase portrait, 𝑦( is the population playing 𝐵, and 𝑦. the fraction playing 𝐶 § Only the diagonal represents feasible population profiles 𝑦( + 𝑦. = 1
15781 Fall 2016: Lecture 22
17
§ Do all ESSs have a corresponding evolutionary end point and do all end points have a corresponding ESS? § A fixed point of the replicator dynamics is asymptotically stable if any small deviations from that state are eliminated by the dynamics as t → ∞ § Theorem: For any two-strategy pairwise contest, a strategy 𝜏 = (𝑦(,𝑦.) is an ESS if and only if the corresponding fixed point in the replicator dynamics is asymptotically stable § Theorem: For any two-strategy pairwise game, the following relations hold for a strategy 𝜏 and the corresponding population state 𝒚: § 𝜏 ∈ Set of ESSs in the symmetric game corresponding to replicator dynamics ⇔ 𝒚 ∈ Set of asymptotically stable fixed points in replicator dynamics § 𝒚 ∈ Set of asymptotically stable fixed points in the replicator dynamics ⟹ 𝜏 ∈ Set of symmetric Nash equilibrium strategies § 𝜏 ∈ Set of symmetric Nash equilibrium strategies ⟹ 𝒚 ∈ Set of fixed points
15781 Fall 2016: Lecture 22
18
§ 𝑇 = 𝐵, 𝐶 , 𝜌 𝐵, 𝐵 = 3 𝜌 𝐶, 𝐶 = 1 𝜌 𝐵, 𝐶 = 𝜌 𝐶, 𝐵 = 0 § ESS are 𝜏 = 1,0 , or = 0,1 everyone either plays 𝐵 or 𝐶 § The mixed strategy 𝜏 = (
( W , X W) that is a fixed point is not an ESS
§ If 𝑦 corresponds to the portion of the population playing 𝐵: § 𝑦̇ = 𝑦 1 − 𝑦 𝜌 𝐵,𝒚 − 𝜌 𝐶,𝒚 = 𝑦 1 − 𝑦 (3𝑦 − (1 − 𝑦)) = 𝑦(1 − 𝑦)(4𝑦 − 1) § Fixed points are 𝑦∗ = 0, 𝑦∗ = 1, 𝑦∗ = 1/4 § In the phase portrait, 𝑦( is the population playing 𝐵, and 𝑦. the fraction playing 𝐶 § All feasible strategies 𝑦L + 𝑦M = 1 are on the diagonal line joining (1,0) and (0,1)
15781 Fall 2016: Lecture 22
19
§ 𝑦̇ = 𝑦 1 − 𝑦 𝜌 𝐵,𝒚 − 𝜌 𝐶,𝒚 = 𝑦(1 − 𝑦)(4𝑦 − 1) § Asymptotic stability of the fixed points? à Linearize the system about the points and check the stability of the trajectories § Fixed point 𝑦∗ = 0: § Let’s consider a population near 𝑦∗ = 0 à 𝑦 𝑢 = 𝑦∗ + ε 𝑢 , ε 𝑢 > 0 § à 𝑦̇ = 𝜁̇ à ε̇ = −ε 1 − ε 1 − 4ε = −ε + 5ε. − 4εX § First-order (linear) approximation: à ignore terms proportional to ε8 for 𝑜 > 1 § à ε̇ ≈ −𝜁, that has solution: ε 𝑢 = ε 0 𝑓de, with ε 𝑢 → 0, as 𝑢 ⟶ ∞ § à Replicator dynamics exponentially decreases to zero small deviations from the fixed point à Fixed point 𝑦∗ = 0 is asymptotically stable § Fixed point 𝑦∗ = 1: § Population near 𝑦∗ = 1 à 𝑦 𝑢 = 𝑦∗ − ε 𝑢 = 1 − ε 𝑢 , ε 𝑢 > 0 § à ε̇ = − 1 − ε 1 − 1 + ε 4 − 4ε − 1 = −(ε − ε.)(3 − 4ε) § Linear approximation: ε̇ ≈ −3𝜁, that has solution: ε 𝑢 = ε 0 𝑓dXe, with ε 𝑢 → 0, as 𝑢 ⟶ ∞ à Fixed point 𝑦∗ = 0 is asymptotically stable § Fixed point 𝑦∗ = 1/4: § Linear approximation: ε̇ ≈
( (g 𝜁, that has solution: ε 𝑢 = ε 0 𝑓e/(g à Fixed
point 𝑦∗ = 0 is unstable (saddle)
15781 Fall 2016: Lecture 22
20
§ Let’s consider the case of 𝑜 pure strategies 𝑇 = 𝑡(,𝑡.,…, 𝑡8 , population state 𝒚 = 𝑦(,𝑦.,…𝑦8 § Replicator dynamics: 𝑦̇$ = 𝜌 𝑡$,𝒚 − 𝜌 ? 𝒚 𝑦$ = 𝑔
$ 𝒚 𝑗 = 1,…𝑜
§ Introducing the constraint ∑ 𝑦$ = 1
8 $7(
, the state vector become becomes 𝒚 = 𝑦(,𝑦.,…𝑦8d`( and the replicator dynamical system becomes: 𝑦̇$ = 𝑔
$ 𝒚 𝑗 = 1, …𝑜 − 1
§ Let’s consider the following pairwise contest game:
0,0 3,3 1,1 3,3 0,0 1,1 1,1 1,1 1,1
A B C A B C
§ Replicator dynamics: 𝑦̇( = 𝑦( 3𝑦. + 𝑦X − 𝜌 ? 𝒚 𝑦̇. = 𝑦. 3𝑦( + 𝑦X − 𝜌 ? 𝒚 𝑦̇X = 𝑦X(1 − 𝜌 ? 𝒚 ) 𝜌 ? 𝒚 = 6𝑦(𝑦. + 𝑦(𝑦X + 𝑦.𝑦X + 𝑦X
15781 Fall 2016: Lecture 22
21
0,0 3,3 1,1 3,3 0,0 1,1 1,1 1,1 1,1
A B C A B C
§ Replicator dynamics: 𝑦̇( = 𝑦( 3𝑦. + 𝑦X − 𝜌 ? 𝒚 𝑦̇. = 𝑦. 3𝑦( + 𝑦X − 𝜌 ? 𝒚 𝑦̇X = 𝑦X(1 − 𝜌 ? 𝒚 ) 𝜌 ? 𝒚 = 6𝑦(𝑦. + 𝑦(𝑦X + 𝑦.𝑦X + 𝑦X § It can be reduced to a two-variables dynamical system: 𝑦( = 𝑦, 𝑦. = 𝑧, 𝑦X= 1 − 𝑦 − 𝑧 𝑦̇ = 𝑦 1 − 𝑦 + 2𝑧 − 𝜌 ? 𝒚 𝑧̇ = 𝑧 1 + 2𝑦 − 𝑧 − 𝜌 ? 𝒚 𝜌 ? 𝒚 = 1 + 4𝑦𝑧 − 𝑦. − 𝑧.
15781 Fall 2016: Lecture 22
22
0,0 3,3 1,1 3,3 0,0 1,1 1,1 1,1 1,1
A B C A B C
𝑦( = 𝑦, 𝑦. = 𝑧, 𝑦X= 1 − 𝑦 − 𝑧 𝑦̇ = 𝑦 1 − 𝑦 + 2𝑧 − 𝜌 ? 𝒚 𝑧̇ = 𝑧 1 + 2𝑦 − 𝑧 − 𝜌 ? 𝒚 𝜌 ? 𝒚 = 1 + 4𝑦𝑧 − 𝑦. − 𝑧. § Fixed points: (0,0), (0,1), (1,0), (1/2,1/2) § Linearization using the Jacobian § Points must be hyperbolic § The entire non-grayed area forms the probability simplex for the problem, and contains all feasible mixed strategies
15781 Fall 2016: Lecture 22
23
𝐾 = 3𝑦. + 𝑧. − 8𝑦𝑧 − 2𝑦 + 2𝑧 −4𝑦. + 2𝑦𝑧 + 2𝑦 2𝑦𝑧 − 4𝑧. + 2𝑧 𝑦. + 3𝑧. − 8𝑦𝑧 + 2𝑦 − 2𝑧 § (0,0) is not hyperbolic § On the invariant lines 𝑦 = 0 and 𝑧 = 0, (0,0) acts as an attractor: if population starts with either 𝑦( = 0 or 𝑦. = 0, it ends up in (0,0,1). § Otherwise it is a repeller, with the population going to (
( . , ( .)
15781 Fall 2016: Lecture 22
24
§ Theorem: If (𝜏∗,𝜏∗) is a symmetric Nash equilibrium of the symmetric game corresponding to the replicator dynamics, then the population state 𝒚∗ = 𝜏∗ is a fixed point of the replicator dynamics § An evolutionary process can produce a rational behavior (Nash equilibrium) in a population of non-rational individuals (mass rationality) § Individuals only need the ability to evaluate the consequences of their actions, compare them to the results obtained by others, and swap to a better strategy for the current situation § Theorem: If 𝒚∗ is an asymptotically stable fixed point of the replicator dynamics, then the symmetric strategy pair (𝜏∗,𝜏∗) with 𝜏∗ = 𝒚∗ is a Nash equilibrium § Theorem: Every ESS corresponds to an asymptotically stable fixed point in the replication dynamics. That is, if 𝜏∗ is an ESS, then the population with state 𝒚∗ = 𝜏∗ is asymptotically stable
15781 Fall 2016: Lecture 22
25
§ E = set of ESSs, AS = set of asymptotically stable fixed points, N = set of Nash equilibrium points, F = set of fixed points
0,0 1, −2 1,1 −2,1 0,0 3,1 1,1 1,3 0,0
A B C A B C
The polymorphic population 𝒚∗ =
( X, ( X, ( X , ( X , ( X , ( X
is asymptotically stable in the replicator dynamics, but the strategy 𝜏∗ = 𝒚∗ is not an ESS § à There may be asymptotically stable fixed points in the replicator dynamics which do not correspond to an ESS
ESS AS Nash
F Σ
15781 Fall 2016: Lecture 22
26
Unique symmetric NE
1 3 ,1 3 , 1 3 , 1 3 ,1 3 , 1 3
§ This mixed Nash equilibrium is not an ESS (we have previously checked this)
0,0 −1,1 1, −1 1, −1 0,0 −1,1 −1,1 1, −1 0,0
R P S R P S
§ Replicator dynamics: 𝜌 ? 𝒚 = 0 𝑦̇( = 𝑦((𝑦. − 𝑦X) 𝑦̇. = 𝑦. 𝑦X − 𝑦( 𝑦̇X = 𝑦X(𝑦( − 𝑦() § Fixed points: (1,0,0), (0,1,0), (0,0,1), mixed NE § The pure strategies fixed points are not stable § The mixed NE (that corresponds to a polymorphic population) is a center: trajectories form closed loops around it § It can be verified considering a Lyapounov function
15781 Fall 2016: Lecture 22
27
§ Fixed points: (1,0,0), (0,1,0), (0,0,1),
( X , ( X , ( X , ( X , ( X , ( X
§ From the equations it can be noted that on the invariant line 𝑦( = 0, 𝑦̇. > 0, while on the line 𝑦. = 0, 𝑦̇( < 0 § Plots with reduced state using the relation 𝑦X = 1 − 𝑦( − 𝑦.
0,0 −1,1 1, −1 1, −1 0,0 −1,1 −1,1 1, −1 0,0
R P S R P S
15781 Fall 2016: Lecture 22
28
29
https://www.youtube.com/watch?v=M4cV0nCIZoc
Automaton, Proc. of IWINAC, LNCS 6687, 2011,
§ Rock-Paper-Scissors Automata: A Simulation of Bacterial diffusion (Bacterial computing) modeled as an evolutionary game