Multi-agent learning The repliato r dynami Gerard Vreeswijk , - - PowerPoint PPT Presentation

Multi-agent learning

Gerard Vreeswijk, Intelligent Software Systems, Computer Science Department, Faculty of Sciences, Utrecht University, The Netherlands.

Wednesday 10th June, 2020

Topics of today

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 2

• rtions

• ntinuous

Topics of today

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 2

■ Symmetric games, symmetric

Nash equilibria in symmetric games, a-symmetric Nash equilibria in symmetric games.

• rtions

• ntinuous

Topics of today

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 2

■ Symmetric games, symmetric

Nash equilibria in symmetric games, a-symmetric Nash equilibria in symmetric games.

■ Evolutionary game theory:

• rtions,

• ntinuous

Topics of today

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 2

■ Symmetric games, symmetric

Nash equilibria in symmetric games, a-symmetric Nash equilibria in symmetric games.

■ Evolutionary game theory:

• rtions,

■ The replicator dynamic.

• ntinuous

Topics of today

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 2

■ Symmetric games, symmetric

Nash equilibria in symmetric games, a-symmetric Nash equilibria in symmetric games.

■ Evolutionary game theory:

• rtions,

■ The replicator dynamic.

• Fitness vector, score-matrix
• ntinuous

Topics of today

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 2

■ Symmetric games, symmetric

Nash equilibria in symmetric games, a-symmetric Nash equilibria in symmetric games.

■ Evolutionary game theory:

• rtions,

■ The replicator dynamic.

• Fitness vector, score-matrix

(= grand table if species are reply rules).

• ntinuous

Topics of today

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 2

■ Symmetric games, symmetric

Nash equilibria in symmetric games, a-symmetric Nash equilibria in symmetric games.

■ Evolutionary game theory:

• rtions,

■ The replicator dynamic.

• Fitness vector, score-matrix

(= grand table if species are reply rules).

• The
• ntinuous

equation.

Topics of today

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 2

■ Symmetric games, symmetric

Nash equilibria in symmetric games, a-symmetric Nash equilibria in symmetric games.

■ Evolutionary game theory:

• rtions,

■ The replicator dynamic.

• Fitness vector, score-matrix

(= grand table if species are reply rules).

• The
• ntinuous

equation.

• The

equation.

Topics of today

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 2

■ Symmetric games, symmetric

Nash equilibria in symmetric games, a-symmetric Nash equilibria in symmetric games.

■ Evolutionary game theory:

• rtions,

■ The replicator dynamic.

• Fitness vector, score-matrix

(= grand table if species are reply rules).

• The
• ntinuous

equation.

• The

equation.

• The

Topics of today

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 2

■ Symmetric games, symmetric

Nash equilibria in symmetric games, a-symmetric Nash equilibria in symmetric games.

■ Evolutionary game theory:

• rtions,

■ The replicator dynamic.

• Fitness vector, score-matrix

(= grand table if species are reply rules).

• The
• ntinuous

equation.

• The

equation.

• The

• Derivation of the DRE from

the DSE.

Topics of today

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 2

■ Symmetric games, symmetric

Nash equilibria in symmetric games, a-symmetric Nash equilibria in symmetric games.

■ Evolutionary game theory:

• rtions,

■ The replicator dynamic.

• Fitness vector, score-matrix

(= grand table if species are reply rules).

• The
• ntinuous

equation.

• The

equation.

• The

• Derivation of the DRE from

the DSE.

• Derivation of the CRE from

the DRE.

Topics of today

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 2

■ Symmetric games, symmetric

Nash equilibria in symmetric games, a-symmetric Nash equilibria in symmetric games.

■ Evolutionary game theory:

• rtions,

■ The replicator dynamic.

• Fitness vector, score-matrix

(= grand table if species are reply rules).

• The
• ntinuous

equation.

• The

equation.

• The

• Derivation of the DRE from

the DSE.

• Derivation of the CRE from

the DRE.

• Properties of the replicator

dynamic, connection with Nash equilibria.

Symmetric games in normal form

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 3

Ways to denote the payoff matrix of a symmetric game

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 4

Symmetric game: only your actions matter, not your role.

Ways to denote the payoff matrix of a symmetric game

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 4

Symmetric game: only your actions matter, not your role. For 2-player games: it is not important whether you are row or column player:

Ways to denote the payoff matrix of a symmetric game

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 4

Symmetric game: only your actions matter, not your role. For 2-player games: it is not important whether you are row or column player:    A1 A2 A3 A1 1, 1 2, 3

−4, 6

A2 3, 2 0, 0 8, −7 A3 6, −4

−7, 8

5, 5   

=

   A1 A2 A3 A1 1, 1 2, 3

−4, 6

A2 0, 0 8, −7 A3 5, 5   

Ways to denote the payoff matrix of a symmetric game

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 4

Symmetric game: only your actions matter, not your role. For 2-player games: it is not important whether you are row or column player:    A1 A2 A3 A1 1, 1 2, 3

−4, 6

A2 3, 2 0, 0 8, −7 A3 6, −4

−7, 8

5, 5   

=

   A1 A2 A3 A1 1, 1 2, 3

−4, 6

A2 0, 0 8, −7 A3 5, 5   

∼

   A1 A2 A3 A1 1 2

−4

A2 3 8 A3 6

−7

5   .

■ As a bi-matrix.

Ways to denote the payoff matrix of a symmetric game

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 4

Symmetric game: only your actions matter, not your role. For 2-player games: it is not important whether you are row or column player:    A1 A2 A3 A1 1, 1 2, 3

−4, 6

A2 3, 2 0, 0 8, −7 A3 6, −4

−7, 8

5, 5   

=

   A1 A2 A3 A1 1, 1 2, 3

−4, 6

A2 0, 0 8, −7 A3 5, 5   

∼

   A1 A2 A3 A1 1 2

−4

A2 3 8 A3 6

−7

5   .

■ As a bi-matrix. ■ As a partially filled bi-matrix.

Ways to denote the payoff matrix of a symmetric game

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 4

Symmetric game: only your actions matter, not your role. For 2-player games: it is not important whether you are row or column player:    A1 A2 A3 A1 1, 1 2, 3

−4, 6

A2 3, 2 0, 0 8, −7 A3 6, −4

−7, 8

5, 5   

=

   A1 A2 A3 A1 1, 1 2, 3

−4, 6

A2 0, 0 8, −7 A3 5, 5   

∼

   A1 A2 A3 A1 1 2

−4

A2 3 8 A3 6

−7

5   .

■ As a bi-matrix. ■ As a partially filled bi-matrix. ■ As a plain matrix.

Hawk vs. Dove

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 5

Symmetric normal-form games

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 6

• Example. Hawk-dove game (share V or threaten [possibly fight: −C]):

H D H

(V − C)/2

V D V/2

Symmetric normal-form games

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 6

• Example. Hawk-dove game (share V or threaten [possibly fight: −C]):

H D H

(V − C)/2

V D V/2 H D H

−2, −2

2, 0 D 0, 2 1, 1 V=2, C=6

Symmetric normal-form games

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 6

• Example. Hawk-dove game (share V or threaten [possibly fight: −C]):

H D H

(V − C)/2

V D V/2 H D H

−2, −2

2, 0 D 0, 2 1, 1 V=2, C=6 Other instantiations: prisoner’s dilemma, chicken (= hawk-dove), matching pennies, stag hunt.

Symmetric normal-form games

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 6

• Example. Hawk-dove game (share V or threaten [possibly fight: −C]):

H D H

(V − C)/2

V D V/2 H D H

−2, −2

2, 0 D 0, 2 1, 1 V=2, C=6 Other instantiations: prisoner’s dilemma, chicken (= hawk-dove), matching pennies, stag hunt.

• Definition. A game is

and payoffs: ui(a1, . . . , ai, . . . , aj, . . . , an) = uj(a1, . . . , aj, . . . , ai, . . . , an). for all i and j.

Symmetric normal-form games

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 6

• Example. Hawk-dove game (share V or threaten [possibly fight: −C]):

H D H

(V − C)/2

V D V/2 H D H

−2, −2

2, 0 D 0, 2 1, 1 V=2, C=6 Other instantiations: prisoner’s dilemma, chicken (= hawk-dove), matching pennies, stag hunt.

• Definition. A game is

and payoffs: ui(a1, . . . , ai, . . . , aj, . . . , an) = uj(a1, . . . , aj, . . . , ai, . . . , an). for all i and j. So a 2-player game G = (A, B) is symmetric iff m = n and B = AT.

Symmetric equilibrium

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 7

Symmetric equilibrium

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 7

• Definition. Let p be a strategy in an n-player symmetric game. If

the n-vector (p, . . . , p) is a NE, p is called a

Symmetric equilibrium

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 7

• Definition. Let p be a strategy in an n-player symmetric game. If

the n-vector (p, . . . , p) is a NE, p is called a

■ !! Symmetric equilibria can be identified with strategies !!

Symmetric equilibrium

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 7

• Definition. Let p be a strategy in an n-player symmetric game. If

the n-vector (p, . . . , p) is a NE, p is called a

■ !! Symmetric equilibria can be identified with strategies !! ■ (Theorem.) Every symmetric game has at least one symmetric

equilibrium.

Symmetric equilibrium

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 7

• Definition. Let p be a strategy in an n-player symmetric game. If

the n-vector (p, . . . , p) is a NE, p is called a

■ !! Symmetric equilibria can be identified with strategies !! ■ (Theorem.) Every symmetric game has at least one symmetric

equilibrium.

■ (Fact.) Symmetric games can have a-symmetric equilibria.

Symmetric equilibrium

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 7

• Definition. Let p be a strategy in an n-player symmetric game. If

the n-vector (p, . . . , p) is a NE, p is called a

■ !! Symmetric equilibria can be identified with strategies !! ■ (Theorem.) Every symmetric game has at least one symmetric

equilibrium.

■ (Fact.) Symmetric games can have a-symmetric equilibria.

For example Hawk-Dove: H D H

−2, −2

2, 0 D 0, 2 1, 1

Symmetric equilibrium

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 7

• Definition. Let p be a strategy in an n-player symmetric game. If

the n-vector (p, . . . , p) is a NE, p is called a

■ !! Symmetric equilibria can be identified with strategies !! ■ (Theorem.) Every symmetric game has at least one symmetric

equilibrium.

■ (Fact.) Symmetric games can have a-symmetric equilibria.

For example Hawk-Dove: H D H

−2, −2

2, 0 D 0, 2 1, 1 Two asymmetric equilibria and one symmetric equilibrium (1/3, 1/3).

Evolutionary game theory

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 8

Evolutionary game theory: the idea

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 9

Evolutionary game theory: the idea

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 10

• rtions

Evolutionary game theory: the idea

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 10

■ There are n, say 5, species.

• rtions

Evolutionary game theory: the idea

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 10

■ There are n, say 5, species. An encounter

between individuals of different species yields payoffs for both.

• rtions

Evolutionary game theory: the idea

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 10

■ There are n, say 5, species. An encounter

between individuals of different species yields payoffs for both. For row: A =       s1 s2 s3 s4 s5 s1 6 7

−1

s2

−1

5

−1

4 7 s3 9 8 9 6 s4

−4 −2

3

−3

s5 3 6

−1

      .

• rtions

Evolutionary game theory: the idea

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 10

■ There are n, say 5, species. An encounter

between individuals of different species yields payoffs for both. For row: A =       s1 s2 s3 s4 s5 s1 6 7

−1

s2

−1

5

−1

4 7 s3 9 8 9 6 s4

−4 −2

3

−3

s5 3 6

−1

      .

■ The population consists of a very large

number of individuals, each playing a pure

• strategy. Individuals interact randomly.

• rtions

Evolutionary game theory: the idea

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 10

■ There are n, say 5, species. An encounter

between individuals of different species yields payoffs for both. For row: A =       s1 s2 s3 s4 s5 s1 6 7

−1

s2

−1

5

−1

4 7 s3 9 8 9 6 s4

−4 −2

3

−3

s5 3 6

−1

      .

■ The population consists of a very large

number of individuals, each playing a pure

• strategy. Individuals interact randomly.

■ We are

interested in

• rtions: p

= (p1, . . . , p5).

Evolutionary game theory: the idea

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 10

■ There are n, say 5, species. An encounter

between individuals of different species yields payoffs for both. For row: A =       s1 s2 s3 s4 s5 s1 6 7

−1

s2

−1

5

−1

4 7 s3 9 8 9 6 s4

−4 −2

3

−3

s5 3 6

−1

      .

■ The population consists of a very large

number of individuals, each playing a pure

• strategy. Individuals interact randomly.

■ We are

interested in

• rtions: p

= (p1, . . . , p5).

■ The

species i is: fi = ∑5

j=1 pjAij

= (Ap)i.

Evolutionary game theory: the idea

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 10

■ There are n, say 5, species. An encounter

between individuals of different species yields payoffs for both. For row: A =       s1 s2 s3 s4 s5 s1 6 7

−1

s2

−1

5

−1

4 7 s3 9 8 9 6 s4

−4 −2

3

−3

s5 3 6

−1

      .

■ The population consists of a very large

number of individuals, each playing a pure

• strategy. Individuals interact randomly.

■ We are

interested in

• rtions: p

= (p1, . . . , p5).

■ The

species i is: fi = ∑5

j=1 pjAij

= (Ap)i.

■ The

¯ f = ∑5

i=1 pi fi

= pTAp.

The replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 11

History of the replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 12

History of the replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 12

■ Defined for a single species by Taylor and Jonker (1978), and named

by Schuster and Sigmund (1983): “Several evolutionary models in distinct biological fields— population genetics, population ecology, early biochemical evo- lution and sociobiology—lead independently to the same class of

History of the replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 12

■ Defined for a single species by Taylor and Jonker (1978), and named

by Schuster and Sigmund (1983): “Several evolutionary models in distinct biological fields— population genetics, population ecology, early biochemical evo- lution and sociobiology—lead independently to the same class of

■ The replicator equation is the first game dynamics studied in

connection with evolutionary game theory (as developed by Maynard Smith and Price).

History of the replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 12

■ Defined for a single species by Taylor and Jonker (1978), and named

by Schuster and Sigmund (1983): “Several evolutionary models in distinct biological fields— population genetics, population ecology, early biochemical evo- lution and sociobiology—lead independently to the same class of

■ The replicator equation is the first game dynamics studied in

connection with evolutionary game theory (as developed by Maynard Smith and Price).

Taylor P.D., Jonker L. “Evolutionarily stable strategies and game dynamics” in: Math. Biosci. 1978;40(1), pp. 145-156.

History of the replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 12

■ Defined for a single species by Taylor and Jonker (1978), and named

by Schuster and Sigmund (1983): “Several evolutionary models in distinct biological fields— population genetics, population ecology, early biochemical evo- lution and sociobiology—lead independently to the same class of

■ The replicator equation is the first game dynamics studied in

connection with evolutionary game theory (as developed by Maynard Smith and Price).

Taylor P.D., Jonker L. “Evolutionarily stable strategies and game dynamics” in: Math. Biosci. 1978;40(1), pp. 145-156. Schuster P., Sigmund K. “Replicator dynamics” in: J. Theor. Biol. 1983, 100(3), pp. 533-538.

The replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 13

• rtion

The replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 13

■ The

decline) due to mutual interaction.

• rtion

The replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 13

■ The

decline) due to mutual interaction.

■ It is assumed that if an individual of species i interacts with an

individual of species j, the expected reward for the individual of type i is a constant aij.

• rtion

The replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 13

■ The

decline) due to mutual interaction.

■ It is assumed that if an individual of species i interacts with an

individual of species j, the expected reward for the individual of type i is a constant aij. Summarised in a

   a11 . . . a1n . . . ... . . . an1 . . . ann   .

• rtion

The replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 13

■ The

decline) due to mutual interaction.

■ It is assumed that if an individual of species i interacts with an

individual of species j, the expected reward for the individual of type i is a constant aij. Summarised in a

   a11 . . . a1n . . . ... . . . an1 . . . ann   .

Proportions

• rtion

The replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 13

■ The

decline) due to mutual interaction.

■ It is assumed that if an individual of species i interacts with an

individual of species j, the expected reward for the individual of type i is a constant aij. Summarised in a

   a11 . . . a1n . . . ... . . . an1 . . . ann   .

Proportions

■ The number of individuals of species i is denoted by qi, or qi(t).

• rtion

The replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 13

■ The

decline) due to mutual interaction.

■ It is assumed that if an individual of species i interacts with an

individual of species j, the expected reward for the individual of type i is a constant aij. Summarised in a

   a11 . . . a1n . . . ... . . . an1 . . . ann   .

Proportions

■ The number of individuals of species i is denoted by qi, or qi(t). ■

pj =Def qj/q is the

• rtion of species i, where q = q1 + · · · + qn.

The replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 13

■ The

decline) due to mutual interaction.

■ It is assumed that if an individual of species i interacts with an

individual of species j, the expected reward for the individual of type i is a constant aij. Summarised in a

   a11 . . . a1n . . . ... . . . an1 . . . ann   .

Proportions

■ The number of individuals of species i is denoted by qi, or qi(t). ■

pj =Def qj/q is the

• rtion of species i, where q = q1 + · · · + qn.

■ So pi ∝ qi and p1 + · · · + pn = 1.

Fitness

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 14

Fitness

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 14

■ The

expected reward when it encounters a random individual in the population.

Fitness

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 14

■ The

expected reward when it encounters a random individual in the population.

■ Example. Suppose

A =   1 3 1 1 2 3 4 1 3   and p =   0.1 0.4 0.5   .

Fitness

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 14

■ The

expected reward when it encounters a random individual in the population.

■ Example. Suppose

A =   1 3 1 1 2 3 4 1 3   and p =   0.1 0.4 0.5   . The

computed as follows: f = Ap =   1 3 1 1 2 3 4 1 3     0.1 0.4 0.5   =   1.8 2.4 2.3   .

Fitness

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 14

■ The

expected reward when it encounters a random individual in the population.

■ Example. Suppose

A =   1 3 1 1 2 3 4 1 3   and p =   0.1 0.4 0.5   . The

computed as follows: f = Ap =   1 3 1 1 2 3 4 1 3     0.1 0.4 0.5   =   1.8 2.4 2.3   .

■ Average fitness:

¯ f (t) =

3

∑

j=1

pi fi(t)

= p(Ap) = 2.29.

Fitness

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 14

■ The

expected reward when it encounters a random individual in the population.

■ Example. Suppose

A =   1 3 1 1 2 3 4 1 3   and p =   0.1 0.4 0.5   . The

computed as follows: f = Ap =   1 3 1 1 2 3 4 1 3     0.1 0.4 0.5   =   1.8 2.4 2.3   .

■ Average fitness:

¯ f (t) =

3

∑

j=1

pi fi(t)

= p(Ap) = 2.29.

■ Fitness of species 1:

f1 =

3

∑

j=1

pja1j

= (Ap)1 = 1.8.

Fitness

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 14

■ The

expected reward when it encounters a random individual in the population.

■ Example. Suppose

A =   1 3 1 1 2 3 4 1 3   and p =   0.1 0.4 0.5   . The

computed as follows: f = Ap =   1 3 1 1 2 3 4 1 3     0.1 0.4 0.5   =   1.8 2.4 2.3   .

■ Average fitness:

¯ f (t) =

3

∑

j=1

pi fi(t)

= p(Ap) = 2.29.

■ Fitness of species 1:

f1 =

3

∑

j=1

pja1j

= (Ap)1 = 1.8.

So species 1 does worse than average.

Fitness

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 14

■ The

expected reward when it encounters a random individual in the population.

■ Example. Suppose

A =   1 3 1 1 2 3 4 1 3   and p =   0.1 0.4 0.5   . The

computed as follows: f = Ap =   1 3 1 1 2 3 4 1 3     0.1 0.4 0.5   =   1.8 2.4 2.3   .

■ Average fitness:

¯ f (t) =

3

∑

j=1

pi fi(t)

= p(Ap) = 2.29.

■ Fitness of species 1:

f1 =

3

∑

j=1

pja1j

= (Ap)1 = 1.8.

So species 1 does worse than average.

■ Species 2 and 3 have

fitness 2.4 and 2.3, respectively.

The continuous replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 15

The continuous replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 16

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ], where ˙ pi(t) is shorthand for the change of pi in time: ˙ pi(t) = p′

i(t) = dpi(t)/dt.

The continuous replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 16

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ], where ˙ pi(t) is shorthand for the change of pi in time: ˙ pi(t) = p′

i(t) = dpi(t)/dt.

Example 1. Suppose the proportion of species 7 at time t is p7(t) = 0.2

The continuous replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 16

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ], where ˙ pi(t) is shorthand for the change of pi in time: ˙ pi(t) = p′

i(t) = dpi(t)/dt.

Example 1. Suppose the proportion of species 7 at time t is p7(t) = 0.2, the fitness of species 7 at time t is f7(t) = 6

The continuous replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 16

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ], where ˙ pi(t) is shorthand for the change of pi in time: ˙ pi(t) = p′

i(t) = dpi(t)/dt.

Example 1. Suppose the proportion of species 7 at time t is p7(t) = 0.2, the fitness of species 7 at time t is f7(t) = 6, and the average fitness at time t is ¯ f (t) = 9.

The continuous replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 16

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ], where ˙ pi(t) is shorthand for the change of pi in time: ˙ pi(t) = p′

i(t) = dpi(t)/dt.

Example 1. Suppose the proportion of species 7 at time t is p7(t) = 0.2, the fitness of species 7 at time t is f7(t) = 6, and the average fitness at time t is ¯ f (t) = 9. How fast does p7 grow on time t?

The continuous replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 16

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ], where ˙ pi(t) is shorthand for the change of pi in time: ˙ pi(t) = p′

i(t) = dpi(t)/dt.

Example 1. Suppose the proportion of species 7 at time t is p7(t) = 0.2, the fitness of species 7 at time t is f7(t) = 6, and the average fitness at time t is ¯ f (t) = 9. How fast does p7 grow on time t?

• Answer. ˙

p7(t)

The continuous replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 16

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ], where ˙ pi(t) is shorthand for the change of pi in time: ˙ pi(t) = p′

i(t) = dpi(t)/dt.

Example 1. Suppose the proportion of species 7 at time t is p7(t) = 0.2, the fitness of species 7 at time t is f7(t) = 6, and the average fitness at time t is ¯ f (t) = 9. How fast does p7 grow on time t?

• Answer. ˙

p7(t) = p7(t)[ f7(t) − ¯ f (t)]

The continuous replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 16

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ], where ˙ pi(t) is shorthand for the change of pi in time: ˙ pi(t) = p′

i(t) = dpi(t)/dt.

Example 1. Suppose the proportion of species 7 at time t is p7(t) = 0.2, the fitness of species 7 at time t is f7(t) = 6, and the average fitness at time t is ¯ f (t) = 9. How fast does p7 grow on time t?

• Answer. ˙

p7(t) = p7(t)[ f7(t) − ¯ f (t)] = 0.2(6 − 9)

The continuous replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 16

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ], where ˙ pi(t) is shorthand for the change of pi in time: ˙ pi(t) = p′

i(t) = dpi(t)/dt.

Example 1. Suppose the proportion of species 7 at time t is p7(t) = 0.2, the fitness of species 7 at time t is f7(t) = 6, and the average fitness at time t is ¯ f (t) = 9. How fast does p7 grow on time t?

• Answer. ˙

p7(t) = p7(t)[ f7(t) − ¯ f (t)] = 0.2(6 − 9) = −0.6.

The continuous replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 16

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ], where ˙ pi(t) is shorthand for the change of pi in time: ˙ pi(t) = p′

i(t) = dpi(t)/dt.

Example 1. Suppose the proportion of species 7 at time t is p7(t) = 0.2, the fitness of species 7 at time t is f7(t) = 6, and the average fitness at time t is ¯ f (t) = 9. How fast does p7 grow on time t?

• Answer. ˙

p7(t) = p7(t)[ f7(t) − ¯ f (t)] = 0.2(6 − 9) = −0.6.

• Example 2. Suppose p5(t) = 0.2, f5(t) = 6, and ¯

f (t) = 4. Same question.

The continuous replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 16

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ], where ˙ pi(t) is shorthand for the change of pi in time: ˙ pi(t) = p′

i(t) = dpi(t)/dt.

Example 1. Suppose the proportion of species 7 at time t is p7(t) = 0.2, the fitness of species 7 at time t is f7(t) = 6, and the average fitness at time t is ¯ f (t) = 9. How fast does p7 grow on time t?

• Answer. ˙

p7(t) = p7(t)[ f7(t) − ¯ f (t)] = 0.2(6 − 9) = −0.6.

• Example 2. Suppose p5(t) = 0.2, f5(t) = 6, and ¯

f (t) = 4. Same question.

• Answer. ˙

p5(t) = p5(t)[ f5(t) − ¯ f (t)] = 0.2(6 − 4) = 0.4.

The dynamics of the replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 17

Relative score matrix A =   1 3 1 1 2 3 4 1 3  , start proportions p =   1/3 1/3 1/3  .

Phase space of the replicator on the previous page

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 18

Circled rest points indicate Nash equilibria of the score-matrix, interpreted as the payoff matrix of a symmetric game in normal form.

A replicator dynamic in a higher dimension

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 19

Rest point, stable point, asymptotically stable point

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 20

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ] is a system of differential equations. We have p = (p1, . . . , pn)

• int

Rest point, stable point, asymptotically stable point

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 20

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ] is a system of differential equations. We have p = (p1, . . . , pn) ∈ ∆n

• int

Rest point, stable point, asymptotically stable point

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 20

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ] is a system of differential equations. We have p = (p1, . . . , pn) ∈ ∆n and ˙ p = ( ˙ p1, . . . , ˙ pn)

• int

Rest point, stable point, asymptotically stable point

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 20

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ] is a system of differential equations. We have p = (p1, . . . , pn) ∈ ∆n and ˙ p = ( ˙ p1, . . . , ˙ pn) ∈ Rn.

• int

Rest point, stable point, asymptotically stable point

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 20

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ] is a system of differential equations. We have p = (p1, . . . , pn) ∈ ∆n and ˙ p = ( ˙ p1, . . . , ˙ pn) ∈ Rn. Definitions:

• int

Rest point, stable point, asymptotically stable point

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 20

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ] is a system of differential equations. We have p = (p1, . . . , pn) ∈ ∆n and ˙ p = ( ˙ p1, . . . , ˙ pn) ∈ Rn. Definitions:

■

p is called a

• int, if ˙

p = 0.

Rest point, stable point, asymptotically stable point

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 20

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ] is a system of differential equations. We have p = (p1, . . . , pn) ∈ ∆n and ˙ p = ( ˙ p1, . . . , ˙ pn) ∈ Rn. Definitions:

■

p is called a

• int, if ˙

p = 0. (“If at p, then stays at p”.)

Rest point, stable point, asymptotically stable point

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 20

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ] is a system of differential equations. We have p = (p1, . . . , pn) ∈ ∆n and ˙ p = ( ˙ p1, . . . , ˙ pn) ∈ Rn. Definitions:

■

p is called a

• int, if ˙

p = 0. (“If at p, then stays at p”.)

■ A rest point p is called

• f p there is another neighborhood U′ of p such that states in U′, if

iterated, remain within U.

Rest point, stable point, asymptotically stable point

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 20

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ] is a system of differential equations. We have p = (p1, . . . , pn) ∈ ∆n and ˙ p = ( ˙ p1, . . . , ˙ pn) ∈ Rn. Definitions:

■

p is called a

• int, if ˙

p = 0. (“If at p, then stays at p”.)

■ A rest point p is called

• f p there is another neighborhood U′ of p such that states in U′, if

iterated, remain within U. (“If close to p, then always close to p.”)

Rest point, stable point, asymptotically stable point

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 20

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ] is a system of differential equations. We have p = (p1, . . . , pn) ∈ ∆n and ˙ p = ( ˙ p1, . . . , ˙ pn) ∈ Rn. Definitions:

■

p is called a

• int, if ˙

p = 0. (“If at p, then stays at p”.)

■ A rest point p is called

• f p there is another neighborhood U′ of p such that states in U′, if

iterated, remain within U. (“If close to p, then always close to p.”)

■ A rest point p is called

such that all proportion vectors in U, if iterated, converge to p.

Rest point, stable point, asymptotically stable point

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 20

The

• ntinuous

˙ pi(t) = pi(t)[ fi(t) − ¯ f (t) ] is a system of differential equations. We have p = (p1, . . . , pn) ∈ ∆n and ˙ p = ( ˙ p1, . . . , ˙ pn) ∈ Rn. Definitions:

■

p is called a

• int, if ˙

p = 0. (“If at p, then stays at p”.)

■ A rest point p is called

• f p there is another neighborhood U′ of p such that states in U′, if

iterated, remain within U. (“If close to p, then always close to p.”)

■ A rest point p is called

such that all proportion vectors in U, if iterated, converge to p. (“if close to p, then convergence to p.”).

Relation with Nash equilibria

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 21

State p is a Nash equilibrium:

∀q : q(Ap) ≤ p(Ap) ⇔ ∀q : q f ≤ p f ⇔ ∀q : q1 f1 + · · · + qn fn ≤ ¯

f . If for all i: fi ≤ ¯ f, then it must be that for all i: fi = ¯ f (check!), which means we have a rest

• point. Such a rest point is called

Relation with Nash equilibria

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 21

State p is a Nash equilibrium:

∀q : q(Ap) ≤ p(Ap) ⇔ ∀q : q f ≤ p f ⇔ ∀q : q1 f1 + · · · + qn fn ≤ ¯

f . If for all i: fi ≤ ¯ f, then it must be that for all i: fi = ¯ f (check!), which means we have a rest

• point. Such a rest point is called

■ Nash equilibrium ⇔

saturated rest point.

Relation with Nash equilibria

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 21

State p is a Nash equilibrium:

∀q : q(Ap) ≤ p(Ap) ⇔ ∀q : q f ≤ p f ⇔ ∀q : q1 f1 + · · · + qn fn ≤ ¯

f . If for all i: fi ≤ ¯ f, then it must be that for all i: fi = ¯ f (check!), which means we have a rest

• point. Such a rest point is called

■ Nash equilibrium ⇔

saturated rest point.

• Proof. ⇒: take pure q. ⇐: if

for all i: fi ≤ f , then no convex combination of those fi can supersede ¯ f.

Relation with Nash equilibria

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 21

State p is a Nash equilibrium:

∀q : q(Ap) ≤ p(Ap) ⇔ ∀q : q f ≤ p f ⇔ ∀q : q1 f1 + · · · + qn fn ≤ ¯

f . If for all i: fi ≤ ¯ f, then it must be that for all i: fi = ¯ f (check!), which means we have a rest

• point. Such a rest point is called

■ Nash equilibrium ⇔

saturated rest point.

• Proof. ⇒: take pure q. ⇐: if

for all i: fi ≤ f , then no convex combination of those fi can supersede ¯ f.

■ Nash equilibrium ⇒ rest

• point. (Trivial.)

Relation with Nash equilibria

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 21

State p is a Nash equilibrium:

∀q : q(Ap) ≤ p(Ap) ⇔ ∀q : q f ≤ p f ⇔ ∀q : q1 f1 + · · · + qn fn ≤ ¯

f . If for all i: fi ≤ ¯ f, then it must be that for all i: fi = ¯ f (check!), which means we have a rest

• point. Such a rest point is called

■ Nash equilibrium ⇔

saturated rest point.

• Proof. ⇒: take pure q. ⇐: if

for all i: fi ≤ f , then no convex combination of those fi can supersede ¯ f.

■ Nash equilibrium ⇒ rest

• point. (Trivial.)

■ Fully mixed rest point ⇒

Nash equilibrium. (Because fully mixed implies saturated.)

Relation with Nash equilibria

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 21

State p is a Nash equilibrium:

∀q : q(Ap) ≤ p(Ap) ⇔ ∀q : q f ≤ p f ⇔ ∀q : q1 f1 + · · · + qn fn ≤ ¯

f . If for all i: fi ≤ ¯ f, then it must be that for all i: fi = ¯ f (check!), which means we have a rest

• point. Such a rest point is called

■ Nash equilibrium ⇔

saturated rest point.

• Proof. ⇒: take pure q. ⇐: if

for all i: fi ≤ f , then no convex combination of those fi can supersede ¯ f.

■ Nash equilibrium ⇒ rest

• point. (Trivial.)

■ Fully mixed rest point ⇒

Nash equilibrium. (Because fully mixed implies saturated.)

■ Strict Nash equilibrium ⇒

asymptotically stable.

Relation with Nash equilibria

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 21

State p is a Nash equilibrium:

∀q : q(Ap) ≤ p(Ap) ⇔ ∀q : q f ≤ p f ⇔ ∀q : q1 f1 + · · · + qn fn ≤ ¯

f . If for all i: fi ≤ ¯ f, then it must be that for all i: fi = ¯ f (check!), which means we have a rest

• point. Such a rest point is called

■ Nash equilibrium ⇔

saturated rest point.

• Proof. ⇒: take pure q. ⇐: if

for all i: fi ≤ f , then no convex combination of those fi can supersede ¯ f.

■ Nash equilibrium ⇒ rest

• point. (Trivial.)

■ Fully mixed rest point ⇒

Nash equilibrium. (Because fully mixed implies saturated.)

■ Strict Nash equilibrium ⇒

asymptotically stable.

■ Limit point in the interior of

∆n ⇒ Nash equilibrium.

Relation with Nash equilibria

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 21

State p is a Nash equilibrium:

∀q : q(Ap) ≤ p(Ap) ⇔ ∀q : q f ≤ p f ⇔ ∀q : q1 f1 + · · · + qn fn ≤ ¯

f . If for all i: fi ≤ ¯ f, then it must be that for all i: fi = ¯ f (check!), which means we have a rest

• point. Such a rest point is called

■ Nash equilibrium ⇔

saturated rest point.

• Proof. ⇒: take pure q. ⇐: if

for all i: fi ≤ f , then no convex combination of those fi can supersede ¯ f.

■ Nash equilibrium ⇒ rest

• point. (Trivial.)

■ Fully mixed rest point ⇒

Nash equilibrium. (Because fully mixed implies saturated.)

■ Strict Nash equilibrium ⇒

asymptotically stable.

■ Limit point in the interior of

∆n ⇒ Nash equilibrium.

■ Asymptotically stable in the

interior of ∆n ⇒ isolated trembling-hand perfect Nash equilibrium.

Not all Nash equilibria are Lyapunov stable

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 22

(1, 0, 0) is Nash but not Lyapunov stable. (The picture is merely

suggestive, since it only contains a few traces of the dynamics.)

Not all Nash equilibria are Lyapunov stable

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 23

(1, 0, 0) is Nash but not Lyapunov stable. (The picture is merely

suggestive, since it only contains a few traces of the dynamics.)

The discrete replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 24

The discrete step equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 25

The discrete step equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 25

■ The

qi(t + 1) =Def qi(t)[1 + β + fi(t)], where 1 is the reproduction factor, β is the

fi(t) indicates the percentage that is added / subtracted due to fitness.

The discrete step equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 25

■ The

qi(t + 1) =Def qi(t)[1 + β + fi(t)], where 1 is the reproduction factor, β is the

fi(t) indicates the percentage that is added / subtracted due to fitness.

■ To prevent negative proportions and sudden extermination, it is

required that fitness, hence scores, must be defined such that 1 + β + fi(t) > 0, for all t and i.

The discrete step equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 25

■ The

qi(t + 1) =Def qi(t)[1 + β + fi(t)], where 1 is the reproduction factor, β is the

fi(t) indicates the percentage that is added / subtracted due to fitness.

■ To prevent negative proportions and sudden extermination, it is

required that fitness, hence scores, must be defined such that 1 + β + fi(t) > 0, for all t and i. This requirement is often left implicit in the literature,

The discrete step equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 25

■ The

qi(t + 1) =Def qi(t)[1 + β + fi(t)], where 1 is the reproduction factor, β is the

fi(t) indicates the percentage that is added / subtracted due to fitness.

■ To prevent negative proportions and sudden extermination, it is

required that fitness, hence scores, must be defined such that 1 + β + fi(t) > 0, for all t and i. This requirement is often left implicit in the literature,

■ The

∆qi(t) = qi(t + 1) − qi(t)

= qi(t)[1 + β + fi(t)] − qi(t) = qi(t)[β + fi(t)].

The discrete replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 26

The discrete replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 26

■ The

pi(t + 1) = pi(t)1 + β + fi(t) 1 + β + ¯ f (t)

The discrete replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 26

■ The

pi(t + 1) = pi(t)1 + β + fi(t) 1 + β + ¯ f (t)

■ The idea of the discrete replicator equation: between two generations,

an individual of type i produces 1 + β + fi(t)

• ffspring. Here, β is the

The discrete replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 26

■ The

pi(t + 1) = pi(t)1 + β + fi(t) 1 + β + ¯ f (t)

■ The idea of the discrete replicator equation: between two generations,

an individual of type i produces 1 + β + fi(t)

• ffspring. Here, β is the

■ The DRE follows from the

qi(t + 1) =Def qi(t)[1 + β + fi(t)].

Derivation of the DRE from the DSE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 27

pi(t + 1) = qi(t + 1) ∑n

j=1 qj(t + 1) =

qi(t)[1 + β + fi(t)] ∑n

j=1 qj(t)[1 + β + fj(t)]

=

1 q(t)qi(t)[1 + β + fi(t)] 1 q(t) ∑n j=1 qj(t)[1 + β + fj(t)]

=

pi(t)[1 + β + fi(t)] ∑n

j=1 pj(t)[1 + β + fj(t)]

=

pi(t)[1 + β + fi(t)] ∑n

j=1 pj(t) + β ∑n j=1 pj(t) + ∑n j=1 pj(t) fj(t)]

= pi(t)[1 + β + fi(t)]

1 + β + ¯ f (t)

= pi(t)1 + β + fi(t)

1 + β + ¯ f (t) .

Properties of the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 28

Properties of the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 28

• Claim. If a species is present, is was present and will be present
• forever. Same for absent.
• Proof. Just look at the

pi(t + 1) = pi(t)1 + β + fi(t) 1 + β + ¯ f (t) and recall that 1 + β + fi(t) > 0 for all t and i, hence 1 + β + ¯ f (t) > 0 for all t. So all the pi are always multiplied by a positive number.

Properties of the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 28

• Claim. If a species is present, is was present and will be present
• forever. Same for absent.
• Proof. Just look at the

pi(t + 1) = pi(t)1 + β + fi(t) 1 + β + ¯ f (t) and recall that 1 + β + fi(t) > 0 for all t and i, hence 1 + β + ¯ f (t) > 0 for all t. So all the pi are always multiplied by a positive number.

■ If pi was 0 it remains 0.

Properties of the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 28

• Claim. If a species is present, is was present and will be present
• forever. Same for absent.
• Proof. Just look at the

pi(t + 1) = pi(t)1 + β + fi(t) 1 + β + ¯ f (t) and recall that 1 + β + fi(t) > 0 for all t and i, hence 1 + β + ¯ f (t) > 0 for all t. So all the pi are always multiplied by a positive number.

■ If pi was 0 it remains 0. ■ If pi was positive it remains positive.

If a species is absent, it will be absent forever

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 29

Phase space of a replicator. Notice that corners, edges, and the interior map into themselves. This is always the case.

If a species is absent, it will be absent forever

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 30

Phase space of a replicator. Notice that corners, edges, and the interior map into themselves. This is always the case.

If a species is absent, it will be absent forever

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 31

Phase space of a replicator. Notice that corners, edges, and the interior map into themselves. This is always the case. q

Properties of the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 32

Properties of the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 32

• Claim. Species i grows if and only if pi(t) > 0 and it has above

average fitness.

Properties of the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 32

• Claim. Species i grows if and only if pi(t) > 0 and it has above

average fitness. Proof. pi(t + 1) > pi(t)

⇔

pi(t)1 + β + fi(t) 1 + β + ¯ f (t) > pi(t)

⇔

1 + β + fi(t) > 1 + β + ¯ f (t), pi(t) > 0

⇔

fi(t) > ¯ f (t), pi(t) > 0

Properties of the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 32

• Claim. Species i grows if and only if pi(t) > 0 and it has above

average fitness. Proof. pi(t + 1) > pi(t)

⇔

pi(t)1 + β + fi(t) 1 + β + ¯ f (t) > pi(t)

⇔

1 + β + fi(t) > 1 + β + ¯ f (t), pi(t) > 0

⇔

fi(t) > ¯ f (t), pi(t) > 0

• Question. What if β is large?

Properties of the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 32

• Claim. Species i grows if and only if pi(t) > 0 and it has above

average fitness. Proof. pi(t + 1) > pi(t)

⇔

pi(t)1 + β + fi(t) 1 + β + ¯ f (t) > pi(t)

⇔

1 + β + fi(t) > 1 + β + ¯ f (t), pi(t) > 0

⇔

fi(t) > ¯ f (t), pi(t) > 0

• Question. What if β is large? Answer. If β is large then the differences in

growth among species is smaller, and the dynamics is slower (“bluer”).

The continuous replicator equation

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 33

Derivation of the CRE from the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 34

Discrete step equation: qi(t + 1) = qi(t)[1 + β + fi(t)]

Derivation of the CRE from the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 34

Discrete step equation: qi(t + 1) = qi(t)[1 + β + fi(t)]

■ Idea: reduce time steps t = 1 to smaller time steps t = δ, where

0 ≤ δ ≤ 1.

Derivation of the CRE from the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 34

Discrete step equation: qi(t + 1) = qi(t)[1 + β + fi(t)]

■ Idea: reduce time steps t = 1 to smaller time steps t = δ, where

0 ≤ δ ≤ 1.

■ The idea is the following:

Per small step t = δ the largest part 1 − δ of species i remains unchanged, while a smaller part δ of species i does change: qi(t + δ) = (1 − δ) qi(t)

• remains

+δ qi(t)(1 + β + fi(t))

• changes

.

Derivation of the CRE from the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 34

Discrete step equation: qi(t + 1) = qi(t)[1 + β + fi(t)]

■ Idea: reduce time steps t = 1 to smaller time steps t = δ, where

0 ≤ δ ≤ 1.

■ The idea is the following:

Per small step t = δ the largest part 1 − δ of species i remains unchanged, while a smaller part δ of species i does change: qi(t + δ) = (1 − δ) qi(t)

• remains

+δ qi(t)(1 + β + fi(t))

• changes

.

■ What if δ = 0?

Derivation of the CRE from the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 34

Discrete step equation: qi(t + 1) = qi(t)[1 + β + fi(t)]

■ Idea: reduce time steps t = 1 to smaller time steps t = δ, where

0 ≤ δ ≤ 1.

■ The idea is the following:

Per small step t = δ the largest part 1 − δ of species i remains unchanged, while a smaller part δ of species i does change: qi(t + δ) = (1 − δ) qi(t)

• remains

+δ qi(t)(1 + β + fi(t))

• changes

.

■ What if δ = 0? What if δ = 1?

Derivation of the CRE from the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 35

We have: qi(t + δ) − qi(t) δ

= (1 − δ)qi(t) + δqi(t)(1 + β + fi(t)) − qi(t)

δ

= . . . = qi(β + fi(t)).

So: ˙ qi = dqi(t) dt

= lim

δ→0

qi(t + δ) − qi(t) δ

= lim

δ→0 qi(β + fi(t))

= qi(β + fi(t)).

Derivation of the CRE from the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 36

pi(t + δ) = qi(t + δ) ∑n

j=1 qj(t + δ)

= (1 − δ)qi(t) + δqi(t)(1 + β + fi(t))

∑n

j=1

(1 − δ)qj(t) + δqj(t)(1 + β + fj(t))

• (/q(t))

= (1 − δ)pi(t) + δpi(t)(1 + β + fi(t))

∑n

j=1

(1 − δ)pj(t) + δpj(t)(1 + β + fj(t))

• (yields proportions)

=

pi(t)[1 + δ(β + fi(t))] ∑n

j=1 pj(t)[1 + δ(β + fj(t))]

(∑ pi(t) = 1)

= pi(t)1 + δ ((β + fi(t))

1 + δ

• β + ¯

f (t) .

Derivation of the CRE from the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 36

pi(t + δ) = qi(t + δ) ∑n

j=1 qj(t + δ)

= (1 − δ)qi(t) + δqi(t)(1 + β + fi(t))

∑n

j=1

(1 − δ)qj(t) + δqj(t)(1 + β + fj(t))

• (/q(t))

= (1 − δ)pi(t) + δpi(t)(1 + β + fi(t))

∑n

j=1

(1 − δ)pj(t) + δpj(t)(1 + β + fj(t))

• (yields proportions)

=

pi(t)[1 + δ(β + fi(t))] ∑n

j=1 pj(t)[1 + δ(β + fj(t))]

(∑ pi(t) = 1)

= pi(t)1 + δ ((β + fi(t))

1 + δ

• β + ¯

f (t) .

■ What if δ = 0?

Derivation of the CRE from the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 36

pi(t + δ) = qi(t + δ) ∑n

j=1 qj(t + δ)

= (1 − δ)qi(t) + δqi(t)(1 + β + fi(t))

∑n

j=1

(1 − δ)qj(t) + δqj(t)(1 + β + fj(t))

• (/q(t))

= (1 − δ)pi(t) + δpi(t)(1 + β + fi(t))

∑n

j=1

(1 − δ)pj(t) + δpj(t)(1 + β + fj(t))

• (yields proportions)

=

pi(t)[1 + δ(β + fi(t))] ∑n

j=1 pj(t)[1 + δ(β + fj(t))]

(∑ pi(t) = 1)

= pi(t)1 + δ ((β + fi(t))

1 + δ

• β + ¯

f (t) .

■ What if δ = 0? What if δ = 1?

Derivation of the CRE from the DRE

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 37

Now pi(t + δ) − pi(t) δ

=

pi(t) 1+δ((β+ fi(t))

1+δ(β+ ¯ f(t)) − pi(t)

δ

= pi(t)

1+δ((β+ fi(t)) 1+δ(β+ ¯ f(t)) − 1

δ multiply w. 1 + δ

• β + ¯

f (t)

• = pi(t)1 + δ ((β + fi(t)) −
• 1 + δ
• β + ¯

f (t)

• δ
• 1 + δ
• β + ¯

f (t)

• = pi(t)

fi(t) − ¯ f (t) 1 + δ(β + ¯ f (t)). So ˙ pi = lim

δ→0

pi(t + δ) − pi(t) δ

= lim

δ→0 pi(t)

fi(t) − ¯ f (t) 1 + δ(β + ¯ f (t)) = pi(t) fi(t) − ¯ f (t) 1 + 0 · C

= pi(t)[ fi(t) − ¯

f (t) ].

Calculating stationary points

• f the replicator

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 38

Finding stationary points of the replicator: example

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 39

Consider the replicator with A =   6 1 6 4 10 1 8 5 1   and p =   x y z   .

Finding stationary points of the replicator: example

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 39

Consider the replicator with A =   6 1 6 4 10 1 8 5 1   and p =   x y z   . Stationary points (fixed points, rest points):        p ∈ ∆2 x((Ap)x − p(Ap)) = 0 y((Ap)y − p(Ap)) = 0 z((Ap)z − p(Ap)) = 0.

Finding stationary points of the replicator: example

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 39

Consider the replicator with A =   6 1 6 4 10 1 8 5 1   and p =   x y z   . Stationary points (fixed points, rest points):        p ∈ ∆2 x((Ap)x − p(Ap)) = 0 y((Ap)y − p(Ap)) = 0 z((Ap)z − p(Ap)) = 0. This is equivalent with       

(x, y, z) ∈ ∆2

Finding stationary points of the replicator: example

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 39

Consider the replicator with A =   6 1 6 4 10 1 8 5 1   and p =   x y z   . Stationary points (fixed points, rest points):        p ∈ ∆2 x((Ap)x − p(Ap)) = 0 y((Ap)y − p(Ap)) = 0 z((Ap)z − p(Ap)) = 0. This is equivalent with       

(x, y, z) ∈ ∆2, i.e., x, y, z ∈ [0, 1] and x + y + z = 1

Finding stationary points of the replicator: example

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 39

Consider the replicator with A =   6 1 6 4 10 1 8 5 1   and p =   x y z   . Stationary points (fixed points, rest points):        p ∈ ∆2 x((Ap)x − p(Ap)) = 0 y((Ap)y − p(Ap)) = 0 z((Ap)z − p(Ap)) = 0. This is equivalent with       

(x, y, z) ∈ ∆2, i.e., x, y, z ∈ [0, 1] and x + y + z = 1

x = 0

Finding stationary points of the replicator: example

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 39

Consider the replicator with A =   6 1 6 4 10 1 8 5 1   and p =   x y z   . Stationary points (fixed points, rest points):        p ∈ ∆2 x((Ap)x − p(Ap)) = 0 y((Ap)y − p(Ap)) = 0 z((Ap)z − p(Ap)) = 0. This is equivalent with       

(x, y, z) ∈ ∆2, i.e., x, y, z ∈ [0, 1] and x + y + z = 1

x = 0 or 6x − 6x2 + y − 5xy − 10y2 + 6z − 14xz − 6yz − z2 = 0

Finding stationary points of the replicator: example

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 39

Consider the replicator with A =   6 1 6 4 10 1 8 5 1   and p =   x y z   . Stationary points (fixed points, rest points):        p ∈ ∆2 x((Ap)x − p(Ap)) = 0 y((Ap)y − p(Ap)) = 0 z((Ap)z − p(Ap)) = 0. This is equivalent with       

(x, y, z) ∈ ∆2, i.e., x, y, z ∈ [0, 1] and x + y + z = 1

x = 0 or 6x − 6x2 + y − 5xy − 10y2 + 6z − 14xz − 6yz − z2 = 0 y = 0

Finding stationary points of the replicator: example

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 39

Consider the replicator with A =   6 1 6 4 10 1 8 5 1   and p =   x y z   . Stationary points (fixed points, rest points):        p ∈ ∆2 x((Ap)x − p(Ap)) = 0 y((Ap)y − p(Ap)) = 0 z((Ap)z − p(Ap)) = 0. This is equivalent with       

(x, y, z) ∈ ∆2, i.e., x, y, z ∈ [0, 1] and x + y + z = 1

x = 0 or 6x − 6x2 + y − 5xy − 10y2 + 6z − 14xz − 6yz − z2 = 0 y = 0 or 4x − 6x2 + 10y − 5xy − 10y2 + z − 14xz − 6yz − z2 = 0

Finding stationary points of the replicator: example

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 39

Consider the replicator with A =   6 1 6 4 10 1 8 5 1   and p =   x y z   . Stationary points (fixed points, rest points):        p ∈ ∆2 x((Ap)x − p(Ap)) = 0 y((Ap)y − p(Ap)) = 0 z((Ap)z − p(Ap)) = 0. This is equivalent with       

(x, y, z) ∈ ∆2, i.e., x, y, z ∈ [0, 1] and x + y + z = 1

x = 0 or 6x − 6x2 + y − 5xy − 10y2 + 6z − 14xz − 6yz − z2 = 0 y = 0 or 4x − 6x2 + 10y − 5xy − 10y2 + z − 14xz − 6yz − z2 = 0 z = 0

Finding stationary points of the replicator: example

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 39

Consider the replicator with A =   6 1 6 4 10 1 8 5 1   and p =   x y z   . Stationary points (fixed points, rest points):        p ∈ ∆2 x((Ap)x − p(Ap)) = 0 y((Ap)y − p(Ap)) = 0 z((Ap)z − p(Ap)) = 0. This is equivalent with       

(x, y, z) ∈ ∆2, i.e., x, y, z ∈ [0, 1] and x + y + z = 1

x = 0 or 6x − 6x2 + y − 5xy − 10y2 + 6z − 14xz − 6yz − z2 = 0 y = 0 or 4x − 6x2 + 10y − 5xy − 10y2 + z − 14xz − 6yz − z2 = 0 z = 0 or 8x − 6x2 + 5y − 5xy − 10y2 + z − 14xz − 6yz − z2 = 0

Finding stationary points of the replicator: example

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 39

Consider the replicator with A =   6 1 6 4 10 1 8 5 1   and p =   x y z   . Stationary points (fixed points, rest points):        p ∈ ∆2 x((Ap)x − p(Ap)) = 0 y((Ap)y − p(Ap)) = 0 z((Ap)z − p(Ap)) = 0. This is equivalent with       

(x, y, z) ∈ ∆2, i.e., x, y, z ∈ [0, 1] and x + y + z = 1

x = 0 or 6x − 6x2 + y − 5xy − 10y2 + 6z − 14xz − 6yz − z2 = 0 y = 0 or 4x − 6x2 + 10y − 5xy − 10y2 + z − 14xz − 6yz − z2 = 0 z = 0 or 8x − 6x2 + 5y − 5xy − 10y2 + z − 14xz − 6yz − z2 = 0 Solve with Maple / Mathematica / SciPy / . . .

Finding stationary points of the replicator: example

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 39

Consider the replicator with A =   6 1 6 4 10 1 8 5 1   and p =   x y z   . Stationary points (fixed points, rest points):        p ∈ ∆2 x((Ap)x − p(Ap)) = 0 y((Ap)y − p(Ap)) = 0 z((Ap)z − p(Ap)) = 0. This is equivalent with       

(x, y, z) ∈ ∆2, i.e., x, y, z ∈ [0, 1] and x + y + z = 1

x = 0 or 6x − 6x2 + y − 5xy − 10y2 + 6z − 14xz − 6yz − z2 = 0 y = 0 or 4x − 6x2 + 10y − 5xy − 10y2 + z − 14xz − 6yz − z2 = 0 z = 0 or 8x − 6x2 + 5y − 5xy − 10y2 + z − 14xz − 6yz − z2 = 0 Solve with Maple / Mathematica / SciPy / . . . (Nash equilibria are blue):

• (1, 0, 0), (0, 1, 0) , (0, 0, 1),

25 71, 20 71, 26 71

• ,

5 7, 0, 2 7

• ,

9 11, 2 11, 0

• .

Finding stationary points of the replicator: example

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 40

Phase space of the replicator as discussed. Circled rest points indicate Nash equilibria in the corresponding symmetric game.

Summary

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 41

Implications

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 42

SN ESS NSS GSS ASS LSS LR NE FP * i * * SN = strict Nash, ESS - evol’y stable strategy, GSS = glob’y stable state, ASS = asymp’y stable state, NSS = neutrally stable strategy, LR = limit of replicator, LSS = Lyapunov stable state, FP = fixed point, * = only if fully mixed, i = isolated NE. Dotted: indirect implication. Blue: game theory; olive: evolutionary game theory; green: the replicator dynamic.

Justifications of the implications

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 43

■ SN ⇒ ESS: cf. slides evolutionary games and, e.g., Th 7.7.12 of Sh&LB. ■ ESS ⇒ NSS: cf. slides evolutionary games and, e.g., Game Theory Evolving (2nd ed.) by H. Gintis. ■ ESS ⇒ NE: cf. slides evolutionary games and, e.g., Sh&LB Th 7.7.11. ■ ESS ⇒∗ GSS: cf., e.g., Th. 12.7 Gintis. ■ ESS ⇒ ASS: cf., e.g., Th. 7.7.13 Sh&LB,

• Th. 12.7 Gintis, Sec. 3.5 (begin) of
• Evol. Game Theory by J.G. Weibull.

■ NSS ⇒ LSS: cf. Sec. 3.5 Weibull. ■ GSS ⇒ ASS: by definition of the two concepts. ■ ASS ⇒ LSS: by definition of the two concepts. ■ ASS ⇒ LR: by definition of the two concepts. ■ ASS ⇒i NE: Th 7.7.8 Sh&LB,

• Th. 12.6 Gintis.

■ LSS ⇒ NE: Th 7.7.6 Sh&LB, 7.2.1(c) Hofbauer & Sigmund. ■ LR ⇒∗ NE: Th. 7.2.1(b) H&S. ■ NE ⇒ FP: Th. 7.2.1(a) H&S, Th 7.7.5 Sh&LB, Th. 12.6 Gintis. ■ LR ⇒ FP: Ch. 6 Weibull.

Sources referred to

Author: Gerard Vreeswijk. Slides last modified on June 10th, 2020 at 14:01 Multi-agent learning: The replicator dynamic, slide 44

Weibull, J. W. (1997). Evolutionary game theory. MIT press. Hofbauer, J., & Sigmund, K. (1998). Evolutionary games and population dynamics. Cambridge university press. Gintis, H. (2001). Game theory evolving: A problem-centered introduction to modeling strategic interaction (2nd ed.). Princeton: Princeton University Press. Shoham, Y., & Leyton-Brown, K. (2008). Multiagent systems: Algorithmic, game-theoretic, and logical foundations. Cambridge University Press. Vreeswijk, G.A.W. (2011). Evolutionary game theory. (Slides)