CM30174 - CM50206 Introduction to Intelligent Agents Semester 1, - - PDF document

What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information

CM30174 - CM50206 Introduction to Intelligent Agents Semester 1, 2010-11

Marina De Vos, Julian Padget

Game Theory / 20101025 / version 0.6

October 25, 2010

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 1 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information

Authors/Credits for this lecture

Primary author: Marina De Vos. Material sourced from Martin J. Osborne and Ariel Rubinstein, A course in game theory (MIT Press, 1994) [1].

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 2 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information

Game Theory

Game Theory is a bag of analytical tools designed to help us understand the phenomena that we observe when decision-makers interact Two basic assumptions:

rationality strategy

Application Domains: multi-agent systems, political competitions, theoretical computer science, distributions of tongue length in bees and tube length in flowers. Game theory uses mathematics to express its ideas formally First publications in this area are from Von Neumann and Morgenstern in 1944

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 4 / 45

What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition The Prisoner’s Dilemma Nash Equilibria Best Response Function Dominated Strategies

Strategic Games (I)

A strategic game models a situation where several agents (called players) independently make a decision about which action to take, out of a limited set of possibilities The result of the actions is determined by the combined effect of the choices separately made by each player. A collection of actions, one for each player, is called a profile. A profile determines the outcome of the game. Each agent has a preference relation, , over the set of

• utcomes.

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 6 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition The Prisoner’s Dilemma Nash Equilibria Best Response Function Dominated Strategies

Strategic Games (II)

Definition A strategic game is a tuple N, (Ai)i∈N, (i)i∈N where N is a finite set of players; for each player i ∈ N, Ai is a nonempty set of actions that are available to her; and for each player i ∈ N, i is a preference relation on A = ×j∈NAj . An element a ∈ A is called a profile. For a profile a we use ai to denote the action of agents i, while a−i denotes the actions taken by the other agents.

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 7 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition The Prisoner’s Dilemma Nash Equilibria Best Response Function Dominated Strategies

An Example

Assume we just have two agents N = {1, 2} with each two actions Ai = {C, D}. Then we have four outcomes or profiles A = {CC, DD, CD, DC} CC ≻1 DD ≻1 CD 1 DC DD ≻2 CC ≻2 CD 2 DC

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 8 / 45

What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition The Prisoner’s Dilemma Nash Equilibria Best Response Function Dominated Strategies

Payoff/Utility and Preference

The preference relation is often too abstract and therefore replaced by a payoff or utility function ui for each agent. ui : A → R such that ui(a) ≥ ui(b) iff a i b, for each a, b ∈ A. For two players this gives a nice representation, called the payoff matrix: C D C 3,1 0,0 D 0,0 1,3 s

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 9 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition The Prisoner’s Dilemma Nash Equilibria Best Response Function Dominated Strategies

The Prisoner’s Dilemma

Two men are collectively charged with a crime and held in separate cells, with no way of meeting or communicating. They are told that: if one confesses and the other does not, the confessor will be freed, and the other will be jailed for three years. if both confess, then each will be jailed for two years Both prisoners know that if neither confesses, then they will each be jailed for one year.

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 10 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition The Prisoner’s Dilemma Nash Equilibria Best Response Function Dominated Strategies

The Prisoner’s Dilemma (II)

Payoff matrix for this game could be: Silent Confess Silent 2,2 0,3 Confess 3,0 1,1 Top Left: If both remain silent, the both are rewarded for mutual cooperation. Top Right: If the first cooperates by remaining silent and the second confesses, the first gets the sucker’s payoff of zero, while the second gets the maximum payoff. Bottom Left: Reverse of Top Right. Bottom Right: If both confess, they both get punished for mutual defection.

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 11 / 45

What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition The Prisoner’s Dilemma Nash Equilibria Best Response Function Dominated Strategies

Nash Equilibrium

Playing a game N, (Ai)i∈N, (i)i∈N consists of each player i ∈ N selecting a single action from the set of actions Ai available to her. Since players are thought to be rational, it is assumed that a player will select an action that leads to a “preferred” profile. Definition A Nash equilibrium of a strategic game N, (Ai)i∈N, (i)i∈N is a profile a∗ satisfying ∀ai ∈ Ai · (a∗

−i, a∗ i ) ≥i (a∗ −i, ai) .

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 12 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition The Prisoner’s Dilemma Nash Equilibria Best Response Function Dominated Strategies

Find the Nash Equilibria (1)

C D A 4,4 1,1 B 3,4 2,4 C D A 1,0 2,1 B 2,1 3,1

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 13 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition The Prisoner’s Dilemma Nash Equilibria Best Response Function Dominated Strategies

Find the Nash Equilibria (2)

C D A 3,3 2,4 B 1,2 4,1 A B A

• 1,-1

1,2 B 2,1

• 1,-1

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 14 / 45

What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition The Prisoner’s Dilemma Nash Equilibria Best Response Function Dominated Strategies

Find the Nash Equilibria (3)

H T H 1,-1

• 1,1

T

• 1,1

1,-1 A B C A 1,3 2,4 1,3 B 1,2 2,1 4,2 C 3,1 1,2 1,4

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 15 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition The Prisoner’s Dilemma Nash Equilibria Best Response Function Dominated Strategies

Nash Equilibria: Problems

Rational agents have no incentive to deviate from a Nash equilibrium. Unfortunately:

Not every interaction scenario has a Nash equilibrium Some interaction scenarios have more than one Nash equilibrium, possibly resulting in an outcome which is not a Nash equilibrium while every agent plays an action belonging to a Nash equilibrium.

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 16 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition The Prisoner’s Dilemma Nash Equilibria Best Response Function Dominated Strategies

Best Response Function

Definition Let G = N, (Ai)i∈N, (i)i∈N be a strategic game. For any a−i ∈ A−1, we define the best response function of player i as the set-valued function Bi(a−i) such that: Bi(a−i) = {ai ∈ Ai | (a−i, ai) i (a−i, a′

i), ∀a′ i ∈ Ai} .

Definition A Nash equilibrium of a strategic game N, (Ai)i∈N, (i)i∈N is a profile a∗ satisfying: a∗

i ∈ Bi(a∗ −i), ∀i ∈ N

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 17 / 45

What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition The Prisoner’s Dilemma Nash Equilibria Best Response Function Dominated Strategies

Strictly Dominated Strategies (I)

Given a particular strategy ai for a player i, there will be a number of possible outcomes. A rational agent will never opt for an action if this action is not part of the best response of an action profile generated by to other agents. Definition An action of player i in a strategic game is strictly dominated if it is not a best response to any belief a−i ∈ A−i of player i. In

• ther words,

∀a−i ∈ A−i · ai / ∈ B(a−i) Strictly dominated action can never be part of a Nash equilibrium.

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 18 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition The Prisoner’s Dilemma Nash Equilibria Best Response Function Dominated Strategies

Strictly Dominated Strategies (II)

Since strictly dominated strategies are not part of any Nash equilibrium they can safely be removed. An example C D A 1,0 2,1 B 2,1 3,0 C D B 2,1 3,0 C B 2,1

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 19 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition The Prisoner’s Dilemma Nash Equilibria Best Response Function Dominated Strategies

Strictly Dominated Strategies (III)

S C S 2,2 0,3 C 3,0 1,1 C D A 2,1 0,0 B 0,0 1,2 D E F A 2,3 3,5 2,2 B 3,2 5,3 4,5 C 4,3 4,4 3,4

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 20 / 45

What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information The Prisoner’s Dilemma Revisited Horizons

The Prisoner’s Dilemma Revisited (I)

Silent Confess Silent 2,2 0,3 Confess 3,0 1,1 The individual action is Confess. This guarantees a payoff of no worse than 2, whereas remaining silent guarantees you no payoff at all. So confessing is the best response to all possible strategies. But intuition says this is not the best outcome: surely they should both remain silent and each will get a payoff of 2!

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 22 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information The Prisoner’s Dilemma Revisited Horizons

The Prisoner’s Dilemma Revisited (II)

This apparent paradox is the fundamental problem of multi-agent interactions. It appears to imply that cooperation will not occur in societies of self-interested agents Real world examples:

nuclear arms reduction free rider systems television licenses

The prisoner’s dilemma is ubiquitous. Can we recover cooperation?

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 23 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information The Prisoner’s Dilemma Revisited Horizons

Arguments for Recovering Cooperation

Conclusions that some have drawn from this analysis:

The game theory notion of rational agents is wrong somehow the dilemma is being formulated wrongly

Arguments to recover cooperation:

We are not all Machiavelli The other prisoner is my twin The shadow of the future ... Having a notion of trust ...

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 24 / 45

What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information The Prisoner’s Dilemma Revisited Horizons

The Iterated Prisoner’s Dilemma

One answer: play the game more than once If you know you will be meeting your opponent again, then the incentive to defect appears to evaporate. Two types of iteration:

Infinite Horizon: The game continues without the players knowing when the iteration will halt Finite Horizon: The players know from the beginning how many iterations will take place.

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 25 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information The Prisoner’s Dilemma Revisited Horizons

Infinite Horizon

An agent confessing in one round can be punished for this lack of cooperation in the next round. Since agents are trying to maintain the highest possibly payoff they will avoid this scenario. Cooperation - remaining silent - is the rational choice in the infinitely repeated prisoner’s dilemma.

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 26 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information The Prisoner’s Dilemma Revisited Horizons

Finite Horizon

But .. . suppose you both know that you will play the exactly n times. On round n − 1, you have an incentive to confess, to gain that extra bit of payoff. There is no chance for your

• pponent to punish you for it.

But this makes round n − 2 the last “real” round, and so you have an incentive to confess there, too This is the backward induction problem. Playing the prisoner’s dilemma with a fixed, finite, pre-determined, commonly known number of rounds, confessing is the best strategy.

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 27 / 45

What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition Nash Equilibria Subgame Perfect equilibria

Extensive Games with Perfect Information (I)

Strategic games model single interaction encounters between agents. Decisions are made or action decided upon before the actual start of the game Extensive games provide an explicit description of the sequential structure of the underlying decision problem encountered by the agents in a strategic situation. An agent can reconsider her plan of action not only at the beginning at the game but also at any point of time when she needs to make a decision. At each point in time an agents is fully aware of what happened before.

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 29 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition Nash Equilibria Subgame Perfect equilibria

Extensive Games with Perfect Information (II)

Definition An extensive game with perfect information is a tuple N, H, P, (i)i∈N with the following components: A set N of players. A prefix-closed set H of finite sequences. Each element of H is called a history; each component of a history is an action chosen by a player. A history h is terminal if ∃ak+1 · (h, ak+1) ∈ H. We use Z to denote this subset. A function P that assigns to each nonterminal history from H \ Z a member of N. For each player i ∈ N a preference relation i on Z.

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 30 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition Nash Equilibria Subgame Perfect equilibria

An Example (I)

An extensive game N, H, P, (i)i∈N for object division: N = {1, 2} ; H consists of ten histories

ǫ, (2, 0), (1, 1), (0, 2), (2, 0) y1, (2, 0) n1, (1, 1) y2, (1, 1) n2, (0, 2) y3, (0, 2) n3 ;

P(ǫ) = 1 and P(h) = 2 for the non-terminal histories (2, 0), (1, 1), (0, 2) ; (2, 0) y1 >1 (1, 1) y2 >1 (0, 2) y3 =1 (2, 0) n1 =1 (1, 1) n2 =1 (0, 2) n2 and (0, 2) y3 >2 (1, 1) y2 >2 (2, 0) y1 =2 (2, 0) n1 =2 (1, 1) n2 =2 (0, 2) n3 .

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 31 / 45

What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition Nash Equilibria Subgame Perfect equilibria

An Example (II)

❜ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ PPPPPPPP P 1 (2, 0) (1, 1) (0, 2) r

• ❅

❅ ❅ 2 y1 n1 r 2, 0 r 0, 0 r

• ❅

❅ ❅ 2 y2 n2 r 1, 1 r 0, 0 r

• ❅

❅ ❅ 2 y3 n3 r 0, 2 r 0, 0

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 32 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition Nash Equilibria Subgame Perfect equilibria

Profiles

Let N, H, P, (i)i∈N be a finite extensive game with perfect information. A strategy for a player i ∈ N is a function that assigns an action of A(h) to each non-terminal history h ∈ (H \ Z) for which P(h) = i . A strategy profile s is a set containing a strategy for each player i ∈ N, i.e. s = (si)i∈N . The outcome O(s) for a strategy profile s is defined as the terminal history which is reached when each player i ∈ N follows the precepts of si. That is, O(s) is the history (a1, . . . , ak) ∈ Z such that, for 0 ≤ l < k, sP((a1,...,al))((a1, . . . , al)) = al+1 .

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 33 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition Nash Equilibria Subgame Perfect equilibria

An Example

Example Reconsider the Object game. Player 1 has to choose an initial action from (2, 0), (1, 1) and (0, 2). So we can identify each of her strategies with one of these possible actions. Player 2 chooses an action after each of the histories (2, 0), (1, 1) and (0, 2). In each case she has two possible actions: yi or ni with i ∈ {1, 2, 3}. Thus, her strategy will be a triple {a1, b2, c3}, with a, b and c equal to either y or n. The interpretation of player 2’s strategy {a1, b2, c3} is that it is a contingency plan: if player 1 chooses (2, 0) then player 2 will play a1; if player 1 opts for (1, 1) then player 2 goes for b2; and when player 1 decides for (0, 2) then player 2 takes c3.

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 34 / 45

What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition Nash Equilibria Subgame Perfect equilibria

Nash Equilibria (I)

Definition A Nash Equilibrium of an extensive game with perfect information N, H, P, (i)i∈N is a strategy profile s∗ such that for every player i ∈ N we havea O((s∗

−i, s∗ i )) i O((s∗ −i, si)) for every strategy si of player i .

a(s−i, si) is the abbreviation for the strategy profile s′ which is such that

si = s′

i and sj = s′ j for all j ∈ N and j = i. De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 35 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition Nash Equilibria Subgame Perfect equilibria

Nash Equilibria (II)

The extensive game with perfect information of objects has nine Nash equilibria: {{(2, 0)}, {y1, y2, y3, }}, {{(2, 0)}, {y1, y2, n3}}, {{(2, 0)}, {y1, n2, y3}}, {{(2, 0)}, {y1, n2, n3}}, {{(1, 1)}, {n1, y2, y3}}, {{(1, 1)}, {n1, y2, n3}}, {{(0, 2)}, {n1, n2, y3}}, {{(2, 0)}, {n1, n2, y3}}, {{(2, 0)}, {n1, n2, n3}} .

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 36 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition Nash Equilibria Subgame Perfect equilibria

Finding Nash Equilibria

Transform the extensive game into a strategic game. The rows and colums are the strategies of each player The cells are filled with the payoffs the players would receive if the strategies were followed. The Nash equilibria of your extensive games are the Nash equilibria of your strategic game.

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 37 / 45

What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition Nash Equilibria Subgame Perfect equilibria

An Other Example

❜ ✟ ✟ ✟ ✟ ✟ ❍❍❍❍ ❍

1 buy bake

r r

(0, 0)

r ✟ ✟ ✟ ✟ ✟ ❍❍❍❍ ❍

2 strawberries cherries

r r

(2, 1)

r ✟ ✟ ✟ ✟ ❍❍❍❍

1 cream no cream

r

(3, 3)

r

(1, 1) The Cake game has two Nash equilibria: {{buy, cream}, {strawberries}} and {{buy, no cream}, {cherries}} .

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 38 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition Nash Equilibria Subgame Perfect equilibria

Motivation For Subgame Perfect Equilibria

❜ ✟ ✟ ✟ ✟ ✟ ❍❍❍❍ ❍

Parent no pet pet

r

2, 0

r r ✟ ✟ ✟ ✟ ✟ ❍❍❍❍ ❍

Child cat spider

r

2, 2

r

0, 1 This game has three Nash equilibria: {{pet}, {cat} and {{no pet}, {cat} and {{no pet}, {spider}

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 39 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition Nash Equilibria Subgame Perfect equilibria

Subgames

Definition A subgame of the extensive game with Γ = N, H, P, (i)i∈N that follows the history h is the extensive game Γ(h) = N, H|h, P|h, (i |h)i, where H|h is the set of sequences h′ of actions for which (h, h′) ∈ H, P|h is defined by P|h(h′) = P((h, h′)) for each h′ ∈ H|h, and i |h is defined by h′ i |hh′′ iff (h, h′) i (h, h′′) .

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 40 / 45

What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition Nash Equilibria Subgame Perfect equilibria

Subgame Perfect Equilibrium

Definition A subgame perfect equilibrium of an extensive game with perfect information Γ = N, H, P, (i)i∈N is a strategy profile s∗ such that for every player i ∈ N and every non-terminal history h ∈ H \ Z for which P(h) = i we have: Oh(s∗

−i|h, s∗ i |h) i |hOh((s∗ −i|h, si|h)

for every strategy si of player i in the subgame Γ(h) .

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 41 / 45 What is Game Theory Strategic Games Repeated Games Extensive Games with Perfect Information Definition Nash Equilibria Subgame Perfect equilibria

An Example

Example Reconsider the Spider game. When we look at the subgame perfect equilibria of this game, we see that {{pet}, {cat} and {{no pet}, {cat} are the only ones. The unintuitive Nash equilibrium {{no pet}, {spider} is no longer accepted.

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 42 / 45 Summary Overview Additional Reading

Summary

Strategic games

Nash Equilibria Dominated Strategies

Repeated Games

Finite and Infinite Horizon Tournament

Extensive Games

Nash Equilibria Subgame Perfect Equilibria

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 44 / 45

Summary Overview Additional Reading

Directed and Additional Reading

Martin J. Osborne and Ariel Rubinstein, A course in game theory (MIT Press, 1994)[1]

• M. J. Osborne and A. Rubinstein.

A Course in Game Theory. The MIT Press, Cambridge, Massachusets, London, Engeland, third edition, 1996.

De Vos/Padget (Bath/CS) CM30174/Game Theory October 25, 2010 45 / 45