Emergent systems Spring-13 The Prisoners Dilemma - - PDF document

1

Umeå University Department of Computing Science

Emergent systems

Spring-13 The Prisoners´ Dilemma

http://www.cs.umu.se/kurser/5DV017

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Last time

❒ Evolutionary computation

❍ Overview

❒ Genetic programming ❒ Aspects of evolution ❒ Classifier systems ❒ General on cooperation ❒ Game theory

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Today

❒ Prisoners’ Dilemma and other dilemmas

❍ Iterated Prisoners’ Dilemma ❍ Ecological models ❍ Spatial models

❒ Short conclusion of the course

2

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

The Prisoners’ Dilemma

❒ Melvin Dresher and Merrill Flood, RAND

Corporation, 1950

❒ Further developed by Albert W. Tucker ❒ Used in philosophy, ethics, biology,

sociology, political science, economics, game theory, computer science, mathematics, ...

❒ ”The E. Coli of social psychology” - Axelrod

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

The Prisoners’ Dilemma

❒ The story

❍ Two criminals were caught, they have

committed a crime but the police have insufficient evidence

❍ They can not communicate with each other ❍ If both plead guilty, each get 10 years ❍ If one plead guilty and accuses the other

• The one that plead guilty goes free
• The accused get 20 years

❍ If no one plead guilty, each get 1 year

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

The Prisoners’ Dilemma

❒ Payoff matrix ❒ Alice rational analysis (dominant strategy)

❍ If Bob cooperate – Defect ❍ If Bob defect – Defect ❍ (Same analysis for Bob)

❒ Dominant strategy equilibrium 10 year each ❒ Make an irrational decision (coop) 1 year each

Bob Cooperate Defect Alice Cooperate

• 1, -1
• 20, 0

Defect 0, -20

• 10, -10

3

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

The Prisoners’ Dilemma

❒ Summary

❍ Individual rationality is not optimal ❍ An example of a dilemma in game theory

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Dilemmas in game theory

❒ ”A situation requiring a choice between

alternatives that are equivalent”

❒ ”Damned if you do, damned if you don’t” ❒ In game theory: Each player acts

rationally, but the result is not desirable

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Dilemmas in game theory

❒ General payoff matrix Bob Cooperate Defect Alice Cooperate CC (R)

Reward

CD (S)

Sucker’s payoff

Defect DC (T)

Temptation to defect

DD (P)

Punishment

4

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Dilemmas

❒ General terms of a dilemma

❍ You will always win if the other cooperates

• CC > CD and DC > DD

❍ Sometimes you win by defecting

• DC > CC or DD > CD

❍ Mutual cooperation is preferable

• CC > DD

❒ 24 permutations but only 3 are dilemmas

Bob Coop Def Alice Coop CC CD Def DC DD

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Dilemmas

❒ Prisoners’ Dilemma

❍ DC > CC > DD > CD ❍ Better to defect no matter what the other do ❍ Nash equilibrium is DD

❒ Chicken

❍ DC > CC > CD > DD ❍ Mutual defect is worst ❍ Two Nash equilibrium, DC and CD

❒ Stag Hunt

❍ CC > DC > DD > CD ❍ Best to cooperate with cooperators ❍ Two Nash equilibrium, CC and DD Bob Coop Def Alice Coop CC CD Def DC DD Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Iterated Prisoners’ Dilemma

❒ Assumptions

❍ No agreements or threats ❍ A player's next move can not be predicted ❍ No way to eliminate players or avoid interaction ❍ No way of changing the payoff ❍ Communication only via direct interaction

5

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Iterated Prisoners’ Dilemma

❒ Axelrod’s experiment (1980)

❍ Intuitive assumption that future interactions

may affect the rationality of the decision

❍ Round-robin tournament for strategies

• All are competing against everyone, including itself
• 200 iterations of PD – Should be unknown!
• Each program/strategy can remember earlier actions
• 14 program were received
• CC = 3, CD = 0, DC = 5, DD = 1
• (Requirement : DC + CD < 2 * CC)

Bob Coop Def Alice Coop 3, 3 0, 5 Def 5, 0 1, 1

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Iterated Prisoners’ Dilemma

❒ Expected payoff for three simple strategies

ALL-C RAND ALL-D Medel ALL-C 3.0 1.5 0.0 1.5 RAND 4.0 2.0 0.5 2.167 ALL-D 5.0 3.0 1.0 3.0

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Iterated Prisoners’ Dilemma

❒ Result

❍ Winner: Rapoport’s Tit-For-Tat (TFT) ❍ Cooperate at first interaction ❍ Then, do what your opponent did the previous move

❒ Second experiment

❍ 62 program ❍ Everyone knew that TFT won last time ❍ TFT won again

❒ Tit-for-Two-Tats

❍ More forgiving than TFT ❍ TFT is better in a noisy environment ❍ Would have won the first experiment, but performed

poorly in the second

6

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Iterated Prisoners’ Dilemma

❒ Characteristics for a successful strategy

❍ Don’t be envious ❍ Be nice ❍ Reciprocate ❍ Don’t be too clever ❍ Be a generalist ❍ Agree with yourself ❍ Be an evolutionarily stable strategy

• John Maynard Smith
• Resistant against invasion by other strategies

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Ecological models

❒ What happens if we allow more successful

strategies to spread in the population at the expense of less successful?

❒ Success is measured as a percentage of

population

❒ Percentage of population = The probability

that a random program uses this strategy

❒ On the board ❒ Simulation

❍ Fig 17.3a

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Pavlov – new strategy

❒ Pavlov (PAV)

❍ ”Win-Stay, Lose-Shift” ❍ Start with cooperation ❍ If the other cooperate, continue with the

current behavior

❍ If the other defect, switch behavior ❍ Fig 17.3b

7

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Noise

❒ An action is misinterpreted for some

reason

❒ TFT

❍ Leads to a cycle of alternating CD and DC ❍ Broken only by a further error

❒ PAV

❍ Can self-correct itself : DC DD CC ❍ Can exploit ALL-C in an environment where

errors can occur

❍ Fig 17.3c

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

IPD and GA

❒ Let the population evolve using genetic

algorithms

❒ Simulation facts :

❍ N = 100 ❍ 50 bouts/individual/generation with each bout

consisting of 20 PD rounds

❍ Random strategies initially ❍ Coding: Strings with five letters, each C or D

First CC CD DC DD

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

IPD and GA (cont.)

❒ Result

8

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Stimulating cooperation in PD

❒ Know something about the other

❍ Iterative ❍ Rumor

❒ Spatial

❍ Interact with neighbors ❍ Example

• MANET
• Society

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

2 x 2 game

❒ 2 players, 2 strategies ❒ Payoff matrix ❒ 12 permutations of R, S, T, P

❍ Fig 1 in the article by Hauert

Coop Def Coop R = 1 S Def T P = 0

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

2 x 2 game

❒

Well mixed populations

❒

Fig 2 in the article by Hauert

a)

Defection: S < 0, T > 1

• Stable fix point = 0 (proportion of cooperators)
• The area for Prisoners’ Dilemma

b) Co-existence: S > 0, T > 1

• Stable fix point = S / (S + T – 1)

c)

Bi-stability: S < 0, T < 1

• Instable fix point = S / (S + T – 1)
• Stable fix points = 0, 1

d) Cooperation: S > 0, T < 1

• Stable fix point = 1

9

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Spatial Prisoners’ Dilemma

❒ The overall payoff determines an

individual's success and is determined by interaction with neighbors

❒ Each individual has a place in a grid ❒ In each generation N individuals are

selected

❒ N = number of locations in the grid ❒ Every individual play against its neighbors

and then each individual get the

• pportunity to update its strategy

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Spatial PD - Variations

❒ Updating of the grid

❍ Synchronous ❍ Asynchronous ❍ Delayed asynchronous

❒ Updating of individuals

❍ Best takes over (deterministic) ❍ Proportional updating

❒ Number of neighbors

❍ von Neumann – 4 ❍ Moore – 8

❒ Initial frequency of cooperators

❍ 0.2, 0.5 or 0.8

❒ Simulation

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Spatial PD - Discussion

❒ Hauert (2001)

❍ Spatial extensions has an effect ❍ Differences in initial frequency of cooperators

will not affect

❍ Increased stochasticity in updating the grid has

surprisingly little impact

❍ Increased stochasticity in updating of

individuals decrease cooperation

10

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Spatial PD – Discussion (cont.)

❒ Nowak et al (1994)

❍ Result (initial frequency of Coop probably 0,5) ❍ With synchronous updating, proportional

updating decreases cooperation compared to deterministic

❍ With proportional updating, asynchronous

updating result in more cooperation than synchronous updating Synk Asynk Best 1,61 1,35 Prop 1,19 1,3

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Spatial PD – Discussion (cont.)

❒ My results

❍ If we ignore the proportional delayed

asynchronous update :

• Von Neumann and asynchronous proportional update

yields the same amount of cooperation as with no stochasticity

• Asynchronous proportional update is almost

insensitive to the initial fraction of cooperators

❍ Proportional delayed asynchronous update is

superior to all other variants

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

11

Emergent Systems, Jonny Pettersson, UmU

Short summary

❒ Four areas in the course, five in the book

❍ (Computation) ❍ Fractals ❍ Chaos ❍ Complex systems ❍ Adaptation

❒ Much (all?) are related

❍ How??

• Reflect!

❍ Try to see the big picture ❍ See systems from new perspectives

21/2 - 13 Emergent Systems, Jonny Pettersson, UmU

Short summary (cont.)

❒ Three themes in the book

❍ The whole is greater than the sum of the parts ❍ The interesting thing is in the middle / at the

border

❍ Science is doomed to uncertainty - but it's a

good thing

21/2 - 13 Emergent Systems, Jonny Pettersson, UmU

The whole is greater than the sum of the parts

21/2 - 13

12

Emergent Systems, Jonny Pettersson, UmU

The interesting thing is in the middle / at the border

❒ Table 24.1 and 24.2

21/2 - 13 Emergent Systems, Jonny Pettersson, UmU

Science is doomed to uncertainty - but it's a good thing

21/2 - 13 Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Summary

❒ Prisoners’ Dilemma and other dilemmas

❍ Iterated Prisoners’ Dilemma ❍ Ecological models ❍ Spatial models

❒ Short conclusion of the course

13

Emergent Systems, Jonny Pettersson, UmU 21/2 - 13

Next time

❒ Monday

❍ Project kick-off - Compulsory attendance

❒ Thursday, 9 - 13

❍ Written exam

❒ Thursday, Mars 21 (New date!)

❍ Lecture - Emergent interaction systems