SLIDE 1
Previously in Game Theory Previously in Game Theory decision makers: - - PowerPoint PPT Presentation
Previously in Game Theory Previously in Game Theory decision makers: - - PowerPoint PPT Presentation
Previously in Game Theory Previously in Game Theory decision makers: choices preferences Previously in Game Theory decision makers: choices preferences solution concepts: best response Nash equilibrium Rock,
SLIDE 2
SLIDE 3
Previously in Game Theory
◮ decision makers:
◮ choices ◮ preferences
SLIDE 4
Previously in Game Theory
◮ decision makers:
◮ choices ◮ preferences
◮ solution concepts:
◮ best response ◮ Nash equilibrium
SLIDE 5
Rock, paper, scissors
SLIDE 6
Rock, paper, scissors
R P S R 0, 0 −1, 1 1, −1 P 1, −1 0, 0 −1, 1 S −1, 1 1, −1 0, 0
SLIDE 7
Learning in games
SLIDE 8
Learning in games Repeated games
SLIDE 9
Learning in games
SLIDE 10
Best Response learning
SLIDE 11
Best Response learning
- 1. Guess what the opponent(s) will play
SLIDE 12
Best Response learning
- 1. Guess what the opponent(s) will play
- 2. Play a Best Response to that guess
SLIDE 13
Best Response learning
- 1. Guess what the opponent(s) will play
- 2. Play a Best Response to that guess
- 3. Observe the play
SLIDE 14
Best Response learning
- 1. Guess what the opponent(s) will play
- 2. Play a Best Response to that guess
- 3. Observe the play
- 4. Update the guess
SLIDE 15
BR learning: Cournot dynamics
SLIDE 16
BR learning: Cournot dynamics
Guess = last action played
SLIDE 17
BR learning: Cournot dynamics
Guess = last action played C D C 2, 2 −1, 3 D 3, −1 0, 0
SLIDE 18
BR learning: Cournot dynamics
Guess = last action played C D C 2, 2 −1, 3 D 3, −1 0, 0 R P S R 0, 0 −1, 1 1, −1 P 1, −1 0, 0 −1, 1 S −1, 1 1, −1 0, 0
SLIDE 19
BR learning: Fictitious play
SLIDE 20
BR learning: Fictitious play
Guess = empirical distribution of play
SLIDE 21
BR learning: Fictitious play
Guess = empirical distribution of play R P S R 0, 0 −1, 1 1, −1 P 1, −1 0, 0 −1, 1 S −1, 1 1, −1 0, 0
SLIDE 22
BR learning: Fictitious play
Guess = empirical distribution of play R P S R 0, 0 −1, 1 1, −1 P 1, −1 0, 0 −1, 1 S −1, 1 1, −1 0, 0 L C R U 0, 0 0, 1 1, 0 M 1, 0 0, 0 0, 1 D 0, 1 1, 0 0, 0
SLIDE 23
Evolutionary learning
SLIDE 24
Evolutionary learning
Action set: A Utility function: u
SLIDE 25
Evolutionary learning
Action set: A Utility function: u p ∈ ∆(A), k ∈ A ˙ pk = pk (u(k, p) − u(p, p))
SLIDE 26
Battle of the Sexes
SLIDE 27
Battle of the Sexes
O F O 3, 2 0, 0 F 0, 0 2, 3
SLIDE 28
Correlated equilibrium (CE)
SLIDE 29
Correlated equilibrium (CE)
a∗ ∈ A =
i Ai is a NE:
∀i, ∀a′
i, ui(a∗ i, a∗ −i) ≥ ui(a′ i, a∗ −i)
SLIDE 30
Correlated equilibrium (CE)
a∗ ∈ A =
i Ai is a NE:
∀i, ∀a′
i, ui(a∗ i, a∗ −i) ≥ ui(a′ i, a∗ −i)
α ∈
i ∆(Ai) is a NE: ∀i, ∀ai, ∀a′ i,
- a−i
ui(ai, a−i)α(a) ≥
- a−i
ui(a′
i, a−i)α(a)
SLIDE 31
Correlated equilibrium (CE)
a∗ ∈ A =
i Ai is a NE:
∀i, ∀a′
i, ui(a∗ i, a∗ −i) ≥ ui(a′ i, a∗ −i)
α ∈
i ∆(Ai) is a NE: ∀i, ∀ai, ∀a′ i,
- a−i
ui(ai, a−i)α(a) ≥
- a−i
ui(a′
i, a−i)α(a)
π ∈ ∆(A) is a CE: ∀i, ∀ai, ∀a′
i,
- a−i
ui(ai, a−i)π(a) ≥
- a−i
ui(a′
i, a−i)π(a)
SLIDE 32
No regret learning
SLIDE 33
No regret learning
ui(k, a−i) − ui(j, a−i)
SLIDE 34
No regret learning
ui(k, a−i) − ui(j, a−i) Ri
jk(t) = t
- τ=0:ai(τ)=j
ui(k, a−i(τ))−ui(j, a−i(τ))
SLIDE 35
No regret learning
ui(k, a−i) − ui(j, a−i) Ri
jk(t) = t
- τ=0:ai(τ)=j
ui(k, a−i(τ))−ui(j, a−i(τ)) Regret matching converges to the correlated equilibria set.
SLIDE 36
Learning in games
SLIDE 37
Learning in games
◮ Best response
SLIDE 38
Learning in games
◮ Best response ◮ Replicator dynamics
SLIDE 39
Learning in games
◮ Best response ◮ Replicator dynamics ◮ No regret
SLIDE 40
Repeated games
SLIDE 41
Markov Decision Process (MDP)
SLIDE 42
Markov Decision Process (MDP)
state space X action space U transition P : X × U → ∆(X) reward r : X × U → R discount factor δ ∈ [0, 1]
SLIDE 43
Markov Decision Process (MDP)
state space X action space U transition P : X × U → ∆(X) reward r : X × U → R discount factor δ ∈ [0, 1] U(x(·), u(·)) =
+∞
- t=0
δtr(x(t), u(t))
SLIDE 44
MDP (continued)
history H ∈ (X, U) policy π : H → ∆(U)
SLIDE 45
MDP (continued)
history H ∈ (X, U) policy π : H → ∆(U) V π(x0) = Eπ [U(x(·), u(·))]
SLIDE 46
MDP (continued)
history H ∈ (X, U) policy π : H → ∆(U) V π(x0) = Eπ [U(x(·), u(·))] V (x0) = max
π
V π(x0)
SLIDE 47
Principle of Optimality
Bellman’s equation: V (x0) = max
u0 [r(x0, u0) + δV (P(x0, u0))]
SLIDE 48
Dynamic Programming
Solving the MDP:
SLIDE 49
Dynamic Programming
Solving the MDP:
◮ knowing P: value iteration
SLIDE 50
Dynamic Programming
Solving the MDP:
◮ knowing P: value iteration ◮ not knowing P: online learning
SLIDE 51
Repeated game
SLIDE 52
Repeated game
Game (I,
i Ai, i ui)
SLIDE 53
Repeated game
Game (I,
i Ai, i ui)
Discount factor δ Ui(a(·)) =
+∞
- t=0
δtui(a(t))
SLIDE 54
Repeated game
Game (I,
i Ai, i ui)
Discount factor δ Ui(a(·)) =
+∞
- t=0
δtui(a(t)) Strategy σ : H →
i ∆(Aix)
Vi(σ) = Eσ [Ui(a(·))]
SLIDE 55
Nash equilibrium
Player i:
◮ choices σi ◮ utility Vi
SLIDE 56
Nash equilibrium
Player i:
◮ choices σi ◮ utility Vi
Nash equilibrium is not strong enough! (Explanation on the whiteboard ⇒)
SLIDE 57
Information structure
SLIDE 58
Information structure
◮ perfect ◮ imperfect
SLIDE 59
Information structure
◮ perfect ◮ imperfect ◮ public ◮ private (beliefs)
SLIDE 60
Folk theorem
Any feasible, strictly individually rational payoff can be sustained by a sequentially rational equilibrium.
SLIDE 61
Folk theorem
Any feasible, strictly individually rational payoff can be sustained by a sequentially rational equilibrium. Holy grail for repeated games.
SLIDE 62
u1 u2
SLIDE 63
u1 u2 DC
SLIDE 64
u1 u2 CC DD DC CD
SLIDE 65
u1 u2 CC DD DC CD
SLIDE 66
u1 u2 CC DD DC CD
SLIDE 67
Research
SLIDE 68
Weakly belief-free equilibria
Characterization of repeated games with correlated equilibria.
SLIDE 69
Repeated games
SLIDE 70
Repeated games
◮ Dynamic programming
SLIDE 71
Repeated games
◮ Dynamic programming ◮ Repeated games
SLIDE 72
Repeated games
◮ Dynamic programming ◮ Repeated games ◮ Folk theorem
SLIDE 73
Learning in games
SLIDE 74
Learning in games Repeated games
SLIDE 75