[PPT] - Empirical-evidence Equilibria in Stochastic Games Nicolas Dudebout PowerPoint Presentation

SLIDE 1

Empirical-evidence Equilibria in Stochastic Games

Nicolas Dudebout

SLIDE 2

Outline

Stochastic games
Empirical-evidence equilibria (EEEs)
Open questions in EEEs

2

SLIDE 3

Stochastic Games

Game theory
Markov decision processes

3

SLIDE 4

Game Theory

Decision making 𝑣∶ 𝒝 → ℝ ⟹ 𝑏∗ ∈ arg max

𝑏∈𝒝

𝑣(𝑏) Game theory 𝑣1 ∶ 𝒝1 × 𝒝2 → ℝ 𝑣2 ∶ 𝒝1 × 𝒝2 → ℝ Nash Equilibrium ⎧ ⎪ ⎨ ⎪ ⎩ 𝑏∗

1 ∈ arg max 𝑏1∈𝒝1

𝑣1(𝑏1, 𝑏∗

2)

𝑏∗

2 ∈ arg max 𝑏2∈𝒝2

𝑣2(𝑏∗

1, 𝑏2)

4

SLIDE 5

Example: Battle of the Sexes

F O F 2, 2 0, 1 O 0, 0 1, 3 Nash equilibria

(𝐺, 𝐺)
(𝑃, 𝑃)
(3/

4𝐺 1/ 4𝑃, 1/ 3𝐺 2/ 3𝑃)

5

SLIDE 6

Markov Decision Process (MDP)

Dynamic 𝑦+ ∼ 𝑔(𝑦, 𝑏) ⟺ 𝑦𝑢+1 ∼ 𝑔(𝑦𝑢, 𝑏𝑢) Stage cost 𝑣(𝑦, 𝑏) History ℎ𝑢 = (𝑦0, 𝑦1, … , 𝑦𝑢, 𝑏0, 𝑏1, … , 𝑏𝑢) Strategy 𝜏 ∶ ℋ → 𝒝 Utility 𝑉(𝜏) = 𝔽𝑔,𝜏[

∞

∑

𝑢=0

𝜀𝑢𝑣(𝑦𝑢, 𝑏𝑢)] Bellman’s equation 𝑉 ∗(𝑦) = max

𝑏∈𝒝 {𝑣(𝑦, 𝑏) + 𝜀𝔽𝑔[𝑉 ∗(𝑦+) | 𝑦, 𝑏]}

Dynamic programming use knowledge of 𝑔 Reinforcement learning learn 𝑔 from repeated interaction

6

SLIDE 7

Markov Decision Process (MDP)

Dynamic 𝑦+ ∼ 𝑔(𝑦, 𝑏) ⟺ 𝑦𝑢+1 ∼ 𝑔(𝑦𝑢, 𝑏𝑢) Stage cost 𝑣(𝑦, 𝑏) History ℎ𝑢 = (𝑦0, 𝑦1, … , 𝑦𝑢, 𝑏0, 𝑏1, … , 𝑏𝑢) Strategy 𝜏 ∶ ℋ → 𝒝 Utility 𝑉(𝜏) = 𝔽𝑔,𝜏[

∞

∑

𝑢=0

𝜀𝑢𝑣(𝑦𝑢, 𝑏𝑢)] Bellman’s equation 𝑉 ∗(𝑦) = max

𝑏∈𝒝 {𝑣(𝑦, 𝑏) + 𝜀𝔽𝑔[𝑉 ∗(𝑦+) | 𝑦, 𝑏]}

Dynamic programming use knowledge of 𝑔 Reinforcement learning learn 𝑔 from repeated interaction

6

SLIDE 8

Markov Decision Process (MDP)

Dynamic 𝑦+ ∼ 𝑔(𝑦, 𝑏) ⟺ 𝑦𝑢+1 ∼ 𝑔(𝑦𝑢, 𝑏𝑢) Stage cost 𝑣(𝑦, 𝑏) History ℎ𝑢 = (𝑦0, 𝑦1, … , 𝑦𝑢, 𝑏0, 𝑏1, … , 𝑏𝑢) Strategy 𝜏 ∶ 𝒴 → 𝒝 Utility 𝑉(𝜏) = 𝔽𝑔,𝜏[

∞

∑

𝑢=0

𝜀𝑢𝑣(𝑦𝑢, 𝑏𝑢)] Bellman’s equation 𝑉 ∗(𝑦) = max

𝑏∈𝒝 {𝑣(𝑦, 𝑏) + 𝜀𝔽𝑔[𝑉 ∗(𝑦+) | 𝑦, 𝑏]}

Dynamic programming use knowledge of 𝑔 Reinforcement learning learn 𝑔 from repeated interaction

6

SLIDE 9

Imperfect Information (POMDP)

Dynamic 𝑥+ ∼ 𝑜(𝑥, 𝑏) Signal 𝑡 ∼ 𝜉(𝑥) History ℎ𝑢 = (𝑡0, 𝑡1, … , 𝑡𝑢, 𝑏0, 𝑏1, … , 𝑏𝑢) Strategy 𝜏 ∶ ℋ → 𝒝 Belief ℙ𝑜,𝜉,𝜏[𝑥 | ℎ]

7

SLIDE 10

Imperfect Information (POMDP)

Dynamic 𝑥+ ∼ 𝑜(𝑥, 𝑏) Signal 𝑡 ∼ 𝜉(𝑥) History ℎ𝑢 = (𝑡0, 𝑡1, … , 𝑡𝑢, 𝑏0, 𝑏1, … , 𝑏𝑢) Strategy 𝜏 ∶ ℋ → 𝒝 Belief ℙ𝑜,𝜉,𝜏[𝑥 | ℎ]

7

SLIDE 11

Imperfect Information (POMDP)

Dynamic 𝑥+ ∼ 𝑜(𝑥, 𝑏) Signal 𝑡 ∼ 𝜉(𝑥) History ℎ𝑢 = (𝑡0, 𝑡1, … , 𝑡𝑢, 𝑏0, 𝑏1, … , 𝑏𝑢) Strategy 𝜏 ∶ Δ(𝒳 ) → 𝒝 Belief ℙ𝑜,𝜉,𝜏[𝑥 | ℎ]

7

SLIDE 12

Stochastic Games

Dynamic 𝑥+ ∼ 𝑜(𝑥, 𝑏1, 𝑏2) Signals { 𝑡1 ∼ 𝜉1(𝑥) 𝑡2 ∼ 𝜉2(𝑥) Histories { ℎ𝑢

1 = (𝑡1 0, 𝑡1 1, … , 𝑡𝑢 1, 𝑏1 0, 𝑏1 1, … , 𝑏𝑢 1)

ℎ𝑢

2 = (𝑡2 0, 𝑡2 1, … , 𝑡𝑢 2, 𝑏2 0, 𝑏2 1, … , 𝑏𝑢 2)

Strategies { 𝜏1 ∶ ℋ1 → 𝒝1 𝜏2 ∶ ℋ2 → 𝒝2 Beliefs { ℙ𝑜,𝜉1,𝜏1,𝜉2,𝜏2[𝑥, ℎ2 | ℎ1] ℙ𝑜,𝜉1,𝜏1,𝜉2,𝜏2[𝑥, ℎ1 | ℎ2]

8

SLIDE 13

Stochastic Games

Dynamic 𝑥+ ∼ 𝑜(𝑥, 𝑏1, 𝑏2) Signals { 𝑡1 ∼ 𝜉1(𝑥) 𝑡2 ∼ 𝜉2(𝑥) Histories { ℎ𝑢

1 = (𝑡1 0, 𝑡1 1, … , 𝑡𝑢 1, 𝑏1 0, 𝑏1 1, … , 𝑏𝑢 1)

ℎ𝑢

2 = (𝑡2 0, 𝑡2 1, … , 𝑡𝑢 2, 𝑏2 0, 𝑏2 1, … , 𝑏𝑢 2)

Strategies { 𝜏1 ∶ ℋ1 → 𝒝1 𝜏2 ∶ ℋ2 → 𝒝2 Beliefs { ℙ𝑜,𝜉1,𝜏1,𝜉2,𝜏2[𝑥, ℎ2 | ℎ1] ℙ𝑜,𝜉1,𝜏1,𝜉2,𝜏2[𝑥, ℎ1 | ℎ2]

8

SLIDE 14

Existing Approaches

(Weakly) belief-free equilibrium
Mean-field equilibrium
Incomplete theories

9

SLIDE 15

Empirical-evidence Equilibria

10

SLIDE 16

Motivation

Agent 1 Nature Agent 2

0. Pick arbitrary strategies
1. Formulate simple but consistent models
2. Design strategies optimal w.r.t. models, then, back to 1.

Empirical-evidence equilibrium is a fixed point:

Strategies optimal w.r.t. models
Models consistent with strategies

11

SLIDE 17

Example: Asset Management

Trading one asset on the stock market Model based on

information published by the company
observed trading activity

Model very different for each agent

12

SLIDE 18

Multiple to Single Agent

Agent 1 Nature Agent 2

13

SLIDE 19

Multiple to Single Agent

Agent 1 Nature Agent 2 Nature 1

13

SLIDE 20

Single Agent Setup

Agent Nature

14

SLIDE 21

Single Agent Setup

𝑦+ ∼ 𝑔(𝑦, 𝑏, 𝑡) Nature

14

SLIDE 22

Single Agent Setup

𝑦+ ∼ 𝑔(𝑦, 𝑏, 𝑡) Nature 𝑡

14

SLIDE 23

Single Agent Setup

𝑦+ ∼ 𝑔(𝑦, 𝑏, 𝑡) 𝑥+ ∼ 𝑜(𝑥, 𝑦, 𝑏) 𝑡 ∼ 𝜉(𝑥) 𝑡

14

SLIDE 24

Example: Asset Management

𝑦+ ∼ 𝑔(𝑦, 𝑏, 𝑡) 𝑥+ ∼ 𝑜(𝑥, 𝑦, 𝑏) 𝑡 ∼ 𝜉(𝑥) 𝑡 State holding 𝑦 ∈ {0 .. 𝑁} Action sell one, hold, or buy one 𝑏 ∈ {−1, 0, 1} Signal price 𝑞 ∈ {Low, High} Stage cost 𝑞 ⋅ 𝑏 Nature 𝑥 represents market sentiment, political climate,

ther traders

15

SLIDE 25

Single Agent Setup

𝑦+ ∼ 𝑔(𝑦, 𝑏, 𝑡) 𝑏 ∼ 𝜏(ℎ) 𝑥+ ∼ 𝑜(𝑥, 𝑦, 𝑏) 𝑡 ∼ 𝜉(𝑥) 𝑡 Model ̂ 𝑡 ̂ 𝑡 consistent with 𝜏 𝜏 optimal w.r.t. ̂ 𝑡

16

SLIDE 26

Single Agent Setup

𝑦+ ∼ 𝑔(𝑦, 𝑏, 𝑡) 𝑏 ∼ 𝜏(ℎ) 𝑥+ ∼ 𝑜(𝑥, 𝑦, 𝑏) 𝑡 ∼ 𝜉(𝑥) 𝑡 Model ̂ 𝑡 ̂ 𝑡 consistent with 𝜏 𝜏 optimal w.r.t. ̂ 𝑡

16

SLIDE 27

Single Agent Setup

𝑦+ ∼ 𝑔(𝑦, 𝑏, 𝑡) 𝑏 ∼ 𝜏(ℎ) 𝑥+ ∼ 𝑜(𝑥, 𝑦, 𝑏) 𝑡 ∼ 𝜉(𝑥) 𝑡 Model ̂ 𝑡 ̂ 𝑡 consistent with 𝜏 𝜏 optimal w.r.t. ̂ 𝑡

16

SLIDE 28

Single Agent Setup

𝑦+ ∼ 𝑔(𝑦, 𝑏, 𝑡) 𝑏 ∼ 𝜏(ℎ) 𝑥+ ∼ 𝑜(𝑥, 𝑦, 𝑏) 𝑡 ∼ 𝜉(𝑥) 𝑡 Model ̂ 𝑡 ̂ 𝑡 consistent with 𝑡 𝜏 optimal w.r.t. ̂ 𝑡

16

SLIDE 29

Single Agent Setup

𝑦+ ∼ 𝑔(𝑦, 𝑏, 𝑡) 𝑏 ∼ 𝜏(ℎ) 𝑥+ ∼ 𝑜(𝑥, 𝑦, 𝑏) 𝑡 ∼ 𝜉(𝑥) 𝑡 Model ̂ 𝑡 ̂ 𝑡 consistent with 𝜏 𝜏 optimal w.r.t. ̂ 𝑡

16

SLIDE 30

Single Agent Setup

𝑦+ ∼ 𝑔(𝑦, 𝑏, ̂ 𝑡) 𝑏 ∼ 𝜏(ℎ) 𝑥+ ∼ 𝑜(𝑥, 𝑦, 𝑏) 𝑡 ∼ 𝜉(𝑥) 𝑡 Model ̂ 𝑡 ̂ 𝑡 consistent with 𝜏 𝜏 optimal w.r.t. ̂ 𝑡

16

SLIDE 31

Single Agent Setup

𝑦+ ∼ 𝑔(𝑦, 𝑏, ̂ 𝑡) 𝑏 ∼ 𝜏(ℎ) 𝑥+ ∼ 𝑜(𝑥, 𝑦, 𝑏) 𝑡 ∼ 𝜉(𝑥) 𝑡 Model ̂ 𝑡 ̂ 𝑡 consistent with 𝜏 𝜏 optimal w.r.t. ̂ 𝑡

16

SLIDE 32

Depth-𝑙 Consistency

Consider a binary stochastic process 𝑡 0100010001001010010110111010000111010101...

0 characteristic: ℙ[𝑡 = 0], ℙ[𝑡 = 1]
1 characteristic: ℙ[𝑡𝑡+ = 00], ℙ[𝑡𝑡+ = 10],

ℙ[𝑡𝑡+ = 01], ℙ[𝑡𝑡+ = 11]

...
𝑙 characteristic: probability of strings of length 𝑙 + 1

Definition Two processes 𝑡 and 𝑡′ are depth-𝑙 consistent if they have the same 𝑙 characteristic

17

SLIDE 33

Depth-𝑙 Consistency

Consider a binary stochastic process 𝑡 0100010001001010010110111010000111010101...

0 characteristic: ℙ[𝑡 = 0], ℙ[𝑡 = 1]
1 characteristic: ℙ[𝑡𝑡+ = 00], ℙ[𝑡𝑡+ = 10],

ℙ[𝑡𝑡+ = 01], ℙ[𝑡𝑡+ = 11]

...
𝑙 characteristic: probability of strings of length 𝑙 + 1

Definition Two processes 𝑡 and 𝑡′ are depth-𝑙 consistent if they have the same 𝑙 characteristic

17

SLIDE 34

Depth-𝑙 Consistency

Consider a binary stochastic process 𝑡 0100010001001010010110111010000111010101...

0 characteristic: ℙ[𝑡 = 0], ℙ[𝑡 = 1]
1 characteristic: ℙ[𝑡𝑡+ = 00], ℙ[𝑡𝑡+ = 10],

ℙ[𝑡𝑡+ = 01], ℙ[𝑡𝑡+ = 11]

...
𝑙 characteristic: probability of strings of length 𝑙 + 1

Definition Two processes 𝑡 and 𝑡′ are depth-𝑙 consistent if they have the same 𝑙 characteristic

17

SLIDE 35

Depth-𝑙 Consistency: Example

1 𝑨∅

0.5 0.5

𝑨0 𝑨1 1

1 1 0.3 0.7 0.3 0.7

18

SLIDE 36

Complete picture

Fix a depth 𝑙 ∈ ℕ 𝑦+ ∼ 𝑔(𝑦, 𝑏, 𝑡) 𝑏 ∼ 𝜏(ℎ) 𝑥+ ∼ 𝑜(𝑥, 𝑦, 𝑏) 𝑡 ∼ 𝜉(𝑥) Model 𝑡 ̂ 𝑡 𝜏 ↦ 𝜈 consistent with 𝜏 𝜈 ↦ 𝜏 optimal w.r.t. 𝜈 𝑨 contains the last 𝑙 observed signals 𝜈(𝑨 = (𝑡1, 𝑡2, … , 𝑡𝑙))[𝑡𝑙+1] = ℙ𝜏[𝑡𝑢+1 = 𝑡𝑙+1 | 𝑡𝑢 = 𝑡𝑙, … , 𝑡𝑢−𝑙+1 = 𝑡1]

19

SLIDE 37

Complete picture

Fix a depth 𝑙 ∈ ℕ 𝑦+ ∼ 𝑔(𝑦, 𝑏, 𝑡) 𝑏 ∼ 𝜏(ℎ) 𝑥+ ∼ 𝑜(𝑥, 𝑦, 𝑏) 𝑡 ∼ 𝜉(𝑥) 𝑨+ ∼ 𝑛𝑙(𝑨) ̂ 𝑡 ∼ 𝜈(𝑨) 𝑡 ̂ 𝑡 𝜏 ↦ 𝜈 consistent with 𝜏 𝜈 ↦ 𝜏 optimal w.r.t. 𝜈 𝑨 contains the last 𝑙 observed signals 𝜈(𝑨 = (𝑡1, 𝑡2, … , 𝑡𝑙))[𝑡𝑙+1] = ℙ𝜏[𝑡𝑢+1 = 𝑡𝑙+1 | 𝑡𝑢 = 𝑡𝑙, … , 𝑡𝑢−𝑙+1 = 𝑡1]

19

SLIDE 38

Complete picture

Fix a depth 𝑙 ∈ ℕ 𝑦+ ∼ 𝑔(𝑦, 𝑏, 𝑡) 𝑏 ∼ 𝜏(𝑦, 𝑨) 𝑥+ ∼ 𝑜(𝑥, 𝑦, 𝑏) 𝑡 ∼ 𝜉(𝑥) 𝑨+ ∼ 𝑛𝑙(𝑨) ̂ 𝑡 ∼ 𝜈(𝑨) 𝑡 ̂ 𝑡 𝜏 ↦ 𝜈 consistent with 𝜏 𝜈 ↦ 𝜏 optimal w.r.t. 𝜈 𝑨 contains the last 𝑙 observed signals 𝜈(𝑨 = (𝑡1, 𝑡2, … , 𝑡𝑙))[𝑡𝑙+1] = ℙ𝜏[𝑡𝑢+1 = 𝑡𝑙+1 | 𝑡𝑢 = 𝑡𝑙, … , 𝑡𝑢−𝑙+1 = 𝑡1]

19

SLIDE 39

Complete picture

Fix a depth 𝑙 ∈ ℕ 𝑦+ ∼ 𝑔(𝑦, 𝑏, 𝑡) 𝑏 ∼ 𝜏(𝑦, 𝑨) 𝑥+ ∼ 𝑜(𝑥, 𝑦, 𝑏) 𝑡 ∼ 𝜉(𝑥) 𝑨+ ∼ 𝑛𝑙(𝑨) ̂ 𝑡 ∼ 𝜈(𝑨) 𝑡 ̂ 𝑡 𝜏 ↦ 𝜈 consistent with 𝜏 𝜈 ↦ 𝜏 optimal w.r.t. 𝜈 𝑨 contains the last 𝑙 observed signals 𝜈(𝑨 = (𝑡1, 𝑡2, … , 𝑡𝑙))[𝑡𝑙+1] = ℙ𝜏[𝑡𝑢+1 = 𝑡𝑙+1 | 𝑡𝑢 = 𝑡𝑙, … , 𝑡𝑢−𝑙+1 = 𝑡1]

19

SLIDE 40

Definition

(𝜏, 𝜈) is an empirical-evidence optimum (EEO) for 𝑙 iff

𝜏 is optimal w.r.t. 𝜈
𝜈 is depth-𝑙 consistent with 𝜏

(𝜏, 𝜈) is an 𝜗 empirical-evidence optimum (𝜗 EEO) for 𝑙 iff

𝜏 is 𝜗 optimal w.r.t. 𝜈
𝜈 is depth-𝑙 consistent with 𝜏

20

SLIDE 41

Definition

(𝜏, 𝜈) is an empirical-evidence optimum (EEO) for 𝑙 iff

𝜏 is optimal w.r.t. 𝜈
𝜈 is depth-𝑙 consistent with 𝜏

(𝜏, 𝜈) is an 𝜗 empirical-evidence optimum (𝜗 EEO) for 𝑙 iff

𝜏 is 𝜗 optimal w.r.t. 𝜈
𝜈 is depth-𝑙 consistent with 𝜏

20

SLIDE 42

Existence Result

Theorem For all 𝑙 and 𝜗, there exists an 𝜗 EEO for 𝑙 Proof sketch Prove continuity of 𝜏 ↦ 𝜈 ↦ 𝜏 𝜏 ∶ 𝒴 × 𝒶 → Δ(𝒝) 𝜏 parametrized over a simplex (convex and compact) Apply Brouwer’s fixed point theorem

21

SLIDE 43

Existence Result

Theorem For all 𝑙 and 𝜗, there exists an 𝜗 EEO for 𝑙 Proof sketch Prove continuity of 𝜏 ↦ 𝜈 ↦ 𝜏 𝜏 ∶ 𝒴 × 𝒶 → Δ(𝒝) 𝜏 parametrized over a simplex (convex and compact) Apply Brouwer’s fixed point theorem

21

SLIDE 44

Complete picture

Fix a depth 𝑙 ∈ ℕ 𝑦+ ∼ 𝑔(𝑦, 𝑏, 𝑡) 𝑏 ∼ 𝜏(𝑦, 𝑨) 𝑥+ ∼ 𝑜(𝑥, 𝑦, 𝑏) 𝑡 ∼ 𝜉(𝑥) 𝑨+ ∼ 𝑛𝑙(𝑨) ̂ 𝑡 ∼ 𝜈(𝑨) 𝑡 ̂ 𝑡 𝜏 ↦ 𝜈 consistent with 𝜏 𝜈 ↦ 𝜏 optimal w.r.t. 𝜈 𝑨 contains the last 𝑙 observed signals 𝜈(𝑨 = (𝑡1, 𝑡2, … , 𝑡𝑙))[𝑡𝑙+1] = ℙ𝜏[𝑡𝑢+1 = 𝑡𝑙+1 | 𝑡𝑢 = 𝑡𝑙, … , 𝑡𝑢−𝑙+1 = 𝑡1]

22

SLIDE 45

Multiagent Setting

𝑦+

𝑗 ∼ 𝑔𝑗(𝑦𝑗, 𝑏𝑗, 𝑡𝑗)

𝑏𝑗 ∼ 𝜏𝑗(𝑦𝑗, 𝑨𝑗) 𝑥+ ∼ 𝑜(𝑥, 𝑦, 𝑏) 𝑡 ∼ 𝜉(𝑥) 𝑨+

𝑗 ∼ 𝑛𝑗,𝑙(𝑨𝑗)

̂ 𝑡𝑗 ∼ 𝜈𝑗(𝑨𝑗) 𝑡𝑗 ̂ 𝑡𝑗 𝑦 = (𝑦1, 𝑦2, … , 𝑦𝑂) 𝑏 = (𝑏1, 𝑏2, … , 𝑏𝑂) 𝑡 = (𝑡1, 𝑡2, … , 𝑡𝑂)

23

SLIDE 46

Empirical-evidence Equilibrium

(𝜏, 𝜈) is an empirical-evidence equilibrium (EEE) for 𝐿 = (𝑙1, 𝑙2, … , 𝑙𝑂) iff

for all 𝑗, 𝜏𝑗 is optimal w.r.t. 𝜈𝑗
for all 𝑗, 𝜈𝑗 is depth-𝑙𝑗 consistent with 𝜏

Theorem For all 𝐿 and 𝜗, there exists an 𝜗 EEE for 𝐿

24

SLIDE 47

Empirical-evidence Equilibrium

(𝜏, 𝜈) is an empirical-evidence equilibrium (EEE) for 𝐿 = (𝑙1, 𝑙2, … , 𝑙𝑂) iff

for all 𝑗, 𝜏𝑗 is optimal w.r.t. 𝜈𝑗
for all 𝑗, 𝜈𝑗 is depth-𝑙𝑗 consistent with 𝜏

Theorem For all 𝐿 and 𝜗, there exists an 𝜗 EEE for 𝐿

24

SLIDE 48

Open Questions

endogenous model depending on action
large number of agents
large 𝑙
relating EEE to other concepts (MFE, optimum)
offline computation
online learning using empirical evidence

25

SLIDE 49

Open Questions

endogenous model depending on action
large number of agents
large 𝑙
relating EEE to other concepts (MFE, optimum)
offline computation
online learning using empirical evidence

25

SLIDE 50

Open Questions

endogenous model depending on action
large number of agents
large 𝑙
relating EEE to other concepts (MFE, optimum)
offline computation
online learning using empirical evidence

25

SLIDE 51

Open Questions

endogenous model depending on action
large number of agents
large 𝑙
relating EEE to other concepts (MFE, optimum)
offline computation
online learning using empirical evidence

25

SLIDE 52

Example: Asset Management

State holdings 𝑦𝑗 ∈ {0 .. 𝑁} Action sell one, hold, or buy one 𝑏𝑗 ∈ {−1, 0, 1} Signal price 𝑞 ∈ {Low, High} Dynamic 𝑦+

𝑗 = 𝑦𝑗 + 𝑏𝑗

Stage cost 𝑞 ⋅ 𝑏𝑗 Nature market trend 𝑐 ∈ {Bull, Bear} 𝑥 = (𝑐, 𝑞) Nature is a sticky bear

26

SLIDE 53

Example: Asset Management

0. Pick arbitrary models 𝜈
1. Design strategies 𝜏 optimal w.r.t. models 𝜈
2. Formulate consistent models 𝜈upd, then, back to 1.

Depth-0 consistency:

𝜈1 = 1
𝜈2 = 0

𝜈𝑢+1

𝑗

= (1 − 𝛽)𝜈𝑢

𝑗 + 𝛽(𝜈𝑢 𝑗,upd − 𝜈𝑢 𝑗)

27

SLIDE 54

Learning Results: Offline

20 40 60 80 100 0.2 0.4 0.6 0.8 1 Time 𝑢 Prediction 𝜈𝑢

𝑗[High]

𝑗 = 1 𝑗 = 2

28

SLIDE 55

Learning Results: Online

20 40 60 80 100 0.2 0.4 0.6 0.8 1 Time 𝑢 Prediction 𝜈𝑢

𝑗[High]

𝑗 = 1 𝑗 = 2

29

SLIDE 56

Empirical-evidence Equilibria

Introduce
Contrast
Compute

30