[PPT] - Comparators for Quantitative games BY Suguman Bansal Swarat PowerPoint Presentation

SLIDE 1

Comparators for Quantitative games

BY

Suguman Bansal Swarat Chaudhuri Moshe Y. Vardi Dagstuhl Seminar March 16, 2017

SLIDE 2

Repeated games

Infinitely many rounds of a base

game

Each agent receives a reward in

every round

Reward of each agent from game is

an aggregation of its rewards from every round

Discounted-sum aggregation
Mean-payoff aggregation . . .

3/16/17 Comparators for quantitative games 2

SLIDE 3

Automated reasoning about repeated games

Existence of rational behavior in a repeated game
Find one rational behavior in repeated games
Find all rational behaviors in a repeated game
Finite representation
What properties hold in repeated game if agents behave rationally
Properties of rational behaviors in repeated games

3/16/17 Comparators for quantitative games 3

SLIDE 4

Strategy in the Repeated games [Rubinstein 1986]

Finite state machines
Strategies is defined for an agent
All other agents comprise of environment
Transitions denote one round of repeated

game w.r.t. agent

Transition on (agent-Action, envt-Action)
Weight on transition is Reward of agent for

that transition

3/16/17 Algorithmic analysis of Regular repeated games 4

𝐷, 𝐸 , 0 𝐷, 𝐷 , 2 𝐸, 𝐷 , 3 𝐸, 𝐸 , 1 𝑡) 𝑡*

Tit-for-tat strategy in Repeated games

SLIDE 5

Game executions [Rubinstein 1986]

3/16/17 Algorithmic analysis of Regular repeated games 5

𝑏, 𝑐 , 2 𝑐, 𝑏 , 3 𝑏, 𝑐 , (2, 3)

S1 S2 (S1, S2)

𝐷, 𝐸 , 0 𝐷, 𝐷 , 2 𝐸, 𝐷 , 3 𝐸, 𝐸 , 1 𝑡) 𝑡* 𝐷, 𝐸 , 0 𝐷, 𝐷 , 2 𝐸, 𝐷 , 3 𝐸, 𝐸 , 1 𝑡) 𝑡*

×

𝐷, 𝐷 , (2, 2) 𝐷, 𝐷 , (2, 2) 𝐷, 𝐷 , (2, 2) 𝐷, 𝐷 , (2, 2)

Synchronize on all agent actions

SLIDE 6

Strategies are Trees

3/16/17 Comparators for quantitative games 6

𝐷, 𝐸 , 0 𝐷, 𝐷 , 2 𝐸, 𝐷 , 3 𝐸, 𝐸 , 1 𝑡) 𝑡* 𝑡) 𝑡) 𝑡* 𝑡) 𝑡* 𝑡) 𝑡*

SLIDE 7

Strategies Tree and their compositions

Every player has a regular set of strategies
Weighted tree automata
Composition of agent strategies results in regular set of game

executions

Compare rewards of player along different game executions

3/16/17 Comparators for quantitative games 7

SLIDE 8

Comparing rewards

Reward of agent in a game execution
Reward sequence of agent 𝑇 = (𝑡2, 𝑡), … )
𝑡4 is reward received in the i-th round of a game execution
Aggregate function 𝑔 ∶ ℕ 8 → ℝ
Reward of agent is 𝑔 𝑇
Which game execution results in a greater reward?
Reward sequences 𝐵 and 𝐶 on different game executions
Is 𝑔 𝐵 ≤ 𝑔(𝐶) ?

3/16/17 Comparators for quantitative games 8

SLIDE 9

Outline of the talk ...

Introduce Comparator [Bansal, Chaudhuri, Vardi (under submission)]
A novel automata-theoretic technique to compare the aggregate of reward

sequences

Comparator for discounted-sum aggregate function
Applications of Comparators [Ongoing work]

3/16/17 Comparators for quantitative games 9

SLIDE 10

Comparator

Comparator for aggregate function 𝑔: ℕ 8 → ℝ is a Büchi automaton
Pair of bounded reward sequences 𝐵, 𝐶 is a word of the comparator

for 𝑔: ℕ 8 → ℝ iff 𝑔 𝐵 ≤ 𝑔(𝐶)

Comparator for Limsup aggregate function
Limsup of sequence of natural numbers is the largest number that occurs

infinitely often

3/16/17 Comparators for quantitative games 10

SLIDE 11

Discounted-sum Comparator

Discounted sum of reward sequence 𝑆 with discount factor 𝑒 > 1 is

𝐸𝑇C 𝑆 = 𝑠2 + 𝑠

)

𝑒 + 𝑠* 𝑒* …

Discounted-sum Comparator with discount-factor 𝑒 > 1
Pair of bounded reward sequences 𝐵, 𝐶 is a word of the DS Comparator

with discount-factor 𝑒 iff 𝐸𝑇C 𝐵 ≤ 𝐸𝑇C 𝐶

3/16/17 Comparators for quantitative games 11

SLIDE 12

DS Comparator : Core Insight – I

Sequence 𝐵 = (𝑏2, 𝑏), 𝑏* … )
Discount factor 𝑒 > 1
𝐸𝑇C 𝐵 = 𝑏2 +

FG C + FH CH + ⋯

= 𝑏2. 𝑏)𝑏* … C = 𝐵C

Use lexicographic ordering of sequences
Works only if 𝑏4 ≤ 𝑒 − 1 for all 𝑗 ≥ 0
Further difficulty when discount-factor is non-integeral

[Akiyama, Frougny, Sakarovitch, IJoM 2008]

Number in base 𝑒

[Chaudhuri, Sankaranarayanan, Vardi , LICS 2013]

3/16/17 Comparators for quantitative games 12

SLIDE 13

DS Comparator : Core Insight – II

Sequence 𝐵 = (𝑏2, 𝑏), 𝑏* … )
Discount factor 𝑒 > 1
𝐸𝑇C(𝐵) = 𝑏2 +

FG C + FH CH + ⋯

= 𝑏2. 𝑏)𝑏* … C = 𝐵C

𝐸𝑇C 𝐵 ≤ 𝐸𝑇C(𝐶) iff 𝐵C ≤ 𝐶C
Find 𝐷 = (𝑑2, 𝑑), 𝑑*, … ), such that
𝐸𝑇C 𝐷 = 𝐷C ≥ 0
𝐵C + 𝐷C = 𝐶C

Arithmetic in base 𝑒

3/16/17 Comparators for quantitative games 13

SLIDE 14

DS Comparator : Core Insight – II (cont..)

Consider (𝑒 = 10)

3/16/17 Comparators for quantitative games 14

A 5 13 6 0 0 0 …. C + 0 8 6 0 0 0 …. B 7 2 2 0 0 0 …. X 2 1 0 0 0 0 ….

=> 𝐸𝑇C 𝐵 + 𝐸𝑇C 𝐷 = 𝐸𝑇C(𝐶)

𝑗 = 0, 𝑏2 + 𝑑2 + 𝑦2 = 𝑐2 𝑗 > 0, 𝑏4 + 𝑑4 + 𝑦4 = 𝑐4 + 𝑒 ⋅ 𝑦 4l)

SLIDE 15

DS Comparator : Finite-state memory?

Upper bound 𝜈 on reward sequences, 𝑒 =

n

> 1 is discount factor
For all bounded reward sequences 𝐵, 𝐶
Can find sequences 𝐷 and 𝑌 that satisfy equations
0 ≤ 𝑑4 ≤ 𝜈 ⋅

C Cl) , where is 𝑑4 of the form r

for integer 𝑛
𝑦4 ≤ 1 +

t Cl), where is 𝑦4 of the form r

for integer 𝑛
Finitely many possibilties for 𝑦4, 𝑑

u pairs

3/16/17 Comparators for quantitative games 15

SLIDE 16

DS Comparator : Construction

𝑏2 + 𝑦2 + 𝑑2 = 𝑐2

𝑏4 + 𝑑4 + 𝑦4 = 𝑐4 + 𝑒 ⋅ 𝑦 4l)

3/16/17 Comparators for quantitative games 16

start 𝑦2, 𝑑2 𝑏2, 𝑐2 𝑦 4l) , 𝑑 4l) 𝑦4, 𝑑4 𝑏4, 𝑐4

Automaton accepts (𝐵, 𝐶) iff 𝐸𝑇C(𝐵) ≤ 𝐸𝑇C(𝐶)

SLIDE 17

So far ...

ü Introduce Comparator [Bansal, Chaudhuri, Vardi (under review)]

ü A novel automata-theoretic technique to compare the aggregate of reward sequences ü Comparator for discounted-sum aggregate function

Applications of Comparators

3/16/17 Comparators for quantitative games 17

SLIDE 18

Application in quantitative graph games

Graph game with vertices 𝑊

2, 𝑊 ) and edges 𝐹

From vertex 𝑤 ∈ 𝑊

4, agent 𝑄4 picks the next vertex

Each agent receives a reward in every vertex
Reward of agent from game is given by aggregation over its reward

sequence

Objective of every player is to receive greater reward than the other

player

Find a winning strategy for player 𝑄4 ?

3/16/17 Comparators for quantitative games 18

SLIDE 19

Quantitative graph game – Solution

If aggregate function 𝑔 ∶ ℕ 8 → ℝ has a comparator

Objective of agent is an 𝜕-regular objective
We know how to solve graph games with 𝜕-regular objectives!
Leverages algorithms from qualitative domain
Quantitative objective converted to qualitative objective
Generic solutions for aggregate functions
As long as their comparator exists

3/16/17 Comparators for quantitative games 19

SLIDE 20

Comparators for repeated games ?

Finite-state representations for strategies in repeated games [Rubinstein

1986]

Regular set of strategies for each agent
Finite-state, weighted-tree automata-based representation for strategies
Finite-state representations of rational behaviors in this repeated

game

Use comparator

3/16/17 Comparators for quantitative games 20

SLIDE 21

Summary

Introduced the idea of Comparators for qualitatively comparison of

aggregate of reward sequences

Comparators have applications in games and beyond
Quantitative graph games
Discounted-sum inclusion problem
Generic algorithms for aggregate functions with comparators
Leverage algorithms and heuristics from qualitative analysis
Potential to use comparators to reason about repeated games

3/16/17 Comparators for quantitative games 21