Synthesis in Multi-Criteria Quantitative Games Mickael Randour - - PowerPoint PPT Presentation

Synthesis in Multi-Criteria Quantitative Games

Mickael Randour Advisors: V´ eronique Bruy` ere & Jean-Fran¸ cois Raskin Mons - 18.04.2014

Private PhD Thesis Defense

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

1 Synthesis in Quantitative Games 2 Beyond Worst-Case Synthesis 3 Multi-Dimension Objectives 4 Window Objectives 5 Conclusion and Future Work

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 1 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

1 Synthesis in Quantitative Games 2 Beyond Worst-Case Synthesis 3 Multi-Dimension Objectives 4 Window Objectives 5 Conclusion and Future Work

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 2 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

General context

Verification and synthesis:

a reactive system to control, an interacting environment, a specification to enforce.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 3 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

General context

Verification and synthesis:

a reactive system to control, an interacting environment, a specification to enforce.

Qualitative and quantitative specifications.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 3 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

General context

Verification and synthesis:

a reactive system to control, an interacting environment, a specification to enforce.

Qualitative and quantitative specifications. Focus on multi-criteria quantitative models

to reason about trade-offs and interplays.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 3 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Synthesis via two-player graph games

system description environment description informal specification model as a game model as winning

• bjectives

synthesis is there a winning strategy ? empower system capabilities

• r weaken

specification requirements strategy = controller no yes Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 4 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Synthesis via two-player graph games

system description environment description informal specification model as a game model as winning

• bjectives

synthesis is there a winning strategy ? empower system capabilities

• r weaken

specification requirements strategy = controller no yes

1 Can one player guarantee

victory?

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 4 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Synthesis via two-player graph games

system description environment description informal specification model as a game model as winning

• bjectives

synthesis is there a winning strategy ? empower system capabilities

• r weaken

specification requirements strategy = controller no yes

1 Can one player guarantee

victory?

2 Can we decide which one?

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 4 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Synthesis via two-player graph games

system description environment description informal specification model as a game model as winning

• bjectives

synthesis is there a winning strategy ? empower system capabilities

• r weaken

specification requirements strategy = controller no yes

1 Can one player guarantee

victory?

2 Can we decide which one? 3 How complex his strategy

needs to be?

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 4 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Quantitative games on graphs

2 2 5 −1 7 −4 Graph G = (S, E, w) with w : E → Z Deterministic transitions Two-player game G = (G, S1, S2)

P1 states = P2 states =

Plays have values

f : Plays(G) → R ∪ {−∞, ∞}

Players follow strategies

λi : Prefsi(G) → D(S) Finite memory ⇒ stochastic output Moore machine M(λi) = (Mem, m0, αu, αn)

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 5 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Quantitative games on graphs

2 2 5 −1 7 −4 Graph G = (S, E, w) with w : E → Z Deterministic transitions Two-player game G = (G, S1, S2)

P1 states = P2 states =

Plays have values

f : Plays(G) → R ∪ {−∞, ∞}

Players follow strategies

λi : Prefsi(G) → D(S) Finite memory ⇒ stochastic output Moore machine M(λi) = (Mem, m0, αu, αn)

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 5 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Quantitative games on graphs

2 2 5 −1 7 −4 Graph G = (S, E, w) with w : E → Z Deterministic transitions Two-player game G = (G, S1, S2)

P1 states = P2 states =

Plays have values

f : Plays(G) → R ∪ {−∞, ∞}

Players follow strategies

λi : Prefsi(G) → D(S) Finite memory ⇒ stochastic output Moore machine M(λi) = (Mem, m0, αu, αn)

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 5 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Quantitative games on graphs

2 2 5 −1 7 −4 Graph G = (S, E, w) with w : E → Z Deterministic transitions Two-player game G = (G, S1, S2)

P1 states = P2 states =

Plays have values

f : Plays(G) → R ∪ {−∞, ∞}

Players follow strategies

λi : Prefsi(G) → D(S) Finite memory ⇒ stochastic output Moore machine M(λi) = (Mem, m0, αu, αn)

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 5 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Quantitative games on graphs

2 2 5 −1 7 −4 Graph G = (S, E, w) with w : E → Z Deterministic transitions Two-player game G = (G, S1, S2)

P1 states = P2 states =

Plays have values

f : Plays(G) → R ∪ {−∞, ∞}

Players follow strategies

λi : Prefsi(G) → D(S) Finite memory ⇒ stochastic output Moore machine M(λi) = (Mem, m0, αu, αn)

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 5 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Quantitative games on graphs

2 2 5 −1 7 −4 Graph G = (S, E, w) with w : E → Z Deterministic transitions Two-player game G = (G, S1, S2)

P1 states = P2 states =

Plays have values

f : Plays(G) → R ∪ {−∞, ∞}

Players follow strategies

λi : Prefsi(G) → D(S) Finite memory ⇒ stochastic output Moore machine M(λi) = (Mem, m0, αu, αn)

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 5 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Quantitative games on graphs

2 2 5 −1 7 −4 Then, (2, 5, 2)ω Graph G = (S, E, w) with w : E → Z Deterministic transitions Two-player game G = (G, S1, S2)

P1 states = P2 states =

Plays have values

f : Plays(G) → R ∪ {−∞, ∞}

Players follow strategies

λi : Prefsi(G) → D(S) Finite memory ⇒ stochastic output Moore machine M(λi) = (Mem, m0, αu, αn)

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 5 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Markov decision processes

1 2 1 2

2 2 5 −1 7 −4 MDP P = (G, S1, S∆, ∆) with ∆: S∆ → D(S)

P1 states = stochastic states =

MDP = game + strategy of P2

P = G[λ2]

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 6 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Markov chains

1 2 1 2 1 4 3 4

2 2 5 −1 7 −4 MC M = (G, δ) with δ: S → D(S) MC = MDP + strategy of P1 = game + both strategies

M = P[λ1] = G[λ1, λ2]

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 7 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Markov chains

1 2 1 2 1 4 3 4

2 2 5 −1 7 −4 MC M = (G, δ) with δ: S → D(S) MC = MDP + strategy of P1 = game + both strategies

M = P[λ1] = G[λ1, λ2]

Event A ⊆ Plays(G)

probability PM

sinit(A)

Measurable f : Plays(G) → R ∪ {−∞, ∞}

expected value EM

sinit(f )

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 7 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Winning semantics and decision problems

Qualitative objectives - φ ⊆ Plays(G)

λ1 surely winning: ∀ λ2 ∈ Λ2, OutsG(sinit, λ1, λ2) ⊆ φ λ1 almost-surely winning: ∀ λ2 ∈ Λ2, PG[λ1,λ2]

sinit

(φ) = 1

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 8 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Winning semantics and decision problems

Qualitative objectives - φ ⊆ Plays(G)

λ1 surely winning: ∀ λ2 ∈ Λ2, OutsG(sinit, λ1, λ2) ⊆ φ λ1 almost-surely winning: ∀ λ2 ∈ Λ2, PG[λ1,λ2]

sinit

(φ) = 1

Quantitative objectives - f : Plays(G) → R ∪ {−∞, ∞}

worst-case threshold problem, µ ∈ Q: ∃? λ1 ∈ Λ1, ∀ λ2 ∈ Λ2, ∀ π ∈ OutsG(sinit, λ1, λ2), f (π) ≥ µ expected value threshold problem (MDP), ν ∈ Q: ∃? λ1 ∈ Λ1, EP[λ1]

sinit

(f ) ≥ ν

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 8 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Classical qualitative objectives

ReachG(T) = {π = s0s1s2 . . . ∈ Plays(G) | ∃ i ∈ N, si ∈ T} BuchiG(T) = {π = s0s1s2 . . . ∈ Plays(G) | Inf(π) ∩ T = ∅} ParityG = {π = s0s1s2 . . . ∈ Plays(G) | Par(π) mod 2 = 0}

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 9 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Classical quantitative objectives and value functions

Total-payoff : TP(π) = lim inf

n→∞ i=n−1

• i=0

w((si, si+1)) Mean-payoff : MP(π) = lim inf

n→∞

1 n

i=n−1

• i=0

w((si, si+1)) Shortest path: truncated sum up to first visit of T ⊆ S Energy: keep the running sum positive at all times

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 10 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Single-criterion models - known results

reachability B¨ uchi parity games sure sem. complexity P-c. UP ∩ coUP P1 mem. pure memoryless P2 mem. mdps almost-sure sem. complexity P-c. P1 mem. pure memoryless TP MP SP EG games worst-case complexity UP ∩ coUP P-c. UP ∩ coUP P1 mem. pure memoryless P2 mem. mdps expected value complexity P-c. n/a P1 mem. pure memoryless

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 11 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Contributions

Shift from single-criterion models to multi-criteria ones.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 12 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

1 Synthesis in Quantitative Games 2 Beyond Worst-Case Synthesis 3 Multi-Dimension Objectives 4 Window Objectives 5 Conclusion and Future Work

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 13 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Combining two classical models

Games → antagonistic adversary → guarantees on worst-case MDPs → stochastic adversary → optimize expected value BWC synthesis → ensure both

∧

Studied value functions Mean-Payoff Shortest Path

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 14 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Example: going to work

home station traffic waiting room work

1 10 9 10 2 10 7 10 1 10

train 2 car 1 back home 1 bicycle 45 delay 1 wait 4 light 20 medium 30 heavy 70 departs 35

Weights = minutes Goal: minimize our expected time to reach “work” But, important meeting in

• ne hour! Requires strict

guarantees on the worst-case reaching time.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 15 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Example: going to work

home station traffic waiting room work

1 10 9 10 2 10 7 10 1 10

train 2 car 1 back home 1 bicycle 45 delay 1 wait 4 light 20 medium 30 heavy 70 departs 35

Optimal expectation strategy: car.

E = 33, WC = 71 > 60.

Optimal worst-case strategy: bicycle.

E = WC = 45 < 60.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 15 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Example: going to work

home station traffic waiting room work

1 10 9 10 2 10 7 10 1 10

train 2 car 1 back home 1 bicycle 45 delay 1 wait 4 light 20 medium 30 heavy 70 departs 35

Optimal expectation strategy: car.

E = 33, WC = 71 > 60.

Optimal worst-case strategy: bicycle.

E = WC = 45 < 60.

Sample BWC strategy: try train up to 3 delays then switch to bicycle.

E ≈ 37.56, WC = 59 < 60. Optimal E under WC constraint Uses finite memory

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 15 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Beyond worst-case synthesis

Formal definition

Given

• G = (G, S1, S2), sinit ∈ S,
• a finite-memory stochastic model λstoch

2

∈ ΛF

2 of the adversary,

• a measurable value function f : Plays(G) → R ∪ {−∞, ∞}, and two

thresholds µ, ν ∈ Q, the beyond worst-case (BWC) problem asks to decide if P1 has a finite-memory strategy λ1 ∈ ΛF

1 such that

∀ λ2 ∈ Λ2, ∀ π ∈ OutsG(sinit, λ1, λ2), f (π) > µ (1) E

G[λ1,λstoch

2

] sinit

(f ) > ν (2) and the BWC synthesis problem asks to synthesize such a strategy if one exists.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 16 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Beyond worst-case synthesis

Formal definition

Given

• G = (G, S1, S2), sinit ∈ S,
• a finite-memory stochastic model λstoch

2

∈ ΛF

2 of the adversary,

• a measurable value function f : Plays(G) → R ∪ {−∞, ∞}, and two

thresholds µ, ν ∈ Q, the beyond worst-case (BWC) problem asks to decide if P1 has a finite-memory strategy λ1 ∈ ΛF

1 such that

∀ λ2 ∈ Λ2, ∀ π ∈ OutsG(sinit, λ1, λ2), f (π) > µ (1) E

G[λ1,λstoch

2

] sinit

(f ) > ν (2) and the BWC synthesis problem asks to synthesize such a strategy if one exists.

Notice the highlighted parts

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 16 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Related work

Common philosophy: avoiding outlier outcomes

1 Our strategies are strongly risk averse

avoid risk at all costs and optimize among safe strategies

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 17 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Related work

Common philosophy: avoiding outlier outcomes

1 Our strategies are strongly risk averse

avoid risk at all costs and optimize among safe strategies

2 Other notions of risk ensure low probability of risked behavior

[WL99, FKR95]

without worst-case guarantee without good expectation

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 17 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Related work

Common philosophy: avoiding outlier outcomes

1 Our strategies are strongly risk averse

avoid risk at all costs and optimize among safe strategies

2 Other notions of risk ensure low probability of risked behavior

[WL99, FKR95]

without worst-case guarantee without good expectation

3 Trade-off between expectation and variance [BCFK13, MT11]

statistical measure of the stability of the performance no strict guarantee on individual outcomes

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 17 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Mean-payoff value function

worst-case expected value BWC complexity NP ∩ coNP P NP ∩ coNP memory memoryless memoryless pseudo-polynomial

Additional modeling power for free

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 18 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Philosophy of the algorithm

Classical worst-case and expected value results and algorithms as nuts and bolts Screw them together in an adequate way

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 19 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Philosophy of the algorithm

Classical worst-case and expected value results and algorithms as nuts and bolts Screw them together in an adequate way Three key ideas

1 To characterize the expected value, look at end-components

(ECs)

2 Winning ECs vs. losing ECs: the latter must be avoided to

preserve the worst-case requirement

3 Inside a WEC, we have an interesting way to play. . .

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 19 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Philosophy of the algorithm

Classical worst-case and expected value results and algorithms as nuts and bolts Screw them together in an adequate way Three key ideas

1 To characterize the expected value, look at end-components

(ECs)

2 Winning ECs vs. losing ECs: the latter must be avoided to

preserve the worst-case requirement

3 Inside a WEC, we have an interesting way to play. . .

= ⇒ Let’s focus on a WEC

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 19 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Inside a WEC

s5 s6 s7

1 2 1 2

1 1 −1 9

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 20 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Inside a WEC

s5 s6 s7

1 2 1 2

1 1 −1 9

Game interpretation Worst-case threshold is µ = 0 All states are winning: memoryless optimal worst-case strategy λwc

1 ∈ ΛPM 1

(G), ensuring µ∗ = 1 > 0

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 20 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Inside a WEC

s5 s6 s7

1 2 1 2

1 1 −1 9

Game interpretation Worst-case threshold is µ = 0 All states are winning: memoryless optimal worst-case strategy λwc

1 ∈ ΛPM 1

(G), ensuring µ∗ = 1 > 0 MDP interpretation Memoryless optimal expected value strategy λe

1 ∈ ΛPM 1

(P) achieves ν∗ = 2

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 20 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

A cornerstone of our approach

s5 s6 s7

1 2 1 2

1 1 −1 9

BWC problem: what kind of thresholds (µ = 0, ν) can we achieve?

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 20 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

A cornerstone of our approach

s5 s6 s7

1 2 1 2

1 1 −1 9

BWC problem: what kind of thresholds (µ = 0, ν) can we achieve?

Key result

For all ε > 0, there exists a finite-memory strategy of P1 that satisfies the BWC problem for the thresholds pair (0, ν∗ − ε). We can be arbitrarily close to the optimal expectation while ensuring the worst-case

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 20 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Combined strategy

s5 s6 s7

1 2 1 2

1 1 −1 9

Outcomes of the form

WC > 0 E =??

K steps > 0 > 0 ≤ 0 L steps compensate > 0 ≤ 0 compensate

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 20 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Combined strategy

s5 s6 s7

1 2 1 2

1 1 −1 9

Outcomes of the form

WC > 0 E =??

K steps > 0 > 0 ≤ 0 L steps compensate > 0 ≤ 0 compensate

What we want

K, L → ∞ E = ν∗ = 2

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 20 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Combined strategy

s5 s6 s7

1 2 1 2

1 1 −1 9

Outcomes of the form

WC > 0 E =??

K steps > 0 > 0 ≤ 0 L steps compensate > 0 ≤ 0 compensate

What we want

K, L → ∞ E = ν∗ = 2

L = linear(K) P( ) → 0 exp. fast! [Tra09, GO02]

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 20 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Shortest path

Strictly positive integer weights, w : E → N0 P1 wants to minimize its total cost up to target

inequalities are reversed

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 21 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Shortest path

Strictly positive integer weights, w : E → N0 P1 wants to minimize its total cost up to target

inequalities are reversed worst-case expected value BWC complexity P P pseudo-poly. / NP-hard memory memoryless memoryless pseudo-poly.

Problem inherently harder than worst-case and expectation. NP-hardness by K th largest subset problem [JK78, GJ79]

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 21 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Key difference with MP case

Useful observation

The set of all worst-case winning strategies for the shortest path can be represented through a finite game. Sequential approach solving the BWC problem:

1 represent all WC winning strategies, 2 optimize the expected value within those strategies.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 22 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

1 Synthesis in Quantitative Games 2 Beyond Worst-Case Synthesis 3 Multi-Dimension Objectives 4 Window Objectives 5 Conclusion and Future Work

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 23 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Multi-dimension games

EG MP MP TP TP

• ne-dim.

complexity NP ∩ coNP P1 mem. pure memoryless P2 mem. k-dim. complexity coNP-c. NP ∩ coNP ?? P1 mem. pure finite pure infinite ?? P2 mem. pure memoryless

Natural extension

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 24 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Multi-dimension games

EG MP MP TP TP

• ne-dim.

complexity NP ∩ coNP P1 mem. pure memoryless P2 mem. k-dim. complexity coNP-c. NP ∩ coNP ?? P1 mem. pure finite pure infinite ?? P2 mem. pure memoryless

Natural extension, increased complexity.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 24 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Multi-dimension games

EG MP MP TP TP

• ne-dim.

complexity NP ∩ coNP P1 mem. pure memoryless P2 mem. k-dim. complexity coNP-c. NP ∩ coNP ?? P1 mem. pure finite pure infinite ?? P2 mem. pure memoryless

Natural extension, increased complexity. Question: what about TP?

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 24 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Multi-dimension games

EG MP MP TP TP

• ne-dim.

complexity NP ∩ coNP P1 mem. pure memoryless P2 mem. k-dim. complexity coNP-c. NP ∩ coNP undec. P1 mem. pure finite pure infinite

• P2 mem.

pure memoryless

Theorem

Total-payoff games are undecidable for k ≥ 5. Reduction from the halting problem in 2CMs. Open for k = 2, 3 and 4.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 24 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Multi-dimension games

EG MP MP TP TP

• ne-dim.

complexity NP ∩ coNP P1 mem. pure memoryless P2 mem. k-dim. complexity coNP-c. NP ∩ coNP undec. P1 mem. pure finite pure infinite

• P2 mem.

pure memoryless

We want finite-memory controllers.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 24 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Multi-dimension games

EG MP TP TP

• ne-dim.

complexity NP ∩ coNP P1 mem. pure memoryless P2 mem. k-dim. complexity coNP-c. undec. P1 mem. pure finite

• P2 mem.

pure memoryless

We want finite-memory controllers. Restrict P1 to finite-memory strategies.

Lemma [CDHR10, VCD+12]

The answer to the worst-case mean-payoff threshold problem is Yes iff the answer to the unknown initial credit problem is Yes.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 24 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Multi-dimension games

EG MP TP TP

• ne-dim.

complexity NP ∩ coNP P1 mem. pure memoryless P2 mem. k-dim. complexity coNP-c. undec. P1 mem. pure finite

• P2 mem.

pure memoryless

Question: precise memory bounds?

exponential memory sufficient and necessary

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 24 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Multi-dimension games

EG MP TP TP

• ne-dim.

complexity NP ∩ coNP P1 mem. pure memoryless P2 mem. k-dim. complexity coNP-c. undec. P1 mem. pure finite

• P2 mem.

pure memoryless

Question: precise memory bounds?

exponential memory sufficient and necessary

Question: efficient synthesis algorithm?

EXPTIME algorithm symbolic and incremental

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 24 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Multi-dimension games

EG MP TP TP

• ne-dim.

complexity NP ∩ coNP P1 mem. pure memoryless P2 mem. k-dim. complexity coNP-c. undec. P1 mem. pure finite

• P2 mem.

pure memoryless

Question: precise memory bounds?

exponential memory sufficient and necessary

Question: efficient synthesis algorithm?

EXPTIME algorithm symbolic and incremental

Results for EG / MP + parity.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 24 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Trading finite memory for randomness

Question: when and how can P1 trade his pure finite-memory strategy for an equally powerful randomized memoryless one? relax to almost-sure semantics

Multi energy Multi MP (parity) MP parity and energy parity

• ne-player

× √ √ two-player × × √

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 25 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

1 Synthesis in Quantitative Games 2 Beyond Worst-Case Synthesis 3 Multi-Dimension Objectives 4 Window Objectives 5 Conclusion and Future Work

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 26 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Why an alternative to MP/TP?

No known polynomial-time algorithm in one-dimension. TP is undecidable in multi-dimension. No timing guarantee

long-run behavior vs. time frames.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 27 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Window objectives: key idea

Sum Time

Window of fixed size sliding along a play defines a local finite horizon. Objective: see a local MP ≥ 0 before hitting the end of the window needs to be verified at every step. Intuition: local deviations from the threshold must be compensated in a parametrized # of steps.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 28 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Multiple variants

Maximal window size fixed or quantified existentially (Bounded Window) Prefix-independent or not

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 29 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Multiple variants

Maximal window size fixed or quantified existentially (Bounded Window) Prefix-independent or not

Conservative approximations in one-dim.

Any window obj. ⇒ BW ⇒ MP ≥ 0 BW ⇐ MP > 0

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 29 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Results overview

• ne-dimension

k-dimension complexity P1 mem. P2 mem. complexity P1 mem. P2 mem. MP / MP NP ∩ coNP mem-less coNP-c. / NP ∩ coNP infinite mem-less TP / TP NP ∩ coNP mem-less undec.

• WMP: fixed

P-c.

• mem. req.

≤ linear(|S| · lmax) PSPACE-h. polynomial window EXP-easy exponential WMP: fixed P(|S|, V , lmax) EXP-c. arbitrary window WMP: bounded NP ∩ coNP mem-less infinite NPR-h.

• window problem

|S| the # of states, V the length of the binary encoding of weights, and lmax the window size.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 30 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Results overview: advantages

• ne-dimension

k-dimension complexity P1 mem. P2 mem. complexity P1 mem. P2 mem. MP / MP NP ∩ coNP mem-less coNP-c. / NP ∩ coNP infinite mem-less TP / TP NP ∩ coNP mem-less undec.

• WMP: fixed

P-c.

• mem. req.

≤ linear(|S| · lmax) PSPACE-h. polynomial window EXPTIME-easy exponential WMP: fixed P(|S|, V , lmax) EXPTIME-c. arbitrary window WMP: bounded NP ∩ coNP mem-less infinite NPR-h.

• window problem

|S| the # of states, V the length of the binary encoding of weights, and lmax the window size. For one-dim. games with poly. windows, we are in P. For multi-dim. games with fixed windows, we are decidable. Window obj. provide timing guarantees.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 30 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

1 Synthesis in Quantitative Games 2 Beyond Worst-Case Synthesis 3 Multi-Dimension Objectives 4 Window Objectives 5 Conclusion and Future Work

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 31 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Summary

Study of multi-criteria quantitative games.

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 32 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Summary

Study of multi-criteria quantitative games.

1 Beyond worst-case synthesis

worst-case and expected value additional modeling power for free in MP case complexity leap for SP

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 32 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Summary

Study of multi-criteria quantitative games.

1 Beyond worst-case synthesis

worst-case and expected value additional modeling power for free in MP case complexity leap for SP

2 Multi-dimension TP, MP and EG + parity

undecidability of TP tight memory bounds for MP and EG + parity

• ptimal synthesis algorithm

memory vs. randomness

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 32 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Summary

Study of multi-criteria quantitative games.

1 Beyond worst-case synthesis

worst-case and expected value additional modeling power for free in MP case complexity leap for SP

2 Multi-dimension TP, MP and EG + parity

undecidability of TP tight memory bounds for MP and EG + parity

• ptimal synthesis algorithm

memory vs. randomness

3 Window objectives

timing guarantees improved tractability

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 32 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Future work

Beyond worst-case extensions

more general games (e.g., stochastic games) multi-dimension percentile performances

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 33 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Future work

Beyond worst-case extensions

more general games (e.g., stochastic games) multi-dimension percentile performances

Mixed objectives

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 33 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Future work

Beyond worst-case extensions

more general games (e.g., stochastic games) multi-dimension percentile performances

Mixed objectives Window objectives

stochastic context synchronous closing (finitary) parity [CHH09]

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 33 / 34

Quantitative Games Beyond Worst-Case Synthesis Multi-Dimension Objectives Window Objectives Conclusion

Thanks!

To my advisors, V´ eronique Bruy` ere and Jean-Fran¸ cois Raskin, to my other co-authors, Krishnendu Chatterjee, Laurent Doyen and Emmanuel Filiot, and to you, members of the jury, for your precious time!

Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 34 / 34

References I

Tom´ as Br´ azdil, Krishnendu Chatterjee, Vojtech Forejt, and Anton´ ın Kucera. Trading performance for stability in Markov decision processes. In LICS, pages 331–340. IEEE Computer Society, 2013. V´ eronique Bruy` ere, Emmanuel Filiot, Mickael Randour, and Jean-Fran¸ cois Raskin. Expectations or guarantees? i want it all! a crossroad between games and mdps. In Fabio Mogavero, Aniello Murano, and Moshe Y. Vardi, editors, Proceedings 2nd International Workshop on Strategic Reasoning, Grenoble, France, April 5-6, 2014, volume 146 of Electronic Proceedings in Theoretical Computer Science, pages 1–8. Open Publishing Association, 2014. V´ eronique Bruy` ere, Emmanuel Filiot, Mickael Randour, and Jean-Fran¸ cois Raskin. Meet Your Expectations With Guarantees: Beyond Worst-Case Synthesis in Quantitative Games. In Ernst W. Mayr and Natacha Portier, editors, 31st International Symposium on Theoretical Aspects of Computer Science (STACS 2014), volume 25 of Leibniz International Proceedings in Informatics (LIPIcs), pages 199–213, Dagstuhl, Germany, 2014. Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik. Krishnendu Chatterjee, Laurent Doyen, Thomas A. Henzinger, and Jean-Fran¸ cois Raskin. Generalized mean-payoff and energy games. In Kamal Lodaya and Meena Mahajan, editors, FSTTCS, volume 8 of LIPIcs, pages 505–516. Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, 2010. Krishnendu Chatterjee, Laurent Doyen, Mickael Randour, and Jean-Fran¸ cois Raskin. Looking at mean-payoff and total-payoff through windows. In Dang Van Hung and Mizuhito Ogawa, editors, ATVA, volume 8172 of Lecture Notes in Computer Science, pages 118–132. Springer, 2013. Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 35 / 34

References II

Krishnendu Chatterjee, Thomas A. Henzinger, and Florian Horn. Finitary winning in omega-regular games. ACM Trans. Comput. Log., 11(1), 2009. Krishnendu Chatterjee, Mickael Randour, and Jean-Fran¸ cois Raskin. Strategy synthesis for multi-dimensional quantitative objectives. In Maciej Koutny and Irek Ulidowski, editors, CONCUR, volume 7454 of Lecture Notes in Computer Science, pages 115–131. Springer, 2012. Krishnendu Chatterjee, Mickael Randour, and Jean-Fran¸ cois Raskin. Strategy synthesis for multi-dimensional quantitative objectives. Acta Informatica, 2013. 35 pages. Jerzy A. Filar, Dmitry Krass, and Kirsten W. Ross. Percentile performance criteria for limiting average Markov decision processes. Transactions on Automatic Control, 40(1):2–10, 1995. Michael R. Garey and David S. Johnson. Computers and intractability: a guide to the Theory of NP-Completeness. Freeman New York, 1979. Peter W. Glynn and Dirk Ormoneit. Hoeffding’s inequality for uniformly ergodic Markov chains. Statistics & Probability Letters, 56(2):143–146, 2002. Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 36 / 34

References III

Donald B. Johnson and Samuel D. Kashdan. Lower bounds for selection in X + Y and other multisets. Journal of the ACM, 25(4):556–570, 1978. Shie Mannor and John N. Tsitsiklis. Mean-variance optimization in Markov decision processes. In Lise Getoor and Tobias Scheffer, editors, ICML, pages 177–184. Omnipress, 2011. Mickael Randour. Automated synthesis of reliable and efficient systems through game theory: A case study. In Thomas Gilbert, Markus Kirkilionis, and Gregoire Nicolis, editors, Proceedings of the European Conference on Complex Systems 2012, Springer Proceedings in Complexity, pages 731–738. Springer, 2013. Mathieu Tracol. Fast convergence to state-action frequency polytopes for MDPs.

• Oper. Res. Lett., 37(2):123–126, 2009.

Yaron Velner, Krishnendu Chatterjee, Laurent Doyen, Thomas A. Henzinger, Alexander Rabinovich, and Jean-Fran¸ cois Raskin. The complexity of multi-mean-payoff and multi-energy games. CoRR, abs/1209.3234, 2012. Congbin Wu and Yuanlie Lin. Minimizing risk models in Markov decision processes with policies depending on target values. Journal of Mathematical Analysis and Applications, 231(1):47–67, 1999. Synthesis in Multi-Criteria Quantitative Games

• M. Randour (advisors: V. Bruy`

ere & J.-F. Raskin) 37 / 34