Compositionality for Markov Reward Chains with Fast Transitions J. - - PowerPoint PPT Presentation

Compositionality for Markov Reward Chains with Fast Transitions

• J. Markovski, A. Sokolova, N. Trˇ

cka, E. P. de Vink presented by Elias Pschernig January 24, 2008

Outline

Introduction Motivation Recapitulation: Markov Chains Aggregation methods Discontinuous Markov reward chains

Ordinary lumping Reduction

Markov reward chains with fast transitions

τ-lumping τ-reduction

Relational properties Parallel composition

Markov Reward Chains

◮ Among most important and wide-spread analytical

performance models

◮ Ever growing complexity of Markov reward chain systems ◮ Compositional generation: Composing a big system from

several small components

◮ State space explosion: Result size is product of sizes of

components

◮ Need aggregation methods... ◮ ...and they should be compositional ◮ We consider special models of Markov reward chains:

Discontinuous Markov reward chains and Markov reward chains with fast transitions

Markov Reward Chains

◮ Among most important and wide-spread analytical

performance models

◮ Ever growing complexity of Markov reward chain systems ◮ Compositional generation: Composing a big system from

several small components

◮ State space explosion: Result size is product of sizes of

components

◮ Need aggregation methods... ◮ ...and they should be compositional ◮ We consider special models of Markov reward chains:

Discontinuous Markov reward chains and Markov reward chains with fast transitions

Recapitulation: Discrete time Markov chains

• 1

0.3

• 0.3
• 0.4
• 2

0.1

• 0.9
• 3

0.9

• 0.1
• Transition probability ma-

trix: 1 2 3 1 0.4 0.3 0.3 2 0.1 0.9 3 0.9 0.1

◮ Graphs with nodes

representing states

◮ Outgoing arrows

determine stochastic behavior of each state

◮ Probabilities only

depend on current state

Continuous time Markov chains

• 1

2

• 1
• 2

2

• 3

1

• ◮ Generator matrix:

1 2 3 1

• 3

1 2 2

• 2

2 3 1

• 1

=Q

Continuous time Markov reward chains

• 1

0.8 r0 λ

• τ
• 2

0.1 r1 µ

• 3

0.1 r2 ν

• ◮ P = (σ, Q, ρ)

◮ σ is a stochastic row initial probability vector (0.8, 0.1, 0.1) ◮ ρ is a state reward vector (r0, r1, r2) ◮ Transition probability matrix

P(t) =

∞

• n=0

Qntn n! = eQt

◮ Rewards are used to measure performance (application

dependent).

Discontinuous Markov reward chains

◮ Markov chains with instantaneous transitions → discontinuous

Markov chains

◮ Discontinuous Markov reward chain: P = (σ, Π, Q, ρ) ◮ Intuition: Π[i, j] denotes probability that a process occupies

two states via an instantaneous transition.

◮ Π = I leads to a standard Markov chain → generalization

• 1
• 2
• 3
• 4

Discontinuous Markov reward chains

Aggregation for discontinuous Markov reward chains

◮ Ordinary lumping ◮ Reduction

Ordinary lumping

◮ We lump P = (σ, Π, Q, ρ) to P = (σ, Π, Q, ρ) ◮ Partition L is an ordinary lumping ◮ P L

→ P

Ordinary lumping

◮ P L

→ P

◮ Partition of the state space into classes ◮ States lumped together form a class ◮ Equivalent transition behavior to other classes (intuitively:

probability of class is sum of probabilities of states)

◮ All states in a class have the same reward, total reward is

preserved

Example

◮ P L

→ P

• 1

0.5 r1 λ

• µ
• 2

0.2 r2

1 2 λ

• 1

2 λ

• 3

0.3 r2 λ

• 4

r3 ρ

• 5

r3 ρ

• {{1},{2,3},{4,5}}
• 1

0.5 r1 λ+µ

• 2, 3

0.5 r2 λ

• 4, 5

r3 ρ

Reduction

◮ We reduce P = (σ, Π, Q, ρ) to P = (σ, I, Q, ρ) ◮ P →r P ◮ Result is unique up to state permutation. ◮ Canonical product decomposition of Π ◮ Reduced states are given by ergodic classes of the original

process (ergodic = each state can be reached from each other state in finite time)

◮ Total reward is preserved

Markov reward chains with fast transitions

Markov reward chains with fast transitions

◮ Definition ◮ Aggregation

Markov reward chains with fast transitions

◮ Adds parameterized (“fast”) transitions to a standard Markov

reward chain.

◮ Uses two generator matrixes Qs and Qf , for slow and fast

transitions.

◮ P = (σ, Qs, Qf , ρ) is a function... ◮ ...where to each τ > 0 a Markov reward chain

Pτ = (σ, I, Qs + τQf , ρ) is assigned

◮ The limit τ → ∞ makes fast transitions instantaneous, and

we end up with a discontinuous Markov reward chain.

Markov reward chains with fast transitions

Aggregation for Markov reward chains with fast transitions

◮ τ-lumping ◮ τ-reduction

τ-lumping

◮ We τ-lump P = (σ, Qs, Qf , ρ) to P = (σ, Qs, Qf , ρ) ◮ Can define it using the limiting discontinuous Markov reward

chain.

◮ P L

P

◮ Not unique

τ-lumping

P (fast transitions)

∞

• L
• P

(lumped fast transitions)

∞

• Q

(discontinuous)

L

• Q

(lumped discontinuous)

τ-reduction

◮ We τ-reduce P = (σ, Qs, Qf , ρ) to R = (σ, I, Q, ρ) ◮ P r R

Example

◮ P r R

• 1

λ

• 2

aτ

• bτ
• 3

µ

• 4

ρ

• 5

τ-reduction

• 1

a a+b λ

• b

a+b λ

• 2, 3

µ

• 2, 4

ρ

• 5

τ-reduction

P (fast transitions)

∞

• r
• Q

(discontinuous)

r

• R = R′

(continuous)

◮ if P r R ◮ and P →∞ Q →r R′ ◮ then R = R′

Relational properties of ordinary lumping and τ-lumping

◮ Reduction works in one step, so no need to look at details of

its relational properties. Lumping:

◮ Need transitivity and strong confluence... ◮ ...to ensure that iterative application yields a uniquely

determined process.

◮ Repeated application of ordinary lumping... ◮ ...can be replaced by single application of composition of

individual lumpings.

◮ For τ-lumping, only the limit is uniquely determined.

Relational properties of ordinary lumping and τ-lumping

◮ Reduction works in one step, so no need to look at details of

its relational properties. Lumping:

◮ Need transitivity and strong confluence... ◮ ...to ensure that iterative application yields a uniquely

determined process.

◮ Repeated application of ordinary lumping... ◮ ...can be replaced by single application of composition of

individual lumpings.

◮ For τ-lumping, only the limit is uniquely determined.

Example

• 1

1 r1 aτ

• 2

r2 bτ

• 3

r3 λ

• cτ
• 4

r4 µ

• {{1, 2},

{3}, {4}}

• 1, 2

1 r2 bτ

• 3

r3 λ

• cτ
• 4

r4 µ

• {{{1, 2}, {3}},

{{4}}}

• 1, 2, 3

1

cr2+br3 b+c b b+c λ

• 4

r4 µ

• {{1, 2, 3},

{4}}

Parallel composition

◮ P1 ≥ P1, P2 ≥ P2 =

⇒ P1 P2 ≥ P1 P2

◮ Aggregate smaller components first... ◮ ...then combine them into the aggregated complete system. ◮ ≥ is semantic preorder. ◮ P ≥ P means P is an aggregated version of P. ◮ is a parallel composition.

Composing discontinuous Markov reward chains

◮ Kronecker sum ⊕ and Kronecker product ⊗ ◮ Parallel composition P1 P2 =

(σ1 ⊗ σ2, Π1 ⊗ Π2, Q1 ⊗ Π2 + Π1 ⊗ Q2, ρ1 ⊗ 1|ρ2| + 1|ρ1| ⊗ ρ2)

◮ If P1 and P2 are discontinuous Markov reward chains, then so

is P1 P2

Composing discontinuous Markov reward chains

◮ Both lumping and reduction are compositional with respect to

the parallel composition of discontinuous Markov reward chains

◮ If P1 L1

→ P1 and P2

L2

→ P2, then P1 P2

L1⊗L2

→ P1 P2.

◮ If P1 →r P1 and P2 →r P2, then P1 P2 →r P1 P2

Composing Markov reward chains with fast transitions

◮ Parallel composition

P1 P2 = (σ1⊗σ2, Qs,1⊕Qs,2, Qf ,1⊕Qf ,2, ρ1⊗1|ρ2|+1|ρ1|⊗ρ2)

◮ If P1 L1

P1 and P2

L2

P2, then P1 P2

L1⊗L2

• P1 P2.

◮ If P1 r P1 and P2 r P2, then P1 P2 r P1 P2

Example of parallel composition

• 1

1 r0 aτ

• λ
• 2

r1 µ

• 3

r2 ν

• 1

π rr bτ

• 2

1−π r4 cτ

• ξ
• 3

= ⇒

• 1

aτ

• bτ
• λ
• 4

µ

• bτ
• 7

bτ

• ν
• 2

cτ

• aτ
• ξ
• λ
• 5

µ

• cτ
• ξ
• 8

cτ

• ξ
• ν
• 3

aτ

• λ
• 6

µ

• 9

ν

Aggregated version of composition

• 1

1 r1 µ

• 2

r2 ν

• 1

1 πr3+(1−π)r4

b b+c ξ

• 2

= ⇒

• 1

1 r1+πr3+ (1−π)r4

b b+c ξ

• µ
• 3

r2+πr3+ (1−π)r4

b b+c ξ

• ν
• 2

r1 µ

• 4

r2 ν

Summary

P1

∞

• L1
• P1

∞

• Q1

L1 Q1

P2

∞

• L2
• P2

∞

• Q2

L2 Q2

= ⇒ P1 P2

∞

• L1⊗L2
• P1 P2

∞

• Q1 Q2

L1⊗L2 Q1 Q2

P1

∞

• r
• Q1

r R1

P2

∞

• r
• Q2

r R2

= ⇒ P1 P2

∞

• r
• Q1 Q2

r R1 R2

Summary

P1

∞

• L1
• P1

∞

• Q1

L1 Q1

P2

∞

• L2
• P2

∞

• Q2

L2 Q2

= ⇒ P1 P2

∞

• L1⊗L2
• P1 P2

∞

• Q1 Q2

L1⊗L2 Q1 Q2

P1

∞

• r
• Q1

r R1

P2

∞

• r
• Q2

r R2

= ⇒ P1 P2

∞

• r
• Q1 Q2

r R1 R2

End

◮ Questions?